Skip to content

Price scraping with Python: build a repeatable price tracker

A price tracker is a small data pipeline: request a product page, pull the title and price out of the HTML, clean the values, store them under a stable product identifier, and compare today's file with yesterday's. This guide walks through that pipeline with Python, starting from a static practice site and moving to JavaScript-rendered category pages, a scraping API, and a snapshot comparison script.

What price scraping is for, and when it pays off

Price scraping means collecting product prices from public pages automatically and turning them into structured rows. It becomes worthwhile when prices move often, when several stores or categories need watching, or when a manual check would mean opening many tabs just to confirm a small change.

Typical uses include watching competitor listings, studying how prices move over time, adjusting your own prices in response, catching discounts, and comparing the same product across marketplaces. Large marketplaces are a common starting point because they hold a lot of product data, but they also tend to have stronger protections.

On the legal side, pages that are openly reachable without an account or private credentials are generally the safer ground, but each site's terms of service still apply. Circumventing access controls, flooding a server with traffic, or collecting private data is where trouble starts. If a target feels questionable, check the terms before building around it.

The basic static-page flow

For a page whose HTML already contains the product data, the sequence is short: send an HTTP request with requests, parse the response with BeautifulSoup, select the product elements, extract the fields you need, and write the rows to a CSV file.

A small uv project is enough to hold this. The setup below creates the project and adds requests, beautifulsoup4, and lxml.

Choosing a practice target and the selectors that matter

books.toscrape.com is a convenient practice target. Each book sits inside an article with the class .product_pod. Inside that block, the title comes from the h3 a tag and the price from p.price_color. The plan is to find every .product_pod element, loop through them, pull the title and price, and write the results to a CSV file.

The complete example below includes basic error handling for network failures and file-write failures, and it decodes the response bytes explicitly to avoid charset issues on that site.

Running the static scraper and reading the output

Run the script with uv run main.py. If the request and parse succeed, a file named books_prices.csv appears in the project folder. The output has two columns, title and price, with the price kept as the visible text from the page, for example £51.77.

This is the easiest starting point because the prices are already in the HTML response: no browser automation and no JavaScript rendering.

When the price is not in the HTML

Many modern product pages load prices with JavaScript after the page opens in a browser. A plain requests call to such a page often returns HTML without the actual prices, or with blank slots where the numbers should be. At that point the parsing code is not at fault; the fully rendered content never arrived.

There are two common ways around this. A headless browser such as Playwright or Selenium loads the page the way a real browser does and executes JavaScript, which gives full control but adds complexity and overhead. A scraping API handles rendering on its side and returns the final HTML, which is usually faster to set up and easier to scale.

What changes when you move from one page to many

Scraping one page is simple. Scraping hundreds or thousands brings three recurring problems. Rate limiting can slow responses, return errors, or block an IP when too many requests arrive in a short window. IP blocking can flag repeated requests from the same address, especially on large e-commerce platforms. Reliability suffers too: connections drop, pages time out, and some requests fail at random.

The practical responses are proxies to rotate IP addresses, rate limiting to space requests out, retries to survive temporary failures, and parallelism to speed things up without overwhelming the target. A common pattern is to loop through many category pages, collect product links, and then fetch each product page individually. Doing all of that by hand gets messy quickly, which is where a scraping API can simplify the setup.

A worked example: three Newegg categories through an API

The next example moves from a static practice page to something more realistic. The scraper fetches three Newegg category pages through a scraping API, asks the API to render JavaScript and wait for a specific selector, and then parses the returned HTML with BeautifulSoup. The three categories are desktop computers, server and workstation systems, and wireless routers. The script extracts product titles, prices, product URLs, and category names, then writes everything to one CSV file.

The API request passes render_js, premium_proxy, and wait_for parameters, and the three category requests run in parallel with a ThreadPoolExecutor. Newegg splits the current price across a strong element and a sup element, so the parser joins the whole and fractional parts manually. Because the API provider handles rendering and proxies, the scraping code stays focused on parsing.

Account creation and credit terms are owner-configured: {{OWNER_SIGNUP_URL}} and {{OWNER_FREE_PLAN_CREDITS}}. The API origin and credential variable are also owner-configured: {{OWNER_API_BASE_URL}} and {{OWNER_API_KEY_NAME}}.

A sample of the resulting CSV looks like this:

Cleaning and normalizing before you store anything

Raw scraped values often look fine until analysis begins. Prices may contain commas or be split across elements, and titles can carry messy spacing. The updated parse_products() function collapses whitespace in titles, removes commas and dots from the price parts, and builds a clean numeric string such as 6299.99. The key idea is to normalize prices into a consistent numeric format and clean titles early, so later comparisons and reports stay usable.

Daily snapshots and a stable product key

For tracking rather than one-off scraping, save a dated snapshot instead of overwriting the same CSV. Each run creates a new file for that day, and each row carries a stable product ID so today's data can be matched against yesterday's.

On Newegg, the product ID at the end of the product URL is the safer identifier to keep. A title can change slightly over time, but the product ID is much better for matching the same item across snapshots. The output filename becomes newegg_prices_<date>.csv, and the parser extracts product_id from URLs such as https://www.newegg.com/p/N82E16883101919. The CSV writer is updated to include the new column.

Snapshot field map:

  • category: the category label used in the run, useful for grouping and filtering.
  • product_id: the stable key used to match the same product across days.
  • title: the cleaned product name, useful for reports but not for matching.
  • price: the normalized numeric price, stored as a string in the CSV and read as a Decimal during comparison.
  • product_url: the absolute product link, useful for opening the listing from a report.

Comparing two snapshots

The comparison script asks for two snapshot filenames, loads both files into dictionaries keyed by product_id, and reports price changes, new products, and removed products. Prices are read as Decimal values, and rows with a missing product ID or an unparseable price are skipped.

Run it with uv run compare_prices.py and enter two filenames, for example newegg_prices_2026-03-18.csv and newegg_prices_2026-03-19.csv. In the sample report, a Dell desktop moved from 859.99 to 879.99 and is labeled as increased, while an HP Pro Mini moved from 599.99 to 499.99 and is labeled as dropped. No new or removed products appeared in that sample. This is enough to run a simple price tracking workflow without adding a database yet.

A pre-flight checklist before you scale

Before adding volume, check the target and the pipeline against these points:

  • Is the price present in the raw HTML, or does it require JavaScript rendering? This decides whether requests and BeautifulSoup are enough or whether you need a headless browser or an API.
  • Does the page expose a stable product identifier in the URL or markup? Without one, snapshot matching becomes unreliable.
  • Are prices split across multiple elements? If so, plan the join and the normalization step before writing rows.
  • Have you set a request spacing and retry policy? Instant retries after a failure tend to make blocks more likely.
  • Do you have pagination coverage? Most category pages do not stop at page one.
  • Is there a deduplication rule? Repeated runs should not store the same product multiple times.
  • Is CSV still enough, or does the volume call for a database or data pipeline?
  • Is the scraper scheduled? Tracking only works if runs happen consistently.

Telling failures apart

Not every empty result means the same thing, and the fix differs by cause:

  • Parsing failure: the HTML arrived but the selectors did not match. Check whether the page structure changed.
  • Missing rendered content: the HTML arrived without the price because JavaScript had not run. Use a headless browser or an API that renders.
  • Rate limiting: the server is slowing or rejecting requests because too many arrived too quickly. Space requests out.
  • IP blocking: repeated requests from one address are being flagged. Rotate proxies and reduce pressure.
  • Transient network failure: a connection dropped or a page timed out. Retry with growing delays.

Logging which pages failed and why is more useful than silently skipping rows.

Frequently asked questions

What Python libraries are used for price scraping?

A common stack is requests for fetching pages and BeautifulSoup or lxml for parsing HTML. CSV or pandas are typical for storing results. For dynamic or protected sites, browser tools or scraping APIs are often added.

How often can I scrape prices from a website?

There is no universal rule. Frequency depends on how often prices change and how strict the site is. Scraping too aggressively can trigger rate limits or blocks, so spacing requests out and keeping a consistent schedule is safer for long-term tracking.

Why do price scraping scripts get blocked?

Blocks usually come from too many requests, missing headers, or trying to read dynamic pages without rendering. Sites detect unusual patterns, so basic scripts often fail unless retries, proxies, or rendering are added.

Can I scrape prices from Amazon using Python?

Technically yes, but it is not simple. Amazon has strong anti-bot protection, so basic scripts often get blocked. Reliable scraping usually requires proxies, proper headers, and sometimes dedicated APIs for handling Amazon pages.

Why use a product ID instead of the title to match snapshots?

Titles can change slightly between runs, which breaks matching. A product ID taken from the product URL stays stable, so the same item can be compared across daily files even when its title is edited.