Skip to content

Build a Python Price Scraper: Extract Product Prices to CSV

A price scraper turns product listings into structured rows you can compare, chart, and export. This guide walks through a Python workflow that renders JavaScript, extracts product names and current prices, and saves results to CSV. It uses a hosted HTML API so you can focus on selectors and data quality rather than browser orchestration.

What a price scraper does

Think of a price scraper as a narrow-purpose collector: it reads a product page and returns the price as a field you can store, not a number buried in a paragraph. Currency, discount labels, and availability often ride along in the same record.

The value shows up in three recurring jobs. Retail teams watch rival listings to see where their own prices sit. Deal-watchers fire an alert when a watched item crosses a threshold. Brands check whether resellers are honoring a Minimum Advertised Price (MAP) agreement.

What makes this harder than a plain page fetch is that price figures are frequently injected after the first HTML arrives, and stores actively discourage automated traffic. A scraper that only grabs static markup will miss the number you actually want, so rendering and request management become part of the design rather than an afterthought.

Why use a hosted HTML API

A hosted HTML API shifts the browser and network plumbing off your machine. Instead of running a headless browser, maintaining a proxy pool, and wiring up CAPTCHA handling yourself, you send a request and receive rendered HTML or structured JSON back.

The Python SDK accepts JavaScript rendering instructions inline, so pages that build their prices after load still return complete markup. Proxy selection and header management are driven by the parameters you pass and the service's defaults, which reduces the amount of infrastructure code in your project.

The practical benefit is a smaller surface to maintain: your script describes what to wait for and which fields to pull, while the service handles the rendering and request logistics. That keeps the pipeline easier to reason about when selectors change or a site tightens its defenses.

Challenges in price scraping

Four problems tend to surface, and each produces a different symptom. Knowing which one you are looking at saves debugging time.

Late-loaded prices. When a storefront fills in prices with JavaScript after the initial response, a plain fetch returns a page whose price slot is empty. Rendering the page before extraction is the fix.

Blocking and challenges. Repeated requests can trigger IP blocks or CAPTCHA pages. Rotating proxies and managed headers make traffic look less mechanical; built-in challenge handling reduces how often a run stalls.

Pacing. Hitting a site too fast invites throttling. Adjusting connection and rendering parameters, adding deliberate delays, and retrying with backoff keeps throughput steady without tripping server-side limits.

Messy values. Symbols, thousands separators, and inconsistent formatting make prices hard to compare. Normalizing the string into a number early, before it reaches your analysis step, prevents silent mismatches later.

Build the price scraper step by step

The walkthrough below targets Target search results for smart watches. It installs the SDK and pandas, sends a rendered request, extracts product name and price with CSS selectors, and exports results to CSV.

Step 1: Install and set up. Use Python 3.6 or newer and install the SDK and pandas. The SDK supports JavaScript rendering and proxy management; pandas structures and exports the data. {{OWNER_API_KEY_INSTRUCTIONS}} {{OWNER_FREE_TRIAL_TERMS}}

Step 2: Send the first request. Import the client and pandas, create a client with your API key, and define a function. A js_scenario dictionary holds JavaScript rendering instructions; the example waits before extraction. The GET call passes js_scenario, premium_proxy, and country_code. JavaScript rendering is enabled by default through a hidden render_js parameter.

Step 3: Parse product price. Use the extract_rules parameter to parse prices from raw HTML with little code. Inspect the page with developer tools to find CSS selectors. The example targets a product card container, then the product title and current price within it. The response is converted to JSON and loaded into a pandas DataFrame.

Step 4: Handle blocks and CAPTCHAs. A hosted API can rotate proxies, manage headers, and handle CAPTCHA challenges on your behalf. Test the premium_proxy and stealth_proxy parameters to see which mode suits a given site. {{OWNER_SUPPORT_COMMITMENTS}}

Python example

url = 'https://www.target.com/s?searchTerm=apple+watch'

Field and control map

The extraction rules define a Search_Result list. Each item contains a Name field and a Price field. The main selector identifies the product card container; the Name selector targets the product title link, and the Price selector targets the current-price span.

Request parameters used in the example:

  • js_scenario: JavaScript rendering instructions, such as a wait before extraction.
  • extract_rules: CSS selector rules that return structured JSON.
  • premium_proxy or stealth_proxy: proxy modes for tougher anti-bot defenses.
  • country_code: sets the target country for the request.

If the response is raw HTML instead of JSON, check whether extract_rules was passed. If the JSON is empty, the selectors likely no longer match the page structure.

Extract multiple product prices

To scale beyond one page, replace the single URL with a list of search URLs. The example uses three Target searches: apple watch, garmin watch, and samsung watch.

Create an empty list before the loop. For each URL, send the GET request with js_scenario, extract_rules, stealth_proxy, and country_code. Convert each JSON response to a DataFrame and append it to the list. After the loop, concatenate the DataFrames with pandas concat and ignore_index=True. Export the combined frame to Price_extraction.csv with index=False.

In the captured example, three pages produced 72 rows of price data, 24 per page. Treat that count as a dated reference from the original run; actual row counts depend on the live page and selectors.

Python example

url_list = ['https://www.target.com/s?searchTerm=apple+watch', 'https://www.target.com/s?searchTerm=garmin+watch', 'https://www.target.com/s?searchTerm=samsung+watch']

Decision checklist before you run

Use this checklist to avoid common failures:

  1. Confirm the page loads prices with JavaScript. If it does, keep js_scenario and JavaScript rendering enabled.
  2. Inspect the product card container and confirm the Name and Price selectors still match. Selector drift is the most common cause of empty JSON.
  3. Decide whether you need premium_proxy or stealth_proxy. Start with the default; add proxy modes when you see blocks or CAPTCHAs.
  4. Set country_code to the market you are monitoring so prices and availability match the right storefront.
  5. Validate the JSON shape before building DataFrames. If Search_Result is missing, the extraction rules did not match.
  6. Export to CSV only after checking row counts and price formatting.

Failure distinction: a block or CAPTCHA usually returns an error page or challenge HTML; a selector mismatch returns valid HTML but empty or missing fields; a JavaScript timing issue returns HTML without the dynamically loaded price.

Build your own price tracker

A price tracker no longer requires manual IP changes, CAPTCHA handling, or browser configuration. A single API request can return rendered HTML or structured data for competitor analysis, product comparison, or market tracking.

Start with a small set of URLs, confirm the selectors, and export to CSV. Then add more pages or connect the output to a spreadsheet for ongoing updates. {{OWNER_PRICING_URL}}

Frequently asked questions

Is it legal to build a price scraper?

Scraping publicly available data is generally legal, but e-commerce platforms may treat automated traffic differently. Avoid collecting personal or non-public information, and review the site's terms and applicable data protection laws before you scrape.

How do I scrape dynamic pricing pages?

Use JavaScript rendering so asynchronously loaded prices appear in the final HTML. A hosted API can execute JavaScript automatically, and proxy rotation helps when the site limits repeated requests.

Can I scrape Amazon for product prices?

Amazon uses strong anti-bot systems, so scraping requires robust proxy rotation and CAPTCHA handling. A hosted API with built-in proxy and CAPTCHA support can make access more consistent, but you should still expect variability and review Amazon's terms.

How do I avoid getting blocked while scraping prices?

Enable premium_proxy or stealth_proxy parameters on GET requests. These settings simulate natural browser traffic and help bypass rate limits and CAPTCHAs. If access still fails, reduce request frequency and review the site's restrictions.