Scrape Amazon Reviews With Python
Amazon review pages are a practical source of customer language, complaint patterns and feature feedback. The hard part is not the analysis — it is getting the review fields out of a JavaScript-heavy product page in a repeatable shape. This guide walks through a Python workflow that targets a product by ASIN, extracts five review fields with CSS selectors, scrolls the page so lazy-loaded reviews appear, and writes the results to CSV. It also shows how to loop over several ASINs so you can compare products instead of studying one listing in isolation.
What the scraper actually collects
Each review record carries five fields, and one product-level field is added so the CSV stays readable when several ASINs are mixed together. The table below maps every field to the selector used in the script and to the meaning a reader should expect in the output. Treat selector names as the contract: if Amazon changes its markup, the field that stops populating tells you which selector to revisit.
Setup: Python, packages and API key
Install Python from the official distribution, then install two packages with pip: the client library that talks to the HTML API, and pandas for shaping and exporting the extracted rows. The client library is initialized once with your API key; that same client object is reused for every request in the script. Keep the key out of committed files and load it from an environment variable or a local secret store.
Defining the extract rules
The extract_rules dictionary is the parsing contract. It names a container selector for the review list, declares that the result is a list, and then maps each output field to a child selector inside each review item. The container is the review list element; the child selectors pick the reviewer name, the star rating text, the posted date, the headline and the body text. Because the rules live in one dictionary, you can adjust a single selector without rewriting the request logic.
Making lazy-loaded reviews appear
Amazon loads additional reviews as the page is scrolled, so a plain GET can return only the first batch. The js_scenario dictionary controls the built-in headless browser: it waits two seconds, scrolls to the bottom of the document, then waits again so the newly triggered content can render. The wait values are in milliseconds. JavaScript rendering is enabled by default, so you do not need to add render_js=true explicitly for this workflow — but if you disable it elsewhere, the scroll instructions will not have the effect you expect.
Sending the first request
A product page is addressed by its ASIN, which is the identifier you need when you are not logged into an Amazon account. The request passes the extract_rules and js_scenario dictionaries as parameters, sets country_code to route through a proxy in the target market, and enables automatic retries so a transient failure does not abort the run. The response body is JSON: the review list appears under the properties key, and the product title appears under product_title when that rule is included.
Turning the response into a CSV
Once the response is parsed, pandas converts the list of review dictionaries into a DataFrame and writes it to a CSV file with the index suppressed. Opening that file in a spreadsheet gives you one row per review and one column per field, which is the shape most downstream analysis expects. If the CSV is empty, check the HTTP status code first, then confirm that the properties key actually contained items — a successful request with zero reviews is a different failure from a rejected request.
Looping over several ASINs
Comparing products requires more than one listing. Change the function signature to accept a list of ASINs, create an empty list to collect results, and move the request inside a for loop. Add the product title rule to extract_rules so each product can be labeled. Because the review rows have five columns, insert a title row with the same five keys and empty strings for the review-specific fields; that keeps the DataFrame rectangular when the title row is appended before the reviews. A short print statement per ASIN reports the status code and how many reviews were extracted, which makes partial failures visible during a batch run.
Worked example: three floor lamps
The example ASIN list in the script contains three floor-lamp products. Running the loop produces a CSV where each product title appears once, followed by its reviews, and the same five review columns are reused across all three products. That layout lets you group by product title and compare average ratings or recurring complaint phrases without joining separate files. The ASINs are illustrative; substitute the products you actually need to study.
Rate limits, scope and legal care
Keep request rates reasonable, use retries rather than hammering a failing endpoint, and avoid extracting content that sits behind a login. Proxying is applied by default to the GET calls in this workflow, which helps keep requests from being flagged. Review the terms that apply to your jurisdiction and to Amazon before running a large collection job, and store only the fields you actually need.
Field and control map
Use this checklist when a field comes back empty or when you adapt the script to a new page layout.
| Output field | Selector | What it captures |
|---|---|---|
| name | .a-profile-name | Reviewer username |
| rating | .review-rating > span | Star rating text |
| date | .review-date | Date the review was posted |
| title | .review-title span:not([class]) | Review headline, plain spans only |
| content | .review-text | Main review body text |
| product_title | span.a-size-large.product-title-word-break | Product title added in the multi-ASIN version |
Selectors reflect the page structure used when the workflow was written; Amazon markup can change and a selector may need updating.
Frequently asked questions
Will Amazon block the scraper?
The HTML API manages proxies and request headers so the workflow can reach JavaScript-heavy review pages without you maintaining a proxy pool. Rate limits still apply, so keep request volume reasonable and rely on the retry setting for transient failures.
Do I need to install a headless browser?
No. JavaScript rendering runs server-side, so you do not install or maintain Puppeteer or Selenium. The js_scenario dictionary is how you tell that remote browser to wait and scroll before extraction.
How many reviews can one run collect?
The loop scales with the ASIN list you supply, subject to rate limits. Each ASIN contributes the reviews that render on its product page after the scroll instructions run, so the practical ceiling per product depends on how many reviews Amazon exposes there.
Can I target Amazon marketplaces outside the US?
Yes. Set country_code to the target market, for example de for Germany. If the localized request still returns the wrong page, adjust the URL in the GET call to the matching Amazon domain.
Why is my CSV empty even though the request succeeded?
A 200 status only means the page was fetched. If properties is missing or empty, the container selector did not match any review items — usually because the page had not finished rendering or the markup changed. Increase the wait instructions, confirm the scroll ran, and re-check the container selector before assuming the request failed.