Skip to content

Scrape Amazon Search Results with Python: API and DIY Workflow

Amazon search pages are a practical source of product identity, pricing, and social-proof signals, but a plain HTTP request often returns a CAPTCHA, a rate-limit response, or HTML that is missing dynamically loaded content. This guide shows a Python workflow that sends a search query to a managed Amazon scraping endpoint and receives typed JSON, then compares that route with a do-it-yourself requests and BeautifulSoup scraper. The goal is a repeatable pipeline: query in, product rows out, saved as CSV.

What the workflow produces

The managed route returns a JSON object whose products key holds a list of product records. Each record can include ASIN, title, price, currency, rating, review count, Prime eligibility, Amazon's Choice and sponsored flags, best-seller status, strikethrough price, sales-volume text, and shipping information. The example keeps a smaller set: ASIN, title, price, currency, rating, reviews_count, is_prime, and a product URL built from the ASIN. That subset is enough to populate a product database or a pricing dashboard, and the ASIN is the stable key for later product-page lookups.

Prerequisites and setup

You need Python 3.10 or newer, the requests library, beautifulsoup4, and an API key for the managed scraping service. Install the two Python packages together, and create a virtual environment if you want to keep the experiment isolated. The guide deliberately uses requests for the API calls as well, so the comparison with the DIY scraper stays on the same HTTP client.

Step 1: Create the scraping function

The core function sends a GET request to the Amazon search endpoint with api_key and query as the only required parameters. The example also passes domain, sort_by, and start_page explicitly so the available controls are visible. Call raise_for_status() so HTTP failures surface immediately, then return the parsed JSON. The endpoint path and parameter names are fixed by the service contract; keep them exactly as supplied in the accompanying code module.

Step 2: Sorting and pagination

The sort_by parameter accepts featured (Amazon's default ranking), price_low_to_high, price_high_to_low, average_review, most_recent, and bestsellers. The start_page parameter selects which results page to process. To collect several pages, loop over page numbers and extend a combined list. A dedicated keyword scraper exists for keyword-based collection, and an ASIN scraper handles product-specific lookups.

Step 3: Choose the fields you keep

The API returns more fields than most pipelines need. The example's extraction function maps each item to a compact dictionary and supplies defaults for currency (USD) and is_prime (false). The product URL is assembled as https://www.amazon.com/dp/{asin}. If you need the full record, keep the additional flags and shipping text; if you only need pricing, drop the social-proof fields. The field map below shows the decision.

Step 4: Location and dynamic content

Amazon serves different results depending on the request origin. A search for a power adapter from the US shows 110V products, while the same search from Germany shows 230V ones. The domain parameter selects the marketplace: com for the US, de for Germany, co.jp for Japan, and so on. The managed service supports more than twenty marketplaces and renders JavaScript internally, which matters because a plain HTTP request can return incomplete HTML when Amazon loads content dynamically. For extraction scenarios beyond fixed fields, AI extraction rules can describe the desired output in natural language.

Step 5: Send the request and handle failures

The request itself is a standard HTTP GET. Amazon pages can be slow and anti-bot systems add latency, so set a generous timeout. Distinguish the failure modes: a 429 means rate limiting, a CAPTCHA page means the request was flagged as automated, a 503 or redirect to a bot-detection screen means the request was blocked, and incomplete HTML usually means JavaScript rendering was missing. The example catches HTTPError and prints a rate-limit message for 429 while re-raising other status errors.

Step 6: Parse and structure the response

Because the endpoint returns JSON, parsing is a dictionary lookup: read the products key from the response. The values are already typed, so there is no need to strip dollar signs or parse rating strings such as "4.5 out of 5 stars". This is the main structural difference from the DIY route, where every field must be extracted from HTML and converted.

Step 7: Save the results to CSV

The save function writes the list of product dictionaries to a CSV file using the keys of the first record as the header. It returns early with a message if the list is empty, and prints the number of saved rows. Running the script with "yoyos" as the query produces rows such as a Duncan Toys Limelight LED Yo-Yo at 9.99 USD with a 4.3 rating and 6,900 reviews, and a Duncan Toys Imperial Yo-Yo six-pack at 25.99 USD with a 4.6 rating and 55 reviews. The ASIN column links each row back to its product page.

DIY scraper vs managed API: a decision checklist

The DIY route gives full control and avoids API costs and vendor dependency, but you take on proxy rotation, CAPTCHA solving, realistic headers and cookie handling, and CSS selectors that can change without notice. The example DIY scraper sends a browser-like User-Agent and Accept-Language header, requests the Amazon search page, selects elements with data-component-type="s-search-result", and pulls title, price, and rating from nested selectors. It may work on a good day and return a CAPTCHA, a 503, or a bot-detection redirect on a bad day. Use this checklist to decide:

  • Do you need more than a few hundred requests? A DIY scraper tends to become unstable at that scale.
  • Can you absorb the engineering time for proxy rotation and CAPTCHA solving? If not, a managed API is usually cheaper than the hours spent maintaining a home-grown scraper.
  • Do you need typed JSON rather than HTML parsing? The API returns typed fields directly.
  • Do you need multiple marketplaces? The API exposes a domain parameter; DIY requires changing the domain and adjusting proxy location.
  • Do you need JavaScript-rendered content? The managed service renders it internally; a plain HTTP client does not.

Field map: what the example keeps and why

The full product record and the compact record differ by design. Use this map to decide which fields your pipeline needs:

  • asin: kept; stable product identifier and the basis for the product URL.
  • title: kept; human-readable product name.
  • price: kept; current price as a number.
  • currency: kept with a USD default; needed for multi-marketplace data.
  • rating: kept; average star rating as a number.
  • reviews_count: kept; total review count as a number.
  • is_prime: kept with a false default; useful for shipping and eligibility filters.
  • url: derived from the ASIN; convenient for linking back to the product page.
  • is_amazons_choice, is_sponsored, best_seller: available but dropped in the example; keep them if you analyze badges or ad placement.
  • price_strikethrough: available but dropped; keep it for discount analysis.
  • sales_volume: available but dropped; keep it for demand signals.
  • shipping_information: available but dropped; keep it for delivery-time analysis.

What you can extract from Amazon search results

Search results carry four categories of data. Product identity includes title, ASIN, URL, thumbnail and main image URL, and brand or manufacturer. Pricing includes current, list, and strikethrough price, currency, Prime eligibility and shipping details, and deal or coupon indicators. Social proof includes average rating, total review count, badges, and estimated sales volume. Listing metadata includes search position, sponsored flag, and availability or stock status. Most use cases map to one category: price monitoring needs pricing data, and competitor monitoring needs social proof.

Limitations to plan around

Amazon personalizes results based on browsing history, location, and purchases, so two users can see different results for the same query. A managed API reduces this variation but does not eliminate it. Amazon also caps search results at around 20 pages, so deep pagination is not available. Prices change frequently, so accurate price tracking requires scraping the same pages often. Plan your refresh cadence and page depth around these limits rather than assuming a complete catalog snapshot.

Common technical challenges

When Amazon detects automated traffic, it presents a CAPTCHA that persists across retries and must eventually be solved. Amazon also runs behavioral analysis on request timing, header ordering, and similar signals, which is hard to imitate without a full headless browser. A fresh session with no cookies is itself a bot signal, because real browsers accumulate and update cookies across requests. Scrapers that do not manage a cookie jar are easy to fingerprint and block. The managed service handles CAPTCHA solving and rendering behind the scenes.

Alternative: describe the extraction in natural language

Traditional scraping relies on predefined extraction rules. An AI extraction route lets you describe the desired output in natural language and lets the model map it to the page content. The example sends a URL, enables JavaScript rendering, and passes an ai_extract_rules parameter describing a list of products with name, price, and rating. This is useful when the page structure is unfamiliar or when you want a quick prototype before committing to fixed selectors.

DIY scraper vs managed API decision checklist
ConsiderationDIY (requests + BeautifulSoup)Managed API
ControlFull control over requests and parsingTrades control for reliability
CostNo API cost, but proxy and CAPTCHA-solver costs may applyAPI cost, often lower than engineering hours at scale
MaintenanceYou maintain proxies, headers, cookies, and CSS selectorsService handles proxies, CAPTCHAs, and rendering
OutputHTML that you parse and convertTyped JSON
ScaleWorks for quick experiments; unstable beyond a few hundred requestsBuilt for consistency at scale
MarketplacesChange domain and adjust proxy locationPass a domain parameter

Comparison reflects the captured guide's stated trade-offs; verify current service capabilities and pricing before relying on them.

Product field map: full record vs example subset
FieldType in exampleKept in exampleUse
asinstringYesStable product identifier and URL basis
titlestringYesProduct name
pricenumberYesCurrent price
currencystring (default USD)YesMulti-marketplace pricing
ratingnumberYesAverage star rating
reviews_countnumberYesReview volume
is_primeboolean (default false)YesShipping eligibility filter
urlstring (derived)YesLink to product page
is_amazons_choicebooleanNoBadge analysis
is_sponsoredbooleanNoAd placement analysis
best_sellerbooleanNoBadge analysis
price_strikethroughnumberNoDiscount analysis
sales_volumestringNoDemand signal
shipping_informationstringNoDelivery-time analysis

Field types and defaults are taken from the captured example record and extraction function; the live API may return additional fields.

Frequently asked questions

Does Amazon's robots.txt allow web scraping?

Partially. Amazon's robots.txt disallows many paths, including account pages, cart, and internal APIs, but product pages and search results are generally not blocked. Respect the disallowed paths and do not treat robots.txt as blanket permission.

Is it legal to scrape Amazon data?

Scraping publicly available data is generally legal under US law, but Amazon's Terms of Service prohibit automated access. The practical risk depends on your scale and use case. For a commercial product that relies on scraped data, consult a lawyer.

Can you scrape Amazon from different country domains?

Yes. A manual scraper must change the Amazon domain and adjust the proxy location. An API-based solution usually accepts an additional configuration parameter for the domain, such as com, de, or co.jp.

What's the best open source tool for scraping Amazon data?

Any HTTP client can perform the basic request, but the details matter: you still need proxy infrastructure and CAPTCHA solving. The choice depends on how much of that infrastructure you are prepared to build and maintain.

How do I scrape Amazon search results in real-time?

You need a headless browser with JavaScript rendering, fast proxies, and efficient parallelization, or a managed service that handles those pieces behind the scenes.

Are official Amazon APIs different than scraping APIs?

Yes. Official APIs are usually the best choice when they expose the data you need. If they do not, scraping is the fallback. Scraping APIs are not designed to mimic official APIs; they serve a different purpose, which is extracting data from pages.

{
    "asin": "B07GSXV39K",
    "title": "MAGICYOYO V3 Yoyo for Kids 8-12 or Above, Responsive Yoyo Professional with Dual Function",
    "price": 17.99,
    "currency": "USD",
    "rating": 4.6,
    "reviews_count": 9400,
    "is_prime": false,
    "is_amazons_choice": false,
    "is_sponsored": false,
    "best_seller": false,
    "price_strikethrough": 21.99,
    "sales_volume": "1K+ bought in past month",
    "shipping_information": "FREE delivery Mon, Apr 13 on $35 of items shipped by Amazon"
}