Scraping Shopify Product Data with an AI Extractor
Shopify stores rarely share a layout: each theme arranges product pages differently, so CSS selectors that work on one shop often fail on the next. This guide takes a different route. It first pulls every product URL from the store's own product sitemap, then asks an AI extraction endpoint to return named fields from each page as structured JSON. The walkthrough uses Python for demonstration and calls a third-party AI extraction API (the provider's Python client is referenced in the supplied module); the same HTTP requests can be issued from any language. The sample store is gymshark.com, and the counts below are dated observations captured at the time of writing.
Step 1 — Collect every product URL from the sitemap
A Shopify storefront normally publishes a product sitemap at https://<domain-name>/sitemap_products_1.xml, and a sitemap index at https://<domain-name>/sitemap.xml that lists any additional sitemaps. Walking categories and pagination by hand is fragile because category names differ per store; the sitemap is the stable starting point.
Fetch the XML, parse it, and keep the text inside each <loc> element. The public XML sitemap sample (module code0001) shows the shape of a product sitemap entry, including the optional image block. The Python snippet in module code0002 uses requests and xmltodict to turn the sitemap into a dictionary and write one URL per line to a text file. For the sample store this produced 2,266 product URLs at the time of writing.
Step 2 — Extract structured fields with an AI query
Instead of writing per-field selectors, describe the fields you want and let the AI query return them. The extraction request in module code0003 (private source example) shows the pattern: pass an ai_selector to scope the region of the page, and an ai_extract_rules JSON object where each key names an output field and each value carries a description and a type.
The fields used in this walkthrough are:
- name — product name, string.
- description — product description, string.
- original_price — price before offers, number.
- offer_price — final price after offers; if no offer is running, return the original price, number.
- in_stock — string, not boolean, because different sizes can have different stock states.
- available_sizes — list of sizes offered.
A representative response (module code0004) for one product, GS Power Crop Tank, returned original_price 36 and offer_price 18, with in_stock listing XS, XXL and L while available_sizes covered XS through XXL. That gap between the two fields is the reason in_stock is kept as a string: a boolean would lose which sizes are actually purchasable.
For each product, save the slug, the original URL, the initial status code reported by the extraction API in its spb-initial-status-code response header, the resolved URL reported in its spb-resolved-url header, and the raw response text. Those headers are what let you tell a live listing apart from a removed or redirected one in the next step.
Step 3 — Keep only live listings and consolidate
When a product page has been removed or redirected to a collections page, the extracted JSON tends to be mostly null. Use two checks to decide whether a record is a real product:
- The initial status code is 200. A 302 (redirect) or 404 (not found) means the listing is no longer live.
- The original URL equals the resolved URL. A difference signals a redirect.
Module code0005 shows the consolidation loop. It reads each saved JSON file, skips records that fail either check, strips stray escape sequences and markdown fences that can appear in LLM output, then parses the remaining response and appends it to a list. For the sample store, 1,838 of the 2,266 URLs passed both checks at the time of writing, and the combined list was written to all-products.json.
A quick reader checklist before you trust the output:
- Every record has a status code of 200.
- The original and resolved URLs match.
- The parsed JSON contains the fields you asked for, not null placeholders.
- The final count is reported alongside the starting URL count so the drop is visible.
Why an AI extractor suits Shopify stores
The sitemap-plus-AI pattern is portable because the URL discovery step relies on Shopify's shared sitemap convention rather than on theme markup, and the extraction step relies on field descriptions rather than on CSS IDs or class names. That portability is a tradeoff, not a guarantee.
Selector-based scrapers are cheap and deterministic: once a selector matches, the same element is returned on every page of the same theme, and a mismatch fails loudly. Description-based extraction is more tolerant of markup churn, but it can also fail quietly — a renamed field, an ambiguous description, or a page region that the ai_selector does not cover can yield plausible-looking values that are wrong. Treat the output as a draft that needs validation, not as a verified record.
Concrete failure modes to watch for:
- A theme update moves the price block outside the ai_selector region, so prices come back null or stale.
- A description like "final price after offers" is interpreted differently on a page that shows a bundle price alongside a single-item price.
- A redirect to a collections page returns a mostly-null record that still parses as JSON, so a naive pipeline keeps it.
Because original_price and offer_price are captured as separate fields, the same output can later feed price-drop or offer monitoring, provided the live-vs-removed checks in Step 3 are applied first.
| Field | Type | What it captures | Reader note |
|---|---|---|---|
| name | string | Product name | Used as a human-readable label |
| description | string | Product description | May include size and material notes |
| original_price | number | Price before offers | Baseline for discount tracking |
| offer_price | number | Final price after offers; equals original when no offer runs | Compare with original_price to see the discount |
| in_stock | string | Sizes currently in stock | String, not boolean, because sizes differ |
| available_sizes | list | All sizes offered | Compare with in_stock to see sold-out sizes |
Field definitions reflect the extraction rules used in the walkthrough; the sample response is a single product captured at the time of writing.
Frequently asked questions
Why is in_stock a string instead of a boolean?
A single product page can show several sizes with different availability. A boolean would collapse that into one yes/no value, so the field is kept as a string listing the sizes that are actually in stock. The available_sizes list separately records every size offered, so you can compare the two to find sold-out sizes.
How do I tell a live product from a removed one?
Check the initial status code reported by the extraction API and compare the original URL with the resolved URL. A 200 status with matching URLs indicates a live listing. A 302 redirect or a 404 means the page is gone or has moved, and the extracted JSON for those records is usually mostly null.
Does this workflow require Python?
No. Python is used only for demonstration. The sitemap fetch and the AI extraction call are plain HTTP requests, so the same steps can be implemented in any language that can send requests and parse JSON.
{
"name": "GS Power Crop Tank",
"description": "Example post details.",
"original_price": 36,
"offer_price": 18,
"in_stock": "XS,XXL,L",
"available_sizes": ["XS", "S", "M", "L", "XL", "XXL"]
}