Scraping WooCommerce Product Data at Scale
WooCommerce catalogues differ widely because each merchant can apply its own theme, custom fields, variations and bot defences. Some details sit in the initial HTML; others appear only after JavaScript runs. This guide shows how to choose fields, handle dynamic pages and build a repeatable extraction workflow.
What a WooCommerce product scraper does
A WooCommerce product scraper visits a store and collects catalogue information automatically, turning page content into structured records. It can work through large catalogues without manual copying, which makes it suitable for price checks, inventory monitoring and market research. Because merchants customise WordPress themes and plugins independently, a useful scraper must tolerate layout differences rather than assume one fixed markup pattern. When several platforms are in scope, a broader eCommerce scraping API can sit alongside WooCommerce-specific logic so each platform keeps its own parsing rules.
Fields worth capturing first
Start with the fields that support a decision. Titles identify items, regular and sale prices support comparison, and SKU connects a listing to inventory systems. Stock status shows availability, descriptions explain features, categories place the item in a hierarchy, and image URLs preserve the main and gallery visuals. For variation-heavy stores, also capture option values such as size or weight, because a single headline price may not describe every variant.
Why teams collect this data
Retailers use fresh catalogue data to react to competitor pricing while protecting margins. Researchers build datasets to spot feature gaps and emerging assortment patterns. Agencies and merchants may export records to CSV or Excel for catalogues, store migration or internal analytics. The common requirement is consistency: the same field should mean the same thing across pages, even when themes differ.
Static HTML or dynamic loading: check before you scale
Many WooCommerce pages are server-rendered, so titles and prices are present in the HTML returned by WordPress. Other stores load content through AJAX or the WooCommerce REST API after the initial response. In that case, a plain HTML fetch can miss prices, stock or variation details. Check whether content appears in the initial document or only after rendering; if it is dynamic, the scraper needs JavaScript rendering and should inspect the rendered DOM or the underlying JSON responses. Static pages can skip rendering to save resources, while dynamic pages should enable it.
A practical extraction workflow
First, prepare the environment and obtain an API key from the configured service. Then describe the desired output in plain language instead of relying only on brittle CSS or XPath selectors. The protected extraction-rules module defines product fields with types and descriptions, including name, price, stock status, description, categories and featured image. A request can combine a natural-language query with those rules, enable JavaScript rendering and wait for dynamic content. Treat non-200 responses and JSON decode failures as per-URL errors, record the source URL with each product, and continue with the remaining items.
Scaling from categories to a full catalogue
For a whole store, work in three passes. Discover main category URLs from the storefront, then collect individual product URLs from each category, and finally extract product details page by page. Filter category links to category paths and product links to product paths, remove duplicates while preserving order, and add a short randomised pause between product requests. Save the collected records as JSON so they can later be converted to CSV or Excel. The protected scale module contains the exact category, link and product rule contracts for this workflow.
Install the documented client, set API_KEY to a key valid for its service endpoint, and set STORE_URL to the WooCommerce store to inspect. This independent example keeps the documented AI rules and example-site selectors .e-n-menu-heading and .product-grid: check or adjust those selectors for your chosen store. It discovers categories and products, deduplicates URLs, skips HTTP/JSON failures and exports products.json. Successful product requests are spaced by a random 2.0–3.5 seconds.
import json
import os
import random
import time
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key=os.environ["API_KEY"])
store_url = os.environ["STORE_URL"]
category_rules = {'category_urls': {'type': 'list',
'description': 'URLs of all main WooCommerce product category pages linked on this store. Only include '
'category pages containing /product-category/. Do not include product pages, cart pages, '
'account pages, or other navigation links.'}}
link_rules = {'product_urls': {'type': 'list',
'description': 'URLs of all individual product pages listed on this category page. Only include product detail '
'URLs containing /product/. Do not include category pages or navigation links.'}}
product_data_rules = {'product_name': {'type': 'string', 'description': 'The full name of the meat or seafood product'},
'price': {'type': 'string', 'description': 'The product price including the dollar sign, e.g. $49'},
'stock_status': {'type': 'string', 'description': 'Whether the product is in stock or out of stock'},
'full_description': {'type': 'string',
'description': 'The full product description including flavour notes, aging process, and butchering '
'details'},
'weight_options': {'type': 'list', 'description': 'List of available weight or size options the customer can choose from'},
'meat_type': {'type': 'string', 'description': 'The type of meat or protein, e.g. Beef, Wagyu, Pork, Chicken, Seafood'},
'cut': {'type': 'string', 'description': 'The specific cut of meat, e.g. Flat Iron, Ribeye, Filet Mignon, Porterhouse'},
'breed_or_grade': {'type': 'string',
'description': 'The breed or quality grade of the meat, e.g. Angus, Wagyu, USDA Prime, Natural'},
'aging': {'type': 'string', 'description': 'How long and how the product has been aged, e.g. "28 days"'},
'categories': {'type': 'list', 'description': 'Breadcrumb or category path this product belongs to, e.g. ["Beef", "Steaks"]'},
'featured_image_url': {'type': 'string', 'description': 'The URL of the main product photo'},
'subscription_available': {'type': 'string',
'description': 'Whether a subscription or recurring delivery option is available for this product'},
'shipping_info': {'type': 'string',
'description': 'Any shipping notes such as ships fresh, ships frozen, or free shipping threshold'}}
def read_result(url, query, rules, selector=None):
options = {"ai_query": query, "ai_extract_rules": rules, "render_js": True, "wait": "3000"}
if selector is not None:
options["ai_selector"] = selector
response = client.get(url, params=options)
if response.status_code != 200:
print("Skipped HTTP", response.status_code, url)
return None
try:
return json.loads(response.content)
except json.JSONDecodeError:
print("Skipped non-JSON result", url)
return None
def list_values(value):
values = value if isinstance(value, list) else value.split(",") if isinstance(value, str) else []
return list(dict.fromkeys(str(item).strip() for item in values if str(item).strip()))
category_data = read_result(store_url, "Extract all main product category URLs from this WooCommerce store", category_rules, ".e-n-menu-heading") or {}
categories = [url for url in list_values(category_data.get("category_urls", []))
if "/product-category/" in url and "/product/" not in url
and len(url.rstrip("/").split("/product-category/")[1].split("/")) == 1]
products = []
seen_products = set()
for category in categories:
link_data = read_result(category, "Extract all product detail URLs from this category page", link_rules, ".product-grid") or {}
for url in list_values(link_data.get("product_urls", [])):
if "/product/" not in url or "/product-category/" in url or url in seen_products:
continue
seen_products.add(url)
product = read_result(url, "Extract all product details from this product page", product_data_rules)
if isinstance(product, dict):
product["source_url"] = url
products.append(product)
time.sleep(random.uniform(2.0, 3.5))
with open("products.json", "w", encoding="utf-8") as file:
json.dump(products, file, ensure_ascii=False, indent=2)
print("Exported", len(products), "products to products.json")WooCommerce compared with Shopify
WooCommerce flexibility is both the main difficulty and the main advantage. Themes and plugins change layouts, so selectors that work on one store may fail on another, but product information is often visible in the source. Shopify, by contrast, uses a more standardised hosted structure, which can make cross-store extraction more consistent, although aggressive rate limiting may apply. Plan for WooCommerce by keeping extraction rules semantic and testing them against several themes rather than tuning only to one example store.
Pre-flight checklist for a reliable run
Before a large run, test one product page and inspect every field. Confirm that prices include the expected currency, stock values use a consistent vocabulary, image URLs resolve, and variation options are not collapsed into a single value. Then test one category end to end, check for duplicate or navigation links, and only then run the full catalogue. Keep a per-URL error log so failed pages can be retried without repeating successful work.
| Field | Why it matters | Watch out for |
|---|---|---|
| Product title | Identifies the listing | Truncation or theme-specific headings |
| Regular and sale price | Supports competitor comparison | Currency symbol and variant pricing |
| SKU | Links listing to inventory | Missing or non-unique identifiers |
| Stock status | Shows availability | Dynamic loading after page render |
| Description | Explains features | Long text and custom fields |
| Categories | Preserves catalogue hierarchy | Breadcrumb versus taxonomy differences |
| Image URLs | Reuses main and gallery media | Lazy-loaded or relative URLs |
Field map derived from the source's common extraction list; exact protected rule definitions remain in code0003.
| Store type | Likely data location | Scraping implication |
|---|---|---|
| Server-rendered | Initial HTML | HTML parsing may be sufficient |
| AJAX or REST-driven | Rendered DOM or JSON response | Enable JavaScript rendering and inspect dynamic content |
| Variation-heavy | Option controls and variant data | Capture each option set, not only the default price |
Diagnostic summary based on the source's description of server-rendered and dynamically loaded WooCommerce pages.
Frequently asked questions
Is it legal to scrape WooCommerce product data?
Publicly visible details such as names and prices are generally treated as usable for market analysis, but the store's terms should be checked first. Personal data and copyrighted images should not be reused commercially without permission or a licence.
Can WooCommerce block scrapers?
Yes. Security plugins and web application firewalls can block automated visits. Rotating residential proxies, varied headers and a human-like pace help a scraper continue consistently.
What is the best data to scrape from WooCommerce stores?
Price, stock level and SKU usually carry the most analytical value because they reveal competitor positioning and inventory health. Titles and descriptions matter when building a comparison catalogue or a new storefront.
How often should WooCommerce product data be scraped?
Cadence follows how fast the market changes. Many teams refresh price and stock every 24 hours, while stores running frequent promotions may need checks every few hours.