Skip to content

Scraping WooCommerce Product Data at Scale

WooCommerce catalogues differ widely because each merchant can apply its own theme, custom fields, variations and bot defences. Some details sit in the initial HTML; others appear only after JavaScript runs. This guide shows how to choose fields, handle dynamic pages and build a repeatable extraction workflow.

What a WooCommerce product scraper does

A WooCommerce product scraper visits a store and collects catalogue information automatically, turning page content into structured records. It can work through large catalogues without manual copying, which makes it suitable for price checks, inventory monitoring and market research. Because merchants customise WordPress themes and plugins independently, a useful scraper must tolerate layout differences rather than assume one fixed markup pattern. When several platforms are in scope, a broader eCommerce scraping API can sit alongside WooCommerce-specific logic so each platform keeps its own parsing rules.

Fields worth capturing first

Start with the fields that support a decision. Titles identify items, regular and sale prices support comparison, and SKU connects a listing to inventory systems. Stock status shows availability, descriptions explain features, categories place the item in a hierarchy, and image URLs preserve the main and gallery visuals. For variation-heavy stores, also capture option values such as size or weight, because a single headline price may not describe every variant.

Why teams collect this data

Retailers use fresh catalogue data to react to competitor pricing while protecting margins. Researchers build datasets to spot feature gaps and emerging assortment patterns. Agencies and merchants may export records to CSV or Excel for catalogues, store migration or internal analytics. The common requirement is consistency: the same field should mean the same thing across pages, even when themes differ.

Static HTML or dynamic loading: check before you scale

Many WooCommerce pages are server-rendered, so titles and prices are present in the HTML returned by WordPress. Other stores load content through AJAX or the WooCommerce REST API after the initial response. In that case, a plain HTML fetch can miss prices, stock or variation details. Check whether content appears in the initial document or only after rendering; if it is dynamic, the scraper needs JavaScript rendering and should inspect the rendered DOM or the underlying JSON responses. Static pages can skip rendering to save resources, while dynamic pages should enable it.

A practical extraction workflow

First, prepare the environment and obtain an API key from the configured service. Then describe the desired output in plain language instead of relying only on brittle CSS or XPath selectors. The protected extraction-rules module defines product fields with types and descriptions, including name, price, stock status, description, categories and featured image. A request can combine a natural-language query with those rules, enable JavaScript rendering and wait for dynamic content. Treat non-200 responses and JSON decode failures as per-URL errors, record the source URL with each product, and continue with the remaining items.

Scaling from categories to a full catalogue

For a whole store, work in three passes. Discover main category URLs from the storefront, then collect individual product URLs from each category, and finally extract product details page by page. Filter category links to category paths and product links to product paths, remove duplicates while preserving order, and add a short randomised pause between product requests. Save the collected records as JSON so they can later be converted to CSV or Excel. The protected scale module contains the exact category, link and product rule contracts for this workflow.

Install the documented client, set API_KEY to a key valid for its service endpoint, and set STORE_URL to the WooCommerce store to inspect. This independent example keeps the documented AI rules and example-site selectors .e-n-menu-heading and .product-grid: check or adjust those selectors for your chosen store. It discovers categories and products, deduplicates URLs, skips HTTP/JSON failures and exports products.json. Successful product requests are spaced by a random 2.0–3.5 seconds.

import json
import os
import random
import time
from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key=os.environ["API_KEY"])
store_url = os.environ["STORE_URL"]
category_rules = {'category_urls': {'type': 'list',
                   'description': 'URLs of all main WooCommerce product category pages linked on this store. Only include '
                                  'category pages containing /product-category/. Do not include product pages, cart pages, '
                                  'account pages, or other navigation links.'}}
link_rules = {'product_urls': {'type': 'list',
                  'description': 'URLs of all individual product pages listed on this category page. Only include product detail '
                                 'URLs containing /product/. Do not include category pages or navigation links.'}}
product_data_rules = {'product_name': {'type': 'string', 'description': 'The full name of the meat or seafood product'},
 'price': {'type': 'string', 'description': 'The product price including the dollar sign, e.g. $49'},
 'stock_status': {'type': 'string', 'description': 'Whether the product is in stock or out of stock'},
 'full_description': {'type': 'string',
                      'description': 'The full product description including flavour notes, aging process, and butchering '
                                     'details'},
 'weight_options': {'type': 'list', 'description': 'List of available weight or size options the customer can choose from'},
 'meat_type': {'type': 'string', 'description': 'The type of meat or protein, e.g. Beef, Wagyu, Pork, Chicken, Seafood'},
 'cut': {'type': 'string', 'description': 'The specific cut of meat, e.g. Flat Iron, Ribeye, Filet Mignon, Porterhouse'},
 'breed_or_grade': {'type': 'string',
                    'description': 'The breed or quality grade of the meat, e.g. Angus, Wagyu, USDA Prime, Natural'},
 'aging': {'type': 'string', 'description': 'How long and how the product has been aged, e.g. "28 days"'},
 'categories': {'type': 'list', 'description': 'Breadcrumb or category path this product belongs to, e.g. ["Beef", "Steaks"]'},
 'featured_image_url': {'type': 'string', 'description': 'The URL of the main product photo'},
 'subscription_available': {'type': 'string',
                            'description': 'Whether a subscription or recurring delivery option is available for this product'},
 'shipping_info': {'type': 'string',
                   'description': 'Any shipping notes such as ships fresh, ships frozen, or free shipping threshold'}}

def read_result(url, query, rules, selector=None):
    options = {"ai_query": query, "ai_extract_rules": rules, "render_js": True, "wait": "3000"}
    if selector is not None:
        options["ai_selector"] = selector
    response = client.get(url, params=options)
    if response.status_code != 200:
        print("Skipped HTTP", response.status_code, url)
        return None
    try:
        return json.loads(response.content)
    except json.JSONDecodeError:
        print("Skipped non-JSON result", url)
        return None

def list_values(value):
    values = value if isinstance(value, list) else value.split(",") if isinstance(value, str) else []
    return list(dict.fromkeys(str(item).strip() for item in values if str(item).strip()))

category_data = read_result(store_url, "Extract all main product category URLs from this WooCommerce store", category_rules, ".e-n-menu-heading") or {}
categories = [url for url in list_values(category_data.get("category_urls", []))
              if "/product-category/" in url and "/product/" not in url
              and len(url.rstrip("/").split("/product-category/")[1].split("/")) == 1]
products = []
seen_products = set()
for category in categories:
    link_data = read_result(category, "Extract all product detail URLs from this category page", link_rules, ".product-grid") or {}
    for url in list_values(link_data.get("product_urls", [])):
        if "/product/" not in url or "/product-category/" in url or url in seen_products:
            continue
        seen_products.add(url)
        product = read_result(url, "Extract all product details from this product page", product_data_rules)
        if isinstance(product, dict):
            product["source_url"] = url
            products.append(product)
        time.sleep(random.uniform(2.0, 3.5))
with open("products.json", "w", encoding="utf-8") as file:
    json.dump(products, file, ensure_ascii=False, indent=2)
print("Exported", len(products), "products to products.json")

WooCommerce compared with Shopify

WooCommerce flexibility is both the main difficulty and the main advantage. Themes and plugins change layouts, so selectors that work on one store may fail on another, but product information is often visible in the source. Shopify, by contrast, uses a more standardised hosted structure, which can make cross-store extraction more consistent, although aggressive rate limiting may apply. Plan for WooCommerce by keeping extraction rules semantic and testing them against several themes rather than tuning only to one example store.

Pre-flight checklist for a reliable run

Before a large run, test one product page and inspect every field. Confirm that prices include the expected currency, stock values use a consistent vocabulary, image URLs resolve, and variation options are not collapsed into a single value. Then test one category end to end, check for duplicate or navigation links, and only then run the full catalogue. Keep a per-URL error log so failed pages can be retried without repeating successful work.

Product field map
FieldWhy it mattersWatch out for
Product titleIdentifies the listingTruncation or theme-specific headings
Regular and sale priceSupports competitor comparisonCurrency symbol and variant pricing
SKULinks listing to inventoryMissing or non-unique identifiers
Stock statusShows availabilityDynamic loading after page render
DescriptionExplains featuresLong text and custom fields
CategoriesPreserves catalogue hierarchyBreadcrumb versus taxonomy differences
Image URLsReuses main and gallery mediaLazy-loaded or relative URLs

Field map derived from the source's common extraction list; exact protected rule definitions remain in code0003.

Static versus dynamic product pages
Store typeLikely data locationScraping implication
Server-renderedInitial HTMLHTML parsing may be sufficient
AJAX or REST-drivenRendered DOM or JSON responseEnable JavaScript rendering and inspect dynamic content
Variation-heavyOption controls and variant dataCapture each option set, not only the default price

Diagnostic summary based on the source's description of server-rendered and dynamically loaded WooCommerce pages.

Frequently asked questions

Is it legal to scrape WooCommerce product data?

Publicly visible details such as names and prices are generally treated as usable for market analysis, but the store's terms should be checked first. Personal data and copyrighted images should not be reused commercially without permission or a licence.

Can WooCommerce block scrapers?

Yes. Security plugins and web application firewalls can block automated visits. Rotating residential proxies, varied headers and a human-like pace help a scraper continue consistently.

What is the best data to scrape from WooCommerce stores?

Price, stock level and SKU usually carry the most analytical value because they reveal competitor positioning and inventory health. Titles and descriptions matter when building a comparison catalogue or a new storefront.

How often should WooCommerce product data be scraped?

Cadence follows how fast the market changes. Many teams refresh price and stock every 24 hours, while stores running frequent promotions may need checks every few hours.