Skip to content

Extracting E-Commerce Product Data with Schema.org and Java

Product pages often carry structured markup that describes the item, its price and its availability. When that markup is present, a scraper can read the same fields across many stores without writing a new selector for every site. This guide shows a Java approach built on HtmlUnit and Jackson, explains the three common Schema.org formats, and maps each target field to the markup that carries it.

What you will need

The example uses HtmlUnit to make the HTTP request and parse the DOM, and Jackson to serialize the extracted object to JSON. Add both dependencies to your Maven pom.xml. The exact dependency blocks are supplied as protected modules and should be used as-is.

Why Schema.org markup helps

Schema.org is a shared vocabulary that web pages can embed to describe their content. Search engines use it to understand whether a page is about a Product, an Article, a Review and so on. For a scraper, the benefit is portability: once you can read a given Schema type, the same extraction logic can work on any site that publishes that type, instead of relying on site-specific XPath or CSS selectors.

Three markup styles appear in practice:

  • JSON-LD, a script block containing a JSON object.
  • RDF-A, attributes such as vocab and typeof on HTML elements.
  • Microdata, attributes such as itemscope, itemtype and itemprop.

The worked example below uses Microdata from a product page. The same fields can appear in JSON-LD or RDF-A, so a production scraper should check which format a target page actually uses before choosing a parser.

Choosing a format and a parser

Use this checklist before writing extraction code:

  1. Inspect the page source for a script tag with type application/ld+json. If present, parse JSON first; it is usually the simplest path.
  2. If no JSON-LD is present, look for itemscope and itemtype attributes. That indicates Microdata.
  3. If you see vocab and typeof attributes, the page uses RDF-A.
  4. Confirm the itemtype points to a Product entity and that an offers block exists.
  5. Decide whether the page needs JavaScript to render the markup. If it does, a headless browser is required; if the markup is in the initial HTML, a lighter client is enough.

HtmlUnit sits between a raw HTTP client and a full headless browser: it parses HTML and can execute JavaScript, but is lighter than Selenium with Headless Chrome. For pages that rely heavily on frontend frameworks such as React or Vue.js, a headless Chrome approach is the safer choice.

Extracting the fields

Start with a small POJO that holds the fields you care about: price, name, SKU, image URL and currency. The supplied Product class module shows the shape. Then load the target URL with HtmlUnit, locate the Product node by its itemtype, and read the child nodes by itemprop.

The supplied extraction module performs these steps: it creates a WebClient, disables CSS and JavaScript because the Schema markup is already in the HTML, fetches the page, selects the element whose itemtype is the Schema.org Product URL, and then reads the image src, the price text, the product name, the currency content attribute and the SKU text. Finally, the Product object is serialized to JSON with Jackson and printed.

A field-to-markup map for the Microdata example:

  • Product name: span with itemprop name.
  • Price: span with itemprop price inside the offers block.
  • Currency: meta with itemprop priceCurrency, read from its content attribute.
  • SKU: span with itemprop sku.
  • Image URL: img src inside the Product node.
  • Availability: link with itemprop availability, pointing to a Schema.org condition URL.

This parser is intentionally minimal. It does not handle every field and it does not loop over multiple offers, which can appear on a single page. Treat it as a starting point and extend it for your own field list.

Avoiding blocks and rate limits

Once extraction works, the next concern is access. Sites use anti-bot measures for several reasons, including protecting performance from heavy automated traffic and stopping spam. Common mechanisms include request-rate limits per second, hour or day, per-IP limits, and behavioral analysis such as checking the timing between requests or whether one IP is making concurrent calls.

Proxies combined with rotating user agents are a common way to spread requests and reach rate-limited pages. Free proxy lists are usually a poor choice because they tend to be slow, unreliable and unclear about proxy location. A paid proxy service or a self-built pool is more dependable. Setting a proxy on HtmlUnit is a short configuration step, shown in the supplied proxy module.

Two failure modes look similar but need different fixes:

  • Missing markup: the page loads but no Product node is found. Check whether the site uses a different Schema format or no Schema at all.
  • Blocked request: the page returns a challenge, a rate-limit response or different content than a normal browser sees. Check request rate, IP reputation and headers before changing selectors.

Where this approach still struggles

Schema.org markup makes product extraction simpler than selector-by-selector scraping, but it does not cover every store. Some sites publish no Schema, some publish it only for certain page types, and some render it only after JavaScript runs. IP blocking and rate limits remain separate problems. A practical pipeline therefore combines format detection, a fallback selector strategy for pages without markup, request pacing, and a rendering option for JavaScript-heavy sites.

Frequently asked questions

Which Schema.org formats can carry product data?

JSON-LD, RDF-A and Microdata are the three common formats. JSON-LD places a JSON object in a script block, RDF-A uses vocab and typeof attributes, and Microdata uses itemscope, itemtype and itemprop attributes. The extraction example uses Microdata.

Why disable JavaScript in HtmlUnit for this scraper?

The Schema markup in the example is present in the initial HTML, so JavaScript execution is unnecessary. Disabling it makes the request lighter. Pages that render markup only after JavaScript runs need a different setup, such as a headless browser.

Does the example handle multiple offers?

No. The supplied parser reads a single offers block and does not loop over multiple offers, even though a page can contain more than one. Extend the parser if your target pages list several offers.

What should I check first if extraction returns nothing?

First confirm the page actually contains a Product node in one of the three Schema formats. If the markup is present but the request returns different content, the issue is likely blocking or rate limiting rather than a selector problem.

<script type="application/ld+json">
{
    "@context": "http://schema.org",
    "@type": "ItemList",
    "url": "http://multivarki.ru?filters%5Bprice%5D%5BLTE%5D=39600",
    "numberOfItems": "315",
    "itemListElement": [
        {
            "@type": "Product",
            "image": "http://img01.multivarki.ru.ru/c9/f1/a5fe6642-18d0-47ad-b038-6fca20f1c923.jpeg",
            "url": "http://multivarki.ru/brand_502/",
            "name": "Brand 502",
            "offers": {
                "@type": "Offer",
                "price": "4399 p."
            }
        },
        {
            "@type": "Product",
            "name": "..."
        }
    ]
}
</script>