Skip to content

Scraper Doesn't See the Data the Browser Shows: Causes and Fixes

A browser and a plain HTML parser can receive the same URL and still produce different results.

OUTPUT FORMATSHTMLJSONMarkdown

If your extraction returns empty fields while the browser view looks complete, the usual reason is that the content is assembled by client-side JavaScript after the initial response. Parsers such as BeautifulSoup and lxml read the markup they are given; they do not run JavaScript. This guide explains the distinction and gives you two practical routes to the data.

Why the browser sees more than your parser

When you open a page in a browser, the browser downloads the HTML, executes any scripts, and then paints the result. An HTML parser stops after the download. If the visible content was inserted by JavaScript, the parser's tree will not contain it, so selectors that work in DevTools can return nothing in your script. Before changing tools, confirm the cause: fetch the raw response and search it for the text you expect. If the text is absent from the raw HTML but present in the rendered page, you are dealing with client-side generation rather than a broken selector.

Two routes to the missing data

There are two established approaches. The first is to use a browser automation framework such as Selenium or Puppeteer, let the page execute its JavaScript, and then extract from the rendered DOM. The second is to look inside the page's script tags: often the data is present as a JavaScript variable and is rendered into the page after load, so you can read the variable directly instead of waiting for rendering. Each route has trade-offs in speed, complexity, and fragility.

Decision checklist

Use this checklist to pick a route and to rule out simpler failures first.

Example: finding a JSON payload in a script tag

Suppose a product page shows a price that your parser cannot find. View the raw HTML and search for the price text or for a likely variable name. If you find a script block containing a JSON object, parse the text between the script tags as JSON and read the field you need. This avoids launching a browser for every request. If the price is not in any script block, move to the rendering route.

Rendering vs. script extraction
ApproachWhat it doesWhen it fitsMain trade-off
Browser automation (Selenium, Puppeteer)Executes page JavaScript, then extracts from the rendered DOMData is generated after load and not present in script tagsSlower and heavier than parsing static HTML
Script tag extractionReads data embedded in JavaScript variables inside <script> tagsData is present in the raw HTML inside a script blockDepends on the page's internal variable structure

Approach suitability depends on where the target data actually appears in the page source; verify by inspecting the raw response.

Frequently asked questions

Why does my scraper return empty results when the browser shows content?

The content is likely generated by client-side JavaScript after the initial HTML loads. Parsers like BeautifulSoup and lxml do not execute JavaScript, so they only see the original markup.

How do I make my scraper execute JavaScript?

Use a browser automation framework such as Selenium or Puppeteer. These tools load the page in a browser engine, run its scripts, and let you extract from the rendered DOM.

Can I avoid a browser and still get the data?

Sometimes. Check the raw HTML for <script> tags containing the data as JavaScript variables. If the values are there, you can parse them directly without rendering the page.