Scraper Doesn't See the Data the Browser Shows: Causes and Fixes
A browser and a plain HTML parser can receive the same URL and still produce different results.
If your extraction returns empty fields while the browser view looks complete, the usual reason is that the content is assembled by client-side JavaScript after the initial response. Parsers such as BeautifulSoup and lxml read the markup they are given; they do not run JavaScript. This guide explains the distinction and gives you two practical routes to the data.
Why the browser sees more than your parser
When you open a page in a browser, the browser downloads the HTML, executes any scripts, and then paints the result. An HTML parser stops after the download. If the visible content was inserted by JavaScript, the parser's tree will not contain it, so selectors that work in DevTools can return nothing in your script. Before changing tools, confirm the cause: fetch the raw response and search it for the text you expect. If the text is absent from the raw HTML but present in the rendered page, you are dealing with client-side generation rather than a broken selector.
Two routes to the missing data
There are two established approaches. The first is to use a browser automation framework such as Selenium or Puppeteer, let the page execute its JavaScript, and then extract from the rendered DOM. The second is to look inside the page's script tags: often the data is present as a JavaScript variable and is rendered into the page after load, so you can read the variable directly instead of waiting for rendering. Each route has trade-offs in speed, complexity, and fragility.
Decision checklist
Use this checklist to pick a route and to rule out simpler failures first.
Example: finding a JSON payload in a script tag
Suppose a product page shows a price that your parser cannot find. View the raw HTML and search for the price text or for a likely variable name. If you find a script block containing a JSON object, parse the text between the script tags as JSON and read the field you need. This avoids launching a browser for every request. If the price is not in any script block, move to the rendering route.
| Approach | What it does | When it fits | Main trade-off |
|---|---|---|---|
| Browser automation (Selenium, Puppeteer) | Executes page JavaScript, then extracts from the rendered DOM | Data is generated after load and not present in script tags | Slower and heavier than parsing static HTML |
| Script tag extraction | Reads data embedded in JavaScript variables inside <script> tags | Data is present in the raw HTML inside a script block | Depends on the page's internal variable structure |
Approach suitability depends on where the target data actually appears in the page source; verify by inspecting the raw response.
Frequently asked questions
Why does my scraper return empty results when the browser shows content?
The content is likely generated by client-side JavaScript after the initial HTML loads. Parsers like BeautifulSoup and lxml do not execute JavaScript, so they only see the original markup.
How do I make my scraper execute JavaScript?
Use a browser automation framework such as Selenium or Puppeteer. These tools load the page in a browser engine, run its scripts, and let you extract from the rendered DOM.
Can I avoid a browser and still get the data?
Sometimes. Check the raw HTML for <script> tags containing the data as JavaScript variables. If the values are there, you can parse them directly without rendering the page.