Scraping Dynamic Content in Python With JavaScript Rendering
A page that looks complete in a browser can arrive almost empty to a plain HTTP client. Prices, reviews, stock levels and search results are often created after scripts run, so the first HTML response may contain only a shell. This guide explains how to detect that pattern, render the page, interact with it, and extract structured data without building every browser and proxy component yourself.
What dynamic means, and how to spot it
Static parsers read only the markup delivered at first response. Dynamic pages instead build the visible document in the browser: scripts fetch data in the background, mutate the DOM, and add more items as the user scrolls or clicks. Booking listings, for example, can require a click before scrolling loads further cards, and disabling JavaScript can stop loading after a limited set.
Use this detection checklist before choosing a tool:
- Open developer tools and inspect how data loads.
- Compare view-source with the rendered page; missing text in view-source is a warning sign.
- Watch the Network panel, filtered to Fetch/XHR, while scrolling — new JSON or GraphQL responses indicate client-side loading.
- Disable JavaScript from the command menu and reload; if content collapses or stops growing, rendering is required.
- Note any sign-in, cookie or consent gate that must be handled before content appears.
Five-step workflow: render, wait, interact, extract
A reliable workflow has five stages: inspect the page, render it in a headless browser, wait for the target element, perform any clicks or scrolls needed to load more, and extract with selectors into structured output. Rendering is enabled by default in the workflow described here; the render_js setting can be turned off for static pages.
For interaction, a scenario is an ordered list of instructions. A fixed wait pauses for a number of milliseconds. A wait_for step pauses until a selector appears. Click steps can open a reviews panel or a next-page control, fill steps can complete a form field, and scroll steps can move by pixels or continue until the end. The Google Play reviews example uses a wait followed by a wait-for-and-click on the ratings button, because not all reviews are present initially.
Infinite scroll, extraction rules and pagination choices
Infinite scroll is handled by a dedicated instruction with a maximum scroll count and a delay between scrolls; a maximum of zero means continue until the end. This removes the need for a manual scrolling loop, with the qualification that it does not work with stealth_proxy.
Structured extraction is separate from interaction. Extraction rules name a container selector, declare a list type, and map child selectors to fields such as name and description. In the reviews example, the container selector groups each review, and the child selectors pick out the reviewer name and review text. The result can be returned as JSON and, with a DataFrame library, saved to CSV.
Decision table — choose the cheaper route first:
- If JSON exists behind XHR/Fetch: call that endpoint directly, reusing headers and cookies copied from developer tools. This is faster and cheaper than rendering.
- If there is no usable JSON endpoint, or the page needs heavy interaction: use full rendering with a wait-for-and-click for pagination or infinite scroll.
Rendering costs more than a static fetch, so test both modes on a sample, batch URLs where possible, and disable rendering when the HTML or JSON endpoint already suffices.
Proxies, retries, screenshots and reliability
At scale, reliability comes from the connection as much as the parser. Premium proxies with a country code route requests through residential addresses and a chosen location; retries absorb transient server errors; device fingerprint randomisation and realistic headers reduce bot signals. A screenshot option can capture the rendered page — optionally a single element or the full page with auto-scrolling — and return it as base64 when JSON output is requested, which is useful for visual checks, SERP review and QA.
Common errors and alternative libraries
Three common failures have distinct fixes:
- A missing key in the extracted output means the container selector matched nothing; inspect the live page and correct the selector.
- A click that yields only the first screenful means the modal or list was not scrolled; add a scroll-to-bottom step for the dialog or use infinite scroll.
- A 403 or invalid-key error points to the API key or quota; check the key and plan limits.
For free, self-hosted automation, Selenium is a long-established, resource-heavy framework suited to testing and one-off tasks; Playwright is a modern Microsoft-built framework with official Python support, fast performance and async support; Puppeteer targets Chrome through Node.js, with only an unofficial Python port. All three automate a browser, but proxies and related infrastructure must be configured separately.
Anti-bot protections, ethics and compliance
Dynamic scraping often triggers CAPTCHAs, IP blocking and fingerprint detection. CAPTCHAs commonly follow data-centre IPs or excessive requests from one address; rotating residential proxies makes traffic resemble ordinary users. Automation frameworks can also be detected through browser signals, so stealth plugins or anti-detect browsers may be needed.
Keep the work compliant: respect robots.txt and terms of service, scrape only publicly allowed pages, rate-limit requests, avoid personal or sensitive data, cache repeat requests, and log jobs so failures and site changes are caught early.
| Situation | Recommended approach |
|---|---|
| JSON exists behind XHR/Fetch | Call the JSON endpoint directly with headers and cookies from DevTools — faster and cheaper than rendering. |
| No JSON endpoint or heavy interactivity | Use full rendering with wait_for_and_click for the next-page button or infinite_scroll. |
Developer tools confirm where data originates and provide headers to reuse.
| Symptom | Likely cause | Fix |
|---|---|---|
| KeyError: 'Products' | The selector matched no review elements, so no data was returned. | Inspect the page and update the container selector. |
| Click does nothing / only first results | The modal was not scrolled, so only visible items loaded. | Add scroll_to_bottom for the dialog or use infinite_scroll. |
| Invalid API key or 403 | Key missing, wrong, or quota exceeded. | Check the key and confirm plan limits. |
Selectors and dialog structure should be confirmed against the live page in developer tools.
Frequently asked questions
How do I know if a site needs JavaScript rendering?
Check whether needed text is absent from view-source, appears only after a delay, or arrives through XHR/Fetch requests in the Network panel. Those signs point to client-side assembly, so a headless browser with rendering enabled is needed.
What is the difference between wait and wait_for in js_scenario?
A fixed wait pauses for a set duration. A wait_for step pauses until a chosen CSS or XPath selector appears, which is safer when load timing varies.
Can an API handle infinite scroll without loops in my code?
Yes — an infinite-scroll instruction can keep scrolling until no new items appear, so your own loop is not required. It does not work together with the stealth_proxy feature.
When should I disable render_js to save credits?
Disable it for static pages that already return complete HTML, or when the underlying JSON/XHR endpoint can be called directly. Benchmark a sample in both modes before deciding.
How do I avoid getting blocked on dynamic sites?
Use rotating proxies, honour rate limits, vary timing, and send realistic headers. Those measures reduce IP bans compared with repeated requests from one address.
Can BeautifulSoup scrape dynamic websites?
Not on its own. It parses HTML that has already arrived; when required data is created later by JavaScript, that data is missing from the initial response.
What is the best Python library for scraping dynamic content?
Selenium, Playwright and Puppeteer can automate a browser, but the overall setup — proxies, rate limits and browser management — still matters. Connecting a library to a scraping API can save time building those parts.
How do you extract data from a JavaScript rendered page?
Let an automated browser run the page scripts, click or scroll as needed, confirm network activity has finished, then extract from the live page.
Is Selenium better than an API for dynamic web scraping?
Selenium suits a proof of concept or occasional one-off task. For larger, repeated runs, an API that manages browsers and proxy rotation is usually more reliable.
How do I bypass captchas when scraping dynamic web pages?
Lower the chance of triggering one by resembling ordinary traffic: premium residential proxies instead of data-centre IPs, plus stealth browser measures that mask automation signals.