Extracting content hidden in a Shadow DOM
Some pages render their visible text inside a shadow root rather than the main document tree.
A conventional scraper that only reads the outer HTML will therefore miss the article, even though the content is present in the browser. The workaround below exposes that hidden markup by running a small JavaScript evaluation before the page is returned, so the extracted HTML can then be parsed like ordinary page content.
Why ordinary scraping misses the content
A shadow root is a separate document fragment attached to a host element. Styles and markup inside it are encapsulated, which is why selectors aimed at the light DOM return nothing for the hidden article body. Inspecting the page confirms the distinction: the host element is visible in the main tree, while the article markup lives under its shadowRoot property.
The MSN article used as the example follows this pattern. Its host is an article element, and the readable content is nested one level inside that host. That structure is the key diagnostic: if the outer page looks almost empty but the browser renders a full article, look for a custom host element with a shadow root before assuming the content is loaded elsewhere.
Exposing the shadow root in the page
The extraction step runs in the page context through an evaluate command. It selects the host element, reads the inner HTML of its shadow root, and assigns that markup to the document body. Replacing the body is deliberate: once the hidden markup sits in the main document, the returned page contains the article in a place that standard parsing can reach.
Protected module code0001 contains the exact evaluation code for this example. Adapt the host selector to the target page; the pattern remains the same — locate the host, read its shadowRoot, and surface that HTML in the main document.
What the response gives you
After the evaluation, the response contains the article markup that was previously encapsulated. From there it can be parsed, cleaned and processed with the same tools used for any other HTML response, without needing a separate shadow-DOM parser.
Use this as a checklist when a page appears empty to a scraper:
- Confirm in browser inspection that the content sits under a shadowRoot rather than the light DOM.
- Identify the host element and its selector.
- Run the evaluate step to copy the shadow root's inner HTML into the document body.
- Parse the returned markup as normal HTML.
- If the selector returns nothing, re-check the host tag name — the example depends on the exact host element used by the page.