Web Scraping With JavaScript and Node.js: A Working Stack Guide
A JavaScript scraper fails in three separate places, and most tutorials only prepare you for one of them. Fetching the HTML is the easy part. Parsing it correctly is the second. Staying unblocked while you repeat the job at volume is the third, and it is the one that quietly ends projects.
This guide walks the whole stack in the order you would actually adopt it: native fetch for static pages, Cheerio for parsing, jsdom for pages that mutate their own markup at load time, and Playwright or Puppeteer for sites that only exist after JavaScript runs. It closes with the honest maintenance math behind proxies, retries, fingerprints and CAPTCHAs, so you can tell when building that layer yourself stops paying for itself.
Five decisions to settle before you write code
Each one, left open, tends to force a rewrite later.
Language. Pick the runtime the rest of your system already uses. If your services, pipelines and team are Node, scraping in Python adds a permanent translation step that no library benchmark repays. The exception is crawl orchestration at large scale: when the scraper *is* the system, with depth-first queues and per-domain politeness rules, Python's Scrapy remains the strongest framework of its kind and Node has no direct equivalent. For the middle ground, which is most readers, JavaScript is the pragmatic choice because it is the language you are already in.
Browser vs. server. Frontend JavaScript cannot scrape anything outside its own origin. The browser blocks the request before it leaves your machine, and a CORS policy line in the target's headers is doing exactly its job. Even same-origin success runs into three walls: no file system or database access, no unattended automation, and no proxy rotation, headless mode or scheduling. Node.js removes all three, which is why scraping in JavaScript effectively means scraping in Node.js.
Parser. Four libraries cover nearly everything, split by one question: do you need a browser? Cheerio parses static markup at parser speed with no DOM and no script execution. jsdom is a Web-spec DOM in Node that can run <script> tags. Playwright drives Chromium, Firefox and WebKit. Puppeteer drives Chromium (and Firefox as of v23) and carries the more mature stealth plugin ecosystem. The comparison table below lays out the trade-offs.
HTTP client. Native fetch is the default: zero bundle cost, built into Node. Move to Got when you start hand-writing retry logic, to Axios when you need interceptors or identical browser/Node behavior, and to Node Crawler when you realize you are crawling a site rather than fetching a page. The client comparison table covers retries, interceptors and bundle size for all six.
Selectors. This is where scrapers die. The element is right there in the DOM, you copy a path, paste it into Cheerio, and get an empty array. The path was correct on the day you copied it.
Choosing a parser and an HTTP client
The parser decision comes down to what the server actually sends. If the data is in the HTML response, Cheerio is the right tool about 80% of the time. If the page rewrites its own markup from an inline script at parse time, jsdom is the lighter alternative to a full browser. If the content only appears after the page fetches more JSON or renders a framework component, you need a real browser engine.
On the client side, the practical progression is: start with native fetch; adopt Got the first time retry semantics matter; adopt Axios the first time you need interceptors; adopt Node Crawler the first time your job becomes site-wide crawling with built-in queues.
Getting selectors that survive a layout change
Chrome's "Copy selector" produces long, position-anchored paths full of :nth-child and wrapper divs. They break the first time the site rearranges its layout. Write the selector by hand instead, anchored to something stable: a semantic class name, an id, or a data- attribute the site uses to wire up its own JavaScript.
On books.toscrape.com, for example, the title links live inside article.product_pod h3 a — three terms, no positional pseudo-classes, no implementation-detail wrappers.
The cheapest test is the browser Console. Run document.querySelectorAll('your-selector') and check the count before you put the selector anywhere near Node. One element when you expected twenty means the selector is wrong. Zero means you are probably looking at a JavaScript-rendered page, and you should skip ahead to the dynamic-pages approach. A count that matches the page is a selector worth shipping.
Setting up the environment
Target a current Node LTS. Node 22 LTS stays in security support through 2027, and a version manager such as nvm lets you install a newer release for one project without breaking a scraper still pinned to an older one. Confirm the version with node --version; anything below v20 should be upgraded. fetch works on v20 (marked experimental there but reliable) and is officially Stability 2 — Stable as of v21.
Scaffold the project with npm init -y, then set "type": "module" so import statements work. Without it you get SyntaxError: Cannot use import statement outside a module, which is a confusing error to chase. Cheerio 1.x is dual ESM/CJS, so the setting is not strictly required, but ESM is the sensible default for new Node code. Then install Cheerio — one dependency is all this setup needs.
If your target is React, Vue or Next.js, skip to the dynamic-pages section and install Playwright instead.
A first working scraper: 20 book titles as JSON
Create scrape.js with the snippet below. It is about a dozen lines and needs no imports beyond Cheerio: fetch is native (no import since Node 18), cheerio.load(html) returns a $ that behaves like jQuery, and .map().get() is the Cheerio idiom for pulling attribute values into a plain array. The selector is the one verified in the Console step above.
Run it with node scrape.js. A JSON array of twenty book titles means you have a working scraper. The rest of this guide is about making it survive contact with reality.
Building a real static-page scraper
The demo target is books.toscrape.com: 1,000 books across 50 pages of 20, with stable markup and no terms-of-service friction. Before you commit to any real target, open the Network tab and ask one question: is the data in HTML I have to parse, or is there an internal JSON API behind the page? Most modern sites power their own frontend with one. Hitting that JSON is faster, less brittle and has fewer moving parts. If there is genuinely only server-rendered HTML, you are on the parsing path.
Each book card is article.product_pod, and the scraper pulls four fields from it. The field map below shows the selector and what each one returns.
Field map for the demo catalog
Two of these fields deserve a warning. The visible title link is shortened with CSS and ends in an ellipsis, so read the title attribute for the complete string rather than the link text. The rating is not stored as a number or an icon: the card carries a class word from star-rating One to star-rating Five, so a lookup table is needed to turn that word into a value.
Fetching, parsing and guarding against silent failures
Start with native fetch plus one error check: if !response.ok, throw with the status code. That single line is the one beginner tutorials skip. Without it, the scraper happily continues past a 500 or a 404 and you spend an hour debugging why the parser sees no elements. With it, the failure is reported at its source.
The per-card parser grabs the full class attribute, splits on spaces, filters out the constant star-rating part, and maps the survivor through a rating table. Using ?? null for an unknown class means a future encoding change returns null in that field instead of crashing the run — and the null is visible in your output. Note also that Cheerio's .map() returns a Cheerio collection, not a plain array; .get() unwraps it.
Pagination, pacing and output
The catalog's page URLs follow /catalogue/page-N.html from N=2 up to N=50, while the first page is served from the site root; the next number past the end, page 51, is a 404. That gives 50 pages of 20 books, or 1,000 records in total. Two decisions in the loop are worth defending.
First, use a sequential for loop with a delay rather than Promise.all. Hitting a site 50 times in parallel from one IP is exactly what production sites rate-limit you for. A 500 ms pause is a reasonable floor for friendly scraping; raise it to 1000–2000 ms for sensitive targets. Second, cap the demo at a handful of pages so tests run quickly, then raise the cap to 50 for the full catalog. For controlled parallel fetching later, bounded concurrency with p-limit is covered in the dynamic-pages section.
For output, write JSON when downstream code or an API will consume it, and CSV when a person will open it in a spreadsheet or BI tool. The hand-rolled CSV writer is fine for four flat fields. For nested objects, custom delimiters or streaming large result sets, use csv-stringify or papaparse. The one CSV detail that bites everyone: any quote character inside a field must be doubled, and the field wrapped in quotes. Skip that and the first title containing a quote breaks every row after it.
What actually breaks, and how to read the failure
Five issues show up on this target, and each one maps to a distinct diagnosis you can reuse elsewhere.
- Truncated titles.
h3 atext is cut off with CSS, so the visible string ends in an ellipsis. The full title sits in thetitleattribute. *Diagnosis: wrong extraction method, not wrong selector.* - Rating as a class word. There is no number and no data attribute — just
star-rating Onethroughstar-rating Five. A naive read returns the literal string. *Diagnosis: value needs a mapping step.* - Whitespace-padded availability. The raw text is full of newlines and spaces. Trimming works, but testing for the presence of the
.instock.availabilityclass is cleaner and survives copy changes such as "In stock", "Available" or "Limited". *Diagnosis: prefer structural signals over text.* - Page 1 is `/`, not `/catalogue/page-1.html`. Both URLs resolve, but the site's own "next" links treat
/as page 1, so matching that convention keeps your loop aligned with the site. *Diagnosis: follow the site's own navigation model.* - `/catalogue/page-51.html` returns 404. The URL pattern has no last-page marker. For unknown-length pagination, parse the "next" link on each page and stop when it is missing rather than guessing an upper bound. *Diagnosis: derive the stop condition from the page.*
A useful distinction to carry forward: an empty result from a static parser usually means either the selector is wrong or the content was never in the server response. Check the page source view first — if the content is absent there but visible in the inspector, no amount of selector tuning will help.
When Cheerio is not enough: jsdom
Cheerio is a parser. It builds a tree and gives you jQuery-like traversal, but it has no DOM, does not execute JavaScript, and does not know what window or document are. Most of the time that is irrelevant. jsdom is for the case where it is not: a full Web-spec DOM in Node with a real document, the ability to run <script> tags, and the answer to "what does this page look like after its load-time scripts finish?"
The boundary in practice: Cheerio when the data is in the HTML the server sends; jsdom when inline scripts mutate the markup at parse time; Playwright or Puppeteer when the content only exists after the page fetches more JSON or renders a single-page app. Most modern sites are either fully server-rendered or fully client-rendered, so jsdom's sweet spot is narrow — older e-commerce pages, some pricing widgets, and sites that ship a small bootstrap script hydrating a static shell from inlined JSON. When you meet one, jsdom is far lighter than a headless browser and far more accurate than Cheerio.
The basic API is the same one you would write in a browser console: new JSDOM(html), then dom.window.document, then standard DOM methods such as querySelector, querySelectorAll, getAttribute and textContent. If that is all you need, use Cheerio instead — it is smaller and faster. jsdom earns its place only with script execution enabled.
Running page scripts safely, and the one place regex is fine
The toggle that turns jsdom from a heavier Cheerio into a lightweight browser substitute is runScripts: 'dangerously'. It is not the default, and forgetting it means jsdom returns the same empty content Cheerio does.
The name is accurate. That option executes arbitrary JavaScript from a remote page inside your Node process. If the page is one you do not control — which is the entire point of scraping — the script can write to your file system, make outbound network calls and consume your CPU. Use it only against pages you trust, or isolate the work in a sandbox or a headless browser.
On parsing: do not parse HTML with regex. HTML is not a regular language, and any pattern that appears to handle nested tags, optional attributes and tolerated malformed markup will break on the first input that deviates from your example. There is exactly one reasonable exception — pulling a single well-defined token from a string you already control, such as /data-product-id="([^"]+)"/ against known markup. The boundary is structural: once a pattern has to locate a descendant, or an element nested inside another element by class, it is doing a parser's job and should be replaced by one.
Dynamic pages: why fetch returns an empty shell
View the raw page source (Cmd-U or Ctrl-U). If the content you want is missing there but present in the inspector, the page is JavaScript-rendered, and fetch plus Cheerio will return the same empty shell every time.
quotes.toscrape.com/js/ is the canonical demonstration. In a browser, quotes appear. In the server response, there is only a shell plus a script block that calls document.write ten times. The quote data is technically present inside an inline var data = [...], but the actual .quote elements exist only after JavaScript runs. You can sometimes regex the inline JSON out, but that is brittle — the next page may fetch from an API instead, and your scraper would need to know which approach each page uses. A tool that executes the page the way a browser does removes that question entirely.
Playwright or Puppeteer?
Playwright is a cross-browser automation tool maintained by Microsoft, with broad language support and first-class multi-tab handling. Puppeteer is Google's Chrome-team tool built primarily for Chrome/Chromium. The comparison table covers browsers, wait strategies, browser contexts, stealth ecosystem and footprint.
Playwright is the default for new projects. Its locator API largely solves the most painful class of bugs — race conditions on dynamic content — and browser contexts make session isolation trivial, which matters when you scrape multiple accounts or rotate cookies. Puppeteer remains the right pick when you are Chromium-only and want the lightest footprint, when you depend on puppeteer-extra-plugin-stealth (still one of the most reliable open-source bot-detection bypasses), or when you are inheriting a Puppeteer codebase and a rewrite buys nothing concrete.
Scraping with Playwright, then Puppeteer
Install Playwright and download Chromium with npx playwright install chromium. That second command is a one-time download of roughly 150 MB; without it, chromium.launch() throws.
Against the same quotes page, the Playwright scraper waits for .quote, then evaluates a function inside the page's own JavaScript context to collect text and author. Four details matter:
page.waitForSelector('.quote')is the line you will forget once.page.goto()resolves when the initial HTML load completes, not when the page's JavaScript has finished mutating the DOM. Without the wait, your evaluation runs against the empty shell and returns an empty array. Playwright's locator API auto-waits; the lower-level$$evaldoes not.page.$$eval(selector, fn)runs your function inside the page, receiving real elements. It is faster than pulling HTML back and re-parsing in Node, and it uses the same DOM API you would write in DevTools.- Browser contexts are Playwright's advantage for stateful scraping: each
browser.newContext()has its own cookies, storage and history. For a one-shot scrape,browser.newPage()is enough. - Locators are the modern API.
page.locator('.quote').all()returns auto-waiting handles you can chain, including.evaluateAll(). For greenfield scrapers, use locators; the$$evalform is used here because it maps one-to-one onto the Puppeteer equivalent.
Puppeteer auto-downloads Chromium during npm install with no extra step. The same scraper differs only in the import and the launcher. Playwright was created by the same engineering team that originally built Puppeteer at Google, so the page-level APIs are nearly identical and migrating either direction is mostly a search-and-replace job.
Scaling to thousands of pages with bounded concurrency
A naive loop pays full page-load latency on every fetch. Spawning one browser per page exhausts memory long before 10,000 pages. The pattern that scales is one browser, many pages, bounded concurrency — p-limit provides the bound.
Three details earn their place in that pattern:
- A concurrency cap of 3 is a reasonable starting point for any target you do not own. Raise it to 10 for your own staging environment. Past 20 you are hammering, and most production sites will throttle, block or fingerprint you.
- `try { ... } finally { await page.close(); }` is what turns a scraper that runs for an hour into one that runs for ten. Pages are not garbage-collected like ordinary JavaScript objects. A thrown error without a close leaks a tab, and enough leaked tabs crash the browser. Always close in
finally. - The stop condition is target-specific. quotes.toscrape.com/js/ has 10 pages of 10 quotes; pages 11 and beyond return an empty rendered shell rather than a 404. For real pagination, parse the "next" link off each page and stop when it is missing, or detect the empty result page and break.
Even tuned correctly, scraping thousands of pages through a headless browser costs 30 to 90 minutes of wall time, several gigabytes of bandwidth, and meaningful CPU and memory. Every blocked request, CAPTCHA and TLS fingerprint mismatch remains your problem.
Retries that do not crash the run
Networks drop, servers return 503s under load, and Cloudflare can return 522s when its origin times out. Without retry handling, any one of these kills a run and you lose everything since the last checkpoint.
The working pattern is a small wrapper around fetch that handles three cases differently. Transient network failures and 5xx responses are retried with exponential backoff. A 429 is treated as an instruction: read Retry-After and wait that long. Permanent 4xx codes are not retried at all. Three judgment calls sit behind it:
- Exponential rather than constant delay. A constant delay either wastes time on easy retries or gives up too fast on hard ones. Backoff of 1s, 2s, 4s, 8s recovers transient failures quickly and gives an overloaded target time to recover.
- Treat 429 differently from 5xx. A 429 tells you exactly how long to wait. Honoring
Retry-Afteris the difference between a polite scraper and one that gets banned for ignoring the signal. - Fail fast on permanent 4xx. A 404 will not become a 200 in eight seconds, and neither will a 401. Retrying them burns your concurrency budget and hides the real problem in your logs.
Wrap your scraping calls in that wrapper instead of raw fetch and you eliminate the whole class of bugs that look like "the scraper ran for an hour and then crashed on a blip."
Proxies, fingerprints and CAPTCHAs: the honest cost
A single IP address is a single rate-limit budget. Push one address hard enough and the sequence is predictable: throttling first, then a block, then a fingerprint. Spreading requests across a pool of addresses keeps any one of them from looking like the same source.
Two proxy flavors matter. Datacenter proxies are cheap — a few dollars per million requests — but easy to detect, since most anti-bot vendors keep blocklists of known datacenter ranges; use them for low-value targets that do not actively defend. Residential proxies route through real consumer ISPs, so requests appear to come from ordinary home users. They are far harder to detect and ten to fifty times more expensive, and they are almost always the right pick for targets that take blocking seriously.
With native fetch, a proxy needs undici's ProxyAgent (undici ships with Node, so no extra install). With Playwright, the proxy is a launch option. The code is the easy part; managing the pool is the project — dead proxies, geographic targeting, sticky sessions for stateful flows, rotation per request versus per session, and retry-on-proxy-failure logic. Most teams end up buying from a proxy provider because the rotation logic is where the maintenance burden actually lives.
Bot detection runs in three layers, and defeating all three compounds over time:
- Headers. The default Node
fetchUser-Agent is a giveaway. Most sites filter on it alone and return a 403 before anything else. A realistic User-Agent,AcceptandAccept-Languagegets you past the cheapest filters and nothing serious. - TLS handshake fingerprints (JA3/JA4). Every HTTPS client has a distinct handshake signature, and Cloudflare and Akamai both fingerprint at this layer. A headless Chrome session started through Playwright or Puppeteer presents a handshake that matches Chrome, which is a strong argument for using a headless browser on blocking-sensitive targets even when the page itself is static HTML. For pure-fetch scrapers, tools like
curl-impersonatereplicate browser fingerprints exactly, at the cost of a non-trivial dependency. - Behavioral signals. Real users move the mouse, scroll erratically, focus inputs before typing and take seconds to click. Headless browsers go straight to extraction in 50 ms.
puppeteer-extra-plugin-stealthhandles many obvious tells — fakingnavigator.webdriver, patching canvas fingerprints, rotating WebGL parameters — and the Puppeteer stealth ecosystem remains more mature than Playwright's community equivalents.
The honest framing: bot detection is an arms race, and you are on the side with fewer engineers. You can win the rounds you fight, but every change at the vendor's end is a fresh debugging session at yours.
On CAPTCHAs, three options exist and one should come first. Preventing the challenge from appearing is the step people tend to skip, and it is usually the one that would have saved them the work: a CAPTCHA is normally the consequence of a bot-detection signal that fired earlier, so realistic headers, a residential proxy, a stealth-patched headless browser and behavioral pacing keep it from being served at all. The cheapest CAPTCHA is the one your scraper never sees. Solver services such as 2Captcha, Anti-Captcha and CapMonster run pools of human and increasingly ML solvers at a few dollars per 1,000 reCAPTCHA solves, with a real latency cost of 15 to 60 seconds per challenge and a corresponding drop in throughput. Solving them yourself with ML is technically interesting and rarely the right call, because modern CAPTCHAs combine behavioral telemetry with the visual challenge. If you are consistently hitting CAPTCHAs, your fingerprint is wrong, not your solver.
When DIY scraping stops paying for itself
Every scraping project reaches a point where the maintenance cost of the proxy pool, retry handlers, User-Agent rotation, headless browser farm, CAPTCHA budget and on-call rotation outweighs the cost of paying someone else to run it.
DIY is still the right call in three cases. One-off scrapes against a single non-blocking page: Cheerio plus fetch is two lines, and adding a managed hop is overhead for a job that does not need it. Internal sites or your own staging: no proxies, no bot detection, just talk to the site directly. Tightly-coupled custom parser work: if you are chasing edge cases that need full control of the parse tree, you want the raw HTML and Cheerio or jsdom, not a JSON envelope.
The signals that the math has flipped:
- You spend more than a few hours a week debugging blocks instead of improving the data pipeline.
- Your proxy and CAPTCHA bill has crept into the high three figures per month.
- You have built, or are about to build, a rotation of engineers to keep scrapers alive.
- One small change at a target site cascades into rewriting selectors and retesting the entire blocking stack.
When two or more are true, the return on building your own infrastructure has inverted.
What a managed scraping API removes from your code
A managed scraping API takes most of the previous section off your plate. The workflows where the math flips fastest share a shape: the value is not in writing the scraper, it is in shipping the system that uses what the scraper returns.
- E-commerce price monitoring across competitors. A brand tracking 50 competitor product pages daily for price and stock changes is maintaining 50 proxy configurations, 50 retry policies and 50 selector inventories.
- Sales and GTM lead enrichment. A sales ops team enriching 5,000 inbound leads with public company pages before assignment. Spinning up a Playwright farm with residential proxies and stealth patches for a weekly batch costs more to maintain than it returns.
- AI agents reading live web data. A research agent pulling fresh sources at query time, or a support agent reading the customer's actual product docs in real time. This is the fastest-growing case and the one where per-request cost matters least, because it is paid inline with high-value interactions.
- Fintech and cybersecurity monitoring. Time-sensitive feeds where reliability matters more than throughput — broker prices, regulatory filings, security advisories. A blocked scraper at the wrong moment is a missed alert.
The integration itself is one HTTP endpoint: pass an API key and the target URL as query parameters and get the page HTML back, with no client library and no auth flow. Proxy rotation, retry logic and header normalization come with it. For JavaScript-rendered pages, a render flag runs a headless browser server-side and returns the post-render HTML in the same response shape.
Selector-based extraction goes further: describe the fields in CSS selectors and get clean JSON back without loading Cheerio locally. The trade-off is flexibility — list extraction, attribute pulls and nested objects are covered, but reshaping data, following links or running per-element conditional logic sends you back to fetching HTML and parsing locally. For most "give me the prices off the page" jobs, that flexibility was overhead you were not using.
For pages whose markup changes weekly, varies per instance, or has no stable structure, natural-language field descriptions let the model do the selector work, including type coercion — a price returned as a number rather than the raw currency string, and a stock flag returned as a boolean. A sibling prompt-style parameter exists for cases where predictable key names do not matter; for production pipelines, structured field descriptions are the safer choice because the key names stay stable.
A decision checklist before you build
Run through these before committing engineering time:
- Is the data in the HTML, an internal JSON API, or only after JavaScript runs? Check the raw source view and the Network tab. This single answer determines your entire stack.
- Does the selector survive a layout change? Anchor to semantic classes, ids or
data-attributes, and verify the count in the Console first. - Is the pagination stop condition derived from the page? Parse the "next" link rather than guessing an upper bound.
- Does every fetch have an `ok` check and a retry wrapper? Fail fast on permanent 4xx, honor
Retry-Afteron 429, back off exponentially on 5xx and network errors. - Are pages closed in `finally`? Leaked tabs crash browsers.
- Is concurrency bounded? Start at 3 for targets you do not own.
- Have you counted the maintenance hours? If blocking work is eating your week and the bill is climbing, the managed option is the cheaper one.
| Scenario | Pick |
|---|---|
| When you want one runtime end-to-end (scrape, process, serve); when the target is a JavaScript-rendered site (Playwright in Node is the same Playwright as in Python, and you've already got Node running), or when you need to scale scraping into a service rather than run it as a script. | JavaScript / Node.js |
| When scraping is the system; you're crawling tens of thousands of sites with complex politeness rules, depth-first queues, and per-domain throttling. From what I've seen, Scrapy is still the best crawl-orchestration framework anyone has written in any language, and Node has nothing equivalent. | Python |
Reflects the author's stated view at the time of writing; treat as a starting heuristic, not a benchmark.
| Library | What it is | When to use it |
|---|---|---|
| Cheerio | Server-side jQuery for HTML; parses static markup at parser speed, no DOM, no JS execution | 80% of the time; static HTML pages |
| jsdom | Web-spec DOM implementation in Node, can run <script> tags | When the page mutates its own markup at parse time, and Cheerio returns the wrong HTML |
| Playwright | Full headless browser automation across Chromium, Firefox, and WebKit | JS-rendered sites (React, Vue, Next.js, anything single-page) |
| Puppeteer | Chrome's official headless library, Chromium-only | When you're Chrome-only and want the extras, particularly the stealth plugin ecosystem |
Library capabilities as described at the time of writing; check current package docs for changes.
| Client | Retries built-in | Interceptors | Bundle size | When to reach for it |
|---|---|---|---|---|
| Native fetch | No | No | 0 (built into Node) | The default |
| Axios | Via plugin | Yes | ~16KB | Interceptors, browser+Node parity |
| Got | Yes (sensible defaults) | Hooks | ~45KB | Retry semantics that work out of the box |
| Ky | Yes (configurable) | Yes | ~7KB | Modern fetch wrapper, lightest of the libraries |
| SuperAgent | Via .retry() | Plugins | ~18KB | Existing codebases, plugin needs |
| Node Crawler | Yes | No | Framework | Crawling at scale with built-in queues |
Bundle sizes and feature flags as stated at the time of writing; verify against current releases.
| Field | Selector | What you pull |
|---|---|---|
| Title | h3 a → title attribute | Full title string |
| Price | .price_color text | "£51.77" |
| In stock | .instock.availability presence | true/false |
| Rating | p.star-rating class attribute | "star-rating Three" (needs parsing) |
Selectors verified against books.toscrape.com at the time of writing; the site's markup may change.
| Cheerio | jsdom | Playwright | |
|---|---|---|---|
| Real DOM | No | Yes | Yes (real browser) |
| Runs inline scripts | No | Yes (opt-in) | Yes |
| Renders SPAs (React, Vue) | No | No | Yes |
| Speed | Fastest | Middle | Slowest |
| Memory footprint | Tiny | Small | Heavy (browser process) |
| When to reach for it | Static HTML | Inline scripts mutating DOM | Anything browser-rendered |
Relative speed and memory are qualitative comparisons, not measured benchmarks.
| Playwright | Puppeteer | |
|---|---|---|
| Maintained by | Microsoft | Google (Chrome team) |
| Browsers supported | Chromium, Firefox, WebKit | Chromium and Firefox (first-class as of v23) |
| Wait strategies | Auto-waiting locators, native event timeouts | Explicit waitForSelector / waitForFunction |
| Browser contexts (isolated sessions) | First-class, lightweight | Possible via incognito, less ergonomic |
| Stealth plugin ecosystem | Smaller, growing | Larger and the most mature in the ecosystem (puppeteer-extra-plugin-stealth) |
| Bundle size | Heavier (one browser engine per platform) | Lighter (Chromium only) |
| When to reach for it | New projects, cross-browser testing, anything where wait strategies matter | Chromium-only work, mature stealth requirements, lightest footprint |
Feature and ecosystem comparisons as described at the time of writing; verify current versions.
Frequently asked questions
Can I scrape React or Next.js sites with Cheerio?
No. Cheerio is an HTML parser, not a JavaScript runtime. When you fetch a React, Vue or Next.js page, the server typically sends an empty shell and JavaScript builds the actual content after the browser loads the page. Cheerio never runs that JavaScript, so it sees the shell and returns nothing useful. Use Playwright or Puppeteer for client-rendered sites, or a managed API if you would rather not run a headless browser yourself.
What's the fastest way to scrape thousands of pages with Node.js?
Keep a single headless browser alive and open many pages inside it, with a hard ceiling on how many run at once. p-limit supplies that ceiling; a value between 3 and 10 suits most targets, depending on how hard they throttle. Launch Chromium once and reuse it, and put page.close() in a finally block so a failed scrape cannot leak a tab and eventually crash the browser. A managed API can also absorb this work.
Do I still need Axios for web scraping?
For most scraping, no. Native fetch ships with Node 18 LTS and later, handles about 90% of scraping calls cleanly, and has no dependency cost. Reach for Axios when you specifically need its interceptor pattern (centralized auth, logging, request/response transforms), when the code must behave identically in browser and Node, or when you are already in an Axios codebase and a second HTTP library is just noise.
How do I handle CAPTCHAs in Node.js?
Avoid them first. Most CAPTCHAs trigger on bot-detection signals — default Node User-Agent, mismatched TLS fingerprint, headless-browser behavioral signature — before the challenge itself fires. Fix the upstream with realistic headers, residential proxies and stealth patches, or skip the maintenance and let a managed API handle proxies, fingerprints and solving in one call.
Is web scraping with JavaScript legal?
Generally yes, with real caveats that vary by jurisdiction and target. Scraping publicly accessible data is broadly legal in the US and EU. Scraping behind a login or paywall typically is not — it can violate the site's terms of service and, in some jurisdictions, statutes such as the CFAA. Respect robots.txt where the target's terms require it, throttle politely so you are not effectively DDoS-ing, and think hard about privacy law (GDPR, CCPA) before scraping personal data.
Why does my Cheerio scraper return an empty array?
Two causes dominate. Either the selector does not match the current markup, or the content was never in the server response. Check the raw page source first: if the content is absent there but visible in the inspector, the page is JavaScript-rendered and no selector change will help — you need jsdom with script execution or a headless browser. If the content is in the source, verify the selector count in the browser Console before touching your Node code.
When should I use jsdom instead of Playwright?
Use jsdom when the page's own inline scripts mutate the markup at parse time and you trust the script source. jsdom is far lighter than a headless browser and far more accurate than Cheerio in that narrow case. If the page fetches additional JSON, renders a framework component, or behaves like a single-page app, use Playwright or Puppeteer instead — jsdom does not render SPAs.
[
"A Light in the Attic",
"Tipping the Velvet",
"Soumission",
"Sharp Objects",
"Sapiens: A Brief History of Humankind",
... fifteen more titles
]