Node.js Fetch API: Requests, Errors, and Scraping Patterns
Modern Node.js ships with a browser-style Fetch API, so many HTTP tasks no longer require an extra client library. This guide shows how to make GET and POST requests, inspect responses and headers, handle failures and timeouts, and extend the same patterns into HTML scraping and proxy routing. It also explains where plain fetch stops being the practical choice.
Quick answer: the working pattern
A reliable fetch call in Node.js follows four steps: send the request, check whether the HTTP status is acceptable, read the body in the correct format, and use the parsed data. The built-in function is available globally from Node.js 18 onward, so no import is needed on modern runtimes.
The essential pattern is shown in the protected code module code0001. It demonstrates a GET request, an explicit response.ok check, and JSON parsing. The same structure applies to most API calls.
If you are on an older runtime, the node-fetch package provides a compatible function. Newer releases of that package are ESM-only, so upgrading Node.js is usually the simpler path for a new project.
What the Fetch API provides in Node.js
Fetch is an interface for sending HTTP requests and reading the resulting resource, whether that resource is JSON, text, a file, or raw HTML. It began in browsers and is now part of Node.js itself.
The central call is fetch(url, options), shown in code0002. The first argument is the destination. The optional second argument carries the HTTP method, request headers, and a request body for methods such as POST or PUT.
The call returns a Promise that resolves to a Response object. That object does not hold the parsed data directly; it represents the HTTP response and provides methods for consuming the body. You choose the reader that matches the content: response.json() for JSON APIs, response.text() for plain text or HTML, and response.arrayBuffer() for binary data.
Before native fetch, Node.js developers commonly used the low-level http and https modules or installed a third-party client such as Axios, got, or request. The core modules remain powerful but require more manual wiring. Native fetch offers a standard, Promise-based model that matches browser usage, which keeps request code consistent across frontend and backend.
Setting up fetch in Node.js
On Node.js 18 or newer, fetch is already present. You can call it directly without imports or extra packages. To confirm your runtime, run the command in code0003 and check that the reported version is at least 18.
If you are tied to an older runtime, you have two realistic choices. The first is to upgrade Node.js when you control the environment; this brings the native API and other improvements. The second is to install node-fetch for legacy or shared environments where upgrading is not yet possible. The minimal import is shown in code0004.
When using node-fetch, keep dependencies deliberate. Prefer the built-in API when your runtime supports it, test against the same Node.js version you deploy, and keep local and production environments consistent. For most modern projects, installing node-fetch is unnecessary.
The examples in this guide use ESM syntax. If your project uses ESM, add "type": "module" to package.json.
GET requests with async/await and .then()
The most common task is a GET request that reads JSON. The httpbingo.org/get endpoint is useful for practice because it echoes back the request it receives, including query parameters and headers.
A basic async/await example appears in code0005. The response is JSON containing fields such as args, headers, and origin. Query parameters appear under args, which makes the endpoint convenient for checking what your code actually sends.
The same operation can be written with Promise chaining, as in code0006. Both styles are valid. Async/await is usually easier to follow once you add error handling, conditional logic, or multiple requests.
To test query parameters, code0007 requests https://httpbingo.org/get?user=john&mode=test and logs data.args. You should see user and mode reflected there. The same pattern applies to real APIs, even though production endpoints will not echo your request.
Fetch does not assume JSON. If a server returns plain text or HTML, response.json() throws. Use response.text() for those responses. Also remember that fetch returns a Promise; the request does not begin until you await it or attach a .then() handler.
For a default GET pattern, code0008 combines a status check with JSON parsing. It throws when response.ok is false, then returns the parsed data. This keeps failures visible rather than allowing partial or invalid data to continue through the application.
Reading responses and headers
Every fetch call resolves to a Response object. The first properties to inspect are response.ok, a boolean that is true for successful responses (typically 2xx status codes); response.status, the numeric HTTP status; and response.statusText, the human-readable status message. Code0009 shows these values for a request to httpbingo.org/get.
After confirming success, read the body. response.json() is appropriate for JSON APIs, while response.text() suits plain text or HTML. These methods read from the response body stream and return a Promise. A body can be consumed only once, so choose the reader that matches the content type.
Headers are available through response.headers, which is a Headers instance rather than a plain object. Use its methods instead of direct property access. Code0011 iterates over all headers, and code0012 reads a specific header with response.headers.get("content-type"). This is useful for content negotiation, caching decisions, and debugging API behavior.
Node.js differs from the browser on CORS. In a browser, you can read only headers the server explicitly exposes. In Node.js, there are no CORS restrictions, so all response headers are inspectable by default. That visibility does not override API rules: rate limits, authentication requirements, and usage policies still apply.
POST, form data, and other HTTP methods
Sending data usually means switching to POST. With fetch, that requires a method, appropriate headers, and a request body. The httpbingo.org/post endpoint echoes whatever you send, which makes it useful for testing.
Code0013 sends JSON. The body must be a string, so JSON.stringify() is required. The Content-Type header tells the server how to interpret the payload. The example also checks response.ok before parsing the response.
For form submissions, use FormData as shown in code0014. In this case you normally do not set Content-Type yourself; fetch supplies it, including the multipart boundary. This approach suits file uploads, multipart forms, and legacy APIs that expect form-encoded data. In modern Node.js, FormData is available globally.
PUT, PATCH, and DELETE follow the same structure. Code0015 shows a DELETE request. For PUT and PATCH, include a body when needed, stringify JSON, set the appropriate headers, and check the response status if the result matters.
Two practical reminders: GET and HEAD requests should not include a request body, even if some servers tolerate it; and never forward user input without validation and sanitization. The API may accept bad input, but your server remains responsible for what it sends.
Errors, timeouts, and retries
A common misconception is that fetch throws for HTTP error statuses. In Node.js, fetch rejects only on network-level problems such as DNS failures, dropped connections, or aborted requests. A 404 or 500 response still resolves to a Response object. You must check response.ok or response.status yourself; otherwise a failed request can be mistaken for a successful one.
A simple rule: if response.ok is false, treat the response as an error; if it is true, proceed to parse the body.
Code0016 shows async/await error handling. When the status is not OK, it attempts to read a small error body with response.text(), falling back to an empty string if that read fails, and throws an error containing the status and body. Reading a small error message can speed up debugging, but avoid logging sensitive data from third-party APIs.
Code0017 applies the same rule with .then() and .catch(). The behavior matches the async/await version; the difference is readability. Promise chains become harder to follow as logic grows, which is why many teams prefer async/await.
Node.js does not impose a request timeout by default. If a request can hang, set one. AbortController is the standards-based way to do this, as shown in code0018. A timer calls controller.abort() after five seconds, and the signal is passed to fetch. When the timer fires, the request is cancelled and the Promise rejects, so execution enters the catch block. Always clear the timer in a finally block.
Retries are not a default for every request. They make sense for temporary failures: network hiccups, timeouts, and 5xx responses from an overloaded upstream. They usually do not make sense for 4xx errors, which often remain until the request changes. A practical rule set:
- Retry on network errors and timeouts.
- Retry on 502, 503, 504, and sometimes other 5xx responses.
- Do not retry on 400, 401, 403, or 404 unless you have a specific reason.
- Add a short backoff and cap the number of attempts.
When retrying, respect rate limits. If the API sends a Retry-After header, follow it. Keep retries small, add delays, and log every failure so patterns are visible in monitoring. Retries without logging and metrics can hide repeated failures.
Fetching HTML and parsing it with Cheerio
Fetch is not limited to APIs. It can download raw HTML, which is often the first step in a scraping workflow. In Node.js, fetch has no browser restrictions and gives full access to the response body.
Code0019 requests a public page and reads the response as text. At that point the HTML is a plain string. To extract data, load it into an HTML parser. Cheerio provides a jQuery-like API for server-side HTML and pairs naturally with fetch for static pages.
Code0020 installs Cheerio, fetches example.com, loads the HTML, and extracts the page title with a CSS selector. The same approach works for prices, product names, links, tables, and other elements you can target with selectors.
Scraping responsibly is part of the task. Check robots.txt, read the site's terms of service, and respect privacy and usage rules. Publicly available HTML does not remove your responsibility as a developer. Also watch request volume: even simple fetch calls can overload a site if sent too aggressively or without delays.
Plain fetch works when the server returns real HTML. Some sites render content only after client-side JavaScript runs, leaving the initial HTML nearly empty. In that case, fetch plus Cheerio has nothing useful to parse. Headless browsers such as Puppeteer load the page, execute scripts, wait for rendering, and return the final HTML. They suit modern web apps, dashboards, and heavy frontend frameworks, but they are heavier and slower and require more infrastructure. For many production teams, a managed browser-based service is easier than maintaining a headless browser fleet.
Proxies and scaling fetch-based scrapers
A small script can pass local tests and still fail at volume. Common problems include IP blocks after repeated requests from one address, rate limits that return 429 responses, CAPTCHAs, geo restrictions that change or hide content, and fragile infrastructure such as retries that multiply traffic or jobs that hang.
An HTTP proxy acts as a middleman. Instead of your scraper contacting the target directly, the proxy forwards the request. This helps with geo routing, spreading traffic across multiple IPs, and reducing immediate blocks. Proxies do not remove legal or ethical obligations: follow local laws, respect the target site's terms, comply with data protection rules, and be careful about what data you collect and store.
Native fetch in Node.js 18+ is powered by Undici, so proxy support differs from older Node HTTP clients. The compatibility details matter:
- In Node 18–23, native fetch does not support an { agent } option and does not automatically honor HTTP_PROXY or HTTPS_PROXY environment variables.
- In Node 24+, proxy environment variables can be respected, but only when enabled with NODE_USE_ENV_PROXY=1 or --use-env-proxy.
To use a proxy with native fetch, configure an Undici dispatcher, usually ProxyAgent or an env-proxy helper. Code0021 sets a global dispatcher so every fetch call uses the proxy unless overridden. Code0022 creates a dispatcher for a single request. Code0023 shows the environment-variable approach for Node 24+ with the required enablement flag.
Production scraping stacks are usually layered: fetch for simple, stable pages; proxy pools for distribution and geo-specific content; backoff and retry rules that respect 429 responses and Retry-After headers; and fallbacks for edge cases. Some teams keep Axios as a fallback client when a particular feature is easier to express there. The main point is to treat scaling as an engineering problem rather than simply increasing loop size.
When to move operational work to a managed scraping service
At some point, plain fetch carries too much operational weight. You may still want simple Node.js code without managing proxies, headless browsers, retries, and anti-bot defenses yourself. A managed scraping API changes the request target: instead of calling the website directly, your fetch call goes to the service and passes the target URL as a parameter. The service handles rotating proxies, JavaScript rendering, retries, and bot defenses while your code focuses on parsing and business logic.
The request model stays familiar: it is still a fetch call, still asynchronous, still predictable. The operational complexity moves out of your application.
A minimal integration is shown in the protected code module code0024. It reads an API key from an environment variable, builds URLSearchParams with the target URL, calls the service endpoint, checks response.ok, and reads the returned HTML as text. From there, the HTML can go into Cheerio or another parser. The code module is retained exactly as supplied.
Managed tooling does not remove responsibility. Fetch only the data you need, avoid collecting sensitive or personal information, respect robots.txt, site terms, and local laws, and log errors and usage so failures are visible.
Once scripts are stable, teams often automate around them. Scheduled scrapes can send results to a spreadsheet, and parsed data can be forwarded to a webhook or internal service. Automation platforms can handle orchestration, notifications, and delivery while developers maintain the core logic.
A decision checklist before you ship
Use this checklist to separate the common failure modes and choose the right level of tooling.
- Did the request fail at the network level, or did the server return an HTTP error? Fetch rejects only for the former. Check response.ok for the latter.
- Does the response body match the reader? Use response.json() for JSON and response.text() for HTML or plain text. A body can be read only once.
- Is there a timeout? If a request can hang, add AbortController and clear the timer in finally.
- Is the retry policy intentional? Retry temporary failures and selected 5xx responses; avoid blind retries on 4xx. Respect Retry-After and cap attempts.
- Are headers being read correctly? response.headers is a Headers instance; use .get() or iterate over it.
- Is the page server-rendered? If the useful content appears only after client-side JavaScript, plain fetch plus Cheerio will not see it; a headless browser or managed service may be required.
- Is a proxy needed? On Node 18–23, native fetch needs an Undici dispatcher. On Node 24+, environment proxies require NODE_USE_ENV_PROXY=1.
- Are legal and ethical rules satisfied? Check robots.txt, terms of service, privacy rules, request volume, and data handling.
- Is failure observable? Log errors and monitor usage so repeated failures do not go unnoticed.
| Name | Where it appears | What it controls or reports |
|---|---|---|
| url | fetch(url, options) | The destination of the request |
| options | fetch(url, options) | Optional object carrying method, headers, and body |
| method | options | HTTP method such as GET, POST, PUT, PATCH, or DELETE |
| headers | options | Request headers, including Content-Type |
| body | options | Request payload; JSON must be stringified |
| signal | options | AbortController signal used to cancel a request |
| dispatcher | options | Undici dispatcher used for proxy routing |
| response.ok | Response | Boolean indicating a successful status, typically 2xx |
| response.status | Response | Numeric HTTP status code |
| response.statusText | Response | Human-readable status message |
| response.headers | Response | Headers instance for reading response headers |
| response.json() | Response | Reads and parses the body as JSON |
| response.text() | Response | Reads the body as plain text or HTML |
| response.arrayBuffer() | Response | Reads the body as binary data |
Field names and behaviors are drawn from the supplied technical examples. This is a reference map, not a runtime test report.
| Failure type | Retry? | Notes |
|---|---|---|
| Network errors | Yes | Temporary connectivity problems can clear on a later attempt |
| Timeouts | Yes | Use a bounded timeout and a capped number of attempts |
| 502, 503, 504 | Yes | Often indicate a temporarily overloaded upstream |
| Other 5xx | Sometimes | Consider the specific service and failure pattern |
| 400, 401, 403, 404 | Usually no | Often permanent until the request changes |
| 429 | Follow rate limits | Respect Retry-After and add delays |
This table summarizes the retry rules described in the guide. It does not guarantee that a given upstream will recover.
| Node.js version | Agent option | Environment proxy variables | Practical approach |
|---|---|---|---|
| 18–23 | Not supported by native fetch | Not honored automatically | Use an Undici dispatcher such as ProxyAgent |
| 24+ | Not the native-fetch mechanism | Can be respected when enabled | Set NODE_USE_ENV_PROXY=1 or use --use-env-proxy, or use a dispatcher |
Version behavior is stated as of the guide's technical description. Verify against your installed Node.js release before deploying.
Frequently asked questions
What Node.js version do I need to use the Fetch API without installing node-fetch?
Node.js 18 or newer includes fetch globally. On Node.js 24 it is available by default, so no extra HTTP client is required.
How do I handle HTTP errors and response.ok when using Fetch in Node.js?
Fetch rejects only on network-level errors. For 4xx and 5xx responses it still resolves with a Response object. Check response.ok or response.status explicitly and throw or handle your own error when the response is not OK.
What is the difference between the Fetch API and Axios for HTTP requests in Node.js?
Fetch ships with Node.js itself and uses the same Request and Response conventions browsers expose. Axios is an add-on package: it bundles conveniences such as automatic JSON handling and extra features, but you take on another dependency and a separate API to learn. If your runtime already provides fetch and you do not need Axios-specific behavior, the built-in option keeps your dependency list shorter.
Can I use Fetch in Node.js for web scraping, and what rules should I follow?
Yes. Fetch can download HTML, and a parser such as Cheerio can extract the parts you need. Access alone does not settle the rules, though: review robots.txt, the site's terms of service, and applicable local law, and steer clear of personal or restricted data. Being able to request a page is not the same as being permitted to use its contents.
How do I send JSON data in a POST request with Fetch in Node.js?
Choose POST as the method, declare application/json in the Content-Type header, and pass the serialized object as the body (JSON.stringify(yourObject)). When the reply arrives, pick response.json() or response.text() according to the format the endpoint actually returns.
How can I set headers and read response headers using Fetch in Node.js?
Send request headers through the headers option in fetch. Read response headers with response.headers.get("content-type") or by looping over response.headers with for (const [key, value] of response.headers).
When should I switch from plain Fetch-based scrapers to a managed scraping API?
Consider switching when scaling problems appear: IP blocks, CAPTCHAs, geo restrictions, JavaScript-heavy pages, or too much time spent maintaining proxies and headless browsers. At that point a managed service can reduce operational work.