Skip to content

Web Scraping with node-fetch

node-fetch brings the browser Fetch API to Node.js, so you can write HTTP requests on the server without learning a separate client. This guide walks through installing node-fetch and cheerio, fetching a page, pulling out the data you need, customizing requests with the options object, and running multiple fetches at once.

Prerequisites

You will get the most from this guide if you are comfortable with ES6 JavaScript and understand promises and async/await. The examples use a package.json file and npm to install dependencies.

What is the Fetch API?

Fetch is a specification that standardizes requests, responses, and the process in between. Both the browser fetch API and node-fetch implement that specification. Compared with XHR, fetch is built around Promises, which removes callback nesting and the verbose XHR API.

One behavior to remember: when a request returns an HTTP status such as 404, the promise from fetch does not reject. The promise only rejects on network-level errors, so a 500 response can still resolve. Check the response status yourself when the outcome matters.

node-fetch brings the same model to the server side. Install it with npm install node-fetch and write requests much as you would in a browser.

Scraping the web with node-fetch and cheerio

node-fetch gives you the HTML as text, but you still need a way to extract specific elements. cheerio provides a JQuery-like API for parsing that HTML and querying the resulting DOM.

If you do not already have a package.json, run npm init to create one, then install both packages: npm install cheerio node-fetch.

The first example fetches a page and prints its HTML. fetch takes a single mandatory argument, the resource URL. It returns a promise that resolves to a Response object once the server sends headers; the body is not available yet at that point. The Response implements Body, a ReadableStream with promise-based methods for consuming the stream. Body.text() is one of those methods, and calling it returns a promise that resolves to the data.

Extracting data with cheerio

Once you have the HTML text, cheerio.load() turns it into a queryable DOM. You can then use CSS selectors to find elements and each() to iterate over a list of nodes. For example, to collect the titles of posts on a page, find the selector for the post list in your browser's developer tools and pass it to cheerio.

In the example below, the class name ._eYtD2XCVieq6emjKBH3m is used to select titles on Reddit's home page. That class name may change, so verify the selector against the current page before relying on it. Inside each() you receive one element, which you can wrap with cheerio again to read its text.

Using the options parameter in node-fetch

fetch takes one mandatory argument and one optional argument: the options object. Use it to customize the request, for example to send cookies or to make a POST request instead of the default GET.

The most common properties are:

  • method — the HTTP method; defaults to GET.
  • headers — headers to send with the request.
  • body — the request body, used for example with POST.

To attach query parameters, the URL module's URLSearchParams class is convenient. To send a POST request, set method to POST; the same applies to PUT or DELETE. To send cookies, use the cookie header.

The example below builds a URL with query parameters, sets a POST method, adds a cookie header, and sends a JSON body.

Making fetch requests in parallel

When you need several URLs at once, running the requests one after another wastes time. HTTP requests are I/O-bound: the computer mostly waits for the server, so the work can be parallelized efficiently with promises.

Because fetch is promise-based, Promise.all lets you start multiple fetches together and wait for all of them. The example below fetches two product pages concurrently and returns a promise that resolves to a list of results.

Conclusion

node-fetch is a good fit for straightforward scraping tasks, but it is an HTTP client, not a browser. Pages that render most of their content with JavaScript, such as single-page applications, are harder to handle this way. Concurrency and other challenging parts still need to be managed in your own code.

Compared with a headless browser, node-fetch is lighter and faster for simple tasks such as submitting a form. If anti-scraping mechanisms block your requests, a managed scraping API can handle proxies and browsers for you.

Frequently asked questions

Does node-fetch reject the promise when the server returns a 404 or 500?

No. The promise resolves once the server responds with headers, even for error status codes. It rejects only on network errors such as connectivity problems, so check the response status when you need to treat a failed request differently.

Why do I need cheerio if node-fetch already downloads the page?

node-fetch returns the HTML as text. cheerio parses that text into a queryable DOM so you can select elements with CSS selectors and extract their content.

How do I fetch several pages at the same time?

Start each fetch and pass the resulting promises to Promise.all. Because HTTP requests are I/O-bound, running them concurrently avoids waiting for each response in sequence.

Can node-fetch scrape a single-page application?

node-fetch only retrieves the initial HTTP response. If the page builds its content with JavaScript in the browser, you will need a headless browser or a managed scraping service that executes JavaScript.