Skip to content

Make concurrent requests in Node.js

Fetching many pages one after another wastes time when most of the wait is spent on network responses.

OUTPUT FORMATSHTMLJSONMarkdown

Running a bounded set of requests together lets Node.js keep other work moving while it waits, without launching every URL at once.

Why concurrency helps scraping

Scraping is typically I/O-bound: the program spends most of its time waiting for remote pages rather than computing. Node.js suits this pattern because its event loop can service other requests while one is still waiting. Concurrency therefore cuts total elapsed time when network latency, not CPU work, dominates.

Promise.all, with a cap

Promise.all runs a group of promises together and settles when the group settles. It is simple and avoids separate operating-system processes. The risk is scale: passing the whole URL list to one Promise.all call starts every request immediately, which can breach the account limit and produce a 429 response. Always bound the group size.

Bounded-batch pattern

Install the client using protected module code0001, then use protected module code0002 for the runnable pattern. In outline, the script creates a client with the owner's API key, defines a per-URL function that fetches HTML and writes it to a uniquely named file, and defines a batching function that slices the URL array into chunks of the configured limit. Each chunk is awaited with Promise.all before the next chunk begins, so simultaneous requests never exceed the limit. Errors for an individual URL are caught and logged without stopping the batch.

Pre-run checklist and trade-offs

Use this checklist before a run: confirm the current concurrency limit in the dashboard; set the batch size to match or stay below it; keep the URL list as long as needed because the batching function handles any length; review per-URL error logging so failures are visible; and watch for a slow URL holding its batch. If that stall matters, switch to a concurrency-control library that starts a new request as soon as a slot frees up instead of waiting for a whole batch.

When to add workers or processes

If your code also parses large HTML responses or performs other CPU-intensive work, the single-threaded event loop may become the bottleneck. In that case consider worker_threads or multiple processes for the heavy computation. Regardless of processing model, keep simultaneous API requests within the account concurrency limit.

Choosing a concurrency approach
SituationRecommended approachWatch out for
Mostly waiting on network responsesBounded Promise.all batchesA slow URL delays its whole batch
Need faster slot reuseConcurrency-control libraryAdded dependency and configuration
Heavy HTML parsing or CPU workworker_threads or multiple processesStill cap simultaneous API requests

Guidance derived from the supplied tutorial units; limits depend on the owner's plan.

Frequently asked questions

How do I know how many requests can run at once?

Check your account dashboard for the limit tied to your plan, then set your batch size at or below that number. Exceeding it can return a 429 response.

Can I add more URLs than fit in one batch?

Yes. The batching function accepts any array length, so you can extend the URL list and it will process it in successive groups of the size you set.

What is the main drawback of fixed batches?

A slow request delays the rest of its batch because the next batch waits for the current one to finish. A concurrency-control library that refills a freed slot immediately can reduce that idle time.

When should I use worker_threads or multiple processes instead?

If parsing or other CPU-intensive work is significant, consider worker_threads or multiple processes. For mostly waiting on network responses, the event-loop approach with bounded Promise.all batches is usually sufficient.