Make concurrent requests in Node.js
Fetching many pages one after another wastes time when most of the wait is spent on network responses.
Running a bounded set of requests together lets Node.js keep other work moving while it waits, without launching every URL at once.
Why concurrency helps scraping
Scraping is typically I/O-bound: the program spends most of its time waiting for remote pages rather than computing. Node.js suits this pattern because its event loop can service other requests while one is still waiting. Concurrency therefore cuts total elapsed time when network latency, not CPU work, dominates.
Promise.all, with a cap
Promise.all runs a group of promises together and settles when the group settles. It is simple and avoids separate operating-system processes. The risk is scale: passing the whole URL list to one Promise.all call starts every request immediately, which can breach the account limit and produce a 429 response. Always bound the group size.
Bounded-batch pattern
Install the client using protected module code0001, then use protected module code0002 for the runnable pattern. In outline, the script creates a client with the owner's API key, defines a per-URL function that fetches HTML and writes it to a uniquely named file, and defines a batching function that slices the URL array into chunks of the configured limit. Each chunk is awaited with Promise.all before the next chunk begins, so simultaneous requests never exceed the limit. Errors for an individual URL are caught and logged without stopping the batch.
Pre-run checklist and trade-offs
Use this checklist before a run: confirm the current concurrency limit in the dashboard; set the batch size to match or stay below it; keep the URL list as long as needed because the batching function handles any length; review per-URL error logging so failures are visible; and watch for a slow URL holding its batch. If that stall matters, switch to a concurrency-control library that starts a new request as soon as a slot frees up instead of waiting for a whole batch.
When to add workers or processes
If your code also parses large HTML responses or performs other CPU-intensive work, the single-threaded event loop may become the bottleneck. In that case consider worker_threads or multiple processes for the heavy computation. Regardless of processing model, keep simultaneous API requests within the account concurrency limit.
| Situation | Recommended approach | Watch out for |
|---|---|---|
| Mostly waiting on network responses | Bounded Promise.all batches | A slow URL delays its whole batch |
| Need faster slot reuse | Concurrency-control library | Added dependency and configuration |
| Heavy HTML parsing or CPU work | worker_threads or multiple processes | Still cap simultaneous API requests |
Guidance derived from the supplied tutorial units; limits depend on the owner's plan.
Frequently asked questions
How do I know how many requests can run at once?
Check your account dashboard for the limit tied to your plan, then set your batch size at or below that number. Exceeding it can return a 429 response.
Can I add more URLs than fit in one batch?
Yes. The batching function accepts any array length, so you can extend the URL list and it will process it in successive groups of the size you set.
What is the main drawback of fixed batches?
A slow request delays the rest of its batch because the next batch waits for the current one to finish. A concurrency-control library that refills a freed slot immediately can reduce that idle time.
When should I use worker_threads or multiple processes instead?
If parsing or other CPU-intensive work is significant, consider worker_threads or multiple processes. For mostly waiting on network responses, the event-loop approach with bounded Promise.all batches is usually sufficient.