Skip to content

Run PHP scraping requests in parallel

When a scraping API permits several operations at once, PHP can use that capacity instead of waiting for one page before starting the next.

OUTPUT FORMATSHTMLJSONMarkdown

A higher concurrent-request allowance means more calls can remain active simultaneously, so a large batch can finish sooner. The practical pattern below separates the single-request work into a reusable function, then starts separate processes for individual URLs.

Why concurrency helps

Parallel execution changes the bottleneck from sequential waiting to the number of requests your account allows at the same time. If the allowance is low, adding more processes will not create additional capacity. Check that limit first, then size the batch accordingly. The example workflow targets two pages, which makes it easy to verify that both processes start, request their assigned URL and print the result before scaling up.

How the example is structured

The protected PHP module, code0001, contains the complete implementation. In outline, it defines a scrape_website function that builds an API query from an API key and target URL, performs a GET request with cURL, checks for a failed transfer, then prints the HTTP status code and response body. A loop over a two-URL array uses pcntl_fork so each child process handles one URL and exits independently. The module uses the configured application base URL, API key placeholder and site base URL values already present in its contract.

Setup and scaling checklist

Before running the example, work through this checklist: confirm the API key is set in the module's configured variable; confirm the base URL points to the intended API origin; replace the two example target URLs with the pages you need; verify that the PHP environment supports process forking and cURL; keep the number of child processes at or below your account's concurrent-request allowance; and check each process output for its HTTP status code and response body. If a request fails, the module stops that process with the cURL error message and error code, so treat a missing response as a per-URL failure rather than assuming the whole batch failed.

Field and control map

A useful way to think about the design is as three layers. The configuration layer holds the base URL and API key. The request layer, scrape_website, knows how to turn one target URL into one API call and how to report success or failure. The orchestration layer holds the list of URLs and forks one process per item. Keeping those layers separate makes it straightforward to change the URL list or the credentials without rewriting the request logic, and to add or remove URLs by editing only the array and loop count.