Skip to content

PHP and a Web Scraping API: Build a First cURL Request

This guide walks through the two pieces of a working PHP scraping client: the reusable request shape and a small tool that writes a page's HTML to disk.

OUTPUT FORMATSHTMLJSONMarkdown

It assumes PHP with the cURL extension available and an API key you can place in the request.

What you will build

Two short scripts. The first shows the general structure of an API request in PHP: initialise cURL, assemble a query string from parameters, send a GET request, and capture the response. The second turns that structure into a small tool that fetches a blog page's HTML and saves it to a local file. By the end you will know which parameters go into the request, which values come back, and how to tell a transport failure apart from an HTTP-level error.

The general request structure

Every request in this pattern follows the same order: create a cURL handle, set the base URL and API key, build the query string from an array of parameters, attach the full URL to the handle, choose GET, ask cURL to return the transfer as a string, execute, check for failure, and close the handle. The two request parameters are api_key and url. The response body is stored in $response. The complete PHP reference for this structure is supplied in the protected module code0001; use it as the canonical contract for parameter names and call order rather than retyping from memory.

Reading the response

Once the transfer succeeds, the response variable holds the body. You can print the HTTP status code with curl_getinfo($ch, CURLINFO_HTTP_CODE) and print the body alongside it. The exact two-line output pattern is supplied in protected module code0002. A useful habit is to read the status code before you act on the body: a 200 means the request reached the target and returned content, while other codes indicate the request was understood but not fulfilled as expected.

A worked example: save a blog page to disk

The second script reuses the same request shape but changes the target URL to a blog page and, after checking the status code, opens a local HTML file in write mode, writes the response body, and closes the file. The full script is retained privately as protected module code0003. The important sequence is: send the request, read the status code, then write the file. Writing before checking the status can leave an error page on disk instead of the content you expected.

Pre-flight checklist

Before running either script, confirm the following: the cURL extension is enabled in your PHP build; your API key is stored outside version control and injected at runtime; the target URL is a full URL including scheme; the query string is built with http_build_query so special characters are encoded; and the script closes the cURL handle on every path, including error paths. After running, confirm the HTTP status code is the one you expect and that the saved file contains HTML rather than an error message.

Failure distinction: transport vs HTTP

The scripts treat a falsy $response as a transport failure and stop with curl_error and curl_errno. That branch covers problems such as DNS resolution, connection refusal, or timeout. It does not cover a request that completed but returned a non-2xx status. To handle that case, read CURLINFO_HTTP_CODE after a successful transfer and branch on the code before parsing or saving the body. Keeping the two checks separate makes debugging much faster: a transport error points at the network or URL, while an HTTP error points at the request parameters or the target's response.

Field and control map

Request inputs: api_key (your credential) and url (the page to fetch). Request method: GET. Transfer behaviour: return the transfer as a string. Outputs: the response body in $response and the HTTP status code via curl_getinfo. File output in the worked example: a local HTML file. These names and their exact usage are fixed by the protected modules code0001, code0002, and code0003; treat them as the contract when adapting the scripts.

Frequently asked questions

Do I need a PHP SDK to call the API?

No. The supplied pattern uses PHP's built-in cURL extension directly. You initialise a handle, set the URL and method, execute, and read the response. The protected module code0001 contains the full request shape.

What is the difference between a cURL error and an HTTP error here?

A cURL error means the transfer itself failed, for example because of a bad host or a timeout; the script stops and reports curl_error and curl_errno. An HTTP error means the transfer completed but the status code was not the one you wanted. Check CURLINFO_HTTP_CODE after a successful transfer to catch that case.

Where should I store the API key?

Keep it out of source control and inject it at runtime, for example from an environment variable. The scripts show the parameter name api_key; the value should come from your own configuration, not a committed literal.

Can I save the response to a different file name?

Yes. The worked example writes to a local HTML file, but the file name is a local choice. Keep the write-after-status-check order so you do not persist an error page.