Skip to content

Apache Forward Proxy Setup for Scraping

A proxy server sits between your client and a destination site: your request goes to the proxy, the proxy forwards it, and the response comes back through the same path. For scraping work, that intermediary can hide your originating IP, give you a different address after a block, or place your request in another region. This guide shows how to turn an Apache installation into a working forward proxy, test it, and point scraping clients at it. It also explains where a self-managed proxy stops being practical.

Why scrapers use a proxy

Three practical reasons come up repeatedly. First, privacy: the destination sees the proxy's address rather than the scraper's. Second, block recovery: if one proxy address is refused, you can switch to another. Third, regional access: a proxy located in a specific country can reach content that is limited to that country. Apache is one of several ways to build such a server yourself; Nginx and dedicated proxy software such as Squid are alternatives. The steps below use Apache because its proxy modules ship with the standard package.

What you need before starting

The walkthrough assumes a machine running a current Ubuntu release and Node.js for the demo clients. Apache commands and package names differ on other operating systems, so consult the platform documentation if you are not on Ubuntu. You do not need to write application code to enable proxying; Apache's modules handle the forwarding.

Step 1: Install Apache and enable the proxy modules

Install the Apache package, then enable the three modules that provide proxying: the core proxy module, the HTTP proxy module, and the CONNECT method module used for HTTPS tunnelling. The exact commands are supplied in the protected code modules below and should be run as shown.

Step 2: Create the proxy VirtualHost

Copy the default site configuration to a new file, then replace its contents with a VirtualHost that turns proxying on. The key directives are ProxyRequests, which makes Apache behave as a forward proxy, and ProxyVia, which adds a Via header so you can trace a request through a chain of proxies. The <Proxy> block controls who may use the proxy; the supplied example allows any host, but you can restrict hosts or add authentication. The complete configuration is in the protected module below.

Step 3: Activate the site and reload Apache

Enable the new site configuration and reload the service so the change takes effect. The commands are supplied exactly in the module below.

Step 4: Verify the proxy with cURL

Send a GET request through localhost on port 80 to a request-echo service. The response should show the request arriving from the proxy's address rather than your machine's, which confirms the forwarding path works. A POST request with a JSON body should likewise arrive intact, with the JSON echoed back. The exact commands and expected responses are in the protected modules below; the IP shown in the sample output is illustrative and will differ on your machine.

Step 5: Route Selenium traffic through the proxy

For a browser-driven scraper, the proxy is configured on the driver rather than in the page request. In the Selenium example, a Firefox options object is built, then the driver is created with a manual proxy mapping that sends HTTPS traffic to localhost on port 80. The script navigates to a target page and collects the text of every h2 element. The setProxy call is the line that actually redirects traffic; without it the browser would connect directly. The full script is retained in a private source module and is not reproduced here.

Step 6: Point a scraping API at your proxy

A hosted scraping service cannot reach a proxy that only listens on your local machine, so the proxy must be exposed through a tunnel. A TCP tunnel opened on port 80 yields a public host and port; that host and port, without the tcp:// prefix, become the value of the own_proxy parameter. The same request can also carry an extract_rules object that selects all h2 elements and returns them as a list. The client code is retained in a private source module; the expected extracted output is supplied in a protected module below.

Where a self-hosted proxy stops being enough

A local Apache proxy has the same public address as the machine it runs on, so it does not by itself solve IP blocking. To get a different address you must deploy the proxy to a remote host, and an unauthenticated proxy exposed to the internet is a serious risk to you and to others. Adding authentication helps but makes securing the server harder. Even then, a single proxy address can be blocked, so you need a pool and a rotation strategy. Those operational burdens are the main reason teams move to a managed proxy service, which supplies rotating addresses and can offer regional or stealth options for difficult targets.

Managed proxy options in a scraping API

When you use a hosted scraping API, proxy selection can be a request parameter rather than a server you maintain. The premium_proxy parameter requests a rotating premium pool that can also reach region-restricted content. The stealth_proxy parameter targets sites that are especially hard to scrape. Both are shown in the protected modules below, alongside the own_proxy parameter used earlier for a self-hosted proxy.

Decision checklist before you commit

Use this checklist to decide whether a self-hosted Apache proxy fits your task.

  • Do you need a different public IP, or only a single forwarding hop? A local proxy does not change your public address.
  • Can you deploy to a remote host and secure it with authentication and firewall rules? If not, do not expose it.
  • Do you have a pool of addresses and a rotation plan for when one is blocked? A single proxy will eventually be refused.
  • Do you need regional addresses or stealth behaviour for difficult targets? Those are managed-service features.
  • Can your scraping client reach the proxy over the public internet, or does it need a tunnel? Hosted services require a reachable endpoint.

If most answers point to operational work you do not want to own, a managed proxy is the simpler path.

Field and control map

The table below maps each configuration element to its role so you can check a setup quickly. It is a reader aid, not a replacement for the exact protected modules.

Apache proxy and client proxy controls
ControlWhere it appearsWhat it does
ProxyRequestsApache VirtualHostTurns Apache into a forward proxy
ProxyViaApache VirtualHostAdds a Via header to trace requests through proxy chains
<Proxy> blockApache VirtualHostDeclares which hosts may use the proxy
setProxySelenium driver setupSends browser HTTPS traffic to the given proxy
own_proxyScraping API requestRoutes the request through a self-hosted proxy host and port
premium_proxyScraping API requestUses a managed rotating premium proxy pool
stealth_proxyScraping API requestUses a managed stealth proxy for difficult targets

Directive names and parameter names are exact; values and availability depend on your Apache version and the API plan you use.

Frequently asked questions

Is this a forward proxy or a reverse proxy?

It is a forward proxy. Setting ProxyRequests to On makes Apache accept requests from clients and forward them to destination servers. A reverse proxy would sit in front of your own servers instead.

Does the Apache proxy handle HTTPS?

The setup enables proxy_connect, which is the module used for HTTPS tunnelling through a forward proxy. The supplied VirtualHost also turns SSL off for the proxy host itself, so the proxy listens on plain HTTP and tunnels HTTPS requests.

Will a local Apache proxy hide my IP address?

No. A proxy running on your own machine shares that machine's public address, so the destination still sees the same IP. To change the visible address you need the proxy hosted elsewhere, or a managed proxy service.

Why do I need a tunnel for the scraping API?

A hosted service makes requests from its own network, so it cannot reach a proxy listening only on your localhost. A TCP tunnel gives the proxy a public host and port that the service can connect to. The own_proxy value uses that host and port without the tcp:// prefix.

Should I add authentication to the proxy?

Yes, if the proxy is reachable from the internet. The supplied example allows any host, which is fine for a local test but unsafe for a public endpoint. Restrict access by host or add authentication before exposing it.

curl -x localhost:80 https://httpbin.org/get # The -x parameter sets the proxy server
{
  "args": {},
  "headers": {
	"Accept": "*/*",
	"Host": "httpbin.org",
	"User-Agent": "curl/8.1.1",
	"X-Amzn-Trace-Id": "Root=1-65094ba7-1676f27d7b6ba692057a7004"
  },
  "origin": "103.44.174.106",
  "url": "https://httpbin.org/get"
}
curl -x localhost:80 -X POST https://httpbin.org/post -d '{"foo": "bar"}' -H 'Content-Type: application/json'
{
  "args": {},
  "data": "{\"foo\": \"bar\"}",
  "files": {},
  "form": {},
  "headers": {
	"Accept": "*/*",
	"Content-Length": "14",
	"Content-Type": "application/json",
	"Host": "httpbin.org",
	"User-Agent": "curl/8.1.1",
	"X-Amzn-Trace-Id": "Root=1-65094d4e-5e06b56f5cd750a1786b61ba"
  },
  "json": {
	"foo": "bar"
  },
  "origin": "103.44.174.106",
  "url": "https://httpbin.org/post"
}
{"headers": ["Render your web page as if it were a real browser .", "Render JavaScript to scrape any website .", "Rotate proxies to bypass rate limiting .", "Simple, transparent pricing.", "Developers are asking...", "Who are we?", "Contact us", "Ready to get started?"]}