Skip to content

How to Set Up a Rotating Proxy in Selenium with Python

Selenium can drive a real browser, but a single outbound IP address still becomes a bottleneck when a site rate-limits or blocks it. A rotating proxy pool spreads requests across multiple addresses. This guide shows how to wire that pool into a Python Selenium script, how to pass proxy credentials that Chrome ignores by default, and how to keep the scraper alive when a proxy fails.

Quick start: install and run the rotating proxy script

Install the pinned Selenium and Selenium Wire versions used in the reference script, then run the rotation example. Replace the sample proxy addresses and placeholder credentials with values from your own proxy provider before running anything.

The complete install commands and the full Python script are supplied as protected code modules code0001 and code0002. They are reproduced exactly, including the pinned versions, the proxy_pool list, the retry count, and the httpbin.org/ip verification step.

Command example

pip install selenium==4.17.2
pip install selenium-wire==5.1.0

Why proxy rotation matters for Selenium scraping

A proxy sits between the client and the target server. The target sees the proxy address rather than the client address, so a single proxy can still be identified and blocked after enough activity. Rotation changes the address between requests or after a time interval, which makes the traffic harder to attribute to one client.

Selenium itself does not rotate proxies. The script must choose an address from a pool and pass it to the browser before navigation. That is the core pattern in every example below.

Prerequisites

You need Python installed, a code editor, and the selenium and selenium-wire libraries. The reference setup used Python 3.12.2, but any supported Python 3 release should work if the library versions are compatible.

Selenium automates the browser. Selenium Wire extends it so proxy authentication and request inspection are possible. Install both before running the scripts.

Five steps to a rotating proxy in Selenium

  1. Choose a proxy provider that supplies a pool of rotating addresses. Confirm whether rotation happens per request or on a time interval.
  2. Collect the connection details: IP address, port, and username/password if the provider requires authentication.
  3. Optionally verify the proxy with cURL or a Python requests call before involving Selenium.
  4. Configure Chrome options. For an unauthenticated proxy, add the --proxy-server argument with a randomly selected address from the working pool.
  5. Navigate to a test URL such as httpbin.org/ip and print the response body. If the returned address matches the selected proxy, the browser is using it.

Adding a rotating proxy pool to Selenium

The reference implementation collects candidate proxies from a public list, tests each one against httpbin.org/ip with a five-second timeout, and keeps only the addresses that respond successfully. It then selects one working proxy at random and passes it through --proxy-server when creating the Chrome driver.

The full implementation is supplied as protected code module code0003. It includes the getProxies, testProxy, and rotateProxy functions exactly as written. The expected output is a printed proxy address that matches the address returned by the test page.

Free proxy lists are unreliable and short-lived. If you use one, expect to test addresses individually and to refresh the pool often. A paid provider usually reduces that maintenance burden.

How often should you rotate?

There is no universal interval. Three factors drive the decision:

  • Anti-scraping pressure. Aggressive sites may require a new address per request or every few requests. Lenient sites may tolerate rotation every few minutes.
  • Pool size. A large pool of high-quality addresses can be rotated less often. A small pool needs faster rotation to avoid reusing the same address.
  • Data volume and task complexity. Large or complex jobs tend to trigger anti-scraping mechanisms sooner, so they usually need more frequent rotation.

Treat these as starting heuristics and measure block rates against your own target.

Configuring authenticated proxies

Commercial and premium proxies often require a username and password. The proxy URL takes the form <PROXY_PROTOCOL>://<USERNAME>:<PASSWORD>@<PROXY_IP_ADDRESS>:<PROXY_PORT>, supplied exactly as protected code module code0004.

Chrome's --proxy-server option ignores embedded credentials by default. Selenium Wire solves this by accepting a proxy dictionary with separate http and https entries that include the credentials. Install Selenium Wire with the command in protected code module code0005, then switch the driver import from selenium.webdriver to seleniumwire.webdriver as shown in protected code module code0006.

On first run, Selenium Wire may produce a Chrome certificate error. Extract the certificate with the command in protected code module code0007 and install it in Chrome. If the credentials are wrong, the proxy responds with HTTP 407 Proxy Authentication Required, and the Selenium script fails with ERR_HTTP_RESPONSE_CODE_FAILURE. Check the username and password before changing anything else.

Command example

pip install selenium-wire

Handling errors and timeouts

Invalid proxies and slow responses should not crash the whole job. The reference pattern wraps the scraping block in a try/except, catches TimeoutException and ProtocolError, and retries with a different random proxy up to three times. It also sets a page-load timeout so a stalled navigation does not hang indefinitely.

The complete retry implementation is supplied as protected code module code0008. It includes the three-attempt loop, the ten-second page-load timeout, and the finally block that closes the driver even when an exception occurs.

A useful distinction: a 407 error points to bad credentials and will not improve by retrying the same proxy. A timeout or protocol error may be transient, so switching to another address is reasonable. Do not retry a credential failure against the same endpoint without changing the credentials.

When to use a managed scraping API instead

Maintaining a proxy pool, testing addresses, handling authentication, and retrying failures is real infrastructure work. A managed scraping API moves that work off your script: you send a request and the service handles proxy selection and browser rendering.

The reference alternative uses a Python SDK and a stealth_proxy parameter set to True. The SDK install command and the quick-start snippet are retained as private source-exact modules code0009 and code0010 and are not reproduced in this public copy. The described behavior is that the request returns HTTP 200 when the service successfully fetches the target page.

Choose a managed API when proxy maintenance is not the core of your project. Choose a self-managed pool when you need direct control over address selection, session stickiness, or provider-specific routing.

Pre-flight checklist

  • Proxy addresses and ports are replaced with your own values.
  • Credentials are stored outside the script, not hard-coded.
  • A test URL confirms the outbound address before the real target is requested.
  • The pool contains more than one working address.
  • A page-load timeout is set.
  • Retries use a different proxy and stop after a fixed count.
  • Credential failures (407) are separated from timeout failures in logs.
  • The driver is closed in a finally block.

Frequently asked questions

Does Selenium rotate proxies automatically?

No. Selenium passes whatever proxy configuration you give it. Rotation is implemented by your script, which selects an address from a pool and supplies it through Chrome options or Selenium Wire before navigation.

Why does Chrome ignore my proxy username and password?

The --proxy-server argument accepts a proxy address but Chrome ignores embedded credentials by default. Use Selenium Wire and pass a proxy dictionary with http and https entries that include the username and password.

What does a 407 error mean in a Selenium proxy setup?

HTTP 407 means Proxy Authentication Required. The proxy rejected the supplied credentials. Selenium typically surfaces this as ERR_HTTP_RESPONSE_CODE_FAILURE. Correct the username or password rather than retrying the same proxy.

How do I fix a Chrome certificate error when using Selenium Wire?

Run python -m seleniumwire extractcert to extract the certificate, then install it in Chrome. The command is supplied exactly in protected code module code0007.

Are free proxy lists suitable for production scraping?

They are unreliable and short-lived. Addresses can stop working quickly, so each one must be tested before use. For sustained scraping, a paid proxy provider or a managed scraping API is usually more stable.