Skip to content

Scrape and Test Free Public Proxies with an AI-Powered Workflow

Public proxy lists are easy to find and hard to trust. The entries change constantly, some protocols are mislabeled, and a proxy that answered a minute ago may already be blocked. This guide shows a repeatable pattern: fetch several list pages, ask an AI extraction step to return each proxy as protocol://ip:port, then test every candidate against a known page and keep only the ones that return the expected content within your timeout. The code examples are in Python, but the same sequence works in any language that can send HTTP requests directly and through HTTP or SOCKS proxies.

What you need before starting

You need three things from each list page: the protocol, the IP address, and the port. Everything else — country, anonymity level, uptime claims — is optional metadata. The extraction step in this workflow is built around those three fields so the output is directly usable by a proxy-aware HTTP client.

For the AI extraction path you need an API key for the extraction service. The Python examples also use the requests library and, for SOCKS proxies, PySocks. If you prefer classic HTML parsing, you can replace the AI extraction call with selectors per site; the testing and reporting stages stay the same.

Step 1: Set up the extraction client

Create a client object with your API key and keep it in one place so every list request uses the same configuration. The exact client initialization is shown in the accompanying code module; treat the key as a secret and load it from an environment variable rather than pasting it into a shared file.

Step 2: Choose the list pages to scrape

Start with a small set of list pages and add more only when you need broader coverage. The accompanying code module lists five example URLs; you can extend the list. Two practical selection criteria: the page should expose protocol, IP, and port in a readable form, and it should update often enough that freshly published entries are still usable. Pages that render their tables with JavaScript may need a rendering wait before extraction.

Python example

PROXY_LISTS = [
    'https://hide.mn/en/proxy-list/',
    'https://gologin.com/free-proxy/',
    'https://proxyscrape.com/free-proxy-list',
    'http://free-proxy.cz/en/',
    'https://gather-proxy.com/',
]

Step 3: Extract proxy entries with an AI request

For each URL, send a request that includes an instruction describing the desired output and a structured rule describing the field to return. The rule names a list field and asks for entries formatted as protocol://ip:port. Because the Python client sends a GET request under the hood, the structured rule must be serialized to a JSON string before it is passed as a query parameter. A rendering wait can be added for pages that need JavaScript.

The response is a JSON string. It may arrive wrapped in Markdown code fences, so the workflow strips backticks and a leading json marker before parsing. After parsing, each entry is lowercased and tagged with the source hostname so you can report results per source later. The exact request parameters and cleanup logic are in the accompanying code modules.

Step 4: Test each proxy against a known page

Send a request through each proxy to a page whose content you can predict, such as a simple example page with a known title. Record the elapsed time and check that the expected string appears in the response body. If the request completes and the string is present, mark the proxy as working. If anything fails, record the failure category rather than a generic error.

The accompanying code module shows the exact request shape, the timeout, and the exception handling. The failure categories are worth understanding because they point to different next actions:

  • proxy_error: the proxy rejected or could not establish the tunnel. Skip it; the endpoint is not usable as configured.
  • timeout: no response within the limit. The proxy may be slow, overloaded, or silently dropping traffic. Retest once with a longer limit before discarding.
  • connection_error: the connection failed before a usable response. Often the host or port is stale.
  • assertion_error: a response arrived but did not contain the expected content. The proxy may be injecting content, redirecting, or returning an error page.
  • value_error: the proxy URL could not be parsed into protocol, IP, and port. Fix the extraction format rather than the proxy.

A proxy that passes once is not guaranteed to pass later. Public proxies are shared, so treat a pass as a short-lived signal and re-test before relying on it.

Step 5: Summarize results by source

Two outputs are useful. The first is a table of working proxies with protocol, IP, port, source, and response time. The second is a per-source summary: how many entries were collected, how many worked, and the average response time among the working ones. The accompanying code module builds both tables and writes them as Markdown.

When comparing sources, look at the working rate alongside the average response time. A source with many entries but a low working rate costs more testing time per usable proxy; a source with a high working rate but slow responses may still be a poor fit for latency-sensitive work. The results below are a reference point, not a current measurement — re-run the workflow before making a decision.

Working proxies table

The table below shows the working-proxy output. It lists protocol, IP, port, source, and response time in seconds. The final row is a placeholder indicating that more entries existed in the full output. Public proxy endpoints change frequently and may no longer respond, so re-test before relying on any entry.

Source-wise statistics

This table summarizes each source: total entries collected, how many worked, and the average response time among working proxies. Use it to understand the shape of the comparison, then re-run the workflow for current numbers.

Comparing proxy sources by working rate and response time

The source-wise statistics show how differently two sources can behave. hide.mn returned 64 entries with 11 working and an average response time of about 6.1 seconds. gologin.com returned 50 entries with 37 working and an average of about 6.3 seconds. The working rate differs sharply: roughly 17% versus 74%. If your goal is usable proxies per test, gologin.com was the stronger starting point. If your goal is raw volume, hide.mn supplied more entries. Neither figure should be treated as current; the point is the comparison method — working rate first, then response time, then volume.

Decision checklist before you rely on a proxy list

  • Does the page expose protocol, IP, and port in a form your extraction step can read?
  • Does the page update often enough that entries are still fresh when you test them?
  • What is the working rate after your own test, not the rate claimed by the list?
  • What is the average response time among working entries?
  • Does the proxy pass a content check, not just a connection check?
  • Have you re-tested recently enough for your task?

If a source fails the content check often, it is not a parsing problem — the proxy is returning something other than the expected page. If it fails the connection check often, the entries are likely stale. If it fails the format check, fix the extraction rule before changing sources.

Why automate this instead of copying lists by hand

Public proxies are volatile: once published, they may be used by many people and blocked by the services you care about. Fetching and testing them on a schedule keeps your pool closer to usable. Automation also makes the comparison honest, because every candidate goes through the same test and the same timeout. The AI extraction step reduces the per-site parsing work, since one instruction can describe the desired output across pages with different HTML. The trade-off is that you still need a validation stage; extraction only gives you candidates.

Frequently asked questions

Do I need the AI extraction feature to use this workflow?

No. The AI request is one way to get protocol, IP, and port out of pages with different HTML. You can replace it with per-site selectors and keep the testing and reporting stages unchanged. The AI path is useful when you want one instruction to work across several list pages without studying each page structure.

Why does the extraction response sometimes arrive wrapped in Markdown?

The extraction service may return the JSON inside a code fence. The workflow strips backticks and a leading json marker before parsing. If parsing still fails, inspect the raw response and adjust the cleanup step rather than changing the proxy test.

What is the difference between a proxy error and a timeout?

A proxy error means the proxy rejected or could not establish the tunnel, so the endpoint is not usable as configured. A timeout means no response arrived within the limit; the proxy may be slow, overloaded, or dropping traffic. A timeout is worth one retest with a longer limit before you discard the proxy.

Why check for a specific string in the response instead of just checking that a response arrived?

A response can arrive from an error page, an injected page, or a redirect. Checking for a known string confirms that the proxy delivered the content you expected, not just that a connection was made. That distinction separates a working proxy from one that merely answers.

How often should I re-test public proxies?

Public proxies are shared and can be blocked soon after they are published. Re-test before each task that depends on them, and treat any earlier pass as a short-lived signal rather than a guarantee. The tables in this guide are a reference point, not a current measurement.

[
  {
    "url": "socks5://68.183.132.69:7497",
    "source": "hide.mn"
  },
  {
    "url": "http://137.184.100.135:80",
    "source": "free-proxy.cz"
  },
  ...some more objects...
]
Working free proxies
protocolipportsourceresponse_time
http46.47.197.2103128hide.mn0.63
http185.191.236.1623128hide.mn0.473
http45.88.192.5618080hide.mn5.663
socks4187.44.211.1184153hide.mn8.442
http119.188.93.25181hide.mn0.811
…more proxies

Endpoints may no longer respond; re-test before use.

Source-wise statistics
nametotalworkingavg_response_time (s)
hide.mn64116.066
gologin.com50376.275
proxyscrape.com1517.159
free-proxy.cz30191.779
gather-proxy.com20133.888

Re-run the workflow for current figures.