User Agent List for Web Scraping: Strings, Rotation and When to Automate
Every HTTP request carries a User-Agent string that tells the server which client is calling. Scrapers borrow real browser identifiers so their traffic looks like ordinary visits instead of a script. The tricky part is not finding a string — it is keeping the string, the surrounding headers, the IP and the rendering context consistent, and refreshing the pool before it goes stale. This guide gives you seed strings for the major browsers, a rotation pattern you can run yourself, and a clear rule for when to hand the job to an API.
What the User-Agent header actually does
The header is a short text field sent with the request. Servers read it for browser detection, analytics bucketing and feature toggles, so the value influences which HTML variant you receive. A minimal example looks like this:
Seed strings for the main browser families
Use these as starting points, not a permanent list. Chrome dominates real-world traffic, so a Chrome-heavy pool is a reasonable default, with Safari, Edge and Firefox mixed in for coverage. Refresh the pool roughly monthly or whenever a major browser version ships. The strings below reflect the versions captured in this guide; check current release numbers before relying on them.
Why a good string still gets blocked
Bot defenses rarely judge the User-Agent alone. They correlate it with Accept-Language, Accept, Sec-CH-UA client hints and compression settings, then with TLS and HTTP/2 fingerprints, JavaScript behaviour (canvas, WebGL, timezone, fonts, headless traits) and IP reputation. A default identifier such as python-requests/2.x or curl/7.x is an obvious automation signal and is quickly rate-limited or challenged on large surfaces. Even a realistic string fails if the rest of the request contradicts it — for example, claiming a Macintosh client while sending Windows-only TLS traits.
A field map for a coherent request
Treat the User-Agent as one field in a bundle that must agree with itself. Use this checklist before blaming the string:
Rotating a pool yourself
A workable manual setup keeps a small active pool of roughly 20–80 strings covering desktop and mobile, pairs each string with a matching header template, and assigns a proxy per request. Cycle through the pool rather than picking at random so coverage stays even, and add jitter to your delays so retries do not look mechanical. The two snippets below show a requests-based cycle and an httpx plus fake-useragent variant; pin library versions and refresh the pool periodically.
When to switch to a managed API
Manual rotation is fine for a small job. Move to a managed service when any of these become true: you exceed a few thousand requests per day or hit many domains; targets need client-side rendering, login flows or infinite scroll; you need SLAs, dashboards, retries and analytics; you want vendor support for robots.txt respect and rate management; or your engineers are spending time on User-Agent debates instead of extraction logic. A managed scraping API typically runs headless browsers, rotates proxies and assigns realistic agents with coherent headers, so you rarely set the header yourself. Confirm the specific capabilities, SLA and compliance posture with the vendor before relying on them.
Worked example: 200 OK with blank HTML
A publisher returned a 200 status with an empty body when requested with a custom User-Agent. The status code looked healthy, so the first instinct — retrying the same request — would not have helped. Switching to a Chrome string for Windows 10 (Win64; x64) with the AppleWebKit/537.36 (KHTML, like Gecko) segment and enabling headless rendering produced the expected content. The lesson: a realistic string only works when the browser context around it is also realistic. If you see the same symptom, log the full response, check for CAPTCHA or JS-challenge markers in the HTML, and try a different browser family before assuming the target is down.
Failure distinctions worth knowing
Different symptoms point to different fixes. A 429 usually means rate limiting — slow down, rotate IPs and add backoff with jitter. A 503 often signals a temporary block or overload — retry later with a different pool entry. An interstitial page containing a CAPTCHA form means the request was flagged — check header coherence and IP reputation before changing the string. A 200 with empty or near-empty HTML usually means the content is rendered client-side — enable JavaScript rendering or fetch the underlying endpoint. Treating all four as 'the User-Agent is wrong' wastes time.
Common mistakes to avoid
Stale version numbers are a red flag — a Chrome/88 string in 2026 stands out. Omitting Sec-CH-UA client hints while claiming a modern Chromium browser looks odd. Hammering thousands of requests through one string and one IP invites bans. Forgetting mobile coverage loses access to layouts and endpoints that only mobile clients see. Retrying without jitter looks robotic. And assuming the User-Agent alone determines success ignores IPs, TLS, JavaScript and cookies.
Legal and ethical baseline
Review the target's terms of service, robots.txt and applicable local law before scraping. Keep request rates modest, cache what you can and collect only the data you need. Some managed scraping APIs offer rate and compliance controls; confirm the specific capabilities with your vendor. The responsibility for lawful use stays with you.
| Situation | Manual pool is fine | Managed API is the better fit |
|---|---|---|
| Request volume | A few hundred to a few thousand per day | Above a few thousand per day or many domains |
| Page type | Static HTML | Client-side rendering, login flows, infinite scroll |
| Operations | You can watch logs yourself | You need documented SLAs, dashboards, retries and analytics from your vendor |
| Compliance | You handle robots.txt and rate limits | Confirm the vendor documents robots.txt and rate-management support |
| Team time | Engineers can maintain the pool | Engineers should focus on extraction logic |
Thresholds reflect the captured guidance; confirm current limits with the provider.
| Symptom | Likely cause | First diagnostic step |
|---|---|---|
| 429 Too Many Requests | Rate limiting | Slow down, rotate IPs, add backoff with jitter |
| 503 Service Unavailable | Temporary block or overload | Retry later with a different pool entry |
| CAPTCHA interstitial | Request flagged | Check header coherence and IP reputation |
| 200 OK with empty HTML | Client-side rendering | Enable JavaScript rendering or fetch the underlying endpoint |
Symptoms are indicative; confirm with full request/response logs.
| Field | What it should match |
|---|---|
| User-Agent | The claimed browser family and version |
| Accept | The claimed browser's typical content types |
| Accept-Language | The claimed locale |
| Accept-Encoding | The claimed browser's compression support |
| Sec-CH-UA | The claimed Chromium version family |
| Sec-Fetch-* | The claimed navigation context |
| Proxy / IP | The claimed geography and network type |
| render_js | Whether the target needs client-side rendering |
| extract_rules | The selectors or JSON rules for structured output |
Field names reflect the captured request contract; confirm current parameter names in the provider documentation.
Frequently asked questions
What is a User-Agent in web scraping?
It is the HTTP header that identifies your client software to the server. Servers use it for feature targeting and analytics; scrapers use realistic values so their traffic resembles ordinary browser visits.
Why do sites block default User-Agents?
Defaults such as python-requests/2.x or curl/7.x are obvious automation signals. A realistic string helps only when paired with proper headers and clean IP reputation.
Which User-Agent strings work best?
Current Chrome-family strings are the safest baseline, followed by Safari (mobile and desktop), Edge and Firefox. Choose based on your target audience and keep the pool fresh.
How often should the pool be refreshed?
Roughly every 30–45 days, or whenever a major browser version ships. Pull from sources that track current browser releases.
How do I rotate User-Agents in Python?
Keep a pool and cycle through it per request, pairing each string with a matching header template. Libraries such as fake-useragent help, but pin versions and refresh the pool periodically.
How does a managed API handle rotation?
A managed scraping API typically handles headers, JavaScript rendering and proxies on its side, so you often do not set a User-Agent yourself. Confirm the exact behavior with your vendor.
What is the difference between a User-Agent and a browser fingerprint?
The User-Agent is a single header string. A fingerprint is the composite of headers, TLS, canvas, fonts, timezone and more. Matching both is what makes a request reliable.
Can mobile User-Agents be used safely?
Yes. Mobile strings often return simpler DOMs or different endpoints. Rotate responsibly, respect site policies, and include Android Chrome and iOS Safari strings in the pool.
Are there legal or ethical concerns with fake User-Agents?
Yes. Review terms of service, robots.txt and local law. Use ethical patterns such as rate limits, caching and data minimisation. A managed API can help enforce rate and compliance constraints programmatically.