Skip to content

User Agent List for Web Scraping: Strings, Rotation and When to Automate

Every HTTP request carries a User-Agent string that tells the server which client is calling. Scrapers borrow real browser identifiers so their traffic looks like ordinary visits instead of a script. The tricky part is not finding a string — it is keeping the string, the surrounding headers, the IP and the rendering context consistent, and refreshing the pool before it goes stale. This guide gives you seed strings for the major browsers, a rotation pattern you can run yourself, and a clear rule for when to hand the job to an API.

What the User-Agent header actually does

The header is a short text field sent with the request. Servers read it for browser detection, analytics bucketing and feature toggles, so the value influences which HTML variant you receive. A minimal example looks like this:

Seed strings for the main browser families

Use these as starting points, not a permanent list. Chrome dominates real-world traffic, so a Chrome-heavy pool is a reasonable default, with Safari, Edge and Firefox mixed in for coverage. Refresh the pool roughly monthly or whenever a major browser version ships. The strings below reflect the versions captured in this guide; check current release numbers before relying on them.

Why a good string still gets blocked

Bot defenses rarely judge the User-Agent alone. They correlate it with Accept-Language, Accept, Sec-CH-UA client hints and compression settings, then with TLS and HTTP/2 fingerprints, JavaScript behaviour (canvas, WebGL, timezone, fonts, headless traits) and IP reputation. A default identifier such as python-requests/2.x or curl/7.x is an obvious automation signal and is quickly rate-limited or challenged on large surfaces. Even a realistic string fails if the rest of the request contradicts it — for example, claiming a Macintosh client while sending Windows-only TLS traits.

A field map for a coherent request

Treat the User-Agent as one field in a bundle that must agree with itself. Use this checklist before blaming the string:

Rotating a pool yourself

A workable manual setup keeps a small active pool of roughly 20–80 strings covering desktop and mobile, pairs each string with a matching header template, and assigns a proxy per request. Cycle through the pool rather than picking at random so coverage stays even, and add jitter to your delays so retries do not look mechanical. The two snippets below show a requests-based cycle and an httpx plus fake-useragent variant; pin library versions and refresh the pool periodically.

When to switch to a managed API

Manual rotation is fine for a small job. Move to a managed service when any of these become true: you exceed a few thousand requests per day or hit many domains; targets need client-side rendering, login flows or infinite scroll; you need SLAs, dashboards, retries and analytics; you want vendor support for robots.txt respect and rate management; or your engineers are spending time on User-Agent debates instead of extraction logic. A managed scraping API typically runs headless browsers, rotates proxies and assigns realistic agents with coherent headers, so you rarely set the header yourself. Confirm the specific capabilities, SLA and compliance posture with the vendor before relying on them.

Worked example: 200 OK with blank HTML

A publisher returned a 200 status with an empty body when requested with a custom User-Agent. The status code looked healthy, so the first instinct — retrying the same request — would not have helped. Switching to a Chrome string for Windows 10 (Win64; x64) with the AppleWebKit/537.36 (KHTML, like Gecko) segment and enabling headless rendering produced the expected content. The lesson: a realistic string only works when the browser context around it is also realistic. If you see the same symptom, log the full response, check for CAPTCHA or JS-challenge markers in the HTML, and try a different browser family before assuming the target is down.

Failure distinctions worth knowing

Different symptoms point to different fixes. A 429 usually means rate limiting — slow down, rotate IPs and add backoff with jitter. A 503 often signals a temporary block or overload — retry later with a different pool entry. An interstitial page containing a CAPTCHA form means the request was flagged — check header coherence and IP reputation before changing the string. A 200 with empty or near-empty HTML usually means the content is rendered client-side — enable JavaScript rendering or fetch the underlying endpoint. Treating all four as 'the User-Agent is wrong' wastes time.

Common mistakes to avoid

Stale version numbers are a red flag — a Chrome/88 string in 2026 stands out. Omitting Sec-CH-UA client hints while claiming a modern Chromium browser looks odd. Hammering thousands of requests through one string and one IP invites bans. Forgetting mobile coverage loses access to layouts and endpoints that only mobile clients see. Retrying without jitter looks robotic. And assuming the User-Agent alone determines success ignores IPs, TLS, JavaScript and cookies.

Legal and ethical baseline

Review the target's terms of service, robots.txt and applicable local law before scraping. Keep request rates modest, cache what you can and collect only the data you need. Some managed scraping APIs offer rate and compliance controls; confirm the specific capabilities with your vendor. The responsibility for lawful use stays with you.

Manual rotation vs managed API
SituationManual pool is fineManaged API is the better fit
Request volumeA few hundred to a few thousand per dayAbove a few thousand per day or many domains
Page typeStatic HTMLClient-side rendering, login flows, infinite scroll
OperationsYou can watch logs yourselfYou need documented SLAs, dashboards, retries and analytics from your vendor
ComplianceYou handle robots.txt and rate limitsConfirm the vendor documents robots.txt and rate-management support
Team timeEngineers can maintain the poolEngineers should focus on extraction logic

Thresholds reflect the captured guidance; confirm current limits with the provider.

Symptom-to-fix map
SymptomLikely causeFirst diagnostic step
429 Too Many RequestsRate limitingSlow down, rotate IPs, add backoff with jitter
503 Service UnavailableTemporary block or overloadRetry later with a different pool entry
CAPTCHA interstitialRequest flaggedCheck header coherence and IP reputation
200 OK with empty HTMLClient-side renderingEnable JavaScript rendering or fetch the underlying endpoint

Symptoms are indicative; confirm with full request/response logs.

Request field checklist
FieldWhat it should match
User-AgentThe claimed browser family and version
AcceptThe claimed browser's typical content types
Accept-LanguageThe claimed locale
Accept-EncodingThe claimed browser's compression support
Sec-CH-UAThe claimed Chromium version family
Sec-Fetch-*The claimed navigation context
Proxy / IPThe claimed geography and network type
render_jsWhether the target needs client-side rendering
extract_rulesThe selectors or JSON rules for structured output

Field names reflect the captured request contract; confirm current parameter names in the provider documentation.

Frequently asked questions

What is a User-Agent in web scraping?

It is the HTTP header that identifies your client software to the server. Servers use it for feature targeting and analytics; scrapers use realistic values so their traffic resembles ordinary browser visits.

Why do sites block default User-Agents?

Defaults such as python-requests/2.x or curl/7.x are obvious automation signals. A realistic string helps only when paired with proper headers and clean IP reputation.

Which User-Agent strings work best?

Current Chrome-family strings are the safest baseline, followed by Safari (mobile and desktop), Edge and Firefox. Choose based on your target audience and keep the pool fresh.

How often should the pool be refreshed?

Roughly every 30–45 days, or whenever a major browser version ships. Pull from sources that track current browser releases.

How do I rotate User-Agents in Python?

Keep a pool and cycle through it per request, pairing each string with a matching header template. Libraries such as fake-useragent help, but pin versions and refresh the pool periodically.

How does a managed API handle rotation?

A managed scraping API typically handles headers, JavaScript rendering and proxies on its side, so you often do not set a User-Agent yourself. Confirm the exact behavior with your vendor.

What is the difference between a User-Agent and a browser fingerprint?

The User-Agent is a single header string. A fingerprint is the composite of headers, TLS, canvas, fonts, timezone and more. Matching both is what makes a request reliable.

Can mobile User-Agents be used safely?

Yes. Mobile strings often return simpler DOMs or different endpoints. Rotate responsibly, respect site policies, and include Android Chrome and iOS Safari strings in the pool.

Are there legal or ethical concerns with fake User-Agents?

Yes. Review terms of service, robots.txt and local law. Use ethical patterns such as rate limits, caching and data minimisation. A managed API can help enforce rate and compliance constraints programmatically.