Web Scraping Headers: Build Requests That Look Like Browser Traffic
Headers are the metadata a client sends with an HTTP request. For scraping, they do more than describe a request: they help a server decide which language, format, compression and session content to return, and whether the request resembles a person using a browser. A default library identity is often the fastest way to be recognised as automated, so the practical task is to replace it with a current, internally consistent browser profile and then test what the server actually receives.
Request headers versus response headers
Request headers travel from client to server and describe what the client wants and who it claims to be. Response headers travel back and describe what was returned, including content type, status and any new cookies. Confusing the two makes debugging harder: if a page returns the wrong language, check the request language preference; if a session is lost, inspect the response Set-Cookie handling and the Cookie value sent on the next request.
Headers can change the result, not only the chance of being blocked. A language preference can select a localised page, an Accept value can request JSON instead of HTML, and a missing Referer can cause some sites to refuse a page. That is why header work belongs in both reliability and data-quality testing.
The core header set
User-Agent: identifies browser and operating system and is usually checked early. Use a current version; an old string can look as suspicious as a library default.
Accept: states the content types the client can handle, which matters for HTML pages and API endpoints.
Accept-Language: states language preference. Keep it plausible for the request context; a mismatch with location can be a warning signal.
Referer: indicates the page the user arrived from. For a product page reached from a category, using that category URL makes the navigation pattern more natural.
Accept-Encoding: lists supported compression such as gzip, deflate and br. Correct handling can reduce transfer size and improve speed.
Cookie: carries session state. Many sites need it for login, prices or personalised content, so preserve cookies across related requests rather than sending a fresh request each time.
The supplied public contract modules illustrate these values: code0002 and code0003 give Chrome and Firefox User-Agent examples, code0004 gives an Accept-Language example, code0005 gives a Referer example, and code0006 gives an Accept-Encoding example. Module code0008 shows a fuller Chrome-style dictionary, while code0010 shows the default python-requests identity that should be replaced.
Copy a realistic set from Chrome DevTools
Copying a working browser set is usually more reliable than inventing one. Open the target site in Chrome, press F12, open the Network tab, reload, select the main document request, and open the Headers tab. Focus first on User-Agent, Accept, Accept-Language, Referer and Cookie. Headers such as Connection or Upgrade-Insecure-Requests can usually be ignored unless a sensitive site checks everything.
To move the set into Python, right-click the request in the Network tab, choose Copy, then Copy as cURL, and convert the command to Python. The resulting dictionary preserves the combination a real browser sent. Before running against the target, send the same set to an echo endpoint such as httpbin.org/headers and compare what the server reports with what you intended to send. Module code0009 illustrates that testing pattern.
Apply and test the headers in Python
For a short script, pass headers directly with the request. For a larger project, define them once as a dictionary so the profile can be reviewed, reused and updated. Always override the library default User-Agent. Module code0007 shows the direct approach and module code0008 shows the separate dictionary approach.
Use this checklist before a run:
- Replace the default library User-Agent with a current browser string.
- Confirm Accept matches the expected response type, HTML or JSON.
- Confirm Accept-Language matches the content you want.
- Set Referer to the previous page when navigation is part of the workflow.
- Preserve Cookie and session state across requests that belong together.
- Send the set to an echo service and inspect the server-side view.
- Check whether a proxy adds headers such as X-Forwarded-For; module code0013 shows how to inspect that.
- Keep the whole set consistent: a Chrome User-Agent should be paired with Chrome-style Accept and related values, not a mixture copied from different browsers.
Advanced consistency checks
Basic headers may not be enough for sites with stronger fingerprinting. Rotate among several realistic User-Agents during long sessions so one identity does not produce an obvious pattern. Preserve header order where a site checks it; module code0012 shows an ordered header list using httpx, because a plain dictionary does not guarantee the order a browser would send.
Review proxy-added headers, refresh browser version strings regularly, and copy complete header sets from one browser rather than mixing values from different sources. Module code0014 contrasts an outdated Chrome string with a current one. These steps address consistency, freshness and order, which are the details that separate a plausible browser profile from a collection of individually valid values.
| Header | What it signals | When it matters most |
|---|---|---|
| User-Agent | Browser and operating system | Every request; replace library defaults |
| Accept | Preferred content types | HTML pages and API responses |
| Accept-Language | Preferred language | International or localised sites |
| Referer | Previous page | Navigation between pages on the same site |
| Accept-Encoding | Supported compression | Performance and browser realism |
| Cookie | Session state | Login, prices and personalised content |
Values should be copied as a coherent set from a real browser and kept current.
Frequently asked questions
What are HTTP headers and why do they matter for scraping?
They are metadata attached to an HTTP request. They describe the client, preferred content and language, where the request appears to come from, and session state. Servers use those signals to decide what content to return and whether the request resembles ordinary browser traffic.
How can headers be configured to reduce detection?
Start with a current browser User-Agent, Accept, Accept-Language, Referer where navigation matters, Accept-Encoding, and Cookie where a session is required. Copy a coherent set from a real browser, keep values consistent with the claimed browser, rotate User-Agents for longer runs, and verify the result with an echo service such as httpbin.org before using it against a target site.
Which headers are essential?
User-Agent identifies the browser and operating system. Accept states preferred content types. Accept-Language states language preference. Referer indicates the previous page. Accept-Encoding lists supported compression methods. Cookie carries session state when login or personalised content is needed.
How can real browser headers be extracted for a scraper?
Open Chrome DevTools with F12, use the Network tab, reload the page, select a request, and inspect its Headers tab. Right-click the request, choose Copy as cURL, then convert that command to Python. That preserves a realistic combination instead of assembling unrelated values by hand.