Direct Answer
Send the same headers a normal browser sends: a real User-Agent, matching Accept and Accept-Encoding, and an Accept-Language that fits the site's country. The point is consistency, not disguise — your headers should agree with each other, with your IP, and with your TLS stack, so you look like the ordinary client you already are.
Why Headers Decide How a Site Answers You
Every HTTP request carries a set of headers that describe your client: what software it is, what content it can handle, what language it prefers, how it found the page. Servers read these to decide what to send back — which template, which language, which compression, sometimes which price.
When you collect public data, the mistake is to think of headers as a costume. They are not a disguise; they are a description. A well-behaved scraper sends headers that accurately describe an ordinary client, because that is what you are: a program making normal requests for public information. The failure mode is not "too little disguise" — it is inconsistency, where one header contradicts another and the request stops looking like anything a real browser would send.
Missing or contradictory headers are one of the reasons a request gets a 403 Forbidden or a challenge page instead of the content. This article covers the headers that matter and, more importantly, the principle that ties them together.
The Headers That Matter
User-Agent
The User-Agent string names your client — the browser or library, its version, and the platform. Most default HTTP libraries send something like python-requests/2.x, which instantly marks the request as automated.
Send an honest, current User-Agent that reflects a real, up-to-date client. Do not invent an implausible one, and do not pin an old version that no real browser still ships. An outdated or contradictory User-Agent is more suspicious than an ordinary one, because it disagrees with the rest of your request.
Accept
Accept tells the server which content types you can handle. A browser asking for a web page sends a rich value like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. A default library often sends */* or nothing. Matching the Accept value a real browser sends for the resource you are requesting keeps the request coherent.
Accept-Language
Accept-Language states your preferred human languages. This is the header that quietly changes what you receive: many sites serve different copy, currency, prices or even redirects based on it — together with your IP's country.
For a German target, send Accept-Language: de-DE,de;q=0.9 so the site treats you as a German visitor. But the header alone is not enough. If you send de-DE from a US datacenter IP, the two signals disagree, and the site may ignore the header, geo-redirect you, or serve a generic international page. To reliably get the Germany-true response — the same page, prices and stock a local user sees — your Accept-Language and your IP should agree. That is why pairing de-DE with a real German mobile IP matters.
Accept-Encoding
Accept-Encoding lists the compression algorithms you support, such as gzip, deflate, br. Real browsers always send this, and servers use it to compress responses. If you claim an encoding your client cannot actually decode, you will get a response you cannot read — so send only what your HTTP library really handles (most handle gzip and deflate; many also handle br).
Referer
Referer tells the server which page linked you to the current one. For a request that a real user would only reach by clicking through a site, an absent or wrong Referer can look off. Where it is genuinely true — you are following a link you actually crawled from — sending an accurate Referer makes the sequence coherent. Do not fabricate a Referer for a page you never came from; a false one is just another inconsistency.
The Consistency Principle
Individual headers matter less than the picture they paint together. A site does not judge your User-Agent in isolation — it looks at the whole fingerprint and asks whether it hangs together.
Three layers have to agree:
- Your headers with each other — a
User-Agentthat claims a current desktop browser but sends a library-styleAccept: */*and noAccept-Encodingcontradicts itself. - Your headers with your IP —
Accept-Language: de-DEfrom a German IP is coherent; the same header from a flagged datacenter range in another country is not. - Your headers with your TLS stack — the way your HTTP client negotiates TLS (its cipher order and extensions, often summarised as a JA3/JA4 fingerprint) also identifies the software. If your
User-Agentclaims one browser but your TLS handshake looks like a scripting library, that mismatch is a strong signal on its own.
This is how websites detect proxies and bots: not by catching a single "bad" header, but by spotting a combination that no real client would ever produce. The defence is not more spoofing — it is fewer contradictions. Send a small, coherent set of real headers, keep them stable within a session, and route them through an IP whose country and reputation match the story your headers tell.
A Minimal, Coherent Header Set
For scraping a German site, a clean baseline looks like this — an honest set that describes a normal German-language browser session:
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "de-DE,de;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
}
proxies = {
"http": "http://USER:PASS@PROXY_HOST:PORT",
"https": "http://USER:PASS@PROXY_HOST:PORT",
}
resp = requests.get("https://www.example.de", headers=headers,
proxies=proxies, timeout=30)
print(resp.status_code)
Keep the User-Agent current, and if you use a real browser tool such as Playwright, let it set its own headers rather than overriding them by hand — a genuine browser is already consistent by default. For the wider setup, see the Python mobile-proxy guide.
Headers Alone Are Not Enough
Correct headers are necessary but not sufficient. If your IP carries poor IP reputation — a datacenter range already associated with abuse — no header set will make you look like a resident. The most convincing "ordinary client" signal is a real German mobile IP from GermanProxy.io, behind mobile carrier-grade NAT (CGNAT), where your traffic shares an address with ordinary phone users. That, combined with coherent headers, is what makes a request indistinguishable from a normal visitor because, in every technical sense, it is one.
Do this the clean way, too: request only public data, respect each site's robots.txt and terms, keep your request rate human-reasonable, and handle any personal data in line with the GDPR. Honest headers and a clean IP are a positioning, not a trick.
FAQ
Which HTTP headers matter most for web scraping?
User-Agent, Accept, Accept-Language, Accept-Encoding and Referer carry the most weight. User-Agent names your client, Accept states the content you expect, Accept-Language signals your locale, and Accept-Encoding lists the compression you support. Keep them coherent with each other and with your IP.
Should I use a fake User-Agent to scrape?
Use an honest, current User-Agent that matches the client and platform you actually run. Randomising to something implausible does more harm than good, because a User-Agent that contradicts your TLS handshake or Accept headers is easier to spot than a plain, consistent one.
Why does Accept-Language matter for German sites?
German sites often serve different prices, copy or redirects based on Accept-Language and your IP's country. Sending Accept-Language: de-DE from a real German mobile IP tells the site you are a local visitor, so you receive the same Germany-true response an ordinary German user would see.
CTA
Coherent headers work best behind an IP that matches them. Get a German mobile proxy — real German 5G IPs, HTTP/SOCKS5/OpenVPN, sticky sessions, fair-use 20 GB/day.
Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or get a dedicated proxy on the Blue network.