Direct Answer
A scraper that works locally but fails on a server almost always has an IP problem. Your home connection has a trusted residential or mobile IP; a cloud server exits from a datacenter IP on a known hosting ASN that anti-bot systems flag on sight. Route the server's requests through a clean German mobile IP for German targets, match your headers, and keep the country consistent — and it behaves like your laptop again.
The Single Most Common Cause: Your Server Has a Datacenter IP
This is one of the most common and most confusing problems in scraping and data work. The exact same code, unchanged, pulls the page cleanly on your laptop and then hits a 403 Forbidden, a CAPTCHA wall, or an empty response the moment it runs on AWS, Hetzner, GCP or any other cloud host. Nothing in your script changed — so the instinct is to hunt for a bug in the code. The bug is not in the code. It is in the network the code runs on.
When you test locally, your requests exit through your home internet connection. That gives you a residential or mobile IP registered to a consumer ISP or a mobile carrier — exactly the kind of address a real person browses from. Sites trust it by default. When you deploy to a server, your requests exit through the data center's IP instead. That address belongs to a hosting provider's published range, tied to an Autonomous System Number (ASN) that any website can look up in milliseconds. A real visitor almost never connects from a hosting ASN, so datacenter traffic gets flagged on sight — before your scraper has done anything at all.
So the failure is not really "on the server." It is on the IP the server hands you. Move the same request onto a trusted IP and the block disappears.
Why the IP Type Decides Everything
Every incoming request carries an IP, and the network behind that IP sets a baseline level of trust before the site reads a single header. That baseline is what IP reputation measures, and it splits cleanly by IP type:
| Where the request runs | IP type | Registered to | Baseline trust |
|---|---|---|---|
| Your laptop at home | Residential / mobile | Consumer ISP or mobile carrier | High |
| A cloud server (AWS, Hetzner, GCP) | Datacenter | Hosting provider ASN | Lowest |
Local development quietly borrows the trust of your home connection. You never see that trust because it is invisible and free — until you deploy and lose it. The cloud server's datacenter IP starts from the bottom of the trust hierarchy, so IP-aware targets apply extra checks or refuse outright. For the full three-way comparison, see mobile vs residential vs datacenter proxies.
The trap is that datacenter IPs work fine against lenient targets — internal tools, your own APIs, sites that do not grade IP type. So your scraper may run flawlessly on a server against one target and fail hard against another. The difference is not your infrastructure; it is whether that specific target scores the network behind the request.
Confirm It Really Is the IP Before You Change Anything
Before rewriting anything, prove the diagnosis. A few minutes of checking saves you from fixing the wrong thing.
Compare the two exit IPs
From your laptop and from the server, fetch a plain IP-echo endpoint from each. You will typically see a residential/mobile IP locally and a hosting-provider IP on the server. That contrast alone usually explains the block.
Isolate the variable
Run the identical request from both machines against the failing target:
- Fails only on the server, works on the laptop → almost certainly the IP type. This is the common case.
- Fails on both → the block is not location-specific; look at headers, request rate, or your account/session next.
- Works on both intermittently → you are likely being rate-limited, not IP-blocked. That is a different fix.
Read the actual response
Do not just log the status code. A 403 during scraping often ships a body explaining the reason — an anti-bot challenge page, a "suspicious traffic" notice, or a Cloudflare interstitial. That body tells you whether you are being blocked by IP, by fingerprint, or by rate. Diagnose first, then fix.
The Fix: Route the Server Through a Clean, Trusted IP
Once you have confirmed it is the IP, the fix is direct: stop letting the server use its raw datacenter IP, and route its requests through an IP the target trusts. For German target sites, that means a real German mobile IP.
A mobile proxy sends your server's traffic out through a genuine German mobile connection, so the destination sees an ordinary local mobile user instead of an AWS or Hetzner address. The reason a mobile IP carries the highest baseline trust is CGNAT — Carrier-Grade NAT: the mobile network maps thousands of real subscribers onto a single public IPv4 address, so that one IP represents a crowd of genuine people. A platform cannot treat it as a lone bot without risking many real, unrelated subscribers — so it is scored generously. Your server now inherits the same kind of trust your laptop had at home.
In practice you point your existing scraper at the proxy's host, port and credentials — the code barely changes. GermanProxy.io provides this as an IT routing / proxy service: you receive a host, port and login for a dedicated German mobile endpoint, with HTTP and SOCKS5 access, sticky sessions plus on-demand rotation, an OpenVPN config, and a fair-use allowance of 20 GB/day. Wiring it in follows the same pattern as any proxy — see the Python (requests / Playwright) and Scrapy guides, or the curl one-liner to test it in one command.
Don't Stop at the IP: Headers, Browser Context, and Geo
Fixing the IP resolves the large majority of "works locally, fails on server" cases. But three secondary differences between your laptop and your server can still cause trouble, and they are worth closing once the IP is clean.
Default headers give you away
Local testing frequently runs through a real browser — Selenium, Playwright, a headed Chrome — which sends a full, natural set of headers. A bare server script using a plain HTTP client sends minimal default headers with a giveaway User-Agent like python-requests/2.x. Match your headers to a real browser: set a realistic browser User-Agent, send a plausible Accept-Language (for German sites, include de), and keep your header order consistent with a real browser. This is not about deception — it is about not sending a request that obviously came from a raw script.
No browser context
Some targets need JavaScript to render, cookies to persist, or a TLS fingerprint that matches a browser. A headless run on the server that skips the browser entirely can fail where your local headed browser quietly succeeded. If the local success came from a real browser, reproduce that on the server — run the same headless browser through the proxy rather than a bare HTTP client. Anti-bot systems also read browser fingerprints, so a consistent browser context matters alongside the IP.
Geo mismatch
If you test from Germany and your server sits in a US region, the target sees a different country — different language, currency, and sometimes a different site entirely, or a geo-block. Keep the exit country consistent with your target. Scraping German sites from a German mobile IP keeps geography, language and IP all pointing the same way, which is exactly what a real local visitor looks like.
Keep It Working: Be a Polite Client
A clean IP gets you in; behaving reasonably keeps you in. Anti-bot systems exist to catch abuse, and legitimate research, monitoring and testing rarely trip them when you pace yourself.
- Match the IP type to the target. Strict, IP-aware German site → real German mobile IP. Lenient or internal target → the datacenter IP is fine.
- Pace requests like a human. Sensible delays protect your reputation far more than swapping IPs does. Bursts are what get 429 Too Many Requests.
- Respect robots.txt and work with public data. Scrape public information, honour crawl rules, and avoid storing personal data (GDPR) — this is our positioning, not an afterthought.
- Hold a session when the task needs continuity. A sticky session across a multi-step flow reads more naturally than switching IPs mid-task for no reason.
FAQ
Why does my scraper work locally but get 403 on the server?
Because the two machines exit from different IP types. Your home connection has a residential or mobile IP that sites trust; your cloud server has a datacenter IP from a known hosting range that anti-bot systems flag on sight. The code is identical — only the network behind it changed, and that is enough to trigger a 403.
How do I fix a scraper that fails only when deployed to AWS?
Route the server's requests through a clean residential or mobile IP instead of the raw datacenter IP AWS gives you. For German target sites, a real German mobile IP reads as an ordinary local user. Then match your headers to a real browser and keep the exit country consistent with the target.
Is it just the IP, or are headers the problem too?
The IP is the dominant cause, but headers matter as a secondary signal. Local testing often runs through a real browser with full headers, while a server script sends bare default headers that stand out. Fix the IP first, then align your User-Agent and header order with a real browser.
CTA
If your scraper runs clean locally and breaks on the server, the fastest fix is a trusted exit IP — route it through a real German 5G mobile IP and test your setup before you rely on it.
Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — compare German mobile proxy plans or start with a dedicated 5G proxy.