Direct Answer
To collect price-comparison data from idealo and Geizhals, request the public offer pages from a real German mobile IP so you see the offers, availability and shipping a German shopper sees, respect each portal's robots.txt and rate limits, hold one IP with a sticky session per run, and store the parsed offers on a schedule to build a comparable price history over time.
Why a German Viewpoint Decides the Data
idealo and Geizhals are Germany's two best-known price-comparison portals, and what they show is location-dependent. The merchants listed for a product, the availability shown, and especially the shipping cost added at the German checkout all depend on where the request appears to come from. Request the same product page from a foreign datacenter range and you may see a shorter merchant list, different shipping, a currency prompt, or a redirect — none of which matches the price a German buyer actually pays.
For competitive price intelligence that decision-makers rely on, that gap matters. If your dataset says a competitor undercuts you by a few euros, but the number came from a non-German view of the offer, you are acting on noise. A real German mobile IP returns the region-true page, so the offers, availability and total-with-shipping in your dataset are the ones a German shopper sees. That is the difference between decision-ready data and a misleading spreadsheet.
Mobile IPs also carry a different reputation profile than datacenter ranges. A German mobile IP is shared across many real subscribers via CGNAT, so it looks like exactly what it is — a genuine German mobile user — rather than a flagged server range. That is why datacenter IPs so often stall on defended portals, a pattern covered in why datacenter proxies get blocked.
Collect Public Offer Data — and Only That
This walkthrough is about public, non-personal offer and price data for legitimate market research: product name, the listed merchants, the price each shows, availability, and shipping. It is not about user accounts, reviews tied to people, or anything behind a login.
Keep the scope clean from the start:
- Public pages only. Product and offer listings that any visitor can open — nothing gated by an account or access control.
- No personal data. Do not collect names, user profiles, or anything that identifies a person. Offer and price fields are enough for price intelligence, and staying non-personal keeps you clear of the GDPR's core concerns.
- Attribute honestly. Identify your client truthfully in the User-Agent instead of impersonating a specific browser you are not.
Public price data can be lawful to collect, but the rules vary by portal and country — see is web scraping legal? EU & GDPR before you scale a run.
Read robots.txt and Set a Polite Cadence
Before your first request, read the portal's robots.txt. It is the site telling you, in plain terms, which paths it does not want automated clients to fetch. Honour it. If a path is disallowed, do not request it — there is almost always a public listing you can use instead, and respecting the file is both the courteous and the defensible choice.
Then set a cadence that a portal would never notice:
- Throttle between requests. Add real delays; do not fire product pages back-to-back. A steady, human-scale pace costs you a little time and saves you from tripping rate limits.
- Cache what you already have. Re-fetch a product only as often as its price realistically changes. Re-pulling the same page every minute is wasteful for you and rude to the portal.
- Back off on pressure. If you start seeing HTTP 429 responses, slow down rather than retry harder — the fix is patience, covered in fix 429 Too Many Requests.
The goal is to be a considerate guest on infrastructure someone else pays for. A polite cadence plus a clean German mobile IP is what keeps a monitoring job running for months instead of getting a single IP flagged in an afternoon.
Hold One IP with a Sticky Session
Comparison portals often render an offer list across a few steps — a region or cookie prompt, a "show all merchants" expansion, pagination through offers. If your IP changes mid-flow, the portal sees a session that jumped networks between clicks, which looks nothing like a real shopper and can reset the page state you were building up.
Use a sticky session so one German mobile IP holds for the whole product's read, then rotate only between products or between runs. That gives each product a clean, self-consistent session while still spreading your overall traffic across the pool. The trade-offs between holding and rotating an IP are laid out in sticky sessions and IP rotation.
A dedicated German mobile connection makes this predictable: you decide when the IP holds and when it rotates, instead of being handed a fresh random IP on every request. That control is exactly what a repeatable comparison pipeline needs.
Structure a Repeatable Comparison Pipeline
The value of price monitoring comes from repetition — the same products, measured the same way, over time. Build the run as a small, boring pipeline you can trust to produce comparable rows:
- Input list. Keep a stable list of the products (or product identifiers) you track on each portal. Comparing idealo and Geizhals is only meaningful when both point at the same product.
- Fetch. Request each public offer page through the German mobile proxy, on a sticky session per product, at your polite cadence. Verify a German mobile IP is actually in the path before a long run — load an IP-echo page and confirm a German mobile IP.
- Parse. Extract the fields you actually need — product, merchant, price, availability, shipping — and nothing more.
- Store with a timestamp. Write each offer with the portal, the product identifier, and a UTC timestamp so every row is comparable across days and across the two portals.
- Schedule. Run it on a cron or workflow schedule at a sensible interval, so a price history accumulates without a human in the loop.
Once the shape is stable, the two portals become two clean feeds you can diff against each other and against your own catalogue. For a full Playwright-based build of this pattern, see price-monitoring scraper with a German mobile proxy; the same discipline applies when you scrape marketplace listings like Amazon, where offer and price fields drive the same kind of decisions.
Why the Portals Are Defended — and the Honest Path
It is worth being clear-eyed about why idealo and Geizhals are bot-protected. Their aggregated offer data is the whole business, and every automated fetch adds load to servers they pay for. So they defend it — with rate limits, browser fingerprinting, and IP-reputation checks that quietly flag ranges known for abuse. That protection is reasonable, and it is not something to treat as an obstacle to defeat.
The legitimate path is the opposite of working around those defences. You understand why the block happens, and then you simply do not trigger it: request public data at a human-scale cadence, respect robots.txt, and come from a clean, dedicated German mobile IP that carries a genuine mobile reputation rather than a flagged datacenter one. A clean IP plus polite behaviour is not a trick — it is being the kind of client the portal has no reason to block.
For a dedicated pipeline that runs on a schedule, a fixed-price mobile connection is also easier to reason about than per-GB residential billing, since a steady monitoring job has a predictable footprint — a comparison drawn out in dedicated mobile vs pay-per-GB residential.
FAQ
Is it legal to scrape idealo or Geizhals price data?
Collecting public offer and price data for market research can be lawful, but it depends on the portal's terms and how you do it. Take only public, non-personal data, respect robots.txt and rate limits, and never work around access controls. Rules vary by site and country, so check each portal's terms and the GDPR before you run at scale.
Why do I need a German IP to compare prices on these portals?
idealo and Geizhals surface offers, availability and shipping costs based on where the request appears to come from. A non-German IP can return a distorted view — different merchants, different shipping, or a redirect. A real German mobile IP returns the region-true page a local shopper sees, so your comparison reflects the actual German market.
Why are price-comparison portals bot-protected?
Their offer data is their core asset, and aggressive scraping raises their infrastructure cost, so portals defend it with rate limits, fingerprinting and IP-reputation checks. The clean path is not to defeat that — it is to behave like a real client: one German mobile IP, a polite cadence, and respect for robots.txt.
CTA
Want region-true idealo and Geizhals data without getting a single IP flagged? Run your pipeline through a dedicated German mobile connection with sticky sessions and a fair-use 20 GB/day allowance — get a German mobile proxy.
Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or start with a dedicated 5G proxy.