Direct Answer
To scrape Indeed.de and StepStone job data in Germany, collect only company and posting fields — job title, employer, location, contract type, salary if published — and never applicant or personal data. Read each site's robots.txt, throttle to polite rates, and request from a real German mobile IP so you see the same Germany-true listings a local candidate would, then structure the output as a clean, analysis-ready dataset.
What This Dataset Is For — and What It Must Never Contain
German job boards are one of the richest open sources of labor-market signal. Aggregate Indeed.de and StepStone postings over time and you can measure salary trends by role, hiring demand by region, which skills employers ask for, and how fast openings appear and disappear — a genuine alternative-data feed for recruitment analytics, workforce planning or economic research.
There is one line that decides whether this is a clean project or a liability, and it comes first, not as a footnote: collect company and posting fields only — never applicant or personal data.
A job posting is published by an employer to be seen. The company name, role title, location, contract type and any stated salary band are business facts about an open position. That is the material you want. What you must leave out is anything that identifies a person: a named recruiter, a contact email, a hiring manager's phone number, or any applicant information you might stumble across. Under the GDPR, personal data is protected regardless of whether it appeared on a public page, and there is no analytics case that justifies pulling it into your store. Aggregate labor-market insight does not need a single name — so don't capture one.
Keep the rule concrete in code: your parser should have an explicit allow-list of fields, and everything outside it is discarded at extraction time, before it ever reaches a database. The rest of this guide assumes that boundary is fixed.
Why a German Mobile Viewpoint Returns Germany-True Listings
Both Indeed.de and StepStone personalize results by the visitor's location and language. Regional filters, which postings surface first, currency and pay formatting, and sometimes the set of results themselves all shift with where the request appears to come from. Scrape from a server in another country and you may get a redirected international variant, a partial result set, or listings ranked for the wrong region — quietly skewing every trend you calculate afterwards.
A real German mobile IP removes that distortion. The request looks like an ordinary candidate browsing from a German mobile network, so the board returns the listings a person in Germany actually sees. For labor-market data where regional accuracy is the whole point, that Germany-true viewpoint is not a nice-to-have — it is the difference between measuring the German market and measuring a localized approximation of it.
Datacenter IPs make this harder in a second way: large job boards routinely flag traffic from hosting ranges, because almost no real candidate browses from a datacenter. That is why datacenter proxies get blocked on exactly this kind of target, while a residential-grade mobile connection reads as a normal visitor. The same mobile viewpoint that gives you accurate German listings also keeps your collection stable enough to run on a schedule — the same reasoning we walk through for retail data in scraping Amazon with mobile proxies.
Robots.txt and Rate-Limit Etiquette
Being a polite client is not just risk management — it is what keeps a data pipeline running for months instead of getting cut off in a week. Two habits do most of the work.
Read robots.txt first, and honor it. Fetch https://de.indeed.com/robots.txt and the StepStone equivalent before you write a single parser. It tells you which paths the site asks automated clients to stay out of. Treat disallowed paths as off-limits, and check each site's terms of service too — respecting both is a core part of whether your collection is defensible. Our overview of whether web scraping is legal in the EU covers how these signals fit the broader legal picture.
Throttle like a human, not a firehose. Space requests out, add jitter so you don't hammer a fixed interval, and keep concurrency low per target. A steady, unremarkable pace serves two goals at once: it is courteous to the site's infrastructure, and it keeps your footprint ordinary. If you do push too hard you will meet an HTTP 429 "Too Many Requests" response — treat that as a clear signal to back off, not to rotate faster and retry harder.
Practical etiquette checklist:
- Fetch and obey
robots.txtand the site's terms before collecting. - Set a truthful, contactable User-Agent where the site allows it.
- Add randomized delays between requests; keep per-target concurrency low.
- Back off immediately on 429 or 503 rather than retrying aggressively.
- Cache pages you have already fetched so you never re-request the same posting needlessly.
Sessions and Rotation for Multi-Page Collection
Job-board scraping is paginated: you walk search-result pages, then follow through to individual postings. How you handle your IP across those requests matters.
For a sequence that belongs together — paging through one search query, or a login-gated flow if you have a legitimate account — hold a sticky session so the same German mobile IP carries the whole sequence and your behavior stays coherent. When you move to a fresh query or a new region, rotating to a new IP spreads your collection naturally across the network. Our guide to sticky sessions and IP rotation explains how to choose per task; the short version is: sticky within a logical session, rotate between them.
Because mobile networks use CGNAT, a single mobile IP already carries many genuine users, so ordinary-paced traffic on it looks exactly like what it is — normal mobile activity. You do not need aggressive rotation to blend in; you need to behave like a real client and let the network do the rest.
Structuring the Dataset
Clean structure at extraction time is what turns scraped pages into something you can actually analyze. Decide the schema before you collect, and map every posting to it.
A workable posting-level schema — company and role fields only:
| Field | Example content | Notes |
|---|---|---|
job_title |
"Data Engineer (m/w/d)" | Normalize whitespace; keep the original language |
company_name |
Employer as listed | A business, not a person |
location |
City / region | Enables demand-by-region analysis |
contract_type |
Full-time, part-time, contract | Standardize to a fixed vocabulary |
salary_range |
If published | Store currency and period; leave null when absent |
posted_date |
Listing date | Powers hiring-velocity metrics |
skills |
Extracted keywords | For skills-demand trends |
source |
indeed_de / stepstone |
Track provenance per row |
listing_url |
Canonical posting URL | For dedup and re-check |
collected_at |
Your fetch timestamp | Essential for any time-series |
A few structuring habits pay off later:
- Deduplicate across sources. The same role often appears on both boards; key on a normalized combination of company, title and location so you don't double-count demand.
- Snapshot over time. Re-collecting on a schedule and stamping
collected_atis what makes salary trends and hiring velocity measurable — a single scrape is a photo, a schedule is a film. - Normalize as you write, not later. Standardize contract types, parse salary strings into numbers plus currency, and keep the raw title too, so downstream analysis is consistent.
- Never add a personal-data column. If a field would identify a person, it does not belong in the schema at all — the allow-list above has no place for one, by design.
On the tooling side, a request-based collector in Python is the natural fit for this — it is lightweight, easy to throttle, and never renders a browser you'd have to lock down. Our mobile proxy with Python guide shows the request setup end to end, and pairs cleanly with the schema above.
FAQ
Is it legal to scrape Indeed and StepStone job postings in Germany?
Scraping public job postings for company and role data is generally defensible in the EU when you collect no personal data, respect each site's terms and robots.txt, and stay within polite rate limits. Job listings about people — recruiter names, applicant details — are personal data under the GDPR and should be left out entirely. See is web scraping legal in the EU for the broader picture.
Why do I need a German IP to scrape German job boards?
Indeed.de and StepStone tailor results to the visitor's location and language — postings, regional filters and pay bands can differ by region. A real German mobile IP returns the listings a candidate in Germany actually sees, so your dataset reflects the true German market rather than a localized or redirected version.
What fields should a job-data scraper collect?
Collect posting-level and company-level fields only: job title, company name, location, contract type, posted date, salary range if published, required skills and the listing URL. Do not collect applicant data, recruiter personal details or anything that identifies an individual — that keeps the dataset GDPR-safe and useful for aggregate analytics.
Related guides
- How to Scrape Google Maps Business Data in Germany
- Scrape ImmobilienScout24 Real-Estate Market Data Responsibly
- Async Web Scraping in Python Through One Mobile IP
- Mobile Proxy for Web Scraping: Keep Runs Region-True at Scale
CTA
Want your German labor-market data to reflect what candidates in Germany actually see? Collect it from a real German mobile IP — browse the German mobile proxy plans.
Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — browse German mobile proxy plans or get a dedicated proxy on the Blue network.