Direct Answer
robots.txt is a plain-text file at a website's root (/robots.txt) that tells automated clients which paths they may crawl and how fast. It is a request the site publishes, not a technical lock or a law. Ignoring it is not automatically illegal, but respecting it is central to responsible, defensible data collection — and crawl etiquette is viewed favourably by EU data-protection authorities.
[!NOTE]
This page is general guidance for building a responsible scraper. It is not legal advice.
What robots.txt Actually Is
The Robots Exclusion Protocol dates back to 1994 and became a formal internet standard (RFC 9309) in 2022. The idea is simple: a site operator publishes a file at their root — for example https://example.de/robots.txt — that lists, in plain text, which parts of the site automated clients are asked to leave alone and which crawlers those requests apply to.
The critical word is asked. robots.txt is a voluntary convention. Nothing in your HTTP client is forced to obey it; the file cannot technically block a request the way a login wall or a firewall can. It is closer to a "staff only" sign on a door than to a locked door. That is exactly why how you treat it says something about you — a well-behaved crawler reads it and honours it, an aggressive one drives straight past.
Because it is a signal of operator intent rather than a lock, robots.txt sits alongside a site's terms of service and rate limits as one of the things a responsible team checks before it starts collecting. It is the first line of good-citizen scraping. A newer sibling aimed at AI crawlers, llms.txt, works on the same voluntary principle — what the data shows about llms.txt in 2026 is worth a read if you build or run crawlers.
How to Read a robots.txt File
A robots.txt file is a list of rule blocks. Each block starts with one or more User-agent lines and is followed by the rules that apply to those agents. Here is a representative example:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Allow: /search/help
User-agent: BadBot
Disallow: /
Crawl-delay: 5
Sitemap: https://example.de/sitemap.xml
User-agent
User-agent names the crawler a block applies to. The value * is a wildcard meaning "every client not named more specifically". If a site lists a block for a specific named bot, that block overrides the * block for that bot — so a well-behaved client reads the whole file and applies the most specific matching block, not just the first one it sees.
Disallow
Disallow lists path prefixes the named agent is asked not to fetch. Disallow: /cart/ covers everything under /cart/. An empty value — Disallow: with nothing after it — means the opposite: nothing is off-limits, the whole site is open to that agent. Disallow: / (a bare slash) asks the agent to stay off the entire site.
Allow
Allow carves an exception out of a disallowed path. In the example above, /search is disallowed but /search/help is explicitly permitted. When a Disallow and an Allow both match a URL, the more specific (longer) rule wins under the standard.
Crawl-delay
Crawl-delay requests a minimum pause, in seconds, between your requests. It is not part of the original standard and not every crawler honours it — but honouring it is good manners and, more practically, it keeps you well under the rate that triggers blocks and 429 Too Many Requests responses. If a site asks for five seconds, give it five seconds.
Sitemap
Sitemap points to an XML sitemap listing the URLs the operator wants discovered. It is a gift to a scraper: instead of crawling blindly, you can read the sitemap and fetch only the public pages the site itself publishes as fair game.
What robots.txt Does and Doesn't Bind
It helps to be precise about what this file is and isn't, because a lot of confusion sits here.
| robots.txt is | robots.txt is not |
|---|---|
| A published request from the site operator | A technical access control (it blocks nothing) |
| A widely-recognised internet convention (RFC 9309) | A law, contract, or on its own a licence |
| Strong evidence of good — or bad — faith | A substitute for checking terms of service or the GDPR |
| A map of what a site is happy to have crawled | A guarantee that everything not disallowed is legal to take |
Two takeaways follow. First, a path being absent from Disallow is not blanket permission — copyright, database rights, a site's terms of service and, where personal data is involved, the GDPR can still apply to data that robots.txt never mentions. Second, ignoring a Disallow is not automatically unlawful, but it removes your best evidence of acting in good faith. If a dispute ever arises, "we respected the operator's stated wishes" is a far stronger position than "the file didn't technically stop us".
For the wider picture of what determines legality, see is web scraping legal in the EU? — the activity decides the outcome, not the tool. And on the tool side, a proxy is neutral infrastructure: see are mobile proxies legal?.
Why Respecting It Is Central to Defensible Collection
EU data-protection authorities and courts increasingly look at how data was collected, not just what was collected. Crawl etiquette — honouring robots.txt, staying under rate limits, not hammering servers — is read as a signal of a considered, proportionate operation rather than an indiscriminate one. That impression matters when you have to explain your data pipeline to a client, an auditor, or a regulator.
There is a practical dividend too. A polite client that respects off-limits paths and reasonable request rates behaves the way ordinary human traffic behaves. That is precisely what keeps your IP reputation intact and reduces the friction — CAPTCHAs, soft blocks, hard bans — that comes from looking like an abusive bot. Being a good citizen and running a reliable scraper are the same discipline, not competing goals. It is also why understanding how websites detect proxies matters: the goal is to look like a normal, well-behaved visitor because you are one.
None of this depends on which network your requests come from. But clean, real German mobile IPs give a well-behaved crawler a solid foundation, because the etiquette and the infrastructure both point the same direction: legitimate traffic, collected responsibly.
The Good-Citizen Checklist
A short, practical routine that keeps data collection defensible:
- Fetch and read
/robots.txtfirst — before your first content request to any new site, and re-check it periodically, since it can change. - Apply the most specific
User-agentblock that matches your client, and honour everyDisallowin it. - Treat
Crawl-delayas a minimum, and default to a gentle rate even where none is stated — one request every few seconds is a sane starting point. - Read the
Sitemapand prefer the public URLs the operator publishes over blind crawling. - Collect only public, non-personal data you actually need, for a defined purpose. Avoid personal data unless you have a clear lawful basis under the GDPR.
- Identify yourself honestly where a site expects it, and never disguise your client to slip past a
Disallow. Emerging standards like Web Bot Auth let a bot prove its identity cryptographically rather than just claim it in a header. - Respect terms of service and rate limits as well as
robots.txt— they are separate signals, and all three count. - Back off when a site pushes back with 429s or 503s. A polite client slows down; it doesn't retry harder.
For the full version of this discipline, see our ethical web scraping checklist.
How GermanProxy.io Fits In
A proxy doesn't read robots.txt for you — that stays your scraper's job — but the provider you route through is part of your compliance story. GermanProxy.io is a registered EU provider with a full imprint, a clear acceptable-use policy and a real abuse-reporting process. We provide routed access to real German mobile IPs for legitimate business use such as market research, price monitoring and SEO analysis, and we don't support unlawful data harvesting. When your own practice is polite by design — reads the file, honours the rules, collects only what it needs — clean infrastructure underneath it keeps the whole operation explainable.
FAQ
Is it illegal to ignore robots.txt when scraping?
robots.txt is a convention, not a law, so ignoring it is not automatically illegal. But it can strengthen a site operator's case against you, and it signals bad faith. Respecting it keeps your collection defensible toward clients, partners and regulators. This is general guidance, not legal advice.
What do User-agent, Disallow and Allow mean in robots.txt?
User-agent names the crawler a rule block applies to (* means all). Disallow lists path prefixes that group is asked not to crawl, and Allow carves out exceptions inside a disallowed path. An empty Disallow means the whole site is open to that agent.
Does respecting robots.txt slow my scraper down?
Only where the site asks it to, via Crawl-delay or by disallowing specific paths. In practice it rarely blocks the public data you actually need, and the politeness it enforces — steady request rates, honoured off-limits areas — is exactly what keeps your IPs healthy and your project low-risk.
CTA
Want infrastructure that fits a responsible, defensible scraping practice? GermanProxy.io routes clean German mobile IPs from a registered EU provider with clear acceptable-use rules. Read the ethical web scraping checklist.
Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or start small with a shared 5G proxy.