Direct Answer
HTTP 429 Too Many Requests means a server is rate-limiting you: you sent more requests in a window than it allows. Fix it by reading the Retry-After header, waiting that long, then retrying with exponential backoff plus jitter, capping concurrency, and spreading legitimate load across rotating mobile IPs so no single IP exceeds the limit.
What HTTP 429 Too Many Requests Means
429 Too Many Requests is a standard HTTP status code (defined in RFC 6585) that a server returns when a client has sent too many requests in a given amount of time. It is the server telling you, plainly, to slow down — not that your request was malformed or unauthorised. The request itself was understood; the rate was the problem.
Three details make 429 different from other errors:
- It is usually temporary. The server is asking you to wait, not refusing you outright. The same request typically succeeds once the window resets.
- It often carries a
Retry-Afterheader. This tells you exactly how long to wait — in seconds, or as an HTTP date. Honouring it is the single most reliable fix. - It is the polite signal in front of harder limits. A client that ignores 429 and keeps hammering a server is the client most likely to see longer cooldowns or an outright block. Respecting the limit keeps your IP reputation intact.
For data-engineering and web-scraping work, 429 is the most common operational error you will hit at scale. The good news: it is also the most predictable one to design around.
Why You Get 429 When Scraping
Rate limits exist so one client cannot degrade a service for everyone else. You hit 429 when your traffic crosses a threshold the server enforces. The common triggers:
- Too many requests per second from one IP. Most public APIs and many websites cap requests per IP — often something like 60–600 requests per minute, though every service sets its own number. Burst past it and the next request returns 429.
- High concurrency. Twenty parallel workers all firing at one host look very different from one sequential client, even at the same total volume. Concurrency, not just total count, drives most 429s.
- A shared IP already near its limit. If you route through a heavily shared pool, other users' traffic counts against the same address. Your modest request load can tip an already-busy IP over the edge — one reason a dedicated mobile connection behaves more predictably.
- Per-account or per-key limits. Some limits attach to an API key or session, not the IP. Rotating IPs alone will not fix those; you have to slow the request rate itself.
The fix is never to defeat the limit — it is to send your legitimate traffic within it. Everything below does exactly that.
How to Fix 429: 5 Steps That Respect the Limit
1. Read and honour the Retry-After header
When a server sends Retry-After, it is handing you the answer. Wait the specified time, then retry. This alone resolves a large share of 429s, because the server is telling you precisely when its window resets.
Retry-After comes in two forms: a number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT). Parse both — calling int() on the date form raises ValueError and crashes your loop:
import time, requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone
def parse_retry_after(value, default=5.0):
if not value:
return default
try:
return float(value) # seconds form
except ValueError:
try: # HTTP-date form
when = parsedate_to_datetime(value)
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
except (TypeError, ValueError):
return default
resp = requests.get(url, proxies=proxies)
if resp.status_code == 429:
time.sleep(parse_retry_after(resp.headers.get("Retry-After")))
resp = requests.get(url, proxies=proxies)
If Retry-After is absent, fall back to backoff (step 2). Never retry a 429 immediately — an instant retry is the fastest way to earn a longer cooldown.
2. Use exponential backoff with jitter
When there is no explicit wait time, increase the delay after each failed attempt instead of retrying at a fixed interval. Exponential backoff doubles the wait each time (e.g. 1s, 2s, 4s, 8s); jitter adds a small random offset so that many retrying clients do not all wake up at the same instant and collide again.
import time, random, requests
# reuse parse_retry_after() from step 1 (handles seconds AND HTTP-date form)
def get_with_backoff(url, proxies, max_retries=5):
resp = None
for attempt in range(max_retries):
resp = requests.get(url, proxies=proxies)
if resp.status_code != 429:
return resp
retry_after = resp.headers.get("Retry-After")
if retry_after:
delay = parse_retry_after(retry_after)
else:
delay = (2 ** attempt) + random.uniform(0, 1) # backoff + jitter
time.sleep(delay)
return resp # give up after max_retries
Always cap the number of retries (here, 5) and the maximum delay. Infinite retries against a rate-limited host help nobody.
3. Cap your concurrency
The most common scraping mistake is firing unbounded parallel requests. Limit how many requests are in flight at once — a semaphore, a worker pool, or a fixed connection count. Start low (2–5 concurrent requests per host) and raise it only while you stay under the limit. Lower concurrency is often a bigger fix than longer delays.
4. Throttle your request rate deliberately
Pace requests to a sustainable rate rather than going as fast as the network allows. A token-bucket limiter or a fixed minimum gap between requests (say, one request every 1–2 seconds per host) keeps you comfortably inside most limits. Frameworks make this easy: see the Scrapy guide (AUTOTHROTTLE and DOWNLOAD_DELAY) and the Crawlee guide (built-in concurrency control).
5. Distribute legitimate load across rotating IPs
When a single IP's per-IP limit is the bottleneck — and your total volume is genuinely legitimate — spread that load across several IPs so no one address exceeds the threshold. Rotating mobile IPs is the clean way to do this: each request batch leaves from a fresh real German mobile IP, so your per-IP rate stays low while your overall throughput rises. Combine sticky sessions for multi-step flows that must stay on one IP with on-demand rotation between independent batches.
This is load distribution, not limit evasion. You are still respecting each server's per-IP budget — you are simply not stacking all your traffic onto one address.
429 Fix Strategies Compared
| Strategy | Fixes which cause | Effort | Best when |
|---|---|---|---|
Honour Retry-After |
Server told you the wait | Low | The header is present |
| Exponential backoff + jitter | No explicit wait given | Low | Transient/bursty 429s |
| Cap concurrency | Too many parallel requests | Low | High-worker scrapers |
| Throttle request rate | Sustained over-rate | Medium | Steady high-volume jobs |
| Rotate across mobile IPs | Per-IP limit is the ceiling | Medium | Legitimate volume above one IP's budget |
No single row is "the" fix. In practice you combine them: honour Retry-After, back off when it is missing, keep concurrency sane, and rotate IPs to distribute load that is legitimately larger than one address can carry.
Why Mobile IPs Help With Rate Limits
Rate limits are commonly enforced per IP, and not all IPs are scored equally. Datacenter ranges are published and easy to identify, so platforms often apply tighter limits to them before you even hit a hard cap. Real mobile IPs read differently.
Thanks to CGNAT, a single mobile IP is shared among thousands of genuine subscribers at once. Because of that, mobile IPs carry higher baseline trust than datacenter ranges, and platforms cannot safely treat one mobile IP as a single automated client. That trust does not raise a published rate limit — but it means well-behaved traffic from a real German mobile IP tends to face friction later than the same traffic from a flagged datacenter range. See the full comparison in mobile vs residential vs datacenter proxies.
GermanProxy.io's 5G mobile proxies give you real German IPs on the Blue and Red Premium Mobile Network, with HTTP and SOCKS5, sticky sessions and on-demand rotation, and fair-use 20 GB/day — exactly the building blocks you need to spread legitimate load without overrunning any one IP's limit.
A Sensible 429-Handling Loop (Putting It Together)
A robust scraper handles 429 in one place, not scattered across the codebase. The order that works:
- On 429, check
Retry-After— if present, wait that long. - Otherwise back off exponentially with jitter, capped at a maximum delay.
- Limit retries (e.g. 5) and log the give-ups instead of looping forever.
- Keep per-host concurrency low and a minimum gap between requests.
- Rotate to a fresh IP between batches when the limit is per-IP and your volume is legitimate.
Wire this through your proxy with a standard endpoint — http://USER:PASS@PROXY_HOST:PORT — and the same logic works in Python requests/Playwright, Node.js, and Scrapy alike. Before you rely on it, test your setup so you know the proxy and the retry loop both behave.
FAQ
What does HTTP 429 Too Many Requests mean?
It means the server is rate-limiting you: you sent more requests in a time window than it allows. The request was valid — only the rate was too high. It is usually temporary, often comes with a Retry-After header, and resolves once you slow down and the window resets.
What is the Retry-After header and should I follow it?
Retry-After tells you how long to wait before retrying, either as a number of seconds or an HTTP date. Yes, always follow it — it is the server stating exactly when its limit resets. Honouring it resolves most 429s and signals that your client is well-behaved, which protects your IP reputation.
How does exponential backoff with jitter fix 429?
Exponential backoff increases the wait after each failed retry (1s, 2s, 4s, 8s) instead of retrying at a fixed pace, giving the server's window time to reset. Jitter adds a small random offset so many clients don't retry in lockstep and collide again. Together they smooth out bursts and stop retry storms.
Do rotating mobile proxies stop 429 errors?
They help when the bottleneck is a per-IP limit and your total volume is legitimate: rotating across real mobile IPs keeps each IP's request rate under the threshold while your overall throughput rises. They do not help with per-account or per-API-key limits — those still need slower request rates. Rotation distributes load; it does not evade a limit.
How many requests per second is safe when scraping?
There is no universal number — every service sets its own limit, often in the range of tens to a few hundred requests per minute per IP. Start conservatively (one request every 1–2 seconds, 2–5 concurrent per host), watch for 429s, and increase only while you stay clear of them.
Is rate-limiting the same as being blocked?
No. A 429 is a temporary "slow down" and usually clears on its own; a block is a harder, longer refusal. Ignoring repeated 429s is one of the surest ways to turn a temporary limit into a longer cooldown — which is exactly why honouring Retry-After and backing off matters.
CTA
Hitting per-IP rate limits on legitimate work? Browse real German 5G mobile proxies with sticky sessions and on-demand rotation, so you can spread genuine load and stay within every server's limit — fair-use 20 GB/day, online in minutes.
Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or try a shared 5G German proxy.