10% off ALL new Proxies – CODE: 10pro

Async Web Scraping in Python Through One Mobile IP

Async Web Scraping in Python Through One Mobile IP

Direct Answer

Run async scraping through one mobile IP by bounding concurrency with an asyncio semaphore, pacing requests per domain, and backing off on HTTP 429. A single clean sticky German mobile IP with moderate, well-spaced concurrency stays reliable far longer than many parallel requests hammering the same target. Treat throughput as a politeness budget, not a race.

Why "One IP, Many Requests" Is the Real Question

Most proxy tutorials show you how to fire concurrent requests. Almost none tell you how many you should — through a single mobile endpoint, against a real target, without becoming the reason that target tightens its defences.

That gap matters, because a dedicated German mobile IP is not an infinite pipe. It is a real connection on a shared mobile cell, with a fair-use 20 GB/day budget and other legitimate traffic around it. If you saturate it with unbounded async requests, three things happen at once: the target starts returning 429s, your latency climbs, and your IP reputation — the thing that made mobile worth paying for — starts to erode.

The good news is that async done carefully is the ideal tool here. asyncio lets you hold many requests in flight without one slow response stalling the rest, while still letting you cap exactly how hard you push. The skill is not maximum concurrency — it is chosen, bounded concurrency.

For the single-request basics this guide builds on, start with the mobile proxy with Python guide.

The Politeness-Budget Mindset

Before any code, adopt one framing: every request is spent from a budget, not extracted for free.

Your budget has three limits, and you should scrape to the tightest of them:

  • The target's tolerance — the rate at which it serves you cleanly, with no 429s and stable latency.
  • Your fair-use 20 GB/day — a hard ceiling on total transfer through your endpoint.
  • The shared cell's health — you are one tenant on a real mobile connection; leaving headroom keeps the IP clean for you and everyone else.

A "politeness budget" means you decide your ceiling in advance and stay under it, rather than pushing until something breaks and calling that the limit. Practically:

  • Prefer a moderate, steady rate you can sustain for hours over a burst that trips defences in minutes.
  • Fetch each URL once, cache the result, and never re-request data you already hold.
  • Request only public pages you actually need — no login-gated or personal data, and honour robots.txt and each site's terms.
  • Collect only what your task requires, and handle anything that could be personal data in line with the GDPR.

This is our positioning, not a footnote: clean, dedicated German mobile IPs are worth protecting, and polite clients keep them reliable.

Bounded Concurrency With httpx and asyncio

httpx speaks async natively and routes through your proxy with a single proxy= argument. The core control is an asyncio.Semaphore — it caps how many requests are ever in flight at once, no matter how many URLs you queue.

import asyncio
import httpx

PROXY = "http://USER:PASS@PROXY_HOST:PORT"   # your sticky endpoint
MAX_CONCURRENCY = 5                           # start low, raise only if healthy

async def fetch(client, sem, url):
    async with sem:                           # never more than N in flight
        resp = await client.get(url, timeout=30)
        resp.raise_for_status()
        return url, resp.text

async def scrape(urls):
    sem = asyncio.Semaphore(MAX_CONCURRENCY)
    async with httpx.AsyncClient(proxy=PROXY) as client:
        tasks = [fetch(client, sem, u) for u in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

# asyncio.run(scrape(my_urls))

Two details make this safe:

  • One AsyncClient, reused — it keeps a connection pool over your single sticky IP, so requests look like one consistent client rather than a stampede of fresh connections.
  • The semaphore is the whole game — MAX_CONCURRENCY = 5 means five requests, not five hundred. Start low. You can always raise it; you cannot un-annoy a target.

socks5:// works in the same proxy= argument if you prefer SOCKS5. Keeping one IP for the whole run is deliberate — see sticky sessions and IP rotation for why a consistent endpoint beats blind rotation.

Per-Domain Rate Limiting

A global concurrency cap is not enough. Ten concurrent requests spread across ten domains is gentle; ten aimed at one domain is a hammer. Pace each domain separately by spacing its requests with a minimum interval.

import asyncio
import time
from collections import defaultdict
from urllib.parse import urlparse

MIN_INTERVAL = 1.0   # seconds between requests to the SAME domain

_last_hit = defaultdict(float)
_domain_locks = defaultdict(asyncio.Lock)

async def pace(url):
    domain = urlparse(url).netloc
    async with _domain_locks[domain]:
        wait = MIN_INTERVAL - (time.monotonic() - _last_hit[domain])
        if wait > 0:
            await asyncio.sleep(wait)
        _last_hit[domain] = time.monotonic()

Call await pace(url) at the top of fetch, inside the semaphore. Now two limits apply together: the semaphore caps total in-flight requests, and the per-domain lock guarantees you never hit one host faster than MIN_INTERVAL. A single mobile IP making one steady, well-spaced request per second to a domain looks like an ordinary user — many bursts a second do not.

Tune MIN_INTERVAL up, never down, when a target looks sensitive. This pacing is also your best defence against triggering challenges in the first place; see how to reduce CAPTCHAs with mobile proxies.

Backing Off on 429

Even a careful crawler will occasionally see HTTP 429 Too Many Requests. That is the target telling you your budget for this moment is spent. The correct response is to slow down, not to rotate around it.

Respect the Retry-After header if present; otherwise back off exponentially with a little jitter so retries do not all fire at once.

import asyncio
import random
import httpx

async def fetch_with_backoff(client, sem, url, attempts=4):
    for attempt in range(attempts):
        async with sem:
            resp = await client.get(url, timeout=30)
        if resp.status_code == 429:
            retry_after = resp.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else (2 ** attempt) + random.random()
            await asyncio.sleep(delay)
            continue
        resp.raise_for_status()
        return url, resp.text
    raise RuntimeError(f"gave up after {attempts} attempts: {url}")

Notice the request happens inside the semaphore, but the backoff sleep happens outside it — so a waiting task releases its slot for others while it cools down. If you see repeated 429s, the honest fix is lowering MAX_CONCURRENCY or raising MIN_INTERVAL, not adding more retries. For the full picture on this status, see how to fix 429 Too Many Requests.

Moderate Concurrency Beats Hammering

Put together, the pattern is small and boring — and that is the point:

Approach What happens
Unbounded async, one IP Fast for seconds, then 429s, climbing latency, damaged IP reputation
Rotate IPs to push past limits Masks the symptom; you are still too fast, and each new IP starts cold
Bounded + paced, one sticky IP Steady throughput for hours, clean reputation, defensible pattern

One clean sticky German mobile IP, held for the whole job, with a low concurrency cap and honest per-domain pacing, will out-collect an aggressive setup over any real time horizon — because it never gives the target a reason to fight back.

Concretely, a sane default: MAX_CONCURRENCY of 3–5, MIN_INTERVAL around one second per domain, Retry-After-aware backoff, and a total-transfer eye on your fair-use 20 GB/day. Measure with your own targets, adjust slowly, and let the IP stay clean.

FAQ

How many concurrent requests can I run through one mobile IP?

There is no fixed number — it depends on the target's tolerance and your fair-use 20 GB/day budget. Start low, around 3–5 concurrent requests per domain, watch for 429 responses and rising latency, and only raise the limit if the target stays healthy. One shared mobile cell rewards patience, not raw parallelism.

Does async scraping get me blocked faster?

Not by itself. Blocks come from request rate and pattern, not from whether your code is async or synchronous. Async simply lets you reach a high rate more easily, so you must add a concurrency limit and per-domain pacing. Well-bounded async through one clean sticky IP is gentler than unbounded threads.

Should I rotate IPs for concurrent scraping or keep one sticky?

For most concurrent jobs, one sticky German mobile IP with paced, bounded concurrency is the reliable choice — it behaves like a single consistent client. Rotate only when the work is genuinely independent and you have a real reason to spread it. Rotating to push past a rate limit usually signals you are simply going too fast.

Related guides

CTA

Build your async pipeline on one clean, dedicated German mobile IP. Get a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, fair-use 20 GB/day.

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or try a shared 5G German proxy.

Telegram Support
Nach oben scrollen