Scrapy Proxy Setup: Route Spiders via German Mobile IPs

Direct Answer

To route Scrapy through a mobile proxy, set the proxy on each request's meta["proxy"] — easiest from an async start() generator or a spider middleware. Scrapy's built-in HttpProxyMiddleware then sends every request through your German mobile IP. With a single dedicated endpoint you don't need any pool-rotation middleware — just your endpoint and credentials.

Why Scrapy Setup Is Simpler Here

Most Scrapy proxy guides assume a rotating pool, so they have you download a third-party middleware to pick a new IP per request. With a dedicated German mobile endpoint, that complexity disappears: you point every request at one endpoint, and rotate on demand at the proxy when you choose to. Scrapy's own HttpProxyMiddleware (enabled by default) does the rest.

Before You Start

You will need four values from your GermanProxy.io account:

Host and port of your endpoint
Username and password

The examples use PROXY_HOST, PORT, USER and PASS as placeholders — replace them with your own.

Option A — Set the Proxy per Request (simplest)

Set meta["proxy"] in your spider. Scrapy reads the credentials straight from the URL. On Scrapy 2.13+ the entry point is the async start() generator (it replaces the now-deprecated start_requests()):

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    PROXY = "http://USER:PASS@PROXY_HOST:PORT"

    async def start(self):
        urls = ["https://www.example.de"]
        for url in urls:
            yield scrapy.Request(
                url,
                callback=self.parse,
                meta={"proxy": self.PROXY},
            )

    def parse(self, response):
        self.logger.info("Status %s", response.status)
        yield {"title": response.css("title::text").get()}

HttpProxyMiddleware is on by default, so no settings change is needed for this approach. On Scrapy 2.12 or older, use def start_requests(self): with the same body instead.

Option B — Apply the Proxy to Every Spider (middleware)

To avoid setting meta on each request, add a tiny middleware. In middlewares.py:

class MobileProxyMiddleware:
    PROXY = "http://USER:PASS@PROXY_HOST:PORT"

    def process_request(self, request, spider):
        request.meta["proxy"] = self.PROXY

Enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "yourproject.middlewares.MobileProxyMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

Now every request from every spider routes through your German mobile IP.

Verify the IP First

Point a quick spider at an IP-echo endpoint and confirm it returns a German mobile IP before you scale up. If it returns your own address, re-check the proxy URL and that HttpProxyMiddleware is enabled.

Sticky Sessions vs. Rotation

Sticky session: keep the same German mobile IP across a multi-request flow.
Rotation on demand: request a fresh IP at the proxy when you want to spread load.

Tune DOWNLOAD_DELAY and CONCURRENT_REQUESTS to a human-reasonable rate. See sticky sessions and IP rotation explained.

Scrape Responsibly

A mobile proxy improves reliability for legitimate data collection — it is not a tool to circumvent access controls or platform rules. Keep your crawl clean and defensible:

Respect robots.txt — keep ROBOTSTXT_OBEY = True and honour each site's terms.
Throttle with DOWNLOAD_DELAY and AutoThrottle so you don't overload servers.
Collect only what you need, and handle any personal data in line with the GDPR.
Set an honest USER_AGENT that identifies your crawler, not a misleading one.

FAQ

Do I need a rotating-proxy middleware?

Not with a dedicated endpoint. Pool middlewares exist to pick an IP from many; one dedicated German endpoint removes that need. Rotate on demand at the proxy instead.

Why do I get a 407 error?

407 Proxy Authentication Required means the username and password didn't reach the proxy. Confirm they're in the meta["proxy"] URL; see how to fix 407 proxy authentication required.

Does ROBOTSTXT_OBEY slow me down?

It makes Scrapy honour robots.txt, which is the responsible default. Keep it on for legitimate public-data collection.

HTTP or SOCKS5?

Scrapy's HttpProxyMiddleware targets HTTP(S) proxies; for SOCKS use an HTTP endpoint or a SOCKS-capable downloader handler. For most Scrapy work, HTTP is simplest — see SOCKS5 vs HTTP proxy.

Get Started

Run your Scrapy spiders on real German mobile IPs. Get a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, fair-use 20 GB/day.

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or try a shared 5G German proxy.