Crawlee Proxy Setup with a Dedicated German Mobile Proxy

How Crawlee Handles Proxies

Crawlee (from the Apify team) centralises proxy handling in one ProxyConfiguration object. You hand it your endpoint, pass it to any crawler — BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler (Python) or CheerioCrawler, PlaywrightCrawler (JS) — and it wires the proxy into every request automatically. With a single dedicated German endpoint you don't manage a pool; you give it one URL and rotate on demand at the proxy when you choose to.

Before You Start

You'll need four values from your GermanProxy.io account:

Host and port of your endpoint
Username and password

The examples use PROXY_HOST, PORT, USER and PASS as placeholders — replace them with your own.

Python (Crawlee for Python)

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration

async def main() -> None:
    proxy_configuration = ProxyConfiguration(
        proxy_urls=["http://USER:PASS@PROXY_HOST:PORT"],
    )

    crawler = BeautifulSoupCrawler(proxy_configuration=proxy_configuration)

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        title = context.soup.title.string if context.soup.title else None
        context.log.info("Title: %s", title)
        await context.push_data({"url": context.request.url, "title": title})

    await crawler.run(["https://www.example.de"])

if __name__ == "__main__":
    asyncio.run(main())

Import paths shifted across Crawlee versions (older builds used from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler). If an import fails, check your installed version's docs — the proxy_configuration= argument is stable.

JavaScript / TypeScript (Crawlee for JS)

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ['http://USER:PASS@PROXY_HOST:PORT'],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    async requestHandler({ $, request, proxyInfo, log }) {
        log.info(`Used proxy: ${proxyInfo?.url}`);
        log.info(`Title: ${$('title').text()}`);
    },
});

await crawler.run(['https://www.example.de']);

proxyInfo.url lets you confirm which endpoint a request used.

Verify the IP First

Point a one-URL crawl at an IP-echo endpoint (e.g. https://api.ipify.org) and confirm the IP changed from your own before scaling up. ipify only shows the address, though — to confirm it is actually a German mobile IP (correct country plus a mobile-network carrier, not a datacentre range), run it through the checks in test your mobile proxy setup. If the echo returns your own address, re-check the proxy URL and credentials. A 407 means auth didn't reach the proxy — see how to fix 407 proxy authentication required.

Sticky Sessions vs. Rotation

Crawlee's SessionPool keeps a stable identity across requests when you enable it — use_session_pool=True in Python, or useSessionPool: true in the crawler options in JavaScript (both default to on). Combined with a dedicated German endpoint, that gives you:

Sticky session: the same German mobile IP across a multi-step flow.
Rotation on demand: a fresh IP requested at the proxy to spread load.

See sticky sessions and IP rotation explained.

Scrape Responsibly

A mobile proxy improves reliability for legitimate data collection — it is not a tool to circumvent access controls or platform rules. Keep your crawl clean and defensible:

Respect robots.txt and each site's terms of use.
Throttle with Crawlee's concurrency and rate-limit settings so you don't overload servers.
Collect only what you need, and handle any personal data in line with the GDPR.
Identify your crawler honestly in the User-Agent.

For the legal side, see is web scraping legal? EU & GDPR.

FAQ

Does Crawlee need a rotating-proxy pool?

No. ProxyConfiguration accepts a single URL just as happily as a list. With one dedicated German endpoint you skip pool management and rotate on demand at the proxy instead.

Which Crawlee crawler should I use?

BeautifulSoupCrawler or ParselCrawler for static HTML; PlaywrightCrawler for JavaScript-heavy pages. The proxy_configuration argument is identical across all of them.

HTTP or SOCKS5?

Pass an http:// URL — that is the common case and works in both Python and JavaScript. SOCKS5 differs by runtime: Crawlee for JavaScript accepts a socks5:// URL, but Crawlee for Python does not — its ProxyConfiguration validates proxy URLs as http/https only and rejects a socks5:// scheme, so use the HTTP endpoint there (tracked upstream in issue #821). Background: SOCKS5 vs HTTP proxy.

Why do I get a 407 error?

407 Proxy Authentication Required means your username and password didn't reach the proxy. Confirm they're in the proxy URL; see the 407 fix guide.

CTA

Run your Crawlee crawlers on real German mobile IPs. Start with a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, fair-use 20 GB/day.

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — browse German mobile proxy plans or try a shared 5G German proxy.