BeautifulSoup Proxy Setup with httpx (German Mobile IP)

Direct Answer

BeautifulSoup only parses HTML — it never makes the request, so the proxy goes on your HTTP client. Pair it with httpx: pass your German mobile endpoint to httpx.Client(proxy="http://USER:PASS@HOST:PORT"), fetch the page, then hand response.text to BeautifulSoup. Every request then leaves through your German mobile IP.

How the Two Pieces Fit Together

A common point of confusion: you can't "set a proxy in BeautifulSoup". BeautifulSoup is a parser — it turns raw HTML into a searchable tree. The network request happens in your HTTP client. We use httpx here because it has a clean sync and async API, HTTP/2 support, and first-class proxy handling. The proxy lives on the httpx client; BeautifulSoup just reads what comes back.

If you're already on requests and Playwright, see how to use a mobile proxy with Python (requests & Playwright). This guide is the modern httpx + parsing equivalent.

Before You Start

You'll need four values from your GermanProxy.io account:

Host and port of your endpoint
Username and password

The examples use PROXY_HOST, PORT, USER and PASS as placeholders — replace them with your own. Install both libraries:

pip install httpx beautifulsoup4

HTTP Proxy (the simplest case)

As of httpx 0.28, the proxy is set with the single proxy= argument (the older proxies= dict was removed and now raises a TypeError):

import httpx
from bs4 import BeautifulSoup

PROXY = "http://USER:PASS@PROXY_HOST:PORT"

with httpx.Client(proxy=PROXY, timeout=30) as client:
    response = client.get("https://www.example.de")
    response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.string if soup.title else "no title")

Every request made through that client now exits via your German mobile IP.

SOCKS5 Proxy

httpx supports SOCKS through an optional dependency:

pip install "httpx[socks]"

Then point the client at a socks5:// URL:

import httpx

PROXY = "socks5://USER:PASS@PROXY_HOST:PORT"

with httpx.Client(proxy=PROXY, timeout=30) as client:
    response = client.get("https://www.example.de")

Not sure which protocol to pick? See SOCKS5 vs HTTP proxy.

Async (for many pages at once)

The same proxy= argument works on httpx.AsyncClient, which is the natural fit when you fetch many German pages concurrently:

import asyncio
import httpx
from bs4 import BeautifulSoup

PROXY = "http://USER:PASS@PROXY_HOST:PORT"

async def fetch(client, url):
    r = await client.get(url)
    r.raise_for_status()
    return BeautifulSoup(r.text, "html.parser")

async def main():
    async with httpx.AsyncClient(proxy=PROXY, timeout=30) as client:
        pages = await asyncio.gather(
            fetch(client, "https://www.example.de/a"),
            fetch(client, "https://www.example.de/b"),
        )
    for soup in pages:
        print(soup.title.string if soup.title else "no title")

asyncio.run(main())

Verify the IP First

Before you scale up, confirm the proxy is actually in the path:

import httpx

with httpx.Client(proxy="http://USER:PASS@PROXY_HOST:PORT") as client:
    print(client.get("https://api.ipify.org").text)

If it returns your own address, re-check the proxy URL and credentials. A 407 means auth didn't reach the proxy — see how to fix 407 proxy authentication required.

Sticky Sessions vs. Rotation

Sticky session: keep the same German mobile IP across a multi-request flow (logins, pagination).
Rotation on demand: request a fresh IP at the proxy to spread load.

Re-use a single httpx.Client to keep your connection (and session) stable. See sticky sessions and IP rotation explained.

Scrape Responsibly

A mobile proxy improves reliability for legitimate data collection — it is not a tool to circumvent access controls or platform rules. Keep your crawl clean and defensible:

Respect robots.txt and each site's terms of use.
Throttle your requests with delays so you don't overload servers.
Collect only what you need, and handle any personal data in line with the GDPR.
Set an honest User-Agent that identifies your client, not a misleading one.

For the legal side, see is web scraping legal? EU & GDPR.

FAQ

Can I set a proxy directly in BeautifulSoup?

No. BeautifulSoup parses HTML; it never makes a request. Set the proxy on your HTTP client (httpx, requests) and pass the response text to BeautifulSoup.

Why httpx instead of requests?

Both work. httpx adds a native async client and HTTP/2, which helps when fetching many German pages at once. The requests approach is covered in our Python requests & Playwright guide.

Why do I get a TypeError about `proxies`?

httpx 0.28 removed the proxies= argument. Use proxy="..." for a single endpoint, or mounts= for per-scheme routing.

Do I need the `[socks]` extra?

Only for SOCKS5. Plain HTTP proxies need no extra package — just proxy="http://...".

Get Started

Fetch German pages on real German mobile IPs. Get a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, fair-use 20 GB/day.

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or try a shared 5G German proxy.