10% off ALL new Proxies – CODE: 10pro

MechanicalSoup Proxy Setup with a German Mobile IP

MechanicalSoup Proxy Setup with a German Mobile IP

The One Thing to Understand First

There is no "proxy setting" in MechanicalSoup itself. MechanicalSoup is a thin layer over the requests library — it drives a requests.Session for you and hands each page to BeautifulSoup. So the proxy doesn't go on the browser object; it goes on the session the browser wraps:

browser.session.proxies = {
    "http":  "http://USER:PASS@PROXY_HOST:PORT",
    "https": "http://USER:PASS@PROXY_HOST:PORT",
}

browser.session is the real requests.Session, exposed as a public attribute. Set .proxies on it once and every request the browser makes — open, follow_link, submit_selected — leaves through your German mobile IP. Credentials sit inline in the URL, so there's no separate authentication call.

With a dedicated German endpoint there's no pool to rotate in code — you point the session at one endpoint and rotate on demand at the proxy when you choose to.

Before You Start

You need four values from your GermanProxy.io account:

  • Host and port of your endpoint
  • Username and password

The examples use PROXY_HOST, PORT, USER and PASS as placeholders — replace them with your own. Install MechanicalSoup — it pulls in requests and beautifulsoup4 automatically:

pip install MechanicalSoup

Step 1 — Set the Proxy on the Session

Create a StatefulBrowser, set the proxy on its session, then browse. The proxy dict is keyed by the target scheme (http / https); both keys can point at the same endpoint:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser()

# Applies to every request this browser makes.
browser.session.proxies = {
    "http":  "http://USER:PASS@PROXY_HOST:PORT",
    "https": "http://USER:PASS@PROXY_HOST:PORT",
}

resp = browser.open("https://www.example.de")
print(resp.status_code)

# browser.page is the parsed BeautifulSoup object of the current page.
page = browser.page
print(page.title.string if page.title else "no title")

browser.open() returns a normal requests.Response, and browser.page gives you the current page as a BeautifulSoup tree to search with .find(...) and friends.

Step 2 — Or Pass a Pre-Configured Session

If you'd rather build the session yourself — to reuse it or set other options — pass it to the constructor:

import requests
import mechanicalsoup

sess = requests.Session()
sess.proxies = {
    "http":  "http://USER:PASS@PROXY_HOST:PORT",
    "https": "http://USER:PASS@PROXY_HOST:PORT",
}

browser = mechanicalsoup.StatefulBrowser(session=sess)
browser.open("https://www.example.de")

Both paths are equivalent — .session is public and injectable. Set the proxy after construction or pass it in; whichever fits your code.

Step 3 — Verify the IP Before You Scale

Confirm the proxy is actually in the path first. Open an IP-echo endpoint and read the raw text — https://api.ipify.org returns just the IP:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser()
browser.session.proxies = {
    "http":  "http://USER:PASS@PROXY_HOST:PORT",
    "https": "http://USER:PASS@PROXY_HOST:PORT",
}

resp = browser.open("https://api.ipify.org")
print(resp.text)  # -> the proxy's egress IP

Read resp.text here, not browser.page — ipify returns plain text, not HTML to parse. If it prints your own address instead of a German mobile IP, the proxy didn't apply. The most common cause is covered next; a 407 means auth didn't reach the proxy — see how to fix 407 proxy authentication required. For a fuller checklist, see validating the proxy connection.

Watch Out: Environment Proxies Can Override You

requests lets the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables override session.proxies. If any are set on the machine — common on shared servers and CI runners — they win, and your carefully set German endpoint is silently ignored. If your IP check shows the wrong address, check the environment first:

  • Clear the variables (unset HTTP_PROXY HTTPS_PROXY), or
  • Force your proxy by passing proxies= on the individual call.

This is a requests behaviour, not a MechanicalSoup bug — but it's the usual reason a MechanicalSoup proxy "doesn't work".

HTTP vs. SOCKS5 in MechanicalSoup

Because MechanicalSoup uses requests, SOCKS5 works the same way it does in requests — with one extra package:

pip install "requests[socks]"

Then use a socks5:// URL in the same dict:

browser.session.proxies = {
    "http":  "socks5://USER:PASS@PROXY_HOST:PORT",
    "https": "socks5://USER:PASS@PROXY_HOST:PORT",
}

Use socks5:// to resolve DNS on your side, or socks5h:// to resolve it at the proxy (matching curl's semantics). For most MechanicalSoup work — form-filling and link-following over HTTP — the plain HTTP endpoint is simplest. Background: SOCKS5 vs HTTP proxy.

Sticky Sessions vs. Rotation

With a dedicated German endpoint you control the IP at the proxy, not in your code:

  • Sticky session: re-use one StatefulBrowser across a multi-step flow — open a page, submit a form, follow a link — and the German mobile IP (and cookies) stay stable. This is MechanicalSoup's whole strength: it holds state between requests.
  • Rotation on demand: when you want a fresh IP, trigger rotation at the proxy. There's no per-request IP pool to manage.

See sticky sessions and IP rotation explained and dedicated vs shared mobile proxy.

Why a Mobile IP for MechanicalSoup at All

MechanicalSoup shines on classic server-rendered sites — login forms, paginated listings, anything you'd navigate by following links and submitting forms. Those flows send many requests from one client, and the egress IP is what platforms judge. Datacenter ranges are widely distrusted, so even a well-behaved script can hit friction on German sites. A real German mobile IP rides on carrier-grade NAT (CGNAT), the same shared-IP space thousands of ordinary phone users sit behind, which is why it carries higher IP reputation.

For the trade-offs against other IP types, see mobile vs residential vs datacenter proxy and what is a mobile proxy.

Scrape Responsibly

A mobile proxy improves reliability for legitimate data collection — it is not a tool to circumvent access controls or platform rules. Keep your MechanicalSoup runs clean and defensible:

  1. Respect each site's terms and its robots.txt; collect only public data you're permitted to use.
  2. Throttle with delays between open / follow_link calls so you don't overload servers.
  3. Collect only what you need, and handle any personal data in line with the GDPR. See is web scraping legal in the EU.
  4. Set an honest User-Agent that identifies your client, not a misleading one.

Whether mobile proxies themselves are lawful to use: are mobile proxies legal.

FAQ

How do I set a proxy in MechanicalSoup?

Set browser.session.proxies on the StatefulBrowser to a dict like {"http": "http://USER:PASS@PROXY_HOST:PORT", "https": "..."}. MechanicalSoup wraps a requests.Session, so the proxy lives on that session and applies to every request the browser makes.

Does the proxy apply to follow_link and submit_selected too?

Yes. browser.session.proxies is session-wide, so browser.open, follow_link and submit_selected all route through the same German mobile endpoint. Set it once after creating the browser.

Can I pass a pre-configured session instead?

Yes. Build a requests.Session, set its proxies, and pass it in: StatefulBrowser(session=sess). Both paths work — .session is public and injectable — so use whichever fits your code.

Does MechanicalSoup support SOCKS5?

Yes, through requests. Install the extra with pip install "requests[socks]" and use a socks5://PROXY_HOST:PORT URL in the proxies dict. Use socks5h:// to resolve DNS at the proxy rather than client-side. See the difference between HTTP and SOCKS5.

Why is my proxy being ignored?

requests lets HTTP_PROXY / HTTPS_PROXY / NO_PROXY environment variables override session.proxies. If those are set on the machine, they win. Clear them or pass proxies= per request to force your German endpoint.

How do I verify the outgoing IP?

Open an IP-echo endpoint and read the response text: browser.open("https://api.ipify.org").text. It should print the proxy's egress IP, not your own address. Don't parse it as HTML — ipify returns plain text.

Related guides

CTA

Run your MechanicalSoup scripts on real German mobile IPs. Get a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, on-demand rotation, fair-use 20 GB/day, from a transparent EU provider. Browse the plans to get started.

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or get a dedicated proxy on the Blue network.

Telegram Support
Nach oben scrollen