10% off ALL new Proxies – CODE: 10pro

CrewAI Proxy Setup with a Dedicated German Mobile Proxy

CrewAI Proxy Setup with a Dedicated German Mobile Proxy

Direct Answer

To route CrewAI through a mobile proxy, set HTTP_PROXY and HTTPS_PROXY on the process that runs the crew. The proxy binds to the agent's web tools — ScrapeWebsiteTool and similar tools use Python's requests, which reads those variables automatically — so your German mobile IP is what fetches the pages. The LLM call keeps its own connection unless you route it through the proxy on purpose. You don't touch how the agents reason or talk to the model; you only change where the crew's web requests come from.

The One Thing Most CrewAI Proxy Guides Get Wrong

CrewAI is a multi-agent framework: you define agents, give them tools, and let a crew work a task together. The common mistake is thinking of "the proxy" as something the whole crew or the LLM sits behind. It isn't.

The proxy binds to the web tool, not the model. When an agent calls ScrapeWebsiteTool to read a German product page, that request is what should leave through your German mobile IP. The reasoning steps — the calls to your LLM — are a separate connection and usually don't need a German exit at all. Get this distinction right and the setup is a two-line change; get it wrong and you route model traffic through a mobile connection for no reason while your scrape still exits from your own address.

So the goal is precise: send the crew's outbound page fetches through the German mobile proxy, and leave everything else alone.

Before You Start

You'll need four values from your GermanProxy.io account:

  • Host and port of your endpoint
  • Username and password

The examples use PROXY_HOST, PORT, USER and PASS as placeholders — replace them with your own. There's no hardware to manage: the proxy is a dedicated mobile connection exposed as a host/port/credentials endpoint.

Install CrewAI with its tools package:

pip install crewai crewai-tools

Step 1 — Set the Proxy on the Crew's Environment

CrewAI's built-in web tools (like ScrapeWebsiteTool) fetch pages with Python's requests, and requests honours HTTP_PROXY / HTTPS_PROXY automatically. Set them before you build the crew:

import os

PROXY = "http://USER:PASS@PROXY_HOST:PORT"
os.environ["HTTP_PROXY"] = PROXY
os.environ["HTTPS_PROXY"] = PROXY

That's the whole binding. Every tool built on requests now exits through your German mobile IP — no per-tool configuration needed for the common case.

Step 2 — Give an Agent a Web Tool and Run the Crew

Here's a runnable crew that scrapes a German page through the proxy. The ScrapeWebsiteTool is the piece that carries the German mobile IP:

import os
from crewai import Agent, Task, Crew
from crewai_tools import ScrapeWebsiteTool

PROXY = "http://USER:PASS@PROXY_HOST:PORT"
os.environ["HTTP_PROXY"] = PROXY
os.environ["HTTPS_PROXY"] = PROXY

# This tool's requests exit through the German mobile IP.
scrape_tool = ScrapeWebsiteTool()

researcher = Agent(
    role="German market researcher",
    goal="Read a German page and summarise the public product data",
    backstory="Collects region-true data from German sites.",
    tools=[scrape_tool],
    verbose=True,
)

task = Task(
    description="Scrape https://www.example.de and report the page title and key public details.",
    expected_output="A short summary of the public data on the page.",
    agent=researcher,
)

crew = Crew(agents=[researcher], tasks=[task], verbose=True)
result = crew.kickoff()
print(result)

The agent decides when to scrape; the tool decides how the request leaves — and with the env vars set, it leaves through your German mobile IP.

Step 3 — Custom Tool With an Explicit Proxy Client

If you write your own CrewAI tool, or want the German exit on one tool only, pass the proxy straight to that tool's HTTP client instead of relying on the environment:

import httpx
from crewai.tools import BaseTool

PROXY = "http://USER:PASS@PROXY_HOST:PORT"

class GermanFetchTool(BaseTool):
    name: str = "german_fetch"
    description: str = "Fetch a URL through the German mobile proxy."

    def _run(self, url: str) -> str:
        # proxy= routes both http:// and https:// through the mobile proxy
        with httpx.Client(proxy=PROXY, timeout=30) as client:
            resp = client.get(url)
            resp.raise_for_status()
            return resp.text

This is the right choice when one crew mixes a German-exit scrape tool with, say, a direct call to your own internal API — only the tool you configure carries the mobile IP.

Verify the IP First

Before a real run, confirm the exit IP is a German mobile address. The quickest check is a one-line request through the same proxy:

curl -x "http://USER:PASS@PROXY_HOST:PORT" https://api.ipify.org?format=json

If it returns your own address, the env vars didn't reach the crew's process or the custom client isn't using the proxy — re-check the URL and that the credentials are in it. Confirm the ASN and geolocation look like a German mobile network, not a datacenter range. A full checklist is in test your mobile proxy setup.

Sticky Sessions vs. Rotation

How the crew should handle IPs depends on the task:

  • Sticky session — keep the same German mobile IP across a multi-step flow (open a page, follow links, extract) so a sequence of tool calls looks like one coherent visitor.
  • Rotation on demand — request a fresh IP at the proxy between runs to spread load across a longer job.

Keep request rates human-reasonable and cap how many tools fire at once. The mechanics are in sticky sessions and IP rotation.

Why a Mobile IP at All

German content is localised. German marketplaces and Google.de serve different results, pricing and stock by country and network. Have your crew fetch a German site from a US datacenter range and it measures the wrong market. A German IP gives it the in-country view the task needs.

Mobile IPs carry high IP reputation. German sites filter datacenter ranges aggressively — often a 403 Forbidden within a handful of requests. Mobile IPs sit in shared CGNAT pools used by thousands of real subscribers, so they carry high IP reputation — the reason a mobile IP keeps working where a datacenter range stalls.

For the trade-offs between IP types, see mobile vs residential vs datacenter proxy and, on cost, dedicated mobile vs pay-per-GB residential. For the wider agent picture, see mobile proxies for AI agents.

Scrape Responsibly

A mobile proxy improves reliability for legitimate data collection — it is not a tool to circumvent access controls or platform rules. A crew runs largely unattended, so the responsibility to keep it clean is higher, not lower:

  1. Respect robots.txt and each site's terms — bake the check into the tool, not just into intentions.
  2. Throttle — add a delay between tool calls and cap concurrency so you don't overload target servers.
  3. Collect only public data you need, and handle any personal data in line with the GDPR.
  4. Identify your client honestly — set a truthful User-Agent; don't impersonate someone else.

Whether a specific task is lawful depends on your jurisdiction and the target's terms — this is general information, not legal advice. See are mobile proxies legal and is web scraping legal in the EU.

[!NOTE]
Note: This article is general information, not legal advice. Whether a specific data-collection task is lawful depends on your jurisdiction, the target site's terms and the data involved.

FAQ

Does the proxy change how CrewAI talks to the LLM?

No. The proxy binds to the crew's web tools — the requests they make to target sites. The LLM call your agents use to reason stays on its own connection unless you deliberately route it through the proxy too. In practice you set HTTP_PROXY / HTTPS_PROXY so the scrape and fetch tools exit through the German mobile IP, and leave the model traffic alone.

Which CrewAI tools honour HTTP_PROXY?

Any tool built on Python's requests — including the built-in ScrapeWebsiteTool — reads HTTP_PROXY and HTTPS_PROXY automatically, because that's requests' default behaviour. For a custom tool that uses httpx or a browser, pass the proxy to that client directly, as in the custom-tool example above.

Why do I get a 407 error?

407 Proxy Authentication Required means the username and password didn't reach the proxy. Confirm they're in the proxy URL you exported (http://USER:PASS@PROXY_HOST:PORT). See how to fix 407 proxy authentication required.

Do I need a rotating pool for CrewAI?

No. Pool middlewares exist to pick an IP from many. With one dedicated German endpoint you point every tool at a single host and rotate on demand at the proxy — nothing to manage in the crew. See dedicated vs shared mobile proxy.

HTTP or SOCKS5?

Tools built on requests work with an HTTP endpoint out of the box, which is why the examples use http://. SOCKS5 and OpenVPN are available too; reach for SOCKS5 only when a specific tool or client needs it — for most CrewAI work, HTTP is simplest.

Can I use a German mobile IP for a custom scraping tool?

Yes. In a custom CrewAI tool, pass the proxy to whatever client the tool uses — requests, httpx or a headless browser — and that tool's traffic exits through the German mobile IP. If you drive a real browser inside a tool, set the proxy on the browser launch; see mobile proxy for AI browser agents.

Related guides

Get Started

Give your crew a real German mobile IP to collect from. Start with a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, on-demand rotation, fair-use 20 GB/day, from a transparent EU provider. Browse the plans to get started.

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — see German 4G/5G mobile proxy options or get a dedicated proxy on the Blue network.

Telegram Support
Nach oben scrollen