Colly Proxy Setup: German Mobile IPs for Go Scrapers

The One Thing Most Colly Proxy Guides Get Wrong

A recurring Colly mistake: building the Proxy-Authorization header by hand. People base64-encode user:pass, set it as a header, confirm it works in cURL — and Colly still returns 407 Proxy Authentication Required. Hand-rolling that header is unreliable in Colly.

The supported path is simpler. Put the credentials in the proxy URL and let Colly parse them:

c.SetProxy("http://USER:PASS@PROXY_HOST:PORT")

Colly runs that string through Go's url.Parse, which lifts the user:pass userinfo into the request, and Go's HTTP transport emits the Proxy-Authorization header for you. No custom transport, no header injection.

With a dedicated German mobile endpoint you also skip RoundRobinProxySwitcher — the rotation helper exists to cycle across many proxies. You point every request at one endpoint and rotate on demand at the proxy when you choose to.

---

Before You Start

You need four values from your GermanProxy.io account:

• Host and port of your endpoint
• Username and password

The examples use PROXY_HOST, PORT, USER and PASS as placeholders — replace them with your own. Install the v2 module (the bare path is the 2019 v1 release):

go get github.com/gocolly/colly/v2

Import it — the package name stays colly, only the path carries the /v2:

import (
    "github.com/gocolly/colly/v2"
    "github.com/gocolly/colly/v2/proxy"
)

---

Step 1 — Set the Proxy on the Collector

This is the complete, runnable pattern. SetProxy returns an error — don't ignore it, a bad URL fails right here:

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Single dedicated endpoint, credentials in the URL.
	if err := c.SetProxy("http://USER:PASS@PROXY_HOST:PORT"); err != nil {
		log.Fatal(err)
	}

	c.OnHTML("title", func(e *colly.HTMLElement) {
		fmt.Println("page title:", e.Text)
	})

	c.OnError(func(r *colly.Response, err error) {
		log.Printf("request failed: %v (status %d)", err, r.StatusCode)
	})

	if err := c.Visit("https://www.example.de"); err != nil {
		log.Fatal(err)
	}
}

Every request this Collector makes now leaves through your German mobile IP. Colly reads the credentials from the URL and Go sends the auth header for you.

---

Step 2 — Verify the IP Before You Scale

Confirm the proxy is actually in the path before you build anything on top of it. Point the Collector at an IP-echo endpoint and print the body from an OnResponse callback — https://api.ipify.org returns the raw IP as plain text:

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()
	if err := c.SetProxy("http://USER:PASS@PROXY_HOST:PORT"); err != nil {
		log.Fatal(err)
	}

	c.OnResponse(func(r *colly.Response) {
		fmt.Printf("outgoing IP: %sn", r.Body)
	})

	if err := c.Visit("https://api.ipify.org"); err != nil {
		log.Fatal(err)
	}
}

Run it once without the proxy and once with it — the printed IP must differ. If it still shows your own address, the proxy didn't apply: most often a custom http.Transport overrode it, or auth failed silently. A persistent 407 points at the credentials — see how to fix 407 proxy authentication required. For a fuller checklist, see test your mobile proxy setup.

---

Step 3 — RoundRobinProxySwitcher (only if you have several endpoints)

If you ever run more than one endpoint, proxy.RoundRobinProxySwitcher cycles a fresh one per request. Its signature is RoundRobinProxySwitcher(ProxyURLs ...string) (colly.ProxyFunc, error), and you attach the returned func with c.SetProxyFunc:

package main

import (
	"fmt"
	"log"

	"github.com/gocolly/colly/v2"
	"github.com/gocolly/colly/v2/proxy"
)

func main() {
	// AllowURLRevisit lets us hit the same URL repeatedly to see rotation.
	c := colly.NewCollector(colly.AllowURLRevisit())

	rp, err := proxy.RoundRobinProxySwitcher(
		"http://USER:PASS@PROXY_HOST:PORT",
		"http://USER:PASS@PROXY_HOST_2:PORT",
	)
	if err != nil {
		log.Fatal(err)
	}
	c.SetProxyFunc(rp)

	c.OnResponse(func(r *colly.Response) {
		fmt.Printf("%sn", r.Body) // each request goes out via the next proxy
	})

	for i := 0; i < 4; i++ {
		if err := c.Visit("https://api.ipify.org"); err != nil {
			log.Println(err)
		}
	}
}

With a single dedicated German endpoint you don't need this — SetProxy from Step 1 is enough. Note that under colly.Async(true) the round-robin index advances per request across goroutines, so rotation isn't strictly one-proxy-per-Visit ordering.

---

HTTP vs. SOCKS5 in Colly

Both SetProxy and RoundRobinProxySwitcher pick the proxy type from the URL scheme — an empty scheme is treated as http:

Aspect	HTTP endpoint	SOCKS5 endpoint
URL form	http://HOST:PORT	socks5://HOST:PORT
Credential auth	Robust — carried in the URL	Weaker across Go versions
Best auth mode	Credentials in URL	Test, or use IP allowlist
Typical use	Default for most scraping	Lower-level / non-HTTP traffic

For most Colly work, the HTTP endpoint with credentials in the URL is the simplest, most reliable path. Colly accepts socks5:// too, but Go's SOCKS5 username/password support has been less robust than HTTP — test it against your endpoint before relying on it. Background: SOCKS5 vs HTTP proxy.

---

Sticky Sessions vs. Rotation

With a dedicated German endpoint you control the IP at the proxy, not in your Go code:

• Sticky session: keep the same German mobile IP across a multi-request crawl — pagination, a logged-in flow — for as long as you need it.
• Rotation on demand: when you want a fresh IP, trigger rotation at the proxy. There's no per-request IP pool to manage in Colly.

This is why a dedicated endpoint keeps the code small: no rotation list, no health-checking, no retry-on-dead-IP loop. See sticky sessions and IP rotation explained and dedicated vs shared mobile proxy.

---

Why a Mobile IP for Colly at All

Colly is fast — a Go crawler can fan out to thousands of pages quickly — and that speed is exactly what draws attention on the egress side. Datacenter IP ranges are widely distrusted, so even a well-behaved crawler can hit friction on German platforms. A real German mobile IP rides on carrier-grade NAT (CGNAT), the same shared-IP space thousands of ordinary phone users sit behind, which is why it carries higher IP reputation.

For the trade-offs against other IP types, see mobile vs residential vs datacenter proxy and what is a mobile proxy.

---

Scrape Responsibly

A mobile proxy improves reliability for legitimate data collection — it is not a tool to circumvent access controls or platform rules. Keep your Colly crawl clean and defensible:

1. Respect each site's terms and its robots.txt; collect only public data you're permitted to use. Colly can honour robots.txt — leave that on.
2. Throttle with a LimitRule (Delay, RandomDelay, Parallelism) so a fast Go crawler doesn't overload servers.
3. Collect only what you need, and handle any personal data in line with the GDPR. See is web scraping legal in the EU.
4. Set an honest User-Agent that identifies your crawler, not a misleading one.

Whether mobile proxies themselves are lawful to use: are mobile proxies legal.

---

FAQ

How do I set a proxy in Colly?

Call c.SetProxy("http://USER:PASS@PROXY_HOST:PORT") on the Collector. Colly parses the user:pass from the URL and Go's transport sends the Proxy-Authorization header for you — no custom transport needed. Import the v2 module: github.com/gocolly/colly/v2.

Why does my manual Proxy-Authorization header return 407?

Hand-adding the base64 Proxy-Authorization header is unreliable in Colly and can still return 407 Proxy Authentication Required. Put the credentials in the SetProxy URL instead — that's the supported path, and Go emits the auth header automatically. See how to fix 407 proxy authentication required.

Do I need RoundRobinProxySwitcher for a dedicated endpoint?

No. RoundRobinProxySwitcher exists to cycle a request across many proxies. With one dedicated German endpoint you point every request at it and rotate on demand at the proxy — no switcher, no health-check loop.

Does Colly support SOCKS5 proxies?

Yes. SetProxy and RoundRobinProxySwitcher pick the proxy type from the URL scheme, so a socks5://PROXY_HOST:PORT URL works. SOCKS5 credential auth is less robust than HTTP across Go versions — test it against your endpoint, or use the HTTP endpoint for credential auth. See HTTP or SOCKS5 — which fits your stack.

Which Colly import path should I use?

Use the v2 module github.com/gocolly/colly/v2 (and github.com/gocolly/colly/v2/proxy for the switcher). The bare github.com/gocolly/colly path is the old v1.2.0 from 2019 — all current work lives under /v2.

How do I verify the proxy is actually in the path?

Visit an IP-echo endpoint like https://api.ipify.org and print r.Body from an OnResponse callback. Run once with and once without the proxy — the IP must differ. If it still shows your own address, the proxy didn't apply.

---

Related guides

• Scrapy Proxy Setup: Route Spiders via German Mobile IPs
• Async Web Scraping in Python Through One Mobile IP
• Scrapling Proxy Setup: German Mobile IP Across Every Fetcher

CTA

Run your Go crawlers on real German mobile IPs. Get a German mobile proxy — HTTP/SOCKS5/OpenVPN, sticky sessions, on-demand rotation, fair-use 20 GB/day, from a transparent EU provider.

---

Get a German mobile proxy: Real 4G/5G IPs from Germany, fair-use 20 GB/day — compare German mobile proxy plans or start with a dedicated 5G proxy.