Scrapy

This page covers Scrapy proxy setup, authentication, and IP rotation for crawls. Scrapy routes requests through a ColdProxy gateway with the built-in HttpProxyMiddleware: no extra package required. Set request.meta["proxy"] (with credentials in the URL) and Scrapy extracts the username/password into a Proxy-Authorization header for you. Below are the per-request pattern, a project-wide middleware, rotating and sticky setups, and the credential-stripping gotcha that trips up rotating-proxy code. For background, see IP whitelisting vs username/password and static vs rotating proxies.

#Your ColdProxy connection details

Proxy URL: http://USERNAME:PASSWORD@gw-2312.coldproxy.com:30000
Host:      gw-2312.coldproxy.com
Port:      30000     (any port in your plan's 30000-34999 allocation)

2312 is your service/package ID from the client area. Residential IPv4 (GB Based) and Residential IPv6 activate instantly after payment clears; Residential IPv4 (Unmetered) and Datacenter IPv6 start automatic setup after payment clears. Allow up to 10 minutes for those two plans and connect as soon as setup completes. See Activation and setup times, then verify with the proxy checker and my IP.

#Per-request proxy via request.meta

The minimal setup: put the full proxy URL (credentials included) in request.meta["proxy"]. HttpProxyMiddleware is enabled by default, parses the credentials, and adds the Proxy-Authorization: Basic ... header automatically.

import scrapy

class IpSpider(scrapy.Spider):
    name = "ip"
    PROXY = "http://USERNAME:PASSWORD@gw-2312.coldproxy.com:30000"

    def start_requests(self):
        yield scrapy.Request(
            "https://api.vipv6proxy.com/api/checker/my-ip",
            meta={"proxy": self.PROXY},
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.json()["ip"])

#Project-wide Scrapy proxy middleware (stable auth for redirects and retries)

For production spiders, set the proxy and a stable Proxy-Authorization header in a small downloader middleware. Setting the header explicitly survives redirects and retries.

# middlewares.py
import base64

class ColdProxyMiddleware:
    HOST = "gw-2312.coldproxy.com:30000"
    USER = "USERNAME"
    PWD = "PASSWORD"

    def process_request(self, request, spider):
        request.meta["proxy"] = f"http://{self.HOST}"
        token = base64.b64encode(f"{self.USER}:{self.PWD}".encode()).decode()
        request.headers["Proxy-Authorization"] = f"Basic {token}"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ColdProxyMiddleware": 610,
    # keep below the built-in HttpProxyMiddleware (default 750)
}

#Geo-targeting and sticky sessions through the username

ColdProxy encodes geo and session controls as tags inside the username, grammar user-key-value-key-value... (full reference: Username tag scheme). Rotation is the default; a sticky IP needs both a session id and a duration together. The tagged username goes in the proxy URL (or the middleware's USER). See rotating and sticky proxies and geo-targeting with proxies.

# Rotating, United States — new IP per request
PROXY = "http://user-country-us:PASSWORD@gw-2312.coldproxy.com:30000"

# Sticky 10 minutes, United States — same IP while the session holds
PROXY = "http://user-country-us-session-481516-time-10m:PASSWORD@gw-2312.coldproxy.com:30000"

#Scrapy rotating proxy sessions per request

To rotate sticky identities across a crawl, vary the session tag in the username per request. Each distinct session id maps to its own sticky IP for the chosen duration.

import scrapy

class RotateSpider(scrapy.Spider):
    name = "rotate"
    TEMPLATE = "http://user-country-us-session-{sid}-time-10m:PASSWORD@gw-2312.coldproxy.com:30000"

    def start_requests(self):
        for i in range(5):
            proxy = self.TEMPLATE.format(sid=481516 + i)  # session IDs are numeric
            yield scrapy.Request(
                "https://api.vipv6proxy.com/api/checker/my-ip",
                meta={"proxy": proxy},
                dont_filter=True,
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.json()["ip"])

#IP whitelisting (no credentials in code)

ColdProxy supports IP authentication: whitelist up to 50 IPs in the client area or via the plan API, and use the gateway with no user:pass at all.

yield scrapy.Request(
    "https://api.vipv6proxy.com/api/checker/my-ip",
    meta={"proxy": "http://gw-2312.coldproxy.com:30000"},
)

#Useful settings

Keep concurrency and rate sensible so you reduce IP blocks during public web data collection without overloading targets. See proxy error codes 407/403/429/502/504 for what the gateway returns under pressure.

# settings.py
CONCURRENT_REQUESTS = 16
DOWNLOAD_DELAY = 0.5
RETRY_ENABLED = True
RETRY_TIMES = 3
AUTOTHROTTLE_ENABLED = True

#Verify and troubleshoot

• Log response.text from https://api.vipv6proxy.com/api/checker/my-ip, or run the proxy checker, to confirm the exit IP and location.
• Connection errors immediately after ordering can have several causes. Check the service status and connection settings. If you bought Residential IPv4 (Unmetered) or Datacenter IPv6, automatic setup may still be running for up to 10 minutes after payment clears; see Activation and setup times.
• For IPv6 plans, many sites are IPv4-only (no AAAA record) and will fail over IPv6; check the target with the IPv6 checker before buying. Confirmed IPv6-ready platforms include Google, YouTube, Facebook, Instagram, Bing, and LinkedIn.

Ready to provision? Compare plans on pricing or browse the proxy products.