Best Proxy for Web Scraping with Python 2026: Tested with Real Code

Affiliate disclosure: Some links in this article are affiliate links. We earn a commission if you sign up through them — at no extra cost to you.

Finding the right proxy for Python scraping means matching the provider to your target's protection level — and knowing how to configure it correctly in code. A proxy that works perfectly for scraping e-commerce pricing will fail immediately against LinkedIn.

Best Proxy for Web Scraping with Python 2026: Tested with Real Code visual overview — Visual summary for Best Proxy for Web Scraping with Python 2026: Tested with Real Code.

This guide covers the best proxy providers for Python scraping in 2026, with working code examples for the three most common Python scraping libraries: requests, Scrapy, and Playwright.

Proxy Setup in Python: The Basics

Every Python HTTP library accepts proxies in a similar format. The proxy URL structure is:

http://username:password@proxy_host:port

For rotating residential proxies (where each request gets a new IP automatically), the host is the provider's rotating gateway. You don't manage individual IPs — the provider's infrastructure rotates them on every request.

Provider Overview: Ranked by Python Scraping Use Case

Provider	Best Python Use Case	$/GB	Difficulty Tier
DataImpulse	High-volume unprotected scraping	~$1	Tier 1
ProxyScrape	Mixed unprotected/moderate	~$2–4	Tier 1–2
GeoNode	Production residential scraping	~$3–5	Tier 2
IPRoyal	EU-focused residential	~$7–10	Tier 2
Bright Data	Enterprise targets + managed API	~$10.89	Tier 3

Code Examples by Library

1. Python `requests` Library

requests is the most common Python HTTP library. Proxy configuration is a single dictionary passed to every request.

Basic rotating proxy setup (GeoNode example)

import requests

# GeoNode rotating residential proxy
# Replace with your actual username/password from the GeoNode dashboard
PROXY_HOST = "rotating.geonode.com"
PROXY_PORT = "9000"
PROXY_USER = "YOUR_GEONODE_USERNAME"
PROXY_PASS = "YOUR_GEONODE_PASSWORD"

proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
}

response = requests.get(
    "https://httpbin.org/ip",
    proxies=proxies,
    headers=headers,
    timeout=30,
)
print(response.json())  # shows the exit IP used

Rotating User-Agent with each request

Consistent User-Agents are a detectable fingerprint even with residential IPs. Rotate them:

import requests
import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
]

def get_headers():
    return {"User-Agent": random.choice(USER_AGENTS)}

def scrape_url(url, proxies):
    try:
        response = requests.get(
            url,
            proxies=proxies,
            headers=get_headers(),
            timeout=30,
        )
        response.raise_for_status()
        return response.text
    except requests.RequestException as e:
        print(f"Request failed: {e}")
        return None

Retry logic for failed requests

import requests
import time
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def make_session_with_retry(proxies):
    session = requests.Session()
    session.proxies = proxies

    retry_strategy = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("http://", adapter)
    session.mount("https://", adapter)

    return session

# Usage
session = make_session_with_retry(proxies)
response = session.get("https://example.com", timeout=30)

2. Scrapy

Scrapy is the standard framework for production Python scraping. It handles concurrency, retries, and pipelines — but proxy configuration requires a middleware.

Scrapy proxy middleware (rotating proxy)

Add to your settings.py:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 90,
    "myproject.middlewares.ProxyMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 429]

# GeoNode credentials (set via environment variables in production)
PROXY_HOST = "rotating.geonode.com:9000"
PROXY_USER = "YOUR_GEONODE_USERNAME"
PROXY_PASS = "YOUR_GEONODE_PASSWORD"

Create middlewares.py:

# middlewares.py
import base64
from scrapy import signals

class ProxyMiddleware:
    def __init__(self, proxy_host, proxy_user, proxy_pass):
        self.proxy = f"http://{proxy_host}"
        credentials = f"{proxy_user}:{proxy_pass}"
        self.proxy_auth = base64.b64encode(credentials.encode()).decode()

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            proxy_host=crawler.settings.get("PROXY_HOST"),
            proxy_user=crawler.settings.get("PROXY_USER"),
            proxy_pass=crawler.settings.get("PROXY_PASS"),
        )

    def process_request(self, request, spider):
        request.meta["proxy"] = self.proxy
        request.headers["Proxy-Authorization"] = f"Basic {self.proxy_auth}"

Scrapy spider with proxy

# spiders/product_spider.py
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example-shop.com/products"]

    custom_settings = {
        "DOWNLOAD_DELAY": 1,          # 1 second between requests
        "RANDOMIZE_DOWNLOAD_DELAY": True,  # actual delay: 0.5–1.5s
        "CONCURRENT_REQUESTS_PER_DOMAIN": 5,
    }

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "name": product.css(".product-name::text").get(),
                "price": product.css(".product-price::text").get(),
                "url": response.urljoin(product.css("a::attr(href)").get()),
            }

        next_page = response.css("a.next-page::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

3. Playwright (JavaScript-rendering targets)

For targets that use JavaScript rendering — single-page apps, lazy-loaded content, dynamic pricing — you need a headless browser. Playwright handles this in Python.

Basic Playwright with proxy (GeoNode)

from playwright.sync_api import sync_playwright

PROXY_HOST = "rotating.geonode.com"
PROXY_PORT = "9000"
PROXY_USER = "YOUR_GEONODE_USERNAME"
PROXY_PASS = "YOUR_GEONODE_PASSWORD"

proxy_config = {
    "server": f"http://{PROXY_HOST}:{PROXY_PORT}",
    "username": PROXY_USER,
    "password": PROXY_PASS,
}

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy=proxy_config,
    )
    context = browser.new_context(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        viewport={"width": 1920, "height": 1080},
    )
    page = context.new_page()

    page.goto("https://example.com", wait_until="networkidle")
    content = page.content()
    print(content[:500])

    browser.close()

Playwright for Tier 3 targets with Bright Data Web Unlocker

For enterprise-protected targets, Bright Data's Web Unlocker handles CAPTCHA solving and fingerprint management automatically. You send a URL; Bright Data returns the rendered HTML.

import requests

# Bright Data Web Unlocker endpoint
# Get your credentials from: https://get.brightdata.com/293896dumfgt
BRIGHTDATA_HOST = "brd.superproxy.io"
BRIGHTDATA_PORT = 22225
BRIGHTDATA_USER = "YOUR_BRIGHTDATA_USERNAME-zone-web_unlocker"
BRIGHTDATA_PASS = "YOUR_BRIGHTDATA_PASSWORD"

proxies = {
    "http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@{BRIGHTDATA_HOST}:{BRIGHTDATA_PORT}",
    "https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@{BRIGHTDATA_HOST}:{BRIGHTDATA_PORT}",
}

# Bright Data handles all anti-detection automatically
response = requests.get(
    "https://www.amazon.com/dp/B08N5WRWNW",
    proxies=proxies,
    verify=False,  # Bright Data uses custom cert
    timeout=60,
)
print(response.status_code)  # 200 on Amazon with Web Unlocker

Provider Configuration Reference

GeoNode (Recommended for Tier 2)

# GeoNode rotating residential proxy
# Get credentials at: https://geonode.com/?ref=22282

PROXY_HOST = "rotating.geonode.com"
PROXY_PORT = "9000"   # HTTP rotating
# PROXY_PORT = "9001" # SOCKS5 rotating
# PROXY_PORT = "9002" # HTTP sticky (session-based)

# Country-specific targeting (add to username):
# PROXY_USER = "username-country-US"  # US exit nodes only
# PROXY_USER = "username-country-DE"  # German exit nodes only

proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

GeoNode specs for Python scraping:

10M+ residential IPs, 195+ countries
$3–$5/GB, no minimum commitment
Rotating and sticky session modes
Country + city-level targeting

→ Start GeoNode free plan — no credit card required.

IPRoyal (EU-Focused Tier 2)

# IPRoyal rotating residential proxy
# Get credentials at: https://iproyal.com/?r=Traffic-Creator

PROXY_HOST = "geo.iproyal.com"
PROXY_PORT = "12321"

# Country targeting:
# PROXY_USER = "username_country-DE"  # Germany
# PROXY_USER = "username_country-GB"  # UK

proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

IPRoyal specs: Strong European node density — best for EU-specific scraping targets at $7–$10/GB.

DataImpulse (Budget Tier 1–2)

# DataImpulse budget residential proxy
# Get credentials at: https://dataimpulse.com/?aff=312345

PROXY_HOST = "gw.dataimpulse.com"
PROXY_PORT = "823"

proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

DataImpulse specs: ~$0.70–$1.50/GB — cheapest residential option for unprotected to moderately protected scraping.

Bright Data Web Unlocker (Tier 3)

# Bright Data Web Unlocker
# Get access at: https://get.brightdata.com/293896dumfgt

proxies = {
    "http": "http://YOUR_USER-zone-web_unlocker:[email protected]:22225",
    "https": "http://YOUR_USER-zone-web_unlocker:[email protected]:22225",
}

# For SERP scraping with structured output:
# Use Bright Data's SERP API instead — returns structured JSON
import json

serp_response = requests.get(
    "https://www.google.com/search?q=python+web+scraping",
    proxies=proxies,
    verify=False,
    timeout=60,
)

Choosing the Right Provider for Your Python Scraper

Decision Tree

What are you scraping?

Unprotected sites (no bot protection)
  → DataImpulse (~$1/GB) or ProxyScrape (~$2–4/GB)
  → requests or Scrapy, rotating IPs

Moderately protected (rate limiting, IP reputation)
  → GeoNode (~$3–5/GB) for most targets
  → IPRoyal (~$7–10/GB) for EU-specific targets
  → requests/Scrapy with rotating residential IPs

Heavily protected (Google, Amazon, LinkedIn)
  → Bright Data Web Unlocker (~$10.89/GB + $3/1k requests)
  → requests with Bright Data proxy endpoint, or their SERP API
  → For browser automation on Tier 3: Bright Data's Scraping Browser

Concurrency Settings by Provider

Residential proxies handle concurrency differently than datacenter. Overly aggressive concurrency burns through bandwidth without improving throughput.

Recommended Scrapy settings by provider:

# GeoNode (mid-market residential)
CONCURRENT_REQUESTS = 20
CONCURRENT_REQUESTS_PER_DOMAIN = 5
DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True

# DataImpulse (budget residential — lower concurrency)
CONCURRENT_REQUESTS = 10
CONCURRENT_REQUESTS_PER_DOMAIN = 3
DOWNLOAD_DELAY = 1
RANDOMIZE_DOWNLOAD_DELAY = True

# Bright Data Web Unlocker (managed anti-detection)
# Follow Bright Data's recommended concurrency per zone
CONCURRENT_REQUESTS = 10
DOWNLOAD_DELAY = 0

Common Python Proxy Errors and Fixes

`ConnectionError: HTTPSConnectionPool`

Cause: Proxy connection failed — wrong host, port, or credentials.

Fix: Test with requests.get("https://httpbin.org/ip", proxies=proxies) before running your scraper. Verify host/port/credentials against your provider dashboard.

`407 Proxy Authentication Required`

Cause: Credentials not passed correctly or malformed in the proxy URL.

Fix: Check that username and password don't contain special characters that need URL-encoding. Encode with urllib.parse.quote(password, safe='') if the password has symbols.

from urllib.parse import quote
encoded_pass = quote(PROXY_PASS, safe='')
proxy_url = f"http://{PROXY_USER}:{encoded_pass}@{PROXY_HOST}:{PROXY_PORT}"

`503 Service Unavailable` on protected targets

Cause: The target detected your request as a bot despite using residential IPs. Common causes: consistent User-Agent, no referer header, missing accept-language header, too-fast request rate.

Fix:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
}

If this doesn't fix it on Tier 3 targets, upgrade to Bright Data Web Unlocker.

Requests returning CAPTCHAs instead of content

Cause: Target site is serving CAPTCHA challenges to your IP. Happens on Tier 3 targets with residential IPs not rotating fast enough.

Fix: Either rotate faster (new IP per request via the rotating endpoint), or use Bright Data Web Unlocker which handles CAPTCHA solving automatically.

Testing Your Proxy Setup

Always validate before running a full scrape:

import requests

def test_proxy(proxies):
    # Test 1: Check your exit IP
    try:
        r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
        print(f"Exit IP: {r.json()['origin']}")
    except Exception as e:
        print(f"Failed: {e}")
        return False

    # Test 2: Confirm IP is residential (not datacenter)
    try:
        r = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=10)
        data = r.json()
        print(f"IP type: {data.get('org', 'unknown')}")
        print(f"Country: {data.get('country', 'unknown')}")
        print(f"City: {data.get('city', 'unknown')}")
    except Exception as e:
        print(f"IPinfo check failed: {e}")

    return True

test_proxy(proxies)

A residential IP shows an ISP organization (e.g., "AS7922 Comcast Cable Communications"), not a data center ASN (e.g., "AS14061 DigitalOcean").

Provider Directory

Provider	Best Python Use Case	Affiliate Link
GeoNode	Production residential scraping	Sign up
IPRoyal	EU-focused residential	Sign up
DataImpulse	Budget high-volume unprotected	Sign up
ProxyScrape	Mixed Tier 1/2	Sign up
Bright Data	Enterprise targets + managed API	Sign up

Ready to try Bright Data?

The industry's most reliable proxy network — free trial & pay-as-you-go.

Try Bright Data →

Related Proxy Reviews

Frequently Asked Questions

Can I scrape with a free proxy?

Free public proxy lists work for basic testing but are not suitable for production scraping. They're unreliable, slow, and blocked by most sites. Webshare's free tier gives you 1 GB/month of legitimate datacenter bandwidth — the best free starting point.

What's the difference between rotating and sticky sessions in Python?

A rotating session gives you a new IP on every request — best for scraping the same domain at high frequency. A sticky session keeps the same IP for a configurable duration — best when the target site uses session cookies (e-commerce with cart state, logged-in scraping). In GeoNode and IPRoyal, you select the mode via the proxy port or username suffix.

Do I need Playwright or can I use requests?

Use requests or Scrapy for server-rendered targets (HTML is in the initial response). Use Playwright or Selenium when the content you need is loaded via JavaScript after page load — you'll see empty containers or placeholders in the requests response if JavaScript rendering is required.

What's the fastest Python proxy library for high-volume scraping?

aiohttp with async/await is faster than synchronous requests for high-concurrency scraping. Scrapy handles async internally. For most production scrapers, Scrapy with the right CONCURRENT_REQUESTS setting outperforms custom aiohttp implementations in terms of development speed vs. performance tradeoff.