Best proxies for Python web scraping in 2026 — with actual code examples for requests, Scrapy, and Playwright. Ranked by target difficulty and developer experience.
Best Proxy for Web Scraping with Python 2026: Tested with Real Code Affiliate disclosure: Some links in this article are affiliate links. We earn a commission if you sign up through them — at no extra cost to you. Finding the right proxy for Python scraping means matching the provider to your target's protection level — and knowing how to configure it correctly in code. A proxy that works perfectly for scraping e-commerce pricing will fail immediately against LinkedIn. Visual summary for Best Proxy for Web Scraping with Python 2026: Tested with Real Code. This guide covers the best proxy providers for Python scraping in 2026, with working code examples for the three most common Python scraping libraries: requests , Scrapy , and Playwright . Proxy Setup in Python: The Basics Every Python HTTP library accepts proxies in a similar format. The proxy URL structure is: http://username:password@proxy_host:port For rotating residential proxies (where each request gets a new IP automatically), the host is the provider's rotating gateway. You don't manage individual IPs — the provider's infrastructure rotates them on every request. Provider Overview: Ranked by Python Scraping Use Case Provider Best Python Use Case $/GB Difficulty Tier DataImpulse High-volume unprotected scraping ~$1 Tier 1 ProxyScrape Mixed unprotected/moderate ~$2–4 Tier 1–2 GeoNode Production residential scraping ~$3–5 Tier 2 IPRoyal EU-focused residential ~$7–10 Tier 2 Bright Data Enterprise targets + managed API ~$10.89 Tier 3 Code Examples by Library 1. Python requests Library requests is the most common Python HTTP library. Proxy configuration is a single dictionary passed to every request. Basic rotating proxy setup (GeoNode example) import requests # GeoNode rotating residential proxy # Replace with your actual username/password from the GeoNode dashboard PROXY_HOST = "rotating.geonode.com" PROXY_PORT = "9000" PROXY_USER = "YOUR_GEONODE_USERNAME" PROXY_PASS = "YOUR_GEONODE_PASSWORD" proxies = { "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", } headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" } response = requests.get( "https://httpbin.org/ip", proxies=proxies, headers=headers, timeout=30, ) print(response.json()) # shows the exit IP used Rotating User-Agent with each request Consistent User-Agents are a detectable fingerprint even with residential IPs. Rotate them: import requests import random USER_AGENTS = [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", ] def get_headers(): return {"User-Agent": random.choice(USER_AGENTS)} def scrape_url(url, proxies): try: response = requests.get( url, proxies=proxies, headers=get_headers(), timeout=30, ) response.raise_for_status() return response.text except requests.RequestException as e: print(f"Request failed: {e}") return None Retry logic for failed requests import requests import time from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry def make_session_with_retry(proxies): session = requests.Session() session.proxies = proxies retry_strategy = Retry( total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504], ) adapter = HTTPAdapter(max_retries=retry_strategy) session.mount("http://", adapter) session.mount("https://", adapter) return session # Usage session = make_session_with_retry(proxies) response = session.get("https://example.com", timeout=30) 2. Scrapy Scrapy is the standard framework for production Python scraping. It handles concurrency, retries, and pipelines — but proxy configuration requires a middleware. Scrapy proxy middleware (rotating proxy) Add to your settings.py : # settings.py DOWNLOADER_MIDDLEWARES = { "scrapy.downloadermiddlewares.retry.RetryMiddleware": 90, "myproject.middlewares.ProxyMiddleware": 100, "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110, } RETRY_TIMES = 3 RETRY_HTTP_CODES = [500, 502, 503, 504, 429] # GeoNode credentials (set via environment variables in production) PROXY_HOST = "rotating.geonode.com:9000" PROXY_USER = "YOUR_GEONODE_USERNAME" PROXY_PASS = "YOUR_GEONODE_PASSWORD" Create middlewares.py : # middlewares.py import base64 from scrapy import signals class ProxyMiddleware: def __init__(self, proxy_host, proxy_user, proxy_pass): self.proxy = f"http://{proxy_host}" credentials = f"{proxy_user}:{proxy_pass}" self.proxy_auth = base64.b64encode(credentials.encode()).decode() @classmethod def from_crawler(cls, crawler): return cls( proxy_host=crawler.settings.get("PROXY_HOST"), proxy_user=crawler.settings.get("PROXY_USER"), proxy_pass=crawler.settings.get("PROXY_PASS"), ) def process_request(self, request, spider): request.meta["proxy"] = self.proxy request.headers["Proxy-Authorization"] = f"Basic {self.proxy_auth}" Scrapy spider with proxy # spiders/product_spider.py import scrapy class ProductSpider(scrapy.Spider): name = "products" start_urls = ["https://example-shop.com/products"] custom_settings = { "DOWNLOAD_DELAY": 1, # 1 second between requests "RANDOMIZE_DOWNLOAD_DELAY": True, # actual delay: 0.5–1.5s "CONCURRENT_REQUESTS_PER_DOMAIN": 5, } def parse(self, response): for product in response.css(".product-card"): yield { "name": product.css(".product-name::text").get(), "price": product.css(".product-price::text").get(), "url": response.urljoin(product.css("a::attr(href)").get()), } next_page = response.css("a.next-page::attr(href)").get() if next_page: yield response.follow(next_page, self.parse) 3. Playwright (JavaScript-rendering targets) For targets that use JavaScript rendering — single-page apps, lazy-loaded content, dynamic pricing — you need a headless browser. Playwright handles this in Python. Basic Playwright with proxy (GeoNode) from playwright.sync_api import sync_playwright PROXY_HOST = "rotating.geonode.com" PROXY_PORT = "9000" PROXY_USER = "YOUR_GEONODE_USERNAME" PROXY_PASS = "YOUR_GEONODE_PASSWORD" proxy_config = { "server": f"http://{PROXY_HOST}:{PROXY_PORT}", "username": PROXY_USER, "password": PROXY_PASS, } with sync_playwright() as p: browser = p.chromium.launch( headless=True, proxy=proxy_config, ) context = browser.new_context( user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", viewport={"width": 1920, "height": 1080}, ) page = context.new_page() page.goto("https://example.com", wait_until="networkidle") content = page.content() print(content[:500]) browser.close() Playwright for Tier 3 targets with Bright Data Web Unlocker For enterprise-protected targets, Bright Data's Web Unlocker handles CAPTCHA solving and fingerprint management automatically. You send a URL; Bright Data returns the rendered HTML. import requests # Bright Data Web Unlocker endpoint # Get your credentials from: https://get.brightdata.com/293896dumfgt BRIGHTDATA_HOST = "brd.superproxy.io" BRIGHTDATA_PORT = 22225 BRIGHTDATA_USER = "YOUR_BRIGHTDATA_USERNAME-zone-web_unlocker" BRIGHTDATA_PASS = "YOUR_BRIGHTDATA_PASSWORD" proxies = { "http": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@{BRIGHTDATA_HOST}:{BRIGHTDATA_PORT}", "https": f"http://{BRIGHTDATA_USER}:{BRIGHTDATA_PASS}@{BRIGHTDATA_HOST}:{BRIGHTDATA_PORT}", } # Bright Data handles all anti-detection automatically response = requests.get( "https://www.amazon.com/dp/B08N5WRWNW", proxies=proxies, verify=False, # Bright Data uses custom cert timeout=60, ) print(response.status_code) # 200 on Amazon with Web Unlocker Provider Configuration Reference GeoNode (Recommended for Tier 2) # GeoNode rotating residential proxy # Get credentials at: https://geonode.com/?ref=22282 PROXY_HOST = "rotating.geonode.com" PROXY_PORT = "9000" # HTTP rotating # PROXY_PORT = "9001" # SOCKS5 rotating # PROXY_PORT = "9002" # HTTP sticky (session-based) # Country-specific targeting (add to username): # PROXY_USER = "username-country-US" # US exit nodes only # PROXY_USER = "username-country-DE" # German exit nodes only proxies = { "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", } GeoNode specs for Python scraping: 10M+ residential IPs, 195+ countries $3–$5/GB, no minimum commitment Rotating and sticky session modes Country + city-level targeting → Start GeoNode free plan — no credit card required. IPRoyal (EU-Focused Tier 2) # IPRoyal rotating residential proxy # Get credentials at: https://iproyal.com/?r=Traffic-Creator PROXY_HOST = "geo.iproyal.com" PROXY_PORT = "12321" # Country targeting: # PROXY_USER = "username_country-DE" # Germany # PROXY_USER = "username_country-GB" # UK proxies = { "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", } IPRoyal specs: Strong European node density — best for EU-specific scraping targets at $7–$10/GB. DataImpulse (Budget Tier 1–2) # DataImpulse budget residential proxy # Get credentials at: https://dataimpulse.com/?aff=312345 PROXY_HOST = "gw.dataimpulse.com" PROXY_PORT = "823" proxies = { "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}", } DataImpulse specs: ~$0.70–$1.50/GB — cheapest residential option for unprotected to moderately protected scraping. Bright Data Web Unlocker (Tier 3) # Bright Data Web Unlocker # Get access at: https://get.brightdata.com/293896dumfgt proxies = { "http": "http://YOUR_USER-zone-web_unlocker:[email protected]:22225", "https": "http://YOUR_USER-zone-web_unlocker:[email protected]:22225", } # For SERP scraping with structured output: # Use Bright Data's SERP API instead — returns structured JSON import json serp_response = requests.get( "https://www.google.com/search?q=python+web+scraping", proxies=proxies, verify=False, timeout=60, ) Choosing the Right Provider for Your Python Scraper Decision Tree What are you scraping? Unprotected sites (no bot protection) → DataImpulse (~$1/GB) or ProxyScrape (~$2–4/GB) → requests or Scrapy, rotating IPs Moderately protected (rate limiting, IP reputation) → GeoNode (~$3–5/GB) for most targets → IPRoyal (~$7–10/GB) for EU-specific targets → requests/Scrapy with rotating residential IPs Heavily protected (Google, Amazon, LinkedIn) → Bright Data Web Unlocker (~$10.89/GB + $3/1k requests) → requests with Bright Data proxy endpoint, or their SERP API → For browser automation on Tier 3: Bright Data's Scraping Browser Concurrency Settings by Provider Residential proxies handle concurrency differently than datacenter. Overly aggressive concurrency burns through bandwidth without improving throughput. Recommended Scrapy settings by provider: # GeoNode (mid-market residential) CONCURRENT_REQUESTS = 20 CONCURRENT_REQUESTS_PER_DOMAIN = 5 DOWNLOAD_DELAY = 0.5 RANDOMIZE_DOWNLOAD_DELAY = True # DataImpulse (budget residential — lower concurrency) CONCURRENT_REQUESTS = 10 CONCURRENT_REQUESTS_PER_DOMAIN = 3 DOWNLOAD_DELAY = 1 RANDOMIZE_DOWNLOAD_DELAY = True # Bright Data Web Unlocker (managed anti-detection) # Follow Bright Data's recommended concurrency per zone CONCURRENT_REQUESTS = 10 DOWNLOAD_DELAY = 0 Common Python Proxy Errors and Fixes ConnectionError: HTTPSConnectionPool Cause: Proxy connection failed — wrong host, port, or credentials. Fix: Test with requests.get("https://httpbin.org/ip", proxies=proxies) before running your scraper. Verify host/port/credentials against your provider dashboard. 407 Proxy Authentication Required Cause: Credentials not passed correctly or malformed in the proxy URL. Fix: Check that username and password don't contain special characters that need URL-encoding. Encode with urllib.parse.quote(password, safe='') if the password has symbols. from urllib.parse import quote encoded_pass = quote(PROXY_PASS, safe='') proxy_url = f"http://{PROXY_USER}:{encoded_pass}@{PROXY_HOST}:{PROXY_PORT}" 503 Service Unavailable on protected targets Cause: The target detected your request as a bot despite using residential IPs. Common causes: consistent User-Agent, no referer header, missing accept-language header, too-fast request rate. Fix: headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate, br", "Connection": "keep-alive", "Upgrade-Insecure-Requests": "1", } If this doesn't fix it on Tier 3 targets, upgrade to Bright Data Web Unlocker. Requests returning CAPTCHAs instead of content Cause: Target site is serving CAPTCHA challenges to your IP. Happens on Tier 3 targets with residential IPs not rotating fast enough. Fix: Either rotate faster (new IP per request via the rotating endpoint), or use Bright Data Web Unlocker which handles CAPTCHA solving automatically. Testing Your Proxy Setup Always validate before running a full scrape: import requests def test_proxy(proxies): # Test 1: Check your exit IP try: r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10) print(f"Exit IP: {r.json()['origin']}") except Exception as e: print(f"Failed: {e}") return False # Test 2: Confirm IP is residential (not datacenter) try: r = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=10) data = r.json() print(f"IP type: {data.get('org', 'unknown')}") print(f"Country: {data.get('country', 'unknown')}") print(f"City: {data.get('city', 'unknown')}") except Exception as e: print(f"IPinfo check failed: {e}") return True test_proxy(proxies) A residential IP shows an ISP organization (e.g., "AS7922 Comcast Cable Communications"), not a data center ASN (e.g., "AS14061 DigitalOcean"). Provider Directory Provider Best Python Use Case Affiliate Link GeoNode Production residential scraping Sign up IPRoyal EU-focused residential Sign up DataImpulse Budget high-volume unprotected Sign up ProxyScrape Mixed Tier 1/2 Sign up Bright Data Enterprise targets + managed API Sign up Related guides Webshare Review 2026: The Best Free Proxy Tier? Tested and Honest 10 Best Residential Proxies 2026: Tested and Ranked Best Proxy for Web Scraping in 2026: Ranked by Target Difficulty Ready to try Bright Data? The industry's most reliable proxy network — free trial & pay-as-you-go. Try Bright Data → Related Proxy Reviews Bright Data Review 2026 IPRoyal Review 2026 Webshare Review 2026 10 Best Residential Proxies 2026 Frequently Asked Questions Can I scrape with a free proxy? Free public proxy lists work for basic testing but are not suitable for production scraping. They're unreliable, slow, and blocked by most sites. Webshare's free tier gives you 1 GB/month of legitimate datacenter bandwidth — the best free starting point. What's the difference between rotating and sticky sessions in Python? A rotating session gives you a new IP on every request — best for scraping the same domain at high frequency. A sticky session keeps the same IP for a configurable duration — best when the target site uses session cookies (e-commerce with cart state, logged-in scraping). In GeoNode and IPRoyal, you select the mode via the proxy port or username suffix. Do I need Playwright or can I use requests? Use requests or Scrapy for server-rendered targets (HTML is in the initial response). Use Playwright or Selenium when the content you need is loaded via JavaScript after page load — you'll see empty containers or placeholders in the requests response if JavaScript rendering is required. What's the fastest Python proxy library for high-volume scraping? aiohttp with async/await is faster than synchronous requests for high-concurrency scraping. Scrapy handles async internally. For most production scrapers, Scrapy with the right CONCURRENT_REQUESTS setting outperforms custom aiohttp implementations in terms of development speed vs. performance tradeoff. Related Articles Best Proxy for Web Scraping 2026 — Ranked by Target Difficulty GeoNode Review 2026: Tested at Production Scale Best Proxy Providers 2026 — Full Comparison Hub Cheapest Residential Proxies 2026