How to Scrape a Website With Python Without Getting Blocked

Almost every “why am I getting blocked?” question comes down to the same thing: your scraper looks like a scraper. Real browsers send certain headers, wait between clicks, and come from residential IPs. Bots blast requests from a datacenter with a Python user-agent. Close that gap and most sites stop caring.

Here’s the order I actually work through, cheapest fix first.

1. Send real headers

The number-one giveaway is a missing or default user-agent. Send a real one, plus the headers a browser would:

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/125.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

r = requests.get("https://example.com", headers=headers, timeout=15)
print(r.status_code)

This alone fixes a surprising number of “403” problems.

2. Slow down and jitter your requests

Humans don’t fire 50 requests a second. Add a delay, and randomize it so the pattern isn’t robotic:

import time, random

for url in urls:
    scrape(url)
    time.sleep(random.uniform(1.5, 4.0))  # be a good guest

Rate-limiting isn’t just politeness — it’s the single best way to keep your IPs alive.

3. Rotate IPs with proxies

Once a site rate-limits per IP, no amount of waiting on a single IP will help. You need to spread requests across many residential IPs. Wiring a rotating proxy into requests is one line:

proxies = {
    "http": "http://user:pass@gate.provider.com:7000",
    "https": "http://user:pass@gate.provider.com:7000",
}
r = requests.get(url, headers=headers, proxies=proxies, timeout=20)

For which provider to use, I broke down the real options in Best Residential Proxies for Web Scraping — for a first project, IPRoyal’s pay-as-you-go is the easiest on-ramp, and Decodo is the best all-round upgrade.

4. Reach for a headless browser only when you must

If the data is rendered by JavaScript, requests sees an empty shell. That’s when you use a headless browser like Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(user_agent=headers["User-Agent"])
    page.goto(url, wait_until="networkidle")
    html = page.content()
    browser.close()

But headless browsers are slow and expensive — they use far more bandwidth and CPU. Before you reach for one, open your browser’s Network tab and look for the underlying JSON/API call. Nine times out of ten the site is fetching data from an endpoint you can hit directly with requests — faster, cheaper, and easier to keep unblocked.

5. Handle blocks gracefully

You will still get blocked sometimes. Detect it, back off, and retry with a fresh IP instead of crashing:

def get(url, tries=3):
    for i in range(tries):
        r = requests.get(url, headers=headers, proxies=proxies, timeout=20)
        if r.status_code == 200:
            return r
        time.sleep(2 ** i)  # exponential backoff
    return None

The stack that survives production

Put together, a scraper that doesn’t get blocked looks like: real headers + randomized delays + rotating residential proxies + the right tool (requests over headless whenever possible) + graceful retries. Nail those five and you’ll get through the vast majority of sites.

The part people underestimate is everything after it works once: keeping it running when the site changes its markup, landing clean deduplicated data somewhere useful, and not waking up to a broken pipeline. That’s the real job.

---

If you’d rather have a scraper built, hardened, and maintained so you never think about blocks again, send us the details — it’s what we do all day.