Table of Contents
Why Robustness Matters
A web scraper that works in development often fails in production. Websites change, networks are unreliable, and servers implement rate limiting. The difference between a hobby project and a production-grade scraper lies in how it handles these inevitable failures.
Building robust scrapers matters for several reasons:
- Data completeness — Missing data due to transient errors can skew analysis and business decisions
- Operational efficiency — Manual intervention for every failure doesn't scale
- Cost optimization — Unnecessary retries and wasted compute add up quickly
- Reputation — Aggressive scrapers that don't handle rate limits get blocked permanently
Key Principle
Design for failure, not success. Assume every request can fail and build systems that degrade gracefully rather than crash completely.
Error Classification
Not all errors are created equal. Effective error handling starts with classifying errors into categories that determine the appropriate response:
Recoverable Errors (Retry)
- Network timeouts — Connection or read timeouts
- DNS resolution failures — Temporary DNS issues
- HTTP 429 — Rate limiting (respect Retry-After header)
- HTTP 500-503 — Server errors, usually temporary
- Connection refused — Server temporarily unavailable
Non-Recoverable Errors (Don't Retry)
- HTTP 404 — Page doesn't exist
- HTTP 403 — Access forbidden (unless using proxy rotation)
- HTTP 410 — Resource permanently removed
- Parse errors — HTML structure changed, needs code update
- Validation errors — Extracted data fails validation
Python
class ErrorClassifier:
RECOVERABLE_STATUS_CODES = {408, 429, 500, 502, 503, 504}
NON_RECOVERABLE_STATUS_CODES = {400, 401, 403, 404, 410, 422}
@classmethod
def is_recoverable(cls, error):
if isinstance(error, requests.exceptions.Timeout):
return True
if isinstance(error, requests.exceptions.ConnectionError):
return True
if isinstance(error, requests.exceptions.HTTPError):
status_code = error.response.status_code
return status_code in cls.RECOVERABLE_STATUS_CODES
return False
@classmethod
def should_rotate_proxy(cls, error):
"""Determine if we should try a different proxy."""
if isinstance(error, requests.exceptions.HTTPError):
return error.response.status_code in {403, 429}
return FalseRetry Strategies & Exponential Backoff
Exponential backoff is the gold standard for retry strategies. Instead of hammering a server that's struggling, you progressively wait longer between attempts, giving it time to recover.
Basic Exponential Backoff
Python
import time
import random
from functools import wraps
def retry_with_backoff(max_retries=3, base_delay=1, max_delay=60):
"""Decorator for retrying functions with exponential backoff."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
retries = 0
while retries < max_retries:
try:
return func(*args, **kwargs)
except Exception as e:
if not ErrorClassifier.is_recoverable(e):
raise # Don't retry non-recoverable errors
retries += 1
if retries >= max_retries:
raise
# Calculate delay with jitter
delay = min(base_delay * (2 ** retries), max_delay)
jitter = random.uniform(0, delay * 0.1)
sleep_time = delay + jitter
print(f"Retry {retries}/{max_retries} after {sleep_time:.2f}s")
time.sleep(sleep_time)
return None
return wrapper
return decorator
@retry_with_backoff(max_retries=5, base_delay=2)
def fetch_page(url):
response = requests.get(url, timeout=30)
response.raise_for_status()
return response.textRespecting Retry-After Headers
When a server sends a Retry-After header, it's telling you exactly when to try again. Always respect this:
Python
def get_retry_delay(response, default_delay):
"""Extract retry delay from response headers."""
retry_after = response.headers.get('Retry-After')
if retry_after is None:
return default_delay
# Retry-After can be seconds or a date
try:
return int(retry_after)
except ValueError:
# Parse as HTTP date
from email.utils import parsedate_to_datetime
retry_date = parsedate_to_datetime(retry_after)
return max(0, (retry_date - datetime.now()).total_seconds())
return default_delayCommon Mistake
Adding jitter (random variation) to retry delays is critical. Without it, multiple scrapers that fail at the same time will all retry simultaneously, creating a "thundering herd" that overwhelms the server again.
Circuit Breaker Pattern
The circuit breaker pattern prevents your scraper from repeatedly hitting a failing service. Like an electrical circuit breaker, it "trips" after too many failures, giving the target time to recover.
Python
from datetime import datetime, timedelta
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Failing, reject requests
HALF_OPEN = "half_open" # Testing if recovered
class CircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=60):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.failures = 0
self.last_failure_time = None
self.state = CircuitState.CLOSED
def can_execute(self):
if self.state == CircuitState.CLOSED:
return True
if self.state == CircuitState.OPEN:
# Check if recovery timeout has passed
if datetime.now() - self.last_failure_time > timedelta(seconds=self.recovery_timeout):
self.state = CircuitState.HALF_OPEN
return True
return False
# HALF_OPEN: allow one request to test
return True
def record_success(self):
self.failures = 0
self.state = CircuitState.CLOSED
def record_failure(self):
self.failures += 1
self.last_failure_time = datetime.now()
if self.failures >= self.failure_threshold:
self.state = CircuitState.OPEN
print(f"Circuit breaker OPEN after {self.failures} failures")
# Usage per domain
circuit_breakers = {}
def get_circuit_breaker(domain):
if domain not in circuit_breakers:
circuit_breakers[domain] = CircuitBreaker()
return circuit_breakers[domain]Logging Best Practices
Good logging is your debugging lifeline. Structure your logs for both human readability and machine parsing:
Structured Logging Setup
Python
import logging
import json
from datetime import datetime
class StructuredLogger:
def __init__(self, name):
self.logger = logging.getLogger(name)
self.logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(message)s'))
self.logger.addHandler(handler)
def _log(self, level, event, **kwargs):
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"level": level,
"event": event,
**kwargs
}
self.logger.log(getattr(logging, level), json.dumps(log_entry))
def info(self, event, **kwargs):
self._log("INFO", event, **kwargs)
def error(self, event, **kwargs):
self._log("ERROR", event, **kwargs)
def warn(self, event, **kwargs):
self._log("WARNING", event, **kwargs)
# Usage
log = StructuredLogger("scraper")
log.info("page_fetched",
url="https://example.com/page/1",
status_code=200,
response_time_ms=245,
items_found=25
)
log.error("fetch_failed",
url="https://example.com/page/2",
error_type="timeout",
retry_count=3
)What to Log
- Request metadata — URL, method, headers (sanitized)
- Response details — Status code, response time, content size
- Extraction results — Items found, validation status
- Errors — Exception type, message, stack trace, retry count
- Performance metrics — Requests per second, queue depth
Monitoring & Alerting
Monitoring gives you visibility into scraper health. Set up dashboards and alerts to catch problems before they become critical.
Key Metrics to Track
Python
from prometheus_client import Counter, Histogram, Gauge
# Define metrics
requests_total = Counter(
'scraper_requests_total',
'Total requests made',
['domain', 'status']
)
request_duration = Histogram(
'scraper_request_duration_seconds',
'Request duration in seconds',
['domain']
)
items_scraped = Counter(
'scraper_items_scraped_total',
'Total items scraped',
['domain', 'item_type']
)
active_tasks = Gauge(
'scraper_active_tasks',
'Currently running scrape tasks'
)
error_rate = Gauge(
'scraper_error_rate',
'Current error rate percentage',
['domain']
)
# Usage in scraper
def scrape_page(url):
domain = urlparse(url).netloc
with request_duration.labels(domain=domain).time():
try:
response = fetch_page(url)
requests_total.labels(domain=domain, status='success').inc()
items = parse_items(response)
items_scraped.labels(domain=domain, item_type='product').inc(len(items))
return items
except Exception as e:
requests_total.labels(domain=domain, status='error').inc()
raiseAlert Conditions
Set up alerts for these conditions:
- High error rate — More than 10% failures in 5 minutes
- Low throughput — Items scraped drops below threshold
- Circuit breaker open — Domain becoming unavailable
- Queue backlog — Pending tasks exceeding capacity
- Data anomalies — Sudden drop in expected data fields
Data Validation & Quality Checks
Scraping data that's incorrect is often worse than scraping no data at all. Implement validation at multiple levels:
Python
from dataclasses import dataclass
from typing import Optional
import re
@dataclass
class Product:
name: str
price: float
url: str
sku: Optional[str] = None
def validate(self):
errors = []
if not self.name or len(self.name) < 2:
errors.append("Invalid product name")
if self.price <= 0 or self.price > 1000000:
errors.append(f"Suspicious price: {self.price}")
if not self.url.startswith('http'):
errors.append("Invalid URL format")
if self.sku and not re.match(r'^[A-Z0-9-]+$', self.sku):
errors.append(f"Invalid SKU format: {self.sku}")
return errors
class DataQualityChecker:
def __init__(self):
self.stats = {
'total': 0,
'valid': 0,
'invalid': 0,
'error_types': {}
}
def check(self, item: Product):
self.stats['total'] += 1
errors = item.validate()
if errors:
self.stats['invalid'] += 1
for error in errors:
self.stats['error_types'][error] = \
self.stats['error_types'].get(error, 0) + 1
return False
self.stats['valid'] += 1
return True
def get_quality_score(self):
if self.stats['total'] == 0:
return 1.0
return self.stats['valid'] / self.stats['total']Data Quality Tip
Track quality scores over time. A sudden drop in quality score often indicates that the target website's structure has changed, requiring scraper updates.
Production Readiness Checklist
Before deploying a scraper to production, verify these requirements:
Error Handling
- All network errors are caught and classified
- Retry logic with exponential backoff implemented
- Circuit breakers configured per domain
- Graceful degradation when services fail
Observability
- Structured logging with correlation IDs
- Metrics exported to monitoring system
- Alerts configured for critical conditions
- Dashboard for real-time visibility
Data Quality
- Input validation on all extracted data
- Schema validation before storage
- Anomaly detection for data patterns
- Duplicate detection mechanism
Operational
- Configuration externalized (not hardcoded)
- Secrets managed securely
- Resource limits defined (memory, CPU)
- Graceful shutdown handling
- Health check endpoints
Let Us Handle the Complexity
Building and maintaining production-grade scrapers requires significant engineering effort. Crawlix provides enterprise-ready scraping infrastructure with built-in reliability.
Talk to Our Team →