Tutorial January 12, 2026 18 min read

Web Scraping with Python Scrapy: From Zero to Production

Learn how to build scalable web crawlers using Scrapy, Python's most powerful scraping framework. From your first spider to production deployment, this guide covers everything you need to know.

Table of Contents

  1. What is Scrapy?
  2. Installation & Project Setup
  3. Understanding Scrapy Architecture
  4. Writing Your First Spider
  5. Selectors & Data Extraction
  6. Item Pipelines
  7. Middleware & Extensions
  8. Deployment Strategies

What is Scrapy?

Scrapy is an open-source Python framework designed for extracting data from websites at scale. Unlike simple HTTP libraries like Requests or parsing tools like Beautiful Soup, Scrapy provides a complete framework for building and managing web crawlers.

Key features that make Scrapy the go-to choice for professional web scraping:

When to Use Scrapy

Choose Scrapy for projects that crawl multiple pages, require data processing pipelines, or need to run in production. For simple one-page extractions, Requests + Beautiful Soup may be sufficient.

Installation & Project Setup

Installing Scrapy

Install Scrapy using pip. We recommend using a virtual environment:

Bash 
# Create and activate virtual environment
python -m venv scraping-env
source scraping-env/bin/activate  # On Windows: scraping-env\Scripts\activate

# Install Scrapy
pip install scrapy

# Verify installation
scrapy version

Creating a New Project

Scrapy provides a command to scaffold a new project with the recommended structure:

Bash 
# Create a new project
scrapy startproject ecommerce_scraper

# Navigate to the project
cd ecommerce_scraper

# Project structure created:
ecommerce_scraper/
    scrapy.cfg              # Deploy configuration
    ecommerce_scraper/
        __init__.py
        items.py            # Item definitions
        middlewares.py      # Custom middleware
        pipelines.py        # Item pipelines
        settings.py         # Project settings
        spiders/            # Spider directory
            __init__.py

Understanding Scrapy Architecture

Scrapy follows a well-defined data flow architecture. Understanding this flow is key to building effective scrapers:

Core Components

Data Flow

  1. Spider generates initial requests
  2. Scheduler queues and prioritizes requests
  3. Downloader fetches pages (through downloader middleware)
  4. Response goes back to Spider (through spider middleware)
  5. Spider parses response, yields Items or new Requests
  6. Items go through Item Pipeline for processing
  7. New Requests go back to Scheduler

Writing Your First Spider

Spiders are classes that define how to crawl a site and extract data. Let's build one step by step:

Basic Spider Structure

Python 
# spiders/products_spider.py
import scrapy

class ProductsSpider(scrapy.Spider):
    # Unique identifier for this spider
    name = 'products'

    # Domains this spider is allowed to crawl
    allowed_domains = ['example-store.com']

    # Starting URLs
    start_urls = ['https://example-store.com/products']

    def parse(self, response):
        """Parse the product listing page."""
        # Extract product links
        for product_link in response.css('.product-card a::attr(href)').getall():
            yield response.follow(product_link, callback=self.parse_product)

        # Follow pagination
        next_page = response.css('.pagination .next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_product(self, response):
        """Parse individual product pages."""
        yield {
            'name': response.css('h1.product-title::text').get(),
            'price': response.css('.price::text').get(),
            'description': response.css('.description::text').get(),
            'sku': response.css('.sku::text').get(),
            'url': response.url,
        }

Running Your Spider

Bash 
# Run spider and output to JSON
scrapy crawl products -o products.json

# Run with specific settings
scrapy crawl products -o products.csv -s LOG_LEVEL=INFO

# Run interactively for testing
scrapy shell 'https://example-store.com/products'

Selectors & Data Extraction

Scrapy supports both CSS selectors and XPath for data extraction. Both are powerful; CSS is often more readable, while XPath offers more features.

CSS Selectors

Python 
# Get single value
title = response.css('h1::text').get()

# Get all values
prices = response.css('.price::text').getall()

# Get attribute
image_url = response.css('img.product::attr(src)').get()

# Nested selection
for product in response.css('.product-card'):
    name = product.css('.name::text').get()
    price = product.css('.price::text').get()

# Combining selectors
response.css('.product-card .name::text, .item .title::text').getall()

XPath Selectors

Python 
# Text content
title = response.xpath('//h1/text()').get()

# Attribute
image_url = response.xpath('//img[@class="product"]/@src').get()

# Contains text
button = response.xpath('//button[contains(text(), "Add to Cart")]')

# Following sibling
price = response.xpath('//label[text()="Price:"]/following-sibling::span/text()').get()

# Select by position
first_product = response.xpath('(//div[@class="product"])[1]')

# Conditional selection
in_stock = response.xpath('//div[@class="stock" and contains(text(), "In Stock")]')
Debugging Selectors

Use scrapy shell to test selectors interactively. It loads a response and lets you experiment with different selectors in real-time.

Item Pipelines

Item Pipelines process items after they're extracted. Common uses include validation, cleaning, deduplication, and storage.

Defining Items

Python 
# items.py
import scrapy
from itemloaders.processors import TakeFirst, MapCompose, Join
from w3lib.html import remove_tags

def clean_price(value):
    """Extract numeric price from string."""
    import re
    match = re.search(r'[\d.]+', value)
    return float(match.group()) if match else None

class ProductItem(scrapy.Item):
    name = scrapy.Field(
        input_processor=MapCompose(str.strip),
        output_processor=TakeFirst()
    )
    price = scrapy.Field(
        input_processor=MapCompose(clean_price),
        output_processor=TakeFirst()
    )
    description = scrapy.Field(
        input_processor=MapCompose(remove_tags, str.strip),
        output_processor=Join(' ')
    )
    sku = scrapy.Field(output_processor=TakeFirst())
    url = scrapy.Field(output_processor=TakeFirst())

Creating Pipelines

Python 
# pipelines.py
from itemadapter import ItemAdapter
import json

class ValidationPipeline:
    """Validate items and drop invalid ones."""

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        # Required fields
        if not adapter.get('name'):
            raise DropItem(f"Missing name in {item}")

        if not adapter.get('price') or adapter['price'] <= 0:
            raise DropItem(f"Invalid price in {item}")

        return item


class DeduplicationPipeline:
    """Remove duplicate items based on SKU."""

    def __init__(self):
        self.seen_skus = set()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        sku = adapter.get('sku')

        if sku in self.seen_skus:
            raise DropItem(f"Duplicate item: {sku}")

        self.seen_skus.add(sku)
        return item


class JsonWriterPipeline:
    """Write items to a JSON Lines file."""

    def open_spider(self, spider):
        self.file = open('products.jsonl', 'w')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(ItemAdapter(item).asdict()) + "\n"
        self.file.write(line)
        return item

Enabling Pipelines

Python 
# settings.py
ITEM_PIPELINES = {
    'ecommerce_scraper.pipelines.ValidationPipeline': 100,
    'ecommerce_scraper.pipelines.DeduplicationPipeline': 200,
    'ecommerce_scraper.pipelines.JsonWriterPipeline': 300,
}

# Lower numbers = higher priority (runs first)

Middleware & Extensions

Middleware lets you customize how Scrapy handles requests and responses. Common use cases include adding headers, rotating proxies, and handling retries.

Custom User-Agent Middleware

Python 
# middlewares.py
import random

class RandomUserAgentMiddleware:
    """Rotate User-Agent for each request."""

    USER_AGENTS = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36',
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101',
    ]

    def process_request(self, request, spider):
        request.headers['User-Agent'] = random.choice(self.USER_AGENTS)
        return None

Proxy Rotation Middleware

Python 
class ProxyMiddleware:
    """Rotate proxies for each request."""

    def __init__(self, proxy_list):
        self.proxies = proxy_list
        self.proxy_index = 0

    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.getlist('PROXY_LIST')
        return cls(proxy_list)

    def process_request(self, request, spider):
        if self.proxies:
            proxy = self.proxies[self.proxy_index % len(self.proxies)]
            request.meta['proxy'] = proxy
            self.proxy_index += 1

    def process_exception(self, request, exception, spider):
        # Remove failed proxy from rotation
        proxy = request.meta.get('proxy')
        if proxy in self.proxies:
            self.proxies.remove(proxy)
            spider.logger.warning(f"Removed failed proxy: {proxy}")

Enable Middleware in Settings

Python 
# settings.py
DOWNLOADER_MIDDLEWARES = {
    'ecommerce_scraper.middlewares.RandomUserAgentMiddleware': 400,
    'ecommerce_scraper.middlewares.ProxyMiddleware': 410,
}

PROXY_LIST = [
    'http://proxy1.example.com:8080',
    'http://proxy2.example.com:8080',
]

Deployment Strategies

Moving from development to production requires proper deployment infrastructure. Here are the main approaches:

1. Scrapyd (Self-Hosted)

Scrapyd is a daemon for running Scrapy spiders. It provides an HTTP API to deploy and schedule spiders:

Bash 
# Install scrapyd and client
pip install scrapyd scrapyd-client

# Start scrapyd daemon
scrapyd

# Deploy your project
scrapyd-deploy default -p ecommerce_scraper

# Schedule a spider via API
curl http://localhost:6800/schedule.json \
    -d project=ecommerce_scraper \
    -d spider=products

2. Docker Deployment

Dockerfile 
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy project
COPY . .

# Run spider
CMD ["scrapy", "crawl", "products", "-o", "/data/products.json"]

3. Cloud Platforms

For managed infrastructure, consider these options:

Production Settings

Python 
# settings.py - Production configuration

# Crawl responsibly
ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 1
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8

# Auto-throttling
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

# Caching (useful for development)
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 86400
HTTPCACHE_DIR = 'httpcache'

# Retry configuration
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

# Logging
LOG_LEVEL = 'INFO'
LOG_FILE = 'scrapy.log'
Rate Limiting

Always implement rate limiting in production. Aggressive scraping can get your IP banned and may impact the target website's performance. Use AUTOTHROTTLE to automatically adjust request rates.

Skip the Infrastructure Hassle

Building and maintaining scraping infrastructure is complex. Crawlix handles proxies, scaling, and reliability so you can focus on the data you need.

Get Started with Crawlix →

Related Articles

The Complete Guide to Web Scraping in 2026

Comprehensive guide covering all web scraping techniques and tools.

Building Robust Web Scrapers

Error handling, retries, and monitoring for production scrapers.

Scraping JavaScript-Heavy Websites

Learn how to extract data from React, Vue, and Angular apps.