📑 Table of Contents
Why Traditional Scraping Fails on JavaScript Sites
If you've ever tried scraping a modern website with Python's requests library only to get empty results, you've encountered the JavaScript rendering problem. Here's what happens:
import requests
from bs4 import BeautifulSoup
# This often returns empty or minimal content
response = requests.get('https://react-app-example.com')
soup = BeautifulSoup(response.text, 'html.parser')
# The products div exists but is empty!
products = soup.select('.product-list')
print(len(products)) # Output: 0The issue is that modern web applications built with React, Vue, Angular, or Next.js render content dynamically using JavaScript. When you fetch the page with a simple HTTP request, you only get the initial HTML shell—not the actual content that JavaScript creates.
The solution? Headless browsers that execute JavaScript just like a real browser.
What is a Headless Browser?
A headless browser is a web browser without a graphical user interface. It can:
- Load web pages and execute JavaScript
- Render CSS and build the DOM
- Handle AJAX requests and API calls
- Execute user interactions (clicks, form fills, scrolling)
- Take screenshots and generate PDFs
Most headless browsers also have a "headed" mode where you can watch the automation happen. This is invaluable for debugging your scraping scripts.
Playwright vs Puppeteer vs Selenium
Three tools dominate the headless browser landscape in 2026. Here's how they compare:
| Feature | Playwright | Puppeteer | Selenium |
|---|---|---|---|
| Browser Support | Chromium, Firefox, WebKit | Chromium only | All major browsers |
| Language Support | Python, JS, .NET, Java | JavaScript/TypeScript | Python, Java, C#, Ruby, JS |
| Auto-waiting | ✅ Excellent | ⚠️ Manual | ⚠️ Manual |
| Best For | Modern web scraping | Chrome automation | Cross-browser testing |
Our recommendation for 2026: Use Playwright for web scraping. Its auto-waiting features, cross-browser support, and excellent Python API make it the best choice for most scraping tasks.
Getting Started with Playwright
Installation
# Install Playwright
pip install playwright
# Download browser binaries
playwright installYour First Scraping Script
from playwright.sync_api import sync_playwright
import json
def scrape_products():
with sync_playwright() as p:
# Launch browser (headless=False to see it work)
browser = p.chromium.launch(headless=True)
page = browser.new_page()
# Navigate to the page
page.goto('https://example-store.com/products')
# Wait for products to load (Playwright auto-waits!)
page.wait_for_selector('.product-card')
# Extract product data
products = []
cards = page.query_selector_all('.product-card')
for card in cards:
product = {
'name': card.query_selector('.product-name').inner_text(),
'price': card.query_selector('.product-price').inner_text(),
'image': card.query_selector('img').get_attribute('src'),
'url': card.query_selector('a').get_attribute('href')
}
products.append(product)
browser.close()
return products
# Run the scraper
products = scrape_products()
print(json.dumps(products, indent=2))Waiting Strategies for Dynamic Content
The most common scraping failures come from not waiting properly for content to load:
# Wait up to 30 seconds for element
page.wait_for_selector('.product-list', timeout=30000)
# Wait for element to be visible (not just in DOM)
page.wait_for_selector('.product-list', state='visible')
# Wait for network to be idle (no requests for 500ms)
page.wait_for_load_state('networkidle')
# Wait for JavaScript condition
page.wait_for_function('window.dataLoaded === true')Handling Single Page Applications
Infinite Scroll
def scroll_and_collect(page, max_items=100):
items = []
previous_count = 0
while len(items) < max_items:
# Scroll to bottom
page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
# Wait for new content
page.wait_for_timeout(2000)
# Collect items
elements = page.query_selector_all('.item')
items = [el.inner_text() for el in elements]
# Check if we've reached the end
if len(items) == previous_count:
break
previous_count = len(items)
return items[:max_items]Performance Optimization Tips
Block Unnecessary Resources
def block_resources(route, request):
if request.resource_type in ['image', 'stylesheet', 'font', 'media']:
route.abort()
else:
route.continue_()
page.route('**/*', block_resources)Use Stealth Mode
context = browser.new_context(
viewport={'width': 1920, 'height': 1080},
user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
locale='en-US',
timezone_id='America/New_York'
)Common Challenges and Solutions
Challenge 1: Elements Not Found
Solution: Check if the element is in an iframe:
# Check for iframes
frames = page.frames
for frame in frames:
element = frame.query_selector('.target-element')
if element:
print('Found in frame:', frame.url)
breakChallenge 2: Bot Detection
Solution: Add realistic delays and human-like behavior:
import random
# Random delay between actions
page.wait_for_timeout(random.randint(1000, 3000))
# Simulate human-like mouse movement
page.mouse.move(100, 200)
page.wait_for_timeout(random.randint(100, 300))
page.click('.button')Crawlix handles headless browser infrastructure, anti-bot bypass, and proxy rotation automatically. Focus on your data, not infrastructure.
Skip the Infrastructure Complexity
Let Crawlix handle JavaScript rendering, anti-bot measures, and scaling. Get clean data delivered to your systems.
Request a Quote →