How to Handle Pagination in Web Scraping

Introduction

Modern e-commerce sites use one of four pagination patterns. The Product Data Scrape team encounters all of them daily across our 50+ supported marketplaces. This guide covers each pattern with working code.

The Four Pagination Patterns

The Four Pagination Patterns

Pattern 1: Numbered Pagination

Walmart, Best Buy, Target use numbered pagination:
async def scrape_walmart_category(category_slug):
    page = 1
    all_products = []
    while True:
        url = f"https://www.walmart.com/browse/{category_slug}?page={page}"
        response = await fetch(url)
        products = parse_products(response)
        if not products:
            break
        all_products.extend(products)
        page += 1
        if page > 1000:
            break
    return all_products

Pattern 4: Infinite Scroll (Hardest)

Flipkart and Myntra load more products as you scroll. Two approaches:

Approach A — Find the underlying API: Open DevTools → Network tab → scroll the page → look for XHR requests with the data.

Approach B — Headless browser scroll simulation: Use Playwright to simulate scrolling and wait for new content to load.

Sample Paginated Response from Product Data Scrape

{
  "query": "wireless headphones",
  "retailer": "walmart_us",
  "page": 1,
  "total_pages": 47,
  "total_results": 2823,
  "results_per_page": 60,
  "next_page_token": "eyJwYWdlIjoyfQ==",
  "products": [
    {
      "product_id": "5037874562",
      "title": "Sony WH-1000XM5 Wireless Headphones",
      "price": 348.00,
      "rating": 4.8
    }
  ],
  "scraped_at": "2026-04-12T09:15:00Z"
}

How Product Data Scrape Helps

Our API endpoints handle pagination automatically. Pass a category or search query and get all results as a single JSON response with pagination cursors.

About Product Data Scrape

Product Data Scrape is the leading provider of managed web scraping services and ready-to-use product datasets. We help 200+ brands, retailers, and AI companies turn the messy public web into clean, structured product data.

Our Services: - Web Scraping API — REST API for developers (1,000 free credits) - Scraper as a Service — Custom scrapers built in 7-10 days - Ready Datasets — 100+ pre-built datasets, free 1,000-row samples in 24 hours

Contact: - Website: https://www.productdatascrape.com - Email: sales@productdatascrape.com