Web Scraping with Node.js: Best Practices for 2026

Introduction

Node.js has become the default choice for web scraping teams that need JavaScript execution and modern async patterns. The Product Data Scrape engineering team has built and operated Node.js-based scrapers across 40+ marketplaces, processing 6 billion+ records last quarter. This guide shares the production patterns we use.

The Modern Node.js Scraping Stack

Async Patterns: The New Default

Node.js is inherently async, but naive Promise.all fires everything at once. Use concurrency limiting for anything past 10 URLs:

import axios from 'axios';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';

const scrapeUrl = async (url) => {
  try {
    const { data } = await axios.get(url, {
      timeout: 15000,
      headers: { 'User-Agent': 'Mozilla/5.0' }
    });
    const $ = cheerio.load(data);
    return parseProduct($);
  } catch (err) {
    return { url, error: err.message };
  }
};

const scrapeMany = async (urls) => {
  const limit = pLimit(10);  // 10 concurrent
  const results = await Promise.allSettled(
    urls.map(url => limit(() => scrapeUrl(url)))
  );
  return results.map(r => r.status === 'fulfilled' ? r.value : { error: r.reason });
};

// Usage
const data = await scrapeMany(['url1', 'url2', 'url3']);

Error Handling Classification

Production scrapers fail in dozens of ways. Catch and categorize errors so the system can respond appropriately — retries for timeouts and 5xx errors, longer waits for 429 rate-limit responses, immediate failures for 404s, and Sentry alerts for unexpected parsing exceptions. Axios's error.response.status and error.code (ECONNRESET, ETIMEDOUT) make this straightforward.

Rate Limiting With Jitter

Constant request intervals are a giveaway to anti-bot systems. Use exponential backoff with jitter — combine p-retry with random delays between attempts. For production, use bottleneck for token-bucket rate limiting. This is one of the core patterns the Product Data Scrape API uses internally.

Sample Data From Product Data Scrape Node.js API

When you use the Product Data Scrape Node.js SDK, results look like this:

import { PDS } from '@productdatascrape/sdk';

const pds = new PDS({ apiKey: process.env.PDS_KEY });
const result = await pds.scrape('https://walmart.com/ip/12345');

// result returns:
{
  "request_id": "req_abc123xyz",
  "status": "success",
  "credits_used": 1,
  "latency_ms": 487,
  "data": {
    "product_id": "WP-12345",
    "retailer": "walmart_us",
    "title": "Apple AirPods Pro (2nd Generation)",
    "brand": "Apple",
    "price": {"current": 199.99, "msrp": 249.00, "currency": "USD"},
    "rating": {"value": 4.7, "count": 8492},
    "availability": "in_stock",
    "scraped_at": "2026-05-15T14:22:00Z"
  }
}

How Product Data Scrape Helps

When you outgrow DIY Node.js scraping (typically around 50K+ SKUs/day or when Amazon/Cloudflare-protected targets enter the picture), the Product Data Scrape Node.js SDK handles all production concerns behind a single REST endpoint — TypeScript-first API, streams support for large datasets, Next.js API route integration, and automatic proxy rotation with residential IPs.

About Product Data Scrape

Product Data Scrape is the leading provider of managed web scraping services and ready-to-use product datasets. We help 200+ brands, retailers, and AI companies turn the messy public web into clean, structured product data.

Our Services: — Web Scraping API — REST API for developers (1,000 free credits) — Scraper as a Service — Custom scrapers built in 7-10 days — Ready Datasets — 100+ pre-built datasets, free 1,000-row samples in 24 hours

Contact: — Website: https://www.productdatascrape.com — Email: info@productdatascrape.com