Fix: Puppeteer Not Working — Navigation Timeout, Chrome Not Found, and Target Closed

The Error

You run your Puppeteer script and immediately hit a timeout:

TimeoutError: Navigation timeout of 30000 ms exceeded.

Or it works locally but fails in Docker or CI:

Error: Failed to launch the browser process!
/root/.cache/puppeteer/chrome/linux-xxx/chrome-linux64/chrome: error while loading shared libraries: libgbm.so.1: cannot open shared object file

Or a page operation crashes mid-run:

Error: Protocol error (Runtime.callFunctionOn): Target closed.

Or you run a scraper in a loop and Node’s memory climbs until the process OOMs an hour later — no error, just a silent crash.

Each of these is a distinct failure mode. None requires switching to a different library.

Why This Happens

Puppeteer controls a real Chrome browser via the Chrome DevTools Protocol. Chrome is a full browser process — it has dependencies, consumes memory, manages network connections, and enforces security policies. Most Puppeteer failures come from one of three sources: Chrome’s environment requirements not being met (common in Docker/CI), timing assumptions that don’t hold for dynamic pages, or resource leaks from pages and browsers not being closed after use.

Fix 1: Navigation Timeout — Adjust waitUntil and Increase Timeouts

The default navigation timeout is 30 seconds. If a page takes longer to load, or if you’re waiting for a SPA to finish its API calls, this fires prematurely.

Increase the timeout globally so you don’t have to set it on every call:

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();

page.setDefaultNavigationTimeout(60_000);  // 60 seconds for all navigation
page.setDefaultTimeout(30_000);            // 30 seconds for all other waits

Choose the right waitUntil option for your page type:

// Static HTML sites — wait for the load event (default)
await page.goto("https://example.com", { waitUntil: "load" });

// Most modern sites — wait until there are ≤2 network connections for 500ms
await page.goto("https://news-site.com", { waitUntil: "networkidle2" });

// SPAs that load all data via API calls — wait until zero network activity
await page.goto("https://react-app.com", { waitUntil: "networkidle0", timeout: 60_000 });

// Just parse the DOM, don't wait for images/scripts
await page.goto("https://example.com", { waitUntil: "domcontentloaded" });

networkidle2 is the most practical default for most sites — it waits until fewer than two connections are open for 500 ms, which handles analytics and ad network requests without timing out. networkidle0 is stricter and better for API-driven apps but will timeout on sites with persistent WebSocket connections.

For click-triggered navigation (clicking a link that loads a new page), use Promise.all to start waiting before the click:

// WRONG — the navigation may complete before waitForNavigation is registered
await page.click("a.next-page");
await page.waitForNavigation();

// CORRECT — register the wait before triggering navigation
const [response] = await Promise.all([
  page.waitForNavigation({ waitUntil: "networkidle2" }),
  page.click("a.next-page"),
]);
console.log("Navigated, status:", response.status());

Pro Tip: For pages that hang indefinitely (some login forms, some SPAs), the domcontentloaded event fires almost immediately even if JS hasn’t executed. Use it to get into the page fast, then waitForSelector for the specific element you need instead of relying on networkidle.

Fix 2: Failed to Launch Browser — Docker and CI Environments

Puppeteer downloads a Chrome binary to ~/.cache/puppeteer/. In Docker containers and CI runners, this binary often fails to start because shared libraries (libgbm, libnss3, libatk, etc.) aren’t present in minimal base images.

The two standard fixes:

Option 1: Use the official Puppeteer Docker image — pre-built with all Chrome dependencies:

FROM ghcr.io/puppeteer/puppeteer:latest

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["node", "index.js"]

Option 2: Install Chrome dependencies manually in a node:slim image:

FROM node:22-slim

RUN apt-get update && apt-get install -y \
    ca-certificates fonts-liberation libasound2 libatk-bridge2.0-0 \
    libatk1.0-0 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 \
    libgbm1 libglib2.0-0 libnss3 libpango-1.0-0 libx11-6 libx11-xcb1 \
    libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 \
    libxkbcommon0 libxrandr2 libxrender1 libxss1 libxtst6 \
    --no-install-recommends \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["node", "index.js"]

You must also pass --no-sandbox in Docker and most CI environments. Chrome’s sandbox requires kernel features (user namespaces) that aren’t available in most container runtimes:

const browser = await puppeteer.launch({
  headless: true,
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

Note: --no-sandbox reduces Chrome’s isolation. This is acceptable in containers where the container itself provides isolation, but don’t run untrusted content without other mitigations.

For CI/CD (GitHub Actions, GitLab CI), set the environment variable to skip redundant Chrome downloads if Chrome is already installed on the runner:

# GitHub Actions — use an action to set up Chrome
- uses: browser-actions/setup-chrome@v1
  
# Or set PUPPETEER_SKIP_CHROMIUM_DOWNLOAD if using system Chrome
- name: Run scraper
  env:
    PUPPETEER_EXECUTABLE_PATH: /usr/bin/google-chrome-stable
  run: node scraper.js

Then in your code:

const browser = await puppeteer.launch({
  executablePath: process.env.PUPPETEER_EXECUTABLE_PATH || undefined,
  headless: true,
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

When executablePath is undefined, Puppeteer uses its downloaded binary. When it’s set (in CI), it uses the system Chrome. The same script works in both environments.

For Docker service startup issues unrelated to Chrome, see Docker daemon not running.

Fix 3: Element Not Found — Selectors and Timing

TimeoutError: Waiting for selector ".data-table" failed: timeout 30000ms exceeded

This means the element didn’t appear in the DOM within the timeout. Either the selector is wrong, the element is hidden, or the content hasn’t loaded yet.

Always use waitForSelector before interacting — never assume an element exists after page.goto() returns:

// WRONG — .results may not be in the DOM yet
await page.goto("https://search-site.com?q=test");
const text = await page.$eval(".results", el => el.textContent);

// CORRECT
await page.goto("https://search-site.com?q=test");
await page.waitForSelector(".results", { visible: true, timeout: 15_000 });
const text = await page.$eval(".results", el => el.textContent);

visible: true waits for the element to be both in the DOM and not hidden. Without it, waitForSelector returns as soon as the element is in the DOM — even if it’s display: none.

Check if an element exists without throwing:

// page.$() returns null if not found (doesn't throw)
const cookieBanner = await page.$(".cookie-consent");
if (cookieBanner) {
  await cookieBanner.click();
}

// page.$$() returns empty array if not found
const items = await page.$$(".product-item");
console.log(`Found ${items.length} products`);

For elements inside iframes, you need the frame context:

// Get the iframe element
const frameElement = await page.waitForSelector("iframe#content");
// Get the frame's context
const frame = await frameElement.contentFrame();
// Now query inside the frame
const button = await frame.waitForSelector(".submit-btn");
await button.click();

Use page.locator() for more reliable interactions (Puppeteer v21+). Locators auto-wait for the element to be ready, visible, and not obscured before acting:

// Locator waits automatically — no separate waitForSelector needed
await page.locator(".submit-button").click();
await page.locator('input[name="email"]').fill("[email protected]");

// Wait for text content
await page.locator("text/Loading complete").wait();

Fix 4: Bot Detection — What Works and What Doesn’t

Puppeteer’s headless Chrome sets navigator.webdriver = true, which is detectable by any JavaScript running in the page. Modern bot protection (Cloudflare, Akamai, DataDome) goes further — it checks TLS fingerprints, browser API behavior, and timing patterns.

Basic mitigation with puppeteer-extra-plugin-stealth:

npm install puppeteer-extra puppeteer-extra-plugin-stealth

const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({
  headless: true,
  args: ["--disable-blink-features=AutomationControlled"],
});

const page = await browser.newPage();

// Set a realistic user agent
await page.setUserAgent(
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
);

await page.goto("https://example.com");

The stealth plugin patches navigator.webdriver, removes the HeadlessChrome user agent string, and spoofs several other automation markers. This defeats basic navigator.webdriver checks.

Manually hide webdriver for finer control:

const page = await browser.newPage();

await page.evaluateOnNewDocument(() => {
  Object.defineProperty(navigator, "webdriver", { get: () => false });
});

evaluateOnNewDocument runs before any page scripts, so this property is already patched when the site’s JS executes.

Be realistic about what this achieves. Stealth works against scripts that only check navigator.webdriver. Modern Cloudflare and enterprise bot detection systems fingerprint the TLS handshake (JA3/JA4), analyze mouse movement timing, check WebGL renderer strings, and score behavior over multiple requests. No Puppeteer plugin bypasses TLS-level fingerprinting — that requires a different approach (residential proxies, headful mode on a desktop browser, or purpose-built services).

Common Mistake: Spending time on stealth configurations when the site is simply returning a 403 due to IP-based blocking. Check the HTTP status code first:

const response = await page.goto("https://example.com");
console.log("Status:", response.status());
// 403 = IP blocked or firewall rule, not just bot detection
// 200 with "verify you're human" = bot challenge in page content

Fix 5: Memory Leaks — Always Close Pages and Browsers

Puppeteer doesn’t garbage-collect Chrome pages or browser processes. Each unclosed page object holds a reference to a Chrome tab. Each unclosed browser object holds the entire Chrome process in memory. In a scraper that processes thousands of URLs, this causes Node to run out of memory and crash without a clear error.

The correct pattern — always use try/finally:

const browser = await puppeteer.launch({ headless: true });

try {
  const page = await browser.newPage();
  try {
    await page.goto("https://example.com");
    // ... your automation ...
  } finally {
    await page.close();  // Always close the page
  }
} finally {
  await browser.close();  // Always close the browser
}

For scraping many URLs, reuse a single browser with one page at a time:

const browser = await puppeteer.launch({ headless: true });

const urls = [/* thousands of URLs */];

for (const url of urls) {
  const page = await browser.newPage();
  try {
    // Block images and CSS to reduce memory usage
    await page.setRequestInterception(true);
    page.on("request", (req) => {
      if (["image", "stylesheet", "font", "media"].includes(req.resourceType())) {
        req.abort();
      } else {
        req.continue();
      }
    });

    await page.goto(url, { waitUntil: "domcontentloaded", timeout: 15_000 });
    const data = await page.$eval("h1", el => el.textContent).catch(() => null);
    console.log(url, data);
  } catch (err) {
    console.error(`Failed: ${url}`, err.message);
  } finally {
    await page.close();  // Critical — close after every URL
  }
}

await browser.close();

Blocking resource types (images, fonts, CSS) reduces memory per page by 30–70% in typical scraping workloads. It’s not always appropriate (some data is image-based) but for text extraction it’s a significant win.

If memory still climbs over a long run, restart the browser periodically:

let browser = await puppeteer.launch({ headless: true });
let pageCount = 0;

for (const url of urls) {
  // Restart Chrome every 100 pages to reset accumulated memory
  if (pageCount > 0 && pageCount % 100 === 0) {
    await browser.close();
    browser = await puppeteer.launch({ headless: true });
  }

  const page = await browser.newPage();
  // ... process url ...
  await page.close();
  pageCount++;
}

await browser.close();

For Node.js heap exhaustion patterns, see JavaScript heap out of memory.

Fix 6: Target Closed / Page Crashed

Error: Protocol error (Runtime.callFunctionOn): Target closed.
Error: Page crashed!

Target closed means the Chrome tab or browser process terminated while your code was trying to communicate with it. This usually happens when: a page runs out of memory and Chrome kills it, you call a method on a page after closing it, or a previous unhandled error terminated the browser.

Detect and handle page errors:

const page = await browser.newPage();

// Listen for page crashes
page.on("error", (err) => {
  console.error("Page crashed:", err.message);
});

// Listen for uncaught exceptions in the page's JS
page.on("pageerror", (err) => {
  console.error("Uncaught JS error in page:", err.message);
});

Wrap every operation in a try/catch and check if the page is still open before retrying:

async function scrapePage(browser, url) {
  const page = await browser.newPage();
  try {
    await page.goto(url, { timeout: 30_000 });
    return await page.$eval(".content", el => el.textContent);
  } catch (err) {
    if (err.message.includes("Target closed") || err.message.includes("Page crashed")) {
      console.warn(`Page crashed on ${url}, skipping`);
      return null;
    }
    throw err;
  } finally {
    await page.close().catch(() => {});  // Ignore close errors if already closed
  }
}

Note .catch(() => {}) on page.close() — if the page already crashed, close() itself will throw. Swallowing that error is correct.

For long-running processes, listen for browser disconnection and relaunch:

let browser = await puppeteer.launch({ headless: true, args: ["--no-sandbox"] });

browser.on("disconnected", async () => {
  console.warn("Browser disconnected, relaunching...");
  browser = await puppeteer.launch({ headless: true, args: ["--no-sandbox"] });
});

Fix 7: Waiting for Dynamic Content

The root problem for most scraping failures is reading data before it’s in the DOM. page.goto() resolving doesn’t mean the data you want is there — SPAs, lazy-loaded lists, and AJAX-driven tables all populate after the initial load.

waitForSelector — wait for a specific element to appear:

await page.goto("https://shop.example.com/products");
await page.waitForSelector(".product-grid .item", { visible: true, timeout: 10_000 });
const items = await page.$$eval(".product-grid .item", els =>
  els.map(el => ({ name: el.querySelector("h3").textContent, price: el.querySelector(".price").textContent }))
);

waitForFunction — wait for any arbitrary condition in the browser:

// Wait until the loading spinner disappears
await page.waitForFunction(
  () => !document.querySelector(".spinner"),
  { timeout: 15_000 }
);

// Wait until a specific number of items are loaded
await page.waitForFunction(
  (count) => document.querySelectorAll(".item").length >= count,
  { timeout: 20_000 },
  50  // Argument passed to the function
);

waitForResponse — wait for a specific API call to complete:

// Start waiting before triggering the request
const dataPromise = page.waitForResponse(
  (res) => res.url().includes("/api/search") && res.status() === 200
);
await page.click("button.search");
const response = await dataPromise;
const data = await response.json();

This is the most reliable pattern for SPA data — you get the API response directly without having to parse it from the DOM.

page.evaluate() — run JavaScript in the page context to extract data:

// Extract structured data from the page's JavaScript state
const state = await page.evaluate(() => {
  // Access React/Redux/Vue internal state if exposed
  return window.__INITIAL_STATE__ || null;
});

// Use DOM APIs not available via Puppeteer's API
const allLinks = await page.evaluate(() =>
  Array.from(document.querySelectorAll("a[href]")).map(a => a.href)
);

Still Not Working?

Certificate Errors on Internal Sites

For HTTPS sites with self-signed certificates (staging environments, internal tools):

const browser = await puppeteer.launch({
  headless: true,
  args: [
    "--no-sandbox",
    "--ignore-certificate-errors",
    "--allow-running-insecure-content",
  ],
});

page.screenshot() Returns Blank Image

The page hasn’t finished rendering when you take the screenshot. Add a wait:

await page.goto(url, { waitUntil: "networkidle2" });
await page.waitForSelector(".main-content", { visible: true });
// Optional: wait for fonts and images
await new Promise(resolve => setTimeout(resolve, 500));
await page.screenshot({ path: "screenshot.png", fullPage: true });

Setting fullPage: true captures the entire scrollable page, not just the viewport.

Slowing Down for Rate-Limited Sites

// Add delay between requests
async function delay(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

for (const url of urls) {
  await page.goto(url, { waitUntil: "domcontentloaded" });
  // ... scrape ...
  await delay(1000 + Math.random() * 2000);  // 1-3 seconds between requests
}

Puppeteer vs Playwright

If you hit persistent bot detection issues or need cross-browser testing (Firefox, WebKit), Playwright not working covers the equivalent errors in Playwright, which has a different detection profile and built-in stealth features. For pure scraping with JS rendering, the two tools are largely interchangeable in capability.

Using puppeteer-core Instead of puppeteer

puppeteer downloads Chrome automatically. puppeteer-core doesn’t — you supply the executable path. Use puppeteer-core when you’re deploying to environments where Chrome is pre-installed and you don’t want the 300MB download in your deployment:

const puppeteer = require("puppeteer-core");

const browser = await puppeteer.launch({
  executablePath: "/usr/bin/google-chrome-stable",
  headless: true,
  args: ["--no-sandbox"],
});

Handling Unhandled Promise Rejections

Puppeteer operations are all async. An unhandled rejection from a timeout or target closed error terminates the Node process. Always catch at the top level:

process.on("unhandledRejection", (reason) => {
  console.error("Unhandled rejection:", reason);
  // Decide whether to exit or continue
});

For deeper patterns around unhandled rejections in Node.js async code, see JavaScript unhandled promise rejection.