Why I Still Reach for Puppeteer in 2026

Every few months someone tells me headless browsers are overkill for scraping and I should just hit the API or parse the raw HTML. Sometimes they are right. But a fair amount of the automated reporting I have built over the years, Python scripts pulling data on a schedule and dropping a CSV into Slack, would not have worked without Puppeteer, and the reasons keep repeating.

The sites that break a simple HTTP request

A plain HTTP request gets you the HTML the server sent before any JavaScript ran. For a static page, that is the whole page. For a lot of internal dashboards, partner portals, and third-party sites I have needed to pull data from, it is an empty shell with a loading spinner, because the actual content gets rendered client-side after a few API calls the browser makes on its own.

You can sometimes reverse-engineer those API calls directly, and when that is stable and well-behaved, it is the better option, fewer moving parts, faster, lighter. But plenty of sites do not expose anything close to a clean API: authenticated sessions tied to cookies set through a login flow with redirects, data that only appears after a user interaction like selecting a filter or scrolling to trigger a lazy load, or endpoints that change their shape often enough that a raw request script breaks weekly. A real browser handles all of that the same way a human would, because it is running the same code the site ships to a human.

What this looked like in practice

The recurring pattern was: log in through the actual login form (so session handling, CSRF tokens, and redirects are someone else’s problem), wait for the specific element that means the data has actually loaded rather than a fixed sleep, extract the data, and write it to a CSV that lands in a Slack channel on a schedule via cron. Nothing about this is glamorous, but it replaced someone manually logging into three or four systems every morning and copying numbers into a spreadsheet.

The honest tradeoffs

Puppeteer is heavier than an HTTP client. It needs a real (or headless) Chromium instance, more memory, more startup time, and it breaks in more interesting ways: a site changes a CSS selector and your script silently returns nothing instead of erroring clearly, or a login flow adds a new consent screen and the whole pipeline stalls. I have learned to treat every selector as something to assert against explicitly, so a broken script fails loudly the same morning instead of quietly reporting stale numbers for a week.

The rule I use now: reach for a direct API call or an HTTP request first, always. Reach for Puppeteer only once that path is genuinely closed off, not because it is the more interesting tool to use. It has earned its place in the toolbox for exactly those cases, and there are still enough of them that I do not expect to stop using it any time soon.