How to capture background requests and responses in Puppeteer
When a page loads, much of its data arrives through background network calls rather than the initial HTML.
Puppeteer exposes those calls through events on the page object, so you can record both what the browser sent and what came back. This guide shows the listener setup, the exact event names, and the ordering rule that decides whether you capture everything or only the tail end of the traffic.
The two events you need
Puppeteer emits a request event each time the page issues a network request and a response event when a reply arrives. You subscribe to both with the page.on() function, passing the event name and a callback that receives the request or response object. Registering both listeners gives you a paired view: the request side tells you the URL, method and headers that went out, and the response side tells you the status and payload that came back.
Order matters: attach listeners before navigation
The listeners must exist before the page starts loading. In the example below, both page.on() calls are registered first, and only then does page.goto run. If you navigate first and attach listeners afterwards, requests fired during the initial load are already gone and your arrays will look emptier than the real traffic. Treat "attach listeners, then navigate" as a fixed sequence rather than a stylistic choice.
Working JavaScript example
The sample launches a headless browser, opens a new page, creates two arrays, and pushes each request and response into the matching array as events fire. After navigation it closes the browser and logs both arrays. Replace the placeholder URL with the page you want to inspect. The complete code is provided as an exact protected module alongside this page.
From raw objects to usable records
The arrays hold live Puppeteer objects, not plain data. Before you store or serialize them, pull out the fields you actually need. A short checklist keeps the output consistent:
- Request side: URL, HTTP method, resource type, and request headers.
- Response side: URL, status code, and response headers.
- Timing: record a timestamp in the callback so you can order events later.
- Matching: pair a response to its request by URL when you need both halves.
Collecting only these fields keeps memory bounded on pages that fire hundreds of calls, and it makes the result easy to write to JSON or a log.
Telling the two failure modes apart
An empty array usually means one of two things, and they need different fixes. If no listener was ever attached, the arrays stay empty no matter how much traffic the page generates. If a listener was attached but after navigation began, you capture only the requests that fired after that point, which often looks like a partial list dominated by late-loading assets. Check the position of your page.on() calls relative to page.goto before assuming the page made no background calls.
Related Puppeteer questions
If you are working through Puppeteer tasks, these companion questions cover nearby ground: loading local files, running Puppeteer in Jupyter notebooks, waiting for a page to load, what Puppeteer is, Playwright versus Puppeteer, who owns Puppeteer, why Puppeteer is needed, finding elements by CSS selector, finding elements by XPath, saving and loading cookies, and taking a screenshot.
Continue learning
For a broader walkthrough of browser-driven scraping, the linked JavaScript web scraping tutorial covers Puppeteer as a headless browser option and puts request capture in the context of a full scraping workflow.
Frequently asked questions
Why are my request and response arrays empty?
Most often the listeners were attached after navigation started, or not attached at all. Register page.on('request') and page.on('response') before calling page.goto so the callbacks are in place when the first network calls fire.
Do I need both the request and the response listener?
Not strictly, but they answer different questions. The request listener records what the page sent; the response listener records what came back, including status codes. Using both lets you pair the two sides of each call.
Can I store the request and response objects directly?
You can push them into arrays as the example does, but they are live objects. For logging or serialization, extract the specific fields you need, such as URL, method, status and headers, rather than keeping everything.