Skip to content

Scrape New York Times Headlines and Article Text with Python

New York Times section pages are a practical starting point for news monitoring, trend research and text datasets. They list current stories with headlines, summaries and links, and each linked article page may expose body paragraphs. The work is less predictable than scraping a small blog: some stories sit behind a paywall, automated requests can be challenged, and parts of a page may depend on JavaScript. This guide walks through a Python workflow that collects section cards, follows each article link, extracts accessible body text, and writes the results to CSV and JSON. It also compares the official Developer API and browser automation so you can choose the lightest method that meets your needs.

Quick answer

Use BeautifulSoup to parse the HTML and a request handler such as {{OWNER_SERVICE_NAME}} to manage proxies and optional JavaScript rendering. Start from a section page, select the article cards, follow each card's link, then read paragraphs from the article body container. Save only records that contain body text. If you need structured metadata rather than full text, check the official Developer API first. If you need full browser behaviour, Playwright is an option, but it carries more setup and maintenance.

What you can do with collected NYT data

The paper's coverage spans politics, business, technology, health, culture, sports and opinion, so a section feed can support several kinds of analysis. Typical uses include watching how a topic's framing shifts over time, monitoring which stories a section is publishing, assembling text for classification or sentiment experiments, and comparing coverage across outlets. A section page gives you a compact index; following the links gives you the article-level detail needed for those tasks.

Three constraints to plan for

Paywalls. Some articles are subscriber-only. A scraper can only work with HTML that the server actually returns, so if the full text is not in the response, no parser can reconstruct it. Public and subscriber pages may also differ in structure.

Anti-bot defences. A high-traffic publisher is more likely to rate-limit or challenge automated requests than a small site. Proxy rotation and request routing reduce the infrastructure you have to build yourself.

JavaScript-rendered content. When a page builds or loads parts of its content in the browser, a plain HTTP request may not return what a visitor sees. In that case you need a rendered browser environment or a service that executes JavaScript.

Choose your method before you write code

Use this checklist to pick the lightest approach that fits the job:

  • Need only metadata such as dates, headlines and links? Try the official Developer API first; it returns structured JSON and avoids anti-bot concerns, but it does not provide full article text and has rate limits.
  • Need full body text from publicly accessible pages? Use an HTML parser plus a request handler that can rotate proxies and optionally render JavaScript.
  • Need to interact with the page, scroll lazy-loaded content or handle heavy client-side rendering? Use browser automation such as Playwright, accepting the extra installation, proxy and maintenance work.
  • Need consistent fields across many pages without writing selectors? Consider an AI extraction option, keeping in mind that it adds extraction cost on top of the request cost.

Set up the project

Create an isolated project folder and virtual environment so the tutorial dependencies do not mix with your global Python setup. The exact commands are supplied in the project setup module. After activating the environment, install the two packages used in the workflow: the {{OWNER_SERVICE_NAME}} client for sending requests and BeautifulSoup for parsing the returned HTML. The install command is supplied in the package install module.

Step 1: Import libraries and set the target section

Import the {{OWNER_SERVICE_NAME}} client, BeautifulSoup and urljoin. Set an API key placeholder and a section URL. The example uses the Opinion section because it lists current editorial articles and links to individual story pages. BeautifulSoup parses the HTML, urljoin turns relative links into absolute URLs, and the client sends each request. The import and configuration code is supplied in the configuration module.

Step 2: Fetch the section page

Define a function that accepts a maximum article count. Inside it, create the client, request the section URL and parse the response. The example request disables JavaScript rendering, blocks images, fonts and stylesheets to keep the request light, and sets a United States country code. The function and its error handling are supplied in the section fetch module.

If the page returns incomplete content or the request is blocked, switch on JavaScript rendering and a premium proxy. Those two parameter changes are supplied in the request fallback module.

Step 3: Loop through the article cards

Select each article element on the section page, then read a heading, a summary paragraph and a link from it. Skip cards that lack a title or a link. Convert the link to an absolute URL with urljoin. Prefer stable structural selectors such as the article tag over generated class names, because the frontend can change. The card loop is supplied in the card loop module.

Step 4: Request each article and extract the body

Inside the same loop, request the article URL with the same lightweight settings. If the request fails, print the status code and continue to the next card rather than stopping the whole run. The article request is supplied in the article request module.

Parse the article HTML and select paragraphs inside the article body container. Join the paragraph text with a blank-line separator so the saved text stays readable. If no body text is found, skip the article so empty records do not enter the output. The body extraction is supplied in the body extraction module.

The article body selector keeps navigation text, newsletter prompts and footer boilerplate out of the result. It is tied to the current layout, so check it when the site changes.

Step 5: Store and return the records

Append a dictionary for each article only after body text has been extracted. The dictionary holds four fields: title, summary, url and body. The summary falls back to an empty string when no summary paragraph exists. Stop once the number of collected articles reaches the maximum. The append and return code is supplied in the record assembly module, and a call that collects five articles is supplied in the call example module.

Step 6: Save the results

Add csv and json to the imports, then write two files: a CSV with a header row and a JSON file with indented, non-ASCII-preserving output. The imports are supplied in the save imports module and the file-writing block is supplied in the save block module. The resulting files contain the same four fields for every collected article.

Field map for the saved records

Each saved record has four fields. Use this map when you adapt the output for a database or analysis pipeline:

  • title: the card heading text, used as the article label.
  • summary: the card's summary paragraph, or an empty string when none is present; useful for previews and short-text features.
  • url: the absolute article URL built from the card link; useful as a stable identifier and for re-fetching.
  • body: the joined article paragraphs; the field that determines whether a record is kept at all.

Optional: AI-assisted extraction

Selector-based extraction is fast and easy to read, but it breaks when the page structure changes. An AI extraction option lets you describe the fields in plain language instead of writing selectors. The simplest form is a prompt that asks for a headline and a one-sentence summary and returns JSON. For a more predictable schema, define named extraction rules and encode them as JSON before sending. Both approaches add extraction credits on top of the normal request cost, so use them where the convenience outweighs the extra spend. The prompt example is supplied in the AI query module and the rules example is supplied in the AI rules module.

Alternative: the official Developer API

If you only need structured metadata, the official Developer API is the cleanest route. It covers Article Search, Top Stories, Most Popular, Archive and Books. It does not return full article text, so it is limited when the body is the goal. To use it, register at the developer portal, create an app, and enable the APIs you need. The documented limits are 5 requests per minute and 500 per day for most endpoints, with the Archive API raised to 2,000 per day. A minimal Article Search call is supplied in the article search module.

Trade-offs: no maintenance overhead, clean JSON and no anti-bot concerns, but no full text, non-commercial use only, rate limits and a required key registration.

Alternative: browser automation

When you need the full article body and the page depends on browser behaviour, Playwright or Selenium can load the page as a real browser would, scroll through lazy-loaded content and read text that a plain HTTP client misses. A minimal Playwright script is supplied in the Playwright module. The trade-off is maintenance: you manage browser installation, contexts, timeouts, proxies, retries and blocking yourself, and every request behaves like a browser session, which is heavier and more expensive at scale.

Trade-offs: full browser control, good for JavaScript-heavy pages and page interaction, but self-managed headless browsers, installation, proxy setup, higher operating cost and heavier deployment.

Putting it together

The workflow in this guide fetches a section page, extracts card data, follows each article link, keeps only records with body text, and writes CSV and JSON output. That structure is reusable for other sections or other publishers with similar card layouts. The main variables are the section URL, the article body selector and the request settings. When a run returns fewer articles than expected, check whether the response was blocked, whether the body selector still matches, or whether the article is paywall-limited.

Method comparison
MethodBest forMain limitation
HTML parser plus request handlerFull body text from accessible pagesSelectors need maintenance when layout changes
Official Developer APIStructured metadata and searchNo full article text; rate limits apply
Browser automationJavaScript-heavy pages and interactionHeavy setup and operating cost
AI extractionConsistent fields without selectorsExtra extraction credits per request

Method suitability is a planning comparison, not a measured benchmark.

Frequently asked questions

Can I scrape NYT without an API?

Yes. Python libraries such as Requests and BeautifulSoup can retrieve and parse pages. The catch is that you handle IP blocks and JavaScript rendering yourself, which takes time and ongoing attention.

How do I handle paywalls on NYT?

Full access to subscriber content generally requires an account. Scraping behind a paywall without permission is often a terms-of-service violation, so check the legal position before you build that into a workflow.

Is it legal to scrape the NYT?

Scraping publicly available articles for personal research is often a grey area. Follow the robots file, avoid personal data, and get legal advice before any commercial use.

Can a scraping API handle both static and dynamic NYT content?

A scraping API that executes JavaScript can process dynamically loaded content and also retrieve static HTML articles through its proxy network.

How do I tell a blocked request from a paywall-limited article?

A blocked request usually returns a non-200 status or a challenge page, so the script logs the status code and moves on. A paywall-limited article returns a normal page but without the full body, so the body selector finds nothing and the record is skipped. Checking the status code first separates the two cases.