Skip to content

Scrape Substack Newsletters and Author Pages

Substack publications are public-facing, but collecting their post metadata, author details, and posting cadence by hand does not scale.

OUTPUT FORMATSHTMLJSONMarkdown

This page explains how an API-based scraper turns a Substack URL into structured JSON or rendered HTML in one request, which fields are available, how the credit system works, and where the practical limits are.

What a Substack scraper returns

A single request against a Substack post or publication page can return the headline, byline, post date, dek summary, section tags, author bio, and subscriber tier pricing. The response is either structured JSON or rendered HTML, depending on the output format you choose. The scraper handles page rendering, proxy rotation, and bot bypass on its side, so you do not need to run headless browsers or manage proxy pools yourself.

Three common jobs for Substack data

The table below maps each common workflow to the fields it depends on and the output mode that suits it best. Publication tracking watches how often a newsletter posts and how much of that output sits behind a paywall; it works from public publication-level signals only and never touches subscriber records. Author research profiles creators using their bio, post count, tier pricing, and category. Cadence analysis measures editorial velocity from publication dates, posting frequency, and the free-versus-paid split. All three draw on the same returned fields; what changes is which fields you keep and how often you refresh them.

Choosing an output mode

The scraper supports three practical output paths. Rendered HTML is the right choice when you need the full page markup and plan to parse it with your own selectors. Structured JSON via extraction rules lets you target specific elements with CSS or XPath and receive clean JSON back. AI-driven extraction via a natural-language specification returns structured JSON without writing selectors, which is useful when the page layout changes or when you are exploring a new publication. Markdown output is also available for feeding Substack posts into an LLM for summarization or research workflows.

Field and control map

The table below maps the main request options to their effect and credit cost. Use it to estimate the cost of a batch before you run it.

Credit costs for common option combinations

The base request costs 1 credit. Enabling JavaScript rendering raises the cost to 5 credits. Premium proxies cost 10 credits without JS rendering or 25 credits with it. Stealth proxy costs 75 credits. These figures come from the published pricing and FAQ on the original page and should be checked against current pricing before committing to a large batch.

Worked example: one publication URL

Suppose you send a request for a Substack publication page with JavaScript rendering enabled and an AI extraction rule asking for the publication name, author bio, and subscriber tier prices. The scraper renders the page, applies the rule, and returns a JSON object with those fields populated. If the page is behind a more aggressive bot check, you would add the stealth proxy option, which raises the credit cost per request. The same pattern works for individual post pages when you need headline, byline, date, and section tags.

Decision checklist before you deploy

Use this checklist to confirm the scraper fits your workflow. First, confirm the pages you need are public-facing and not behind a login. Second, decide whether you need raw HTML, selector-based JSON, or AI-extracted JSON. Third, estimate your monthly request volume and multiply by the credit cost of your chosen options. Fourth, check whether you need geotargeting or screenshots, since those add cost. Fifth, review the target site's terms of service before running at scale. Sixth, plan how you will store and refresh the data, since Substack pages change as new posts are published.

What the scraper does not do

The scraper returns public page data only. It does not access subscriber lists, private posts, or post-login content. It does not guarantee that every field will be present on every page — a publication without paid tiers will not return tier pricing, and a post without section tags will not return them. Data-handling and privacy commitments are set by the provider; confirm them with the operating company before relying on them for compliance purposes.

Getting started

Create an account to receive an API key and a starter credit allowance. Install the SDK for your stack — Python, Node, Go, PHP, or Ruby — or send a GET request directly with the target URL. The dashboard includes a request builder for testing without writing code. Code samples are available in Python, Node.js, Ruby, PHP, Go, and cURL.

Request options and credit costs
OptionEffectCredit cost
Base requestFetch a public Substack page1 credit
JavaScript renderingRender JS-heavy pages5 credits
Premium proxy (no JS)Route through premium proxies10 credits
Premium proxy + JSPremium proxies with rendering25 credits
Stealth proxyHigher-tolerance bot bypass75 credits
GeotargetingRoute through 200+ countries (40+ on classic proxies)Included in plan
Screenshots & extraction rulesFull-page screenshots and CSS/XPath extractionIncluded in plan

Credit costs are taken from the published FAQ and pricing on the original page. Check current pricing before committing to a large batch.

Workflow to field and output mode
WorkflowFields it depends onSuggested output mode
Publication trackingPost titles, paid-content frequency, author profile data (public signals only, no subscriber data)Selector-based JSON, refreshed on a schedule
Author researchAuthor bio, post count, subscriber tier pricing, publication categoryAI-extracted JSON when layouts vary across publications
Cadence analysisPublication dates, posting frequency, free-versus-paid splitSelector-based JSON with date fields normalized
LLM summarizationFull post body textMarkdown output

Field availability depends on what renders on the public page; a publication without paid tiers will not return tier pricing.

Frequently asked questions

Do I need a Substack account to scrape Substack?

No. You sign up for an API key with the scraper provider, send the Substack URL, and the provider handles rendering and proxy rotation. You do not need a Substack login.

Is scraping Substack legal?

Public-facing pages can usually be scraped, but you should review the target site's terms of service before deploying and avoid post-login content. Legal rules vary by jurisdiction and by site.

What data can I extract from Substack?

Anything that renders on the public page. You can use CSS selectors via extraction rules or a natural-language specification via AI extraction to get structured JSON back. Typical fields include headline, byline, post date, dek summary, section tags, author bio, and subscriber tier pricing.

How do I start scraping Substack?

Sign up, get your API key, and send a GET request with the target URL. The provider renders the page, rotates proxies, and returns the data. New accounts receive a starter credit allowance to test the service.

Will I get blocked when scraping Substack?

The provider rotates residential proxies and uses a real headless browser so requests resemble normal traffic. If a target becomes aggressive, enable the stealth proxy option on the request.

How much does it cost to scrape Substack?

New accounts receive a starter credit allowance with no card required. Paid plans are available for higher volume and concurrency. The credit cost per request depends on options: 5 credits for JavaScript rendering, 25 for premium proxy with JS, and 75 for stealth proxy.