Skip to content

Claude Scraper API for Public Shared Chats

Public Claude shared-chat pages carry a lot of usable signal: how people phrase prompts, which model variants appear, what answers get shared, and which sources the model cites.

OUTPUT FORMATSHTMLJSONMarkdown

This page covers how a hosted scraping API returns that data as structured JSON or rendered HTML, what request parameters control the result, and how the plan tiers compare.

What the Claude Scraper API returns

Send a URL from Claude's public shared-chat pages to the endpoint. The service renders the page's JavaScript, rotates proxies, and returns either rendered HTML ready for parsing or structured JSON when extraction rules are supplied. The fields available on public shared chats include prompt text, response text, model variant, citation URLs, and the turn-by-turn structure of the conversation. Nothing behind a login is in scope.

Three jobs this endpoint fits

Most Claude scraping work falls into one of three patterns. Each one maps to a different output shape and a different set of request parameters.

Collecting public shared chats

Pull publicly shared conversations to study prompt phrasing, model variant selection, and which response formats get shared most often. The turn-by-turn structure is the useful unit here: it lets you compare how a conversation develops rather than treating each page as a single blob of text.

Brand and competitor mention monitoring

Scan public shared chats for your brand, product, or competitor names to see where AI users discuss your category. A practical setup is to keep a short list of terms, run each shared-chat URL through the endpoint, and store the matched turn alongside the URL and the model variant. That gives you a dated record you can re-check rather than a one-off count.

Citation tracking

Extract the sources, papers, and URLs that appear in shared answers. This is useful for mapping which references recur across conversations and for spot-checking whether a cited page actually supports the claim next to it.

Field map for a public shared chat

The table below is a reader aid, not a schema contract. It shows which field answers which question so you can decide whether to use CSS extraction rules or AI extraction rules.

Request parameters that change the result

The endpoint accepts a small set of parameters that materially change what comes back. Use this map to decide which ones your job actually needs.

Blocked request vs. empty page

These two failures look similar in a log but need different fixes. A blocked request usually returns an error status or a challenge page instead of the shared-chat content. An empty page returns a normal response but no prompt or response text, which usually means the URL is not a public shared chat or the page rendered before the content loaded. Check the status code first, then check whether the returned HTML contains the conversation container before you change any request parameters.

Setup path

Create an account to get an API key and starting credits. Install the SDK for your stack (Python, Node, Go, PHP, or Ruby) or send a plain GET request. Test the request in the dashboard request builder first, then copy the generated snippet into your codebase. The builder produces snippets in Python, Node, Go, PHP, Ruby, and cURL.

Plan comparison

The plan table below is a dated reference captured on 2026-11-15. Prices, credit allowances, concurrency limits, and feature inclusions can change; check the current pricing page before committing. All prices are exclusive of VAT. Credits are valid for one month and do not roll over. Requests start at 1 credit each; enabling JS rendering costs 5 credits per successful request; premium proxies cost 10 or 25 credits depending on whether JS rendering is also enabled.

Why use a hosted endpoint instead of your own browser fleet

Running your own headless browsers means managing proxy pools, handling IP-based blocking, and keeping rendering stable as target pages change. A hosted endpoint moves that work off your plate: proxy rotation, headless rendering, and CAPTCHA handling are handled on the service side, and the response is either rendered HTML or structured JSON. The trade-off is that you depend on the service's rendering and extraction behaviour rather than controlling it directly.

Compliance notes

Public-facing pages can usually be scraped, but review the target site's terms of service before deploying and avoid post-login content. The service states that it does not collect or store personal data from scraped sites unless the user explicitly requests it. GDPR and CCPA compliance claims are the service's own; verify them against your own legal requirements.

Field map for a public Claude shared chat
FieldWhat it answersTypical extraction method
Prompt textWhat the user askedCSS selector or AI extraction rule
Response textWhat Claude answeredCSS selector or AI extraction rule
Model variantWhich model produced the answerCSS selector or AI extraction rule
Citation URLsWhich sources the answer referencesCSS selector or AI extraction rule
Turn structureHow the conversation developsCSS selector or AI extraction rule

Reader aid only; the actual schema is defined by the request parameters you send.

Request parameters that change the result
ParameterWhat it doesWhen to use it
render_jsRenders JavaScript before returning the pageWhen the shared chat loads its content dynamically
stealth_proxyRoutes the request through stealth proxiesWhen a target shows aggressive blocking
country_codeSets the request's geolocationWhen content varies by region
extract_rulesApplies CSS selectors to return structured dataWhen you know the page structure
ai_extract_rulesApplies natural-language extraction rulesWhen the page structure is unstable or unknown

Parameter names and behaviour are as supplied; confirm current documentation before relying on any single parameter.

Plan comparison (captured 2026-11-15)
PlanPriceAPI creditsConcurrent requestsPriority supportAccount managerTeam management
Hobby$19 /mo75,00025Not includedNot includedNot included
Freelance$49 /mo250,00050Not includedNot includedNot included
Startup$99 /mo1,000,000100Not includedNot includedNot included
Business$249 /mo3,000,000200IncludedIncludedIncluded
Business +$599 /mo8,000,000400IncludedIncludedIncluded
Enterprise 14$999 /mo14,000,000500IncludedIncludedIncluded
Enterprise 24$1,599 /mo24,000,000600IncludedIncludedIncluded
Enterprise 41$2,399 /mo41,000,000700IncludedIncludedIncluded
Enterprise 69$3,599 /mo69,000,000800IncludedIncludedIncluded
Enterprise 120$5,799 /mo120,000,000900IncludedIncludedIncluded

Dated reference captured 2026-11-15. Prices exclude VAT. Credits are valid for one month and do not roll over. JS rendering costs 5 credits per successful request; premium proxies cost 10 or 25 credits depending on JS rendering. Check current pricing before purchase.

Frequently asked questions

Do I need an account to scrape Claude Web?

No external account is needed on the target side. You sign up for an API key, which comes with starting credits and no card requirement, then send the URL and the service handles the rest.

Is scraping Claude Web legal?

Public-facing pages can usually be scraped, but review the target site's terms of service before deploying and avoid post-login content.

What data can I extract from Claude Web?

Any content that renders on the public page is fair game. For structured JSON output, target elements with CSS selectors through extract_rules, or hand the field descriptions to ai_extract_rules and let AI-driven extraction build the result.

How do I start scraping Claude Web?

Sign up, grab your API key, and send a GET request with the target URL. The service renders the page, rotates proxies, and returns the data. Starting credits are available for testing.

Will I get blocked when scraping Claude Web?

Each request goes out through rotating residential proxies and a real headless browser, so it looks like a normal visitor. When a target starts blocking aggressively, switch on stealth_proxy for that request.

Can I scrape Claude Web without writing code?

Yes. The dashboard request builder lets you paste a URL, set parameters, and copy the generated snippet in Python, Node, Go, PHP, Ruby, or cURL.

How do I choose between extract_rules and ai_extract_rules?

Use extract_rules when you know the page's CSS structure and want predictable, cheap extraction. Use ai_extract_rules when the structure is unstable or you want to describe the fields in natural language. The AI path is more flexible but less deterministic, so it is a better fit for exploratory work than for a fixed daily pipeline.

What does a blocked request look like compared with an empty page?

A blocked request typically returns an error status or a challenge page instead of the shared-chat content. An empty page returns a normal response but no prompt or response text, which usually means the URL is not a public shared chat or the page rendered before the content loaded. Check the status code first, then check whether the returned HTML contains the conversation container before changing request parameters.