Claude Scraper API for Public Shared Chats
Public Claude shared-chat pages carry a lot of usable signal: how people phrase prompts, which model variants appear, what answers get shared, and which sources the model cites.
This page covers how a hosted scraping API returns that data as structured JSON or rendered HTML, what request parameters control the result, and how the plan tiers compare.
What the Claude Scraper API returns
Send a URL from Claude's public shared-chat pages to the endpoint. The service renders the page's JavaScript, rotates proxies, and returns either rendered HTML ready for parsing or structured JSON when extraction rules are supplied. The fields available on public shared chats include prompt text, response text, model variant, citation URLs, and the turn-by-turn structure of the conversation. Nothing behind a login is in scope.
Three jobs this endpoint fits
Most Claude scraping work falls into one of three patterns. Each one maps to a different output shape and a different set of request parameters.
Collecting public shared chats
Pull publicly shared conversations to study prompt phrasing, model variant selection, and which response formats get shared most often. The turn-by-turn structure is the useful unit here: it lets you compare how a conversation develops rather than treating each page as a single blob of text.
Brand and competitor mention monitoring
Scan public shared chats for your brand, product, or competitor names to see where AI users discuss your category. A practical setup is to keep a short list of terms, run each shared-chat URL through the endpoint, and store the matched turn alongside the URL and the model variant. That gives you a dated record you can re-check rather than a one-off count.
Citation tracking
Extract the sources, papers, and URLs that appear in shared answers. This is useful for mapping which references recur across conversations and for spot-checking whether a cited page actually supports the claim next to it.
Field map for a public shared chat
The table below is a reader aid, not a schema contract. It shows which field answers which question so you can decide whether to use CSS extraction rules or AI extraction rules.
Request parameters that change the result
The endpoint accepts a small set of parameters that materially change what comes back. Use this map to decide which ones your job actually needs.
Blocked request vs. empty page
These two failures look similar in a log but need different fixes. A blocked request usually returns an error status or a challenge page instead of the shared-chat content. An empty page returns a normal response but no prompt or response text, which usually means the URL is not a public shared chat or the page rendered before the content loaded. Check the status code first, then check whether the returned HTML contains the conversation container before you change any request parameters.
Setup path
Create an account to get an API key and starting credits. Install the SDK for your stack (Python, Node, Go, PHP, or Ruby) or send a plain GET request. Test the request in the dashboard request builder first, then copy the generated snippet into your codebase. The builder produces snippets in Python, Node, Go, PHP, Ruby, and cURL.
Plan comparison
The plan table below is a dated reference captured on 2026-11-15. Prices, credit allowances, concurrency limits, and feature inclusions can change; check the current pricing page before committing. All prices are exclusive of VAT. Credits are valid for one month and do not roll over. Requests start at 1 credit each; enabling JS rendering costs 5 credits per successful request; premium proxies cost 10 or 25 credits depending on whether JS rendering is also enabled.
Why use a hosted endpoint instead of your own browser fleet
Running your own headless browsers means managing proxy pools, handling IP-based blocking, and keeping rendering stable as target pages change. A hosted endpoint moves that work off your plate: proxy rotation, headless rendering, and CAPTCHA handling are handled on the service side, and the response is either rendered HTML or structured JSON. The trade-off is that you depend on the service's rendering and extraction behaviour rather than controlling it directly.
Compliance notes
Public-facing pages can usually be scraped, but review the target site's terms of service before deploying and avoid post-login content. The service states that it does not collect or store personal data from scraped sites unless the user explicitly requests it. GDPR and CCPA compliance claims are the service's own; verify them against your own legal requirements.
| Field | What it answers | Typical extraction method |
|---|---|---|
| Prompt text | What the user asked | CSS selector or AI extraction rule |
| Response text | What Claude answered | CSS selector or AI extraction rule |
| Model variant | Which model produced the answer | CSS selector or AI extraction rule |
| Citation URLs | Which sources the answer references | CSS selector or AI extraction rule |
| Turn structure | How the conversation develops | CSS selector or AI extraction rule |
Reader aid only; the actual schema is defined by the request parameters you send.
| Parameter | What it does | When to use it |
|---|---|---|
| render_js | Renders JavaScript before returning the page | When the shared chat loads its content dynamically |
| stealth_proxy | Routes the request through stealth proxies | When a target shows aggressive blocking |
| country_code | Sets the request's geolocation | When content varies by region |
| extract_rules | Applies CSS selectors to return structured data | When you know the page structure |
| ai_extract_rules | Applies natural-language extraction rules | When the page structure is unstable or unknown |
Parameter names and behaviour are as supplied; confirm current documentation before relying on any single parameter.
| Plan | Price | API credits | Concurrent requests | Priority support | Account manager | Team management |
|---|---|---|---|---|---|---|
| Hobby | $19 /mo | 75,000 | 25 | Not included | Not included | Not included |
| Freelance | $49 /mo | 250,000 | 50 | Not included | Not included | Not included |
| Startup | $99 /mo | 1,000,000 | 100 | Not included | Not included | Not included |
| Business | $249 /mo | 3,000,000 | 200 | Included | Included | Included |
| Business + | $599 /mo | 8,000,000 | 400 | Included | Included | Included |
| Enterprise 14 | $999 /mo | 14,000,000 | 500 | Included | Included | Included |
| Enterprise 24 | $1,599 /mo | 24,000,000 | 600 | Included | Included | Included |
| Enterprise 41 | $2,399 /mo | 41,000,000 | 700 | Included | Included | Included |
| Enterprise 69 | $3,599 /mo | 69,000,000 | 800 | Included | Included | Included |
| Enterprise 120 | $5,799 /mo | 120,000,000 | 900 | Included | Included | Included |
Dated reference captured 2026-11-15. Prices exclude VAT. Credits are valid for one month and do not roll over. JS rendering costs 5 credits per successful request; premium proxies cost 10 or 25 credits depending on JS rendering. Check current pricing before purchase.
Frequently asked questions
Do I need an account to scrape Claude Web?
No external account is needed on the target side. You sign up for an API key, which comes with starting credits and no card requirement, then send the URL and the service handles the rest.
Is scraping Claude Web legal?
Public-facing pages can usually be scraped, but review the target site's terms of service before deploying and avoid post-login content.
What data can I extract from Claude Web?
Any content that renders on the public page is fair game. For structured JSON output, target elements with CSS selectors through extract_rules, or hand the field descriptions to ai_extract_rules and let AI-driven extraction build the result.
How do I start scraping Claude Web?
Sign up, grab your API key, and send a GET request with the target URL. The service renders the page, rotates proxies, and returns the data. Starting credits are available for testing.
Will I get blocked when scraping Claude Web?
Each request goes out through rotating residential proxies and a real headless browser, so it looks like a normal visitor. When a target starts blocking aggressively, switch on stealth_proxy for that request.
Can I scrape Claude Web without writing code?
Yes. The dashboard request builder lets you paste a URL, set parameters, and copy the generated snippet in Python, Node, Go, PHP, Ruby, or cURL.
How do I choose between extract_rules and ai_extract_rules?
Use extract_rules when you know the page's CSS structure and want predictable, cheap extraction. Use ai_extract_rules when the structure is unstable or you want to describe the fields in natural language. The AI path is more flexible but less deterministic, so it is a better fit for exploratory work than for a fixed daily pipeline.
What does a blocked request look like compared with an empty page?
A blocked request typically returns an error status or a challenge page instead of the shared-chat content. An empty page returns a normal response but no prompt or response text, which usually means the URL is not a public shared chat or the page rendered before the content loaded. Check the status code first, then check whether the returned HTML contains the conversation container before changing request parameters.