Skip to content

Scrape Publicly Shared OpenAI Chats at Scale

Publicly shared OpenAI chat pages are a useful source for prompt research, citation mapping, and LLM-ready text.

OUTPUT FORMATSHTMLJSONMarkdown

This page explains how a scraping API can fetch those pages, which fields they expose, and how to choose between rendered HTML, Markdown, and structured JSON output. It also covers the practical limits you should plan around: concurrency, credit costs, and the difference between public shared chats and anything behind a login.

Three ways teams use shared-chat data

Most teams working with public OpenAI shared chats fall into one of three patterns. The first is collecting the chats themselves to study how people phrase prompts, which model variants appear, and what gets shared most often. The second is following the citations inside shared responses to see which domains the model leans on and to audit factual grounding. The third is analyzing prompt structure across many conversations to spot recurring patterns and common failure modes. Each pattern needs the same underlying fetch: a public shared-chat URL rendered in a real browser, then parsed into a usable shape.

How a request flows

The workflow is deliberately short. You create an account, receive an API key, and get a starting credit balance. You install a client for your stack or send a plain HTTP request. You pass a public OpenAI shared-chat URL along with any rendering or extraction options. The service renders the page, rotates proxies as needed, and returns rendered HTML, Markdown, or parsed JSON depending on what you asked for. Advanced options let you turn on JavaScript rendering, route through stealth proxies when a page challenges the request, set a country code for geotargeting, and describe extraction rules in natural language.

Fields you can expect from a public shared chat

Public shared chats are not a fixed schema, but the fields most teams care about are consistent: the prompt text, the response text, citation URLs, the model variant, and the turn-by-turn conversation structure. Treat these as targets for your extraction rules rather than guaranteed keys. A practical field map looks like this: prompt text and response text map to the message blocks in the rendered page; citation URLs map to the links inside responses; model variant maps to the label shown near the conversation; turn structure maps to the ordered sequence of messages. If a page changes layout, the extraction rule is what you update, not the whole pipeline.

Choosing an output format

Rendered HTML is the most faithful capture and is useful when you want to inspect the page as a browser sees it. Markdown is a clean text form that feeds well into LLM workflows, such as prompt-pattern analysis or research pipelines. Structured JSON is the best fit when downstream systems need named fields rather than raw text. You can define those fields with CSS selectors mapped to field names, or describe them in natural language and let the extraction step return parsed JSON. Pick the format that matches the next stage of your pipeline rather than converting later.

Concurrency, throttling, and cost planning

Two different limits matter, and they are easy to confuse. The first is your plan's concurrency limit, which caps how many requests can run at the same time. The second is throttling from the target site, which can happen even when you are well under your concurrency cap. If a target starts rejecting aggressive patterns, spacing out requests or using multiple sessions is the usual remedy. Cost is separate again: each request consumes credits, and the amount depends on the options you enable. JavaScript rendering, premium proxies, and stealth proxies each add to the per-request credit cost. Plan a batch by multiplying the per-request cost by the number of pages, then check that the total fits your monthly credit allowance.

Pre-flight checklist

Before you run a batch, confirm the following. The URLs point to public shared chats, not post-login conversations. You have reviewed the target site's terms of service for your use case. Your extraction rules or natural-language spec name the fields you actually need. You know which options are enabled, because each one changes the credit cost. Your concurrency setting matches your plan. You have a small sample run to confirm the output shape before scaling up. If any of these is unclear, resolve it on a handful of pages first.

Plan comparison (captured source values)
PlanMonthly priceAPI creditsConcurrent requestsPriority supportAccount managerTeam management
Hobby$19 /mo75,00025Not includedNot includedNot included
Freelance$49 /mo250,00050Not includedNot includedNot included
Startup$99 /mo1,000,000100Not includedNot includedNot included
Business$249 /mo3,000,000200IncludedIncludedIncluded
Business +$599 /mo8,000,000400IncludedIncludedIncluded
Enterprise 14$999 /mo14,000,000500IncludedIncludedIncluded
Enterprise 24$1,599 /mo24,000,000600IncludedIncludedIncluded
Enterprise 41$2,399 /mo41,000,000700IncludedIncludedIncluded
Enterprise 69$3,599 /mo69,000,000800IncludedIncludedIncluded
Enterprise 120$5,799 /mo120,000,000900IncludedIncludedIncluded

Captured source values. All prices are exclusive of VAT. JavaScript rendering, rotating and premium proxies, geotargeting, and screenshots and extraction rules are listed as included on every plan in the captured source. Credit validity and per-request credit costs are described in the FAQ below.

Per-request credit costs (captured source values)
OptionCredits per successful request
Base request1
JavaScript rendering5
Premium proxy without JS rendering10
Premium proxy with JS rendering25
Stealth proxy75

Captured source values. Credit cost per request depends on the options enabled. API credits are valid for one month and do not roll over.

Frequently asked questions

Do I need an OpenAI account to scrape shared chats?

No external account is required. You sign up for an API key, which comes with a starting credit balance and no card requirement, then send the public shared-chat URL and let the service handle rendering and proxies.

Is scraping OpenAI shared chats legal?

Public-facing pages can usually be scraped, but you should review the target site's terms of service before deploying and avoid post-login content. Legality depends on your jurisdiction and use case, so treat this as a starting point rather than legal advice.

What rate limits apply?

The API enforces concurrency limits per plan rather than per-target rate limits. If the target site throttles aggressive request patterns, space out requests or use multiple sessions. Your plan's concurrency cap and the target's throttling are separate constraints.

How do I get structured JSON from a shared chat?

You can supply extract_rules that map CSS selectors to field names, or describe the fields you want in plain language through ai_extract_rules. Either way the response arrives as parsed JSON. Which fields are available depends on what the public shared-chat page exposes.

Which programming languages can I use?

Any language that can send an HTTP request. SDKs are published for Python, Node.js, Go, PHP, Ruby, and Java, and plain cURL works too. Pick whatever your stack already uses.

How much does it cost to scrape a shared chat?

New accounts start with free credits and no card requirement. Paid plans start at $19/month for higher volume and concurrency. Credit cost per request depends on the options enabled: 5 credits for JavaScript rendering, 25 for a premium proxy with JavaScript rendering, and 75 for a stealth proxy. API credits are valid for one month and do not roll over.