PubMed Scraper API for Research Citations
PubMed article and citation pages are rendered with JavaScript and protected by bot detection.
This API puts a managed proxy and headless-browser layer in front of those pages so your client can stay a thin HTTP call. You send a PubMed URL on pubmed.ncbi.nlm.nih.gov and receive rendered HTML or structured JSON ready to parse.
What you can extract from PubMed
The scraper is built around three common research workflows: pulling article records, mapping author collaboration networks, and tracking keyword or MeSH term coverage across papers. For each article page, the documented fields are title, authors, affiliations, abstract, publication date, journal, MeSH keywords, citation count, and PMID. You can request those as rendered HTML and parse them yourself, or define extraction rules so the response arrives as parsed JSON.
Field map and extraction approach
Before you write a scraper, decide which fields you need and how each should be captured. The table below maps the documented PubMed fields to a practical extraction approach and a validation check. Use it as a starting schema for extract_rules or ai_extract_rules.
Decision checklist before your first request
Use this checklist to choose the right request controls for PubMed. Each item corresponds to a documented parameter or plan capability.
- Do you need the page after JavaScript runs? If yes, set render_js=true. The API runs the page in a real headless browser and returns post-render HTML.
- Is the target returning aggressive blocks or CAPTCHAs? Enable stealth_proxy on the request. Residential proxies rotate by default, but stealth_proxy adds the stronger layer when needed.
- Does the content vary by geography? Pass country_code to route through a residential IP in that country.
- Do you need structured output rather than HTML? Use extract_rules with CSS selectors mapped to field names, or ai_extract_rules with a natural-language specification. The response comes back as parsed JSON.
- Are you hitting concurrency limits? Limits are set per plan, not per target. Space out requests or use multiple sessions if the target throttles aggressive patterns.
- Do you need a visual record? Capture a full-page screenshot for verification.
Worked example: from PubMed URL to a JSON record
Suppose you are building a literature-review dataset and want one record per article. Start with the documented field list: title, authors, affiliations, abstract, publication date, journal, MeSH keywords, citation count, and PMID. Define a JSON schema with those keys, then map each key to a CSS selector via extract_rules, or describe the fields in natural language via ai_extract_rules. Send the PubMed article URL with render_js=true so the page is fully rendered before extraction. The response is parsed JSON, so your downstream pipeline can load it without an HTML parsing step. Validate each record by checking that PMID is present and that the title and abstract are non-empty; if a field is missing, inspect the rendered HTML to confirm the selector still matches the current page structure.
Two different failure modes and the right control for each
A failed request can come from two different sources, and the fix is different for each. If you are sending more simultaneous requests than your plan allows, you are hitting the API's concurrency limit. The documented remedy is to space out requests or use multiple sessions. If the target itself is throttling aggressive patterns or returning blocks, the remedy is to enable stealth_proxy or adjust request pacing. The API does not impose per-target rate limits; concurrency limits are per plan. Distinguishing these two cases prevents you from changing the wrong control.
How the request flow works
The documented flow has four steps. First, create an account to get an API key and free starting credits. Second, install a client SDK for your stack; each PubMed scrape becomes a single function call. Third, send a GET request with your API key and a PubMed URL on pubmed.ncbi.nlm.nih.gov; JavaScript renders, residential proxies rotate, and rendered HTML comes back. Fourth, add advanced parameters when needed: render_js for JavaScript, stealth_proxy for CAPTCHAs, country_code for geo-targeting, and ai_extract_rules for natural-language data extraction. Code samples are available for Python, Node.js, Ruby, PHP, Go, and cURL, and the dashboard includes a request builder for testing without writing code.
Plan capability reference
The table below lists the plan tiers and their documented capabilities. All prices are exclusive of VAT. API credits are valid for one month and do not roll over; requests start at 1 credit each. JavaScript rendering costs 5 API credits per successful request. Premium proxies cost 10 or 25 API credits per successful request depending on whether JS rendering is enabled. Geotargeting covers 200+ countries and territories (40+ on classic proxies).
| Field | Extraction approach | Validation check |
|---|---|---|
| Title | extract_rules CSS selector or ai_extract_rules | Non-empty string |
| Authors | extract_rules list selector or ai_extract_rules | At least one author |
| Affiliations | extract_rules list selector or ai_extract_rules | Matches author count where available |
| Abstract | extract_rules selector or ai_extract_rules | Non-empty text |
| Publication date | extract_rules selector or ai_extract_rules | Parseable date |
| Journal | extract_rules selector or ai_extract_rules | Non-empty string |
| MeSH keywords | extract_rules list selector or ai_extract_rules | List of terms |
| Citation count | extract_rules selector or ai_extract_rules | Integer or null |
| PMID | extract_rules selector or ai_extract_rules | Present and unique per record |
Field availability depends on the individual PubMed page; validate selectors against rendered HTML.
Frequently asked questions
Will I get blocked when scraping PubMed?
Requests go out through rotating residential proxies and a real headless browser, so they look like normal traffic. If a target gets aggressive, enable stealth_proxy on the request.
Can I scrape PubMed without writing code?
Yes. The dashboard request builder lets you paste a URL, set parameters, and copy the generated snippet in Python, Node, Go, PHP, Ruby, or cURL.
How do I handle JavaScript when scraping PubMed?
Pass render_js=true on the request. The page runs in a real headless browser, the JavaScript executes server-side, and the post-render HTML is returned.
Can I scrape PubMed from a specific country?
Yes. Pass country_code on the request to route through a residential IP in that country. This is useful when content varies by geography.
What's the rate limit for scraping PubMed?
Limits are set by plan concurrency, not by target. If a site slows down aggressive request patterns, add spacing between calls or spread the work across multiple sessions.
How do I get structured JSON from PubMed?
Pass extract_rules with CSS selectors mapped to field names, or use ai_extract_rules with a natural-language specification. The response comes back as parsed JSON.