Scrape Semrush Public Pages With Rendering Built In
Semrush exposes a slice of its data on public pages: authority scores, keyword previews, backlink counts, traffic estimates and competitor lists.
This page explains how to route those public URLs through a managed scraping API so that JavaScript rendering, proxy rotation and anti-bot handling happen on the provider's side, and how to shape the response into HTML, structured JSON or Markdown.
What this scraper does
The target is Semrush's public surface — the pages anyone can load without a paid Semrush seat. You hand the API a Semrush URL; it fetches the page with a real browser, executes the JavaScript, rotates proxies to dodge IP blocks, and returns the rendered result.
On a typical public Semrush page the fields you can expect to see include an Authority Score, a preview of top organic keywords, a backlink count, a traffic estimate and a competitor list. What actually appears depends on the page and on Semrush's own rendering, so treat the field list as a starting map rather than a guarantee.
Output shape is your choice: raw rendered HTML, structured JSON via extraction rules, or clean Markdown for downstream LLM use.
Three workflows teams build
The public Semrush surface supports a handful of recurring jobs. Three that come up most often:
- Competitor authority tracking. Read Authority Score, top organic keyword previews and backlink counts from the free Domain Overview view, then assemble a leaderboard across competitors without buying Semrush seats for each analyst.
- Keyword ranking extraction. Collect the ranking positions a competitor holds for a keyword set, so you can spot market shifts and adjust your own SEO plan with evidence rather than guesswork.
- Backlink profile analysis. Gather referring-domain counts, backlink totals and top anchor texts from a competitor's public profile, then use the gaps to prioritise outreach and authority-building work.
Each of these is a read of public data — nothing behind a login — and each benefits from the same managed rendering and proxy layer.
How a request flows through the API
The path from zero to a first response is short:
- Create an account. You get an API key immediately. The source page advertises 1,000 free credits to start — see the pricing note below before relying on that figure.
- Pick a client. SDKs exist for Python, Node.js, Go, PHP, Ruby and Java; plain cURL works too. Each scrape becomes a single function call, so you don't manage headless Chrome or proxy pools yourself.
- Send the Semrush URL. POST or GET the target URL. The API renders the public page, handles anti-bot measures, and returns rendered HTML — or structured JSON if you supply extraction rules.
- Add options only when needed.
render_jsfor JavaScript-heavy pages,stealth_proxywhen CAPTCHAs appear,country_codefor geo-targeted views, andai_extract_ruleswhen you'd rather describe the data in natural language than write selectors.
Request options and what each one buys you
Use this as a decision checklist before you send a request:
- Default request (no options). Cheapest path. Works when the Semrush page returns useful HTML without JavaScript and isn't blocking you. Start here.
- `render_js`. Turn on when the data you want only appears after client-side rendering. Costs more credits per successful request (see the worked example below).
- `stealth_proxy`. Turn on when you see CAPTCHAs or repeated blocks. This is the most expensive option per request, so reserve it for pages that actually need it.
- `extract_rules`. Supply CSS selectors mapped to field names when you know the page structure and want parsed JSON back instead of HTML.
- `ai_extract_rules`. Supply a natural-language description of the fields you want when selectors are brittle or you're exploring a new page layout. Returns structured JSON without you writing selectors.
- `country_code`. Use when the Semrush view you need is geo-specific.
- Screenshot / Markdown modes. Screenshots suit visual reports of competitor profiles; Markdown suits feeding page content into an LLM for summaries.
A practical rule: only add an option when a plain request has demonstrably failed to return the data you need. Each option raises the credit cost of the request.
Worked example: what a Semrush request costs in credits
Three reference credit costs per successful request apply here, and they stack in a way worth walking through.
Suppose you want a competitor's public Domain Overview page and the data only appears after JavaScript runs. A plain request with render_js costs 5 credits. If that same page also throws CAPTCHAs and you add stealth_proxy, the cost rises to 75 credits per successful request. A middle case — premium proxy with JS rendering — sits at 25 credits.
So a batch of 1,000 rendered pages costs roughly 5,000 credits on the cheap path, 25,000 on the premium-proxy path, or 75,000 on the stealth path. That spread is why the decision checklist above matters: turning on stealth for every request can burn through a plan's credit allowance many times faster than necessary.
These figures are reference values, not a current offer. Confirm against the live pricing page before you size a plan.
Telling the failure modes apart
When a Semrush scrape comes back empty or wrong, the cause usually falls into one of four buckets. Diagnosing which one saves you from adding expensive options blindly.
- Blocked. You get a CAPTCHA page, a 403, or an interstitial instead of the data. Fix: add
stealth_proxy. This is the only case where stealth is the right answer. - JavaScript not rendered. The HTML arrives but the fields you want are empty placeholders. Fix: add
render_js. If the page is JS-heavy, this is expected. - Wrong selector. The page rendered fine and the data is present in the HTML, but your
extract_rulesreturned nulls. Fix: inspect the rendered HTML and correct the selector, or switch toai_extract_rules. - Data genuinely absent. The public page doesn't show the field you expected — for example, a metric that only appears behind a paid Semrush seat. No request option will conjure it. Fix: change the target page or accept the gap.
A quick triage: fetch the page with rendering on and no extraction rules first. If the raw HTML contains your data, the problem is your selector. If it doesn't, the problem is rendering or blocking. If neither, the data isn't public.
Setup path
Six SDKs are published — Python, Node.js, Go, PHP, Ruby and Java — plus raw cURL examples, so you can stay in whatever stack you already use. The source page claims docs-to-first-request in under five minutes; that's a reasonable target for a single URL, though a production pipeline with extraction rules and retries will take longer to tune.
Start with one Semrush URL and a plain request. Confirm you get HTML back. Then add render_js if the fields are missing, extract_rules once you know the selectors, and stealth_proxy only if you hit blocks. That order keeps your credit spend low while you learn the page.
Plan tiers and what changes between them
The plan ladder runs from a $19/month Hobby tier with 75,000 API credits and 25 concurrent requests up to enterprise tiers at $5,799/month with 120,000,000 credits and 900 concurrent requests. Across the ladder, JavaScript rendering, rotating and premium proxies, geotargeting, and screenshots/extraction rules are included at every tier; priority email support, a dedicated account manager and team management only appear from the Business tier upward.
These are reference values, not a current offer from this site. Prices, credit allowances and concurrency figures change; check the live pricing page before you commit. The signup offer is 1,000 free credits with no card required, and plans can be cancelled at any time.
What actually differs between tiers, in practice: the credit allowance (which caps how many requests you can make per month at a given per-request cost), the concurrency limit (which caps how many requests run at once), and whether you get priority support and team management. If you're unsure which tier fits, the worked example above lets you estimate monthly credits from your request mix.
Legal and data-handling notes
Scraping public-facing pages is generally permissible, but check the target site's terms of service before you deploy and stay away from anything behind a login. The provider does not collect or store personal data from scraped sites unless the user explicitly requests it; verify that against the current data-processing terms if you handle personal data.
Semrush Scraper API FAQs
See the FAQ entries below for account requirements, legality, extractable data, JSON output, language support and cost.
| Control | Type | Label / tooltip text | Notes |
|---|---|---|---|
| Standard / Enterprise toggle | button | Standard | aria-pressed=true in default state |
| Standard / Enterprise toggle | button | Enterprise | aria-pressed=false in default state |
| API Credits | button | API credits are valid for one month and don't roll over. Requests start at 1 credit each. | Tooltip |
| Concurrent requests | button | The number of requests that can be sent at the same time. | Tooltip |
| JavaScript rendering | button | Each successful request that enables JS rendering costs 5 API credits | Tooltip |
| Rotating & Premium Proxies | button | Each successful request that enables Premium proxies costs 10 or 25 API credits, depending on whether JS rendering is disabled or not. | Tooltip |
| Geotargeting | button | Route your requests through any of 200+ countries and territories (40+ on classic proxies) | Tooltip |
| Screenshots & Extraction Rules | button | Capture full-page screenshots and pull structured data using CSS or XPath extraction rules. | Tooltip |
| Team Management | button | Team management allows you to invite multiple users to your organization and assign them a custom quota of credits and concurrency. | Tooltip |
| Get in touch | button | Get in touch! | Opens sales chat |
This table reproduces the request and plan controls exactly as supplied in the protected form-controls contract. Prices and credit costs are the captured reference values and should be checked against the current pricing page before purchase.
Frequently asked questions
Do I need an account to scrape Semrush?
You don't need a Semrush account. You do need an API key from the scraping provider — the source page advertises 1,000 free credits on signup with no card required, which you should confirm against the provider's current offer. Once you have the key, send the Semrush URL and the provider handles rendering and anti-bot.
Is scraping Semrush legal?
Pages that anyone can load without signing in are generally fair game for collection, but read Semrush's terms of service before you put anything into production, and keep your requests to content that sits outside a login.
What data can I extract from Semrush?
Any field that shows up on the public page. Point extract_rules at it with CSS selectors, or describe it in plain language through ai_extract_rules, and the response comes back as structured JSON.
How do I get structured JSON from Semrush?
Map CSS selectors to your field names in extract_rules, or hand ai_extract_rules a natural-language description of what you want. Both routes return the result as parsed JSON.
Which programming languages can I use to scrape Semrush?
Any. The provider publishes SDKs for Python, Node.js, Go, PHP, Ruby and Java. Plain cURL works too. Pick whatever your stack already uses.
How much does it cost to scrape Semrush?
Signup includes 1,000 free credits with no card required, and paid plans start at $19/month for higher volume and concurrency. Per-request credit cost depends on the options you enable: 5 credits for JS rendering, 25 for premium proxy with JS, 75 for stealth proxy. Confirm against the live pricing page before sizing a plan.