AWS Public Page Scraper for Structured Data
AWS public pages mix server-rendered text with JavaScript-driven pricing tables and active bot detection.
A hosted scraping API can absorb the proxy rotation and headless-browser work so your client stays a thin HTTP call. This page explains what the endpoint returns, which request controls matter, and where the limits are.
What the AWS scraper endpoint does
The endpoint accepts a public AWS URL and returns the page after server-side rendering. You can request raw rendered HTML, structured JSON built from CSS or XPath rules, or AI-extracted JSON from a natural-language specification. Screenshots and LLM-ready Markdown are available as separate output modes.
Typical targets are service catalog pages, per-region pricing tables, feature lists, free-tier eligibility statements, and documentation pages. The API does not authenticate into AWS consoles or private accounts; it works on public pages only.
Request controls and what each one changes
The useful controls for AWS pages fall into four groups. Use this map to decide which parameters to set before writing parsing code.
- Rendering: render_js=true runs the page in a real headless browser and returns post-render HTML. This is the control that makes JavaScript pricing tables and dynamic service lists visible.
- Proxy and geo: stealth_proxy routes through residential infrastructure for pages with CAPTCHA or aggressive bot checks. country_code selects a residential IP in a specific country when content varies by geography.
- Extraction: extract_rules maps CSS selectors to field names and returns parsed JSON. ai_extract_rules accepts a natural-language spec for fields such as service names, pricing tiers, regions, and feature descriptions.
- Output mode: screenshot capture returns a full-page image; Markdown output returns documentation text suitable for retrieval pipelines.
A practical order is to start with render_js=true, inspect the returned HTML, then add extract_rules once selectors are stable. Move to ai_extract_rules when the page structure changes often or when you want field definitions in plain language.
Decision checklist before you commit
Use these questions to decide whether this endpoint fits the task or whether a different approach is cheaper.
- Is the target page public? If the data sits behind an AWS login or console session, this endpoint is not the right tool.
- Does the page need JavaScript? If yes, budget for render_js=true and the associated credit cost.
- Is the page protected by CAPTCHA or IP blocking? If yes, plan for stealth_proxy and possibly country_code.
- Do you need field-level output or just the page? Choose extract_rules or ai_extract_rules for JSON; choose HTML or Markdown for downstream parsing.
- How many concurrent requests will you run? Concurrency is capped per plan, not per target.
- Will the target throttle aggressive patterns? If so, space requests or use multiple sessions rather than assuming the API imposes a per-target rate limit.
- Do you need a visual record? Add screenshot capture when you want to confirm how the page rendered.
Worked example: a regional pricing page
Suppose you want the on-demand hourly price, reserved-instance discount, and free-tier note for one AWS service in a specific region. A workable sequence is:
- Send the public pricing URL with render_js=true so the pricing table is populated.
- Add country_code if the page shows different regional content based on visitor location.
- Add extract_rules with selectors for the service name, price-per-hour, reservation term, and free-tier text, or use ai_extract_rules with those field names in plain language.
- Parse the returned JSON and store the region alongside the values so later comparisons are meaningful.
This example uses the captured scenario of monitoring AWS pricing pages by service and region. It does not assert a specific price or availability; those change and should be read from the page at request time.
Limits and a common confusion
Two different limits are often conflated. The API enforces concurrency limits per plan: the number of requests that can be in flight at the same time. It does not impose a per-target rate limit on AWS. If AWS itself throttles aggressive request patterns, the fix is to space out requests or use multiple sessions, not to look for a higher per-target quota.
Credit consumption also varies by feature. A plain request starts at one credit. JavaScript rendering costs five credits per successful request. Premium proxies cost ten or twenty-five credits depending on whether JavaScript rendering is enabled. Credits are valid for one month and do not roll over. These are the published contract values; check the current plan page before sizing a budget.
Outputs and integration paths
The endpoint returns rendered HTML by default. With extraction rules or AI extraction, the response is parsed JSON. Markdown output is available for documentation pages that will feed a retrieval or LLM pipeline. Screenshots return a full-page image.
Client options include SDKs for Python, Node.js, Go, PHP, Ruby, and Java, plus plain cURL. The dashboard request builder lets you paste a URL, set parameters, and copy a generated snippet in those languages, so a first request does not require writing scraper code. For workflow automation, the API can be called from n8n or Make scenarios.
Public AWS data this endpoint can return
Across public AWS pages, the fields most often requested are service titles, region availability, pricing tiers, feature descriptions, free-tier eligibility, and documentation links. The endpoint returns the page content; the shape of the JSON depends on the extraction rules you supply. If you need a stable schema across many pages, define the field names once and reuse the same rule set, then validate a sample of pages after any AWS layout change.
| Plan | Monthly price | API credits | Concurrent requests | Priority support | Account manager | Team management |
|---|---|---|---|---|---|---|
| Hobby | $19 /mo | 75,000 | 25 | Not included | Not included | Not included |
| Freelance | $49 /mo | 250,000 | 50 | Not included | Not included | Not included |
| Startup (Recommended) | $99 /mo | 1,000,000 | 100 | Not included | Not included | Not included |
| Business | $249 /mo | 3,000,000 | 200 | Included | Included | Included |
| Business + | $599 /mo | 8,000,000 | 400 | Included | Included | Included |
| Enterprise 14 | $999 /mo | 14,000,000 | 500 | Included | Included | Included |
| Enterprise 24 | $1,599 /mo | 24,000,000 | 600 | Included | Included | Included |
| Enterprise 41 (Recommended) | $2,399 /mo | 41,000,000 | 700 | Included | Included | Included |
| Enterprise 69 | $3,599 /mo | 69,000,000 | 800 | Included | Included | Included |
| Enterprise 120 | $5,799 /mo | 120,000,000 | 900 | Included | Included | Included |
Prices are exclusive of VAT. Credits are valid for one month and do not roll over. Requests start at 1 credit each; JavaScript rendering costs 5 credits per successful request; premium proxies cost 10 or 25 credits depending on JavaScript rendering. This is a dated reference captured from the publisher's plan display; check current pricing before purchase.
Frequently asked questions
Can I scrape AWS without writing code?
Yes. The dashboard request builder lets you paste a URL, set parameters, and copy the generated snippet in Python, Node, Go, PHP, Ruby, or cURL. You still need a place to run the snippet or workflow, but you do not have to write the scraping logic yourself.
How do I handle JavaScript when scraping AWS?
Pass render_js=true on the request. The page runs in a real headless browser, JavaScript executes server-side, and the response contains the post-render HTML. This is the control that makes dynamic pricing tables and service lists visible.
Can I scrape AWS from a specific country?
Yes. Pass country_code on the request to route through a residential IP in that country. This matters when AWS shows different content or regional offers based on visitor location.
What's the rate limit for scraping AWS?
The API enforces concurrency limits per plan, not per-target rate limits. If AWS throttles aggressive request patterns, space out requests or use multiple sessions. The plan table above lists the concurrency figure for each tier.
How do I get structured JSON from AWS?
Pass extract_rules with CSS selectors mapped to field names, or use ai_extract_rules with a natural-language specification. The response comes back as parsed JSON. For pricing pages, define fields such as service name, price-per-hour, region, and free-tier eligibility.
Which programming languages can I use to scrape AWS?
Any language that can make an HTTP request. Published SDKs cover Python, Node.js, Go, PHP, Ruby, and Java, and plain cURL works too. Pick whatever your stack already uses.
Does the endpoint log into AWS or read private account data?
No. It works on public AWS pages. Data behind a console login or private account is outside its scope.
What happens if AWS changes its page layout?
Selector-based extract_rules can break when the layout changes. AI extraction from a natural-language spec is more tolerant of structural changes, but you should still validate a sample of pages after a redesign. Reusing one rule set across many pages makes breakage easier to spot.