AWS Lambda Web Scraping: Build a Serverless Scraper (Python + Java)
AWS Lambda is a good home for scraping work that is short, repeatable and easy to split into small jobs. It is a poor home for crawlers that need to stay alive for hours or hold a browser session open. This guide walks through the decision first, then the build: a static HTML scraper in Python deployed with AWS SAM, an optional API-backed variant that writes to S3, and a Java 21 equivalent for JVM teams. Every code block, SAM template and cost formula is provided as a protected module and reproduced exactly.
Is Lambda the right tool for this scrape?
Lambda runs code only when something triggers it. That trigger can be an HTTP request through API Gateway, a schedule through EventBridge, a message in SQS, an upload to S3, or an event from another AWS service. For scraping, the useful question is not 'can Lambda fetch a page' but 'does this job fit the shape Lambda is built for'.
A job fits when each unit of work is short, when URLs are independent of each other, and when a failed unit can be retried on its own. Price monitoring, simple ETL, content checks and scheduled scrapes every few hours are typical fits. A job does not fit when it needs a long-lived browser session, hours of continuous crawling, or heavy anti-bot handling; those belong on Step Functions, Fargate, EC2 Spot, AWS Batch, a proxy setup, or a scraping API.
Use this checklist before writing code:
- Can one page be fetched and parsed inside a single invocation? If yes, Lambda is a candidate.
- Are the URLs independent, so a failure on one does not require rerunning the whole crawl? If yes, queue them.
- Does the target return the data in the initial HTML? If yes, start with requests and BeautifulSoup.
- Does the data appear only after JavaScript runs, or does the site require interaction? Plan for a browser or a scraping API.
- Does the target block datacenter IPs? Expect 403s, captcha pages, rate limiting or bot-check redirects, and budget for proxies or an API.
- Does the job need to run longer than 15 minutes? Split it or move it off Lambda.
The good-fit and poor-fit table below is a quick reference for that decision.
Four parts of a Lambda scraping architecture
Do not build one large function that does everything. Split the design into four parts and each part becomes testable and replaceable.
- Trigger — decides when the scraper runs. EventBridge Scheduler for time-based jobs, SQS for URL queues, API Gateway for on-demand requests.
- Scraping logic — the Lambda handler that fetches and parses.
- Output layer — S3, a database, or another durable store.
- Monitoring and secrets — CloudWatch logs, X-Ray traces, SSM Parameter Store or Secrets Manager for keys.
Three patterns cover most jobs:
- Scheduled scrape. EventBridge Scheduler invokes the function, the function fetches, parses, writes to S3 and exits. Few moving parts, easy to reason about, and a failed run can be inspected and rerun on its own.
- Queue-based scrape. A producer puts URLs into SQS and Lambda processes them in batches. This is usually the safer production pattern for larger jobs because SQS gives buffering, retries and concurrency control, and a single failed URL can be retried without restarting the crawl.
- Workflow scrape. Step Functions coordinates several functions — one fetches listing pages, another scrapes detail pages, another cleans or stores records. Use this when the handler is turning into a full application or the job has branches and enrichment steps.
Lambda limits to design around
Lambda is not a long-running crawler host. AWS publishes current quotas in its Lambda quotas documentation, and the values below are the ones that shape a scraping design. New AWS accounts start with reduced concurrency and memory quotas, and AWS raises them automatically based on usage.
The limits that matter most for scraping are the 15-minute maximum runtime, the package size, the memory allocation and the concurrency ceiling. If a job needs more than 15 minutes of continuous work, split it with SQS or Step Functions, or move the workload to a service built for long-running processes.
Static HTML path: Python, requests and BeautifulSoup
Start with the simplest useful version: fetch a static page, parse it with BeautifulSoup, return structured data. This keeps the Lambda package small and avoids a browser runtime entirely — no Playwright, no Chromium, no container image.
The demo target is Books to Scrape, a site built for scraping practice. The scraper extracts book titles, prices, availability, ratings and product URLs. The prerequisites are Python 3.13, Java 21 and Maven, an AWS account, the AWS CLI, the AWS SAM CLI, Docker (or OrbStack on macOS) and a hosted scraping API account for the later API-backed variant.
Why AWS SAM rather than Serverless Framework? SAM keeps the deployment close to AWS itself: functions, schedules, permissions, S3 buckets and API endpoints are defined in one template.yaml and deployed through CloudFormation. Serverless Framework remains a valid tool, especially for teams already using it, but SAM is a cleaner fit for an all-AWS stack and avoids the login and licensing changes introduced in Serverless Framework v4.
The handler is built in five steps:
- Define defaults and reuse an HTTP session at module level.
- Normalize Lambda events so direct invocation, API Gateway and EventBridge all produce one payload shape.
- Fetch the HTML with raise_for_status() and an encoding fallback.
- Parse each article.product_pod card and resolve relative links with urljoin().
- Return a JSON response, mapping upstream request errors to 502 and other failures to 500.
Keeping the parser outside the handler means it can be tested without AWS. The SAM template sets a 30-second timeout, active tracing, arm64 architecture, 256 MB memory and a daily schedule that is disabled until the scraper is verified.
What a successful run looks like
The test event sends a URL and a limit. A successful invocation returns statusCode 200, a count and a JSON body of items. In the captured example the response reports three items, each with title, price, availability, rating and a full product URL, for example 'A Light in the Attic' at £51.77, in stock, rating Three. The same event can be run locally with sam local invoke and then in the cloud with aws lambda invoke; the CloudWatch log stream confirms the deployed function ran.
When something goes wrong, distinguish the failure before changing code:
- 502 upstream_request_failed — the target returned an HTTP error or the request failed. Check the URL, the target's availability and whether the response is a block page.
- 500 scrape_failed — the fetch succeeded but parsing or another step raised an exception. Check selectors and the response encoding.
- A 200 with empty items — the page loaded but the CSS selectors no longer match. This is a parser problem, not a network problem.
- A captcha page or 403 with a normal status — the target is blocking the request. This is an IP or anti-bot problem, not a Lambda bug.
When BeautifulSoup is not enough
BeautifulSoup only parses the HTML it receives. It does not execute JavaScript, wait for client-side rendering, click buttons, solve captchas or behave like a browser. The scenario table below separates the cases where a plain HTTP request is enough from the cases that need a browser or a scraping API.
You can run Playwright or Chromium on Lambda, but package size and cold starts become harder to manage. If a real browser is genuinely required, Playwright is usually the easiest route. If you would rather not own a browser stack at all, a scraping API removes that maintenance.
Some older options are best avoided. PhantomJS is no longer a modern choice, even though many older tutorials still point to it. Selenium still works, but Playwright is usually cleaner for new browser automation. The original chrome-aws-lambda package is no longer the safest default because Chromium, Puppeteer and Lambda runtimes move quickly; puppeteer-core with @sparticuz/chromium, or a Lambda container image with Playwright, is a safer bet.
Stealth plugins are not a durable answer to bot protection. They patch obvious browser fingerprints, but they do not change the signals that harder targets actually weigh. As of early 2025, a lot of sophisticated bot defenses were already responding to typical Puppeteer stealth setups with captcha challenges or 403 blocks. Those defenses also weigh signals beyond the browser fingerprint, such as how reputable the IP is, how a session behaves, how a visitor interacts with the page, and what request history looks like. Treat stealth as a temporary patch, not a permanent fix.
A short rule of thumb: requests and BeautifulSoup for static HTML, Playwright when rendering is truly needed, and a scraping API when rendering, proxies and anti-bot handling are all required.
AWS IPs get blocked — plan for it
Moving a scraper to Lambda does not make it invisible. AWS publishes its IP ranges, and many sites can identify traffic from cloud providers. Running from Lambda can even make blocking more likely, because the request now comes from a known datacenter range.
Common failure modes include HTTP 403 responses, captcha pages, rate limiting, redirects to bot-check pages, and content that differs from what a normal browser sees. Cloudflare-protected sites usually look at more than the user agent: IP reputation, device fingerprints, browser, cookies, sessions, interaction patterns and IP history all feed the decision. The practical implication is that a proxy layer or a hosted scraping API is often a design requirement rather than an afterthought.
API-backed Lambda that writes to S3
The static scraper works because Books to Scrape returns data in the initial HTML. For harder pages you may need JavaScript rendering, proxy handling, country targeting or extraction options. The second function delegates that work to a hosted scraping API and stores the result.
It does three things:
- Read the API key from SSM Parameter Store (or from an environment variable for local runs).
- Call the API with parameters taken from the Lambda event.
- Write the response to S3 and return the S3 location.
The event parameters map to request options as follows:
- url — target page; defaults to the demo URL.
- render_js — boolean, converted to a string; defaults to false.
- country_code, wait, wait_for, wait_browser, timeout, device, session_id, tag — passed through as strings when present.
- premium_proxy, stealth_proxy, block_ads, block_resources — booleans, converted to strings when present.
- extract_rules, ai_extract_rules, js_scenario — JSON-encoded when present.
- output_bucket — falls back to the OUTPUT_BUCKET environment variable.
- output_key — falls back to a dated path under scrapes/.
The API key is sent in the Authorization header. The SAM template gives this function 512 MB and a 60-second timeout, and the IAM policy is deliberately narrow: read one SSM parameter and write objects into one S3 bucket. The bucket uses a retain policy so collected output is not deleted automatically if the stack is removed. The schedule is again disabled until the function is verified.
The test event sets render_js to false because the sandbox page does not need rendering, and asks for the page's h1 text through extract_rules. A successful response returns statusCode 200 and an s3_uri; the saved object contains the source URL, status code, content type and body.
Cold starts, SnapStart and arm64
A cold start happens when Lambda creates a new execution environment before running the function. For small Python scrapers this is rarely the main bottleneck; network delays, retries, browser startup and slow targets usually matter more.
The practical guidance is short. Keep Python dependencies small, reuse HTTP sessions outside the handler and avoid loading large files at import time. For Java, keep initialization predictable and use SnapStart when startup time becomes noticeable. SnapStart applies to published versions, not to $LATEST, so publish a version and point an alias at it. arm64 is usually a good default for simple Python scrapers when dependencies support it. Browser scraping deserves extra testing, because Playwright, Chromium, native libraries and container images can change cold start behaviour.
What does this actually cost?
Lambda pricing comes down to requests and duration, with duration billed in GB-seconds (memory multiplied by runtime). The free tier provides 1 million requests and 400,000 GB-seconds each month. The three worked examples below use the captured formulas exactly.
Example 1 — a daily static scrape. A 256 MB function running 3 seconds once per day uses 30 × 3 × 0.25 = 22.5 GB-seconds, well within the free tier.
Example 2 — one million static pages per month at 256 MB and 2 seconds per page uses 1,000,000 × 2 × 0.25 = 500,000 GB-seconds. After the free tier, 100,000 GB-seconds are billable at 100,000 × $0.0000133334 = $1.33, and requests still cost $0 because they are covered by the free tier.
Example 3 — one million headless browser pages per month at 2 GB and 10 seconds per page uses 1,000,000 × 10 × 2 = 20,000,000 GB-seconds. After the free tier, 19,600,000 × $0.0000133334 = $261.33.
These figures exclude CloudWatch logs, S3 storage, data transfer, proxies and scraping API credits. Treat the unit prices as the captured reference values and confirm current AWS pricing before budgeting.
Java on Lambda
Python is the primary path here, but Java remains a good option for teams already on the JVM or wanting stronger structure around parsing, data models and API responses. The Java version uses Java 21, Maven, java.net.http.HttpClient, Jsoup, Jackson and Lambda SnapStart.
The logic mirrors the Python scraper: receive a Lambda event or API Gateway request, fetch the Books to Scrape page with the built-in HttpClient, parse book cards with Jsoup and return the records as JSON. The SAM resource sets 512 MB, a 30-second timeout, AutoPublishAlias: live and SnapStart applied to published versions. AutoPublishAlias matters because SnapStart applies to published versions, not to $LATEST. Build locally with Maven, invoke through the live alias, and if an HTTP API route is exposed, open the generated endpoint to see the JSON response.
Production checklist
Before running a Lambda scraper in production, check these eight things.
- Keep HTTP timeouts below the Lambda timeout. If the function timeout is 30 seconds, the outbound request should usually time out earlier, for example at 10 or 20 seconds, leaving time for parsing, S3 writes, logging and a clean error response.
- Reuse clients at module level. Create requests.Session(), boto3 clients, database clients and browser objects outside the handler so warm invocations skip repeated setup.
- Keep credentials out of environment variables. Plain, non-sensitive settings are fine as environment variables, but API keys and other secrets should be kept in SSM Parameter Store or Secrets Manager. The Lambda environment should reference only the parameter name or secret ARN, not the secret value.
- Handle SQS partial batch failures. Enable ReportBatchItemFailures on the event source mapping and return only the failed message IDs so Lambda does not retry the whole batch.
- Route exhausted retries to a dead letter queue. Retrying on its own will not make a persistently failing job visible. Once retries are used up, move the job to a DLQ so you can examine the payload, find broken selectors or blocked targets, and replay only the jobs that are safe to retry.
- Use CloudWatch Logs Insights and X-Ray. Log the target URL, status code, duration, retry count and output key, and use X-Ray to separate Lambda time from network latency, S3 writes, SSM calls and downstream API calls.
- Cap reserved concurrency. Lambda can scale quickly, but the target site may not appreciate it; reserved concurrency gives a hard upper bound and protects both the target and your account.
- Respect robots.txt, terms of service and privacy laws. Scraping law is fact-specific, and cases such as hiQ v. LinkedIn, Meta v. Bright Data and Van Buren v. United States show why public access, authentication, authorization and data use all matter. Review the target's rules before scraping.
Where to go next
Lambda is a strong fit for short, scheduled or event-driven scrapers that do not need server management. Its limits show up when pages need long browser sessions or anti-bot handling, and that is the point at which a hosted scraping API or a different compute service becomes the sensible next step. Start with the static path, prove the parser, then add rendering, proxies or an API only when the target demands it.
| Good fits | Poor fits |
|---|---|
| Scraping a static product-listing page on a schedule | A crawler that needs to run for hours |
| Processing one URL, or a small batch of URLs, from SQS | A logged-in browser session that must stay alive between requests |
| Calling a scraping API and writing the response to S3 | A high-volume always-on crawler (ECS, Fargate, EC2 Spot, or AWS Batch would be cheaper) |
| Retrying failed URLs independently without rerunning the whole crawl | A crawler that handles blocks, captcha handling, or browser fingerprint management |
This is a design-fit reference, not a benchmark of any specific site.
| Limit | Value |
|---|---|
| Maximum runtime | 900 seconds, or 15 minutes |
| Memory | 128 MB to 10,240 MB |
| /tmp storage | 512 MB to 10,240 MB |
| Default regional concurrency | 1,000 concurrent executions |
| ZIP package size | 50 MB zipped, 250 MB unzipped |
| Container image size | 10 GB uncompressed |
| Synchronous request and response payload | 6 MB each |
Quotas are published by AWS and change over time; new accounts start with reduced concurrency and memory quotas that AWS raises automatically based on usage. Check the current Lambda quotas documentation before sizing a workload.
| Scenario | Use BeautifulSoup | Use a browser or scraping API |
|---|---|---|
| Data availability | The target returns the data in the initial HTML | The data appears only after JavaScript rendering |
| Request type | The page can be fetched with a normal HTTP request | The site requires interaction before the data loads |
| Data extraction | You can identify the fields with stable CSS selectors | You need screenshots or the fully rendered DOM |
| Browser features | You do not need cookies, sessions, or browser actions | You need sessions or browser actions |
| Blocking behavior | — | The site employs anti-bot measures |
Use this as a triage table; a single target can move between columns as its rendering or protection changes.
| Topic | What to do |
|---|---|
| Python scrapers | Keep dependencies small, reuse HTTP sessions outside the handler, and avoid loading large files during import. |
| Java scrapers | Keep initialization predictable and use SnapStart when startup time becomes noticeable. |
| SnapStart | Use it through a published version or alias, not only $LATEST. It is especially useful for Java functions. |
| arm64 | Use it when your dependencies support it. For simple Python scrapers, it is usually a good default. |
| Browser scraping | Test carefully. Playwright, Chromium, native libraries, and container images can change cold start behavior. |
Cold start impact varies by runtime, package size and target latency; measure on your own workload.
Frequently asked questions
Is BeautifulSoup illegal?
No. BeautifulSoup is an open-source Python library for parsing HTML. Whether a particular scrape is lawful depends on the target's terms, robots.txt, privacy law and how the data is used, not on the parsing library.
How do you use Lambda for web scraping?
Create a Lambda handler that fetches one or a few URLs, parses the responses and stores the results. Use EventBridge for schedules, SQS for queues and S3 for storage.
Can AWS Lambda run locally?
Yes. With the AWS SAM CLI you can test functions locally using sam local invoke, or run local API routes with sam local start-api.
What is the maximum runtime of an AWS Lambda function?
An AWS Lambda function has a hard ceiling of 15 minutes, or 900 seconds. When a job needs longer, either split it into smaller pieces using SQS or Step Functions, or move the workload to a service built for longer-running processes.
Should I use Java or Python for Lambda scraping?
Python is generally the more modern and popular choice for Lambda scraping because of its simplicity and support. Java is a reasonable choice if your team already works with the JVM or benefits from features such as SnapStart.
{
"url": "https://books.toscrape.com/",
"limit": 10
}
{
"statusCode": 200,
"headers": {
"Content-Type": "application/json"
},
"body": {
"source_url": "https://books.toscrape.com/",
"count": 3,
"items": [
{
"title": "A Light in the Attic",
"price": "£51.77",
"availability": "In stock",
"rating": "Three",
"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
},
{
"title": "Tipping the Velvet",
"price": "£53.74",
"availability": "In stock",
"rating": "One",
"url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
},
{
"title": "Soumission",
"price": "£50.10",
"availability": "In stock",
"rating": "One",
"url": "https://books.toscrape.com/catalogue/soumission_998/index.html"
}
]
}
}
{
"url": "https://books.toscrape.com/",
"limit": 10
}
{
"url": "https://books.toscrape.com/",
"render_js": false,
"extract_rules": {
"title": "h1"
}
}
{
"source_url": "https://books.toscrape.com/",
"status_code": 200,
"content_type": "application/json; charset=utf-8",
"body": "{\"title\":\"All products\"}"
}