Skip to content

Kotlin Web Scraping: Fetch, Parse, and Extract Data

Kotlin is a practical language for web scraping because it runs on the JVM, interoperates fully with Java libraries, and adds null safety that helps when page structures change. This guide walks through a complete scraper using OkHttp for HTTP requests and Jsoup for HTML parsing, then shows how to add pagination, retries, and safe field extraction. It also explains when a static HTML scraper is enough and when JavaScript rendering or a scraping API becomes necessary.

Quick answer

A minimal Kotlin scraping stack has three parts: an HTTP client to fetch the page, an HTML parser to query the response, and a data model to hold the extracted fields. OkHttp and Jsoup cover most static sites. If the target page renders content with JavaScript or blocks automated requests, add a headless browser or route requests through a scraping API that handles rendering and anti-bot friction.

What Kotlin web scraping involves

Web scraping in Kotlin means requesting a public page, reading its HTML, and selecting the values you need. Typical uses include price tracking, job listing collection, content monitoring, and dataset assembly. The workflow is predictable: send an HTTP request, parse the returned HTML into a queryable document, extract fields, and store the results as JSON, CSV, or database rows. Kotlin's JVM foundation means the same mature Java libraries used in Java scrapers are available without adaptation. Null safety is especially useful here because scraped pages often omit elements or change class names between runs.

Choosing the right Kotlin scraping tools

Tool choice should follow the target site's behavior, not the other way around. Use the table below as a starting decision guide.

Tool selection by site behavior

The table maps common page conditions to the smallest stack that can handle them. Start with the simplest row that matches your target and escalate only when the extracted HTML is missing data or requests are being rejected.

Parsing HTML with Jsoup in Kotlin

Jsoup loads HTML into a Document object and lets you query it with CSS selectors. In Kotlin, the common pattern is to call selectFirst for a single element and then read either its text or an attribute such as href, src, or title. Using selectFirst instead of select().first() avoids an exception when the selector does not match. The same approach extends to product links and image sources: read the attribute, then convert relative paths to absolute URLs with absUrl.

Other parsing and rendering options

Jsoup covers most HTML extraction needs, including CSS selectors and XPath-style queries. If you are working inside an existing Java parsing stack, TagSoup, NekoHTML, or the JDK's DOM and XPath APIs are alternatives, though they tend to be more verbose. For JavaScript-rendered pages, HtmlUnit is lighter than a full browser, while Playwright and Selenium handle modern sites more reliably. Playwright is often faster and more consistent for rendering; Selenium has broader ecosystem support and more examples. Helper libraries such as kotlinx.serialization for JSON output, a CSV writer for exports, and retry or rate-limiting utilities become useful as the scraper grows beyond a few pages.

Step-by-step Kotlin scraping tutorial

The walkthrough uses Books to Scrape, a practice site with a stable layout. The goal is to fetch the homepage, parse every book card in the list, and print each book's title, price, availability, rating, product URL, and image URL. The flow is linear so each step can be checked independently.

Step 1: Create the project and add dependencies

Use a Gradle Kotlin DSL project and add OkHttp for HTTP requests and Jsoup for HTML parsing. The example works in a Kotlin CLI application, a backend service, or any JVM project. Java 11 or newer is recommended because modern HTTP clients and libraries expect it.

Step 2: Fetch the page HTML

Send a GET request to https://books.toscrape.com/ and read the response body as a string. That string is the input to the parser. Check the response status before parsing, and treat an empty body as a failure rather than passing it to Jsoup.

Step 3: Parse the HTML and select book cards

Books are listed inside an ol.row container. Each book is an li element wrapping an article.product_pod. The selector ol.row > li targets only the book cards and avoids unrelated list items on the page.

Field and selector map for each book card

The table below shows where each field lives in the card markup and how the example handles a missing value. It is a practical reference for adapting the scraper to a different site: replace the selector, keep the null-safe read pattern.

Step 4: Extract fields and print results

Loop over every li inside ol.row and extract each field with a CSS selector. When a selector does not match, fall back to a safe default instead of letting the scraper crash. The title is read from the title attribute rather than the visible link text, because the visible text is often truncated. Availability text is normalized by collapsing whitespace. The rating is encoded in the class name of p.star-rating, so the scraper checks for One through Five. Product and image links are converted to absolute URLs with absUrl.

Adding pagination

Most real sites spread data across multiple pages. Pagination usually works by adding a page number to the URL or by following a next link in the HTML. The loop stays the same: fetch a page, extract data, find the next page, and stop when no next link exists. A key detail is parsing each page with its own URL as the base, so relative pagination links resolve correctly. A short pause between requests avoids hitting the site too quickly.

Error handling and troubleshooting

Scraping fails for predictable reasons: timeouts, changed HTML, blocked requests, and empty responses. A good scraper expects these and fails in a controlled way, producing useful logs instead of crashing. Check status codes, set timeouts, and retry only temporary failures such as 429 or 5xx responses. Keep retries bounded so the site is not hammered.

Common failure modes and what to do

The table separates failures by symptom so you can fix the right layer. A 429 or 403 is a request-behavior problem, not a selector problem. Empty HTML on a dynamic page means the content never arrived in the response. A selector that returns nothing usually means the layout changed. If the HTML does not match what the browser shows, dump the response to a file and inspect it before changing selectors, because the server may have returned a JavaScript-required page, a cookie-consent page, or a bot check.

Make parsers null-safe and resilient

Markup shifts without warning, so a parser that treats every selector as guaranteed will fail the moment a price disappears or a class is renamed. Two habits prevent most of that pain: query with selectFirst so a missing match returns null instead of throwing, and chain safe calls with a field-appropriate default. Wrapping those reads in small helpers that substitute a default for blank text or attributes keeps extraction readable and stops one absent element from aborting the whole run.

Log what matters

When a run fails, you need quick answers. Log the URL, the status code, and which selector failed. If parsing breaks, save a small HTML snippet to a file so you can debug without re-running requests. Absolute URLs matter here too: sites often return relative paths such as catalogue/... or media/cache/..., and printing them as-is makes the output hard to use. Parse with a base URL and use absUrl so every link is complete.

Resilient scraper checklist

Use this checklist before scaling a Kotlin scraper beyond a few pages.

Scrape responsibly

Collect public data and check the site's terms of service and robots.txt. Even where robots.txt is not law, it signals what the operator expects. Avoid aggressive request rates and unnecessary load. Responsible scraping keeps projects stable and reduces legal and ethical risk.

Conclusion

Kotlin is a strong choice for web scraping when you want clean code, strong typing, and access to the Java ecosystem. With OkHttp and Jsoup you can go from a simple request to structured data in a few clear steps. Start small, scrape responsibly, and add rendering or a scraping API only when the target site requires it. Keeping the scraper simple, readable, and easy to maintain matters more than adding tools you do not need.

Tool selection by site behavior
Site conditionMinimum stackWhy
Static HTML with data in the responseOkHttp + JsoupThe HTML already contains the fields; CSS selectors are enough.
JavaScript-rendered contentHeadless browser (HtmlUnit, Playwright, or Selenium) + parserThe raw response is missing data that only appears after scripts run.
Bot protection, CAPTCHAs, or rate limitsScraping API or proxy/rendering layer + parserRequest behavior and rendering are handled before HTML is returned.
Large project needing timeouts, retries, and headersKtor Client or OkHttp with explicit configurationA client that supports these cleanly saves maintenance pain later.

Tool suitability depends on the target site; this table is a starting decision guide, not a guarantee for any specific site.

Field and selector map for the Books to Scrape card
FieldSelector or sourceExtraction detailMissing-value behavior
Titlearticle.product_pod h3 aRead the title attribute, not the visible link text, to avoid truncation.Blank title causes the card to be skipped.
Pricearticle.product_pod .product_price .price_colorRead text content and trim.Empty string.
Availabilityarticle.product_pod .product_price .instock.availabilityRead text, collapse whitespace, and trim.Empty string.
Ratingarticle.product_pod p.star-ratingRead class names and match One through Five."Unknown".
Product URLhref attribute of article.product_pod h3 aConvert with absUrl using the page base URL.Empty string.
Image URLsrc attribute of article.product_pod .image_container imgConvert with absUrl using the page base URL.Empty string.

Selectors match the practice site's layout at the time of writing; other sites require their own selectors.

Common failure modes and what to do
SymptomLikely causeFirst action
429 Too Many RequestsRequest rate is too highSlow down, add delays, rotate IPs, or use an API that handles rate limiting.
403 ForbiddenHeaders look bot-like or IP is blockedImprove request behavior, use realistic headers, or rotate proxies.
Empty HTML on a dynamic pageContent is rendered by JavaScriptUse a headless browser or a scraping API that renders pages.
Selectors return nothingPage layout changedRe-check the HTML structure and update selectors.
HTML does not match the browserServer returned a JavaScript-required, cookie-consent, or bot-check pageDump the HTML to a file and inspect it before changing selectors.

These are common symptoms; the correct fix depends on the specific response received.

Resilient scraper checklist
CheckWhy it matters
Set connect and read timeoutsPrevents a stalled request from blocking the run.
Retry only 429 and 5xx responses, with a capHandles temporary failures without hammering the site.
Use selectFirst and safe callsAvoids crashes when a selector stops matching.
Provide sensible defaults for missing fieldsKeeps partial records usable.
Parse with a base URL and use absUrlProduces complete links instead of relative paths.
Log URL, status code, and failed selectorMakes failures diagnosable without guesswork.
Save HTML snippets for debuggingLets you inspect the actual response without re-fetching.
Check terms of service and robots.txtKeeps scraping within the site operator's expectations.

A practical checklist, not a guarantee that every site will be scrapeable.

Frequently asked questions

Is Kotlin good for web scraping?

Yes. Kotlin runs on the JVM, so it can use mature Java libraries such as Jsoup and OkHttp. Its null safety and concise syntax make scrapers easier to write, debug, and maintain as they grow.

Do Kotlin developers use Java scraping libraries?

Yes. Kotlin is fully interoperable with Java, so Jsoup, OkHttp, Selenium, and Playwright work without adaptation. That lets you reuse proven libraries while keeping Kotlin's cleaner syntax.

Can Kotlin scrape JavaScript websites?

Yes, but it usually needs extra tooling. Headless browsers such as Playwright or HtmlUnit can render pages before parsing. A scraping API can also handle JavaScript rendering and blocking automatically.

How do I parse HTML in Kotlin?

Jsoup is the usual choice. Load the HTML into a Document, query elements with CSS selectors, and read text or attributes while handling missing elements safely.

When should I use a scraping API instead of a plain HTTP client?

Use a plain HTTP client when the data is present in the initial HTML response. Consider a scraping API when the page requires JavaScript rendering, returns CAPTCHAs or 403 responses, or applies rate limits that would require proxy rotation and request-behavior tuning.

Why do my scraped links come out as relative paths?

Many sites return href and src values as relative paths. Parse the HTML with the page URL as the base and use absUrl to resolve them into full URLs. This also matters for pagination, where the next link is often relative.