Skip to content

Python Web Crawler Guide: From First Script to Scrapy

A Python web crawler is a script that starts from one or more URLs, downloads pages, finds links, and keeps moving through a site. The first version can be small: fetch a page, parse it, follow the next link, save what you found. The harder questions come later: how many pages should it visit, how should it behave when a request fails, and when does a simple loop stop being enough? This guide follows that progression. It begins with a requests and BeautifulSoup crawler, then shows how Scrapy adds scheduling, retries, throttling, pipelines, and structured exports.

What a Python crawler actually does

A crawler is a link-following loop. It begins with seed URLs, fetches the HTML, extracts links, adds unseen URLs to a queue, and repeats until the queue is empty or a limit is reached. The crawler discovers pages; a scraper extracts specific values from those pages, such as titles, prices, or reviews. The same script often does both jobs.

The basic loop is:

  1. Start from one or more seed URLs.
  2. Fetch the page HTML.
  3. Pull out links.
  4. Add new links to a queue.
  5. Stop when nothing new remains or a crawl limit is reached.

A useful crawler adds three things to that loop: a visited set so it does not revisit the same URL, a per-domain limit so it does not wander forever, and a delay so it does not overload the server.

Where crawlers are useful

Search engines use crawlers to discover pages, follow links, and store page content so it can be indexed. Smaller teams build similar internal crawlers to make documentation searchable.

SEO work is another common use: crawling partner sites or your own site to map backlinks, find broken internal links, and check whether important pages are buried too deep. Price monitoring uses crawlers to check product pages for pricing, plans, discounts, stock, and shipping changes. Research projects use them to collect public listings, statements, job posts, or policy pages. Image downloading is a simpler case: fetch binary content and save it locally.

For JavaScript-heavy product pages, plain HTML fetching may not expose the data. A rendering service or a browser automation tool becomes part of the crawl path.

Starter crawler with requests and BeautifulSoup

The first working crawler can use requests to download HTML and BeautifulSoup to parse it. A clean learning target is the Hacker News front page: the structure is static and the story titles sit inside elements with the titleline class.

The script has three jobs: fetch the page, parse the story titles and links, and print the first ten results. The fetch step should include a timeout and a User-Agent header. The parse step selects .titleline, finds the first anchor inside each node, and reads its text and href. The output is a numbered list of titles and URLs.

That pattern — fetch, parse, repeat — is the foundation. Once it works, add filtering. For example, keep only external http or https links and skip internal item?id= links. Save the filtered results to a file so the crawl output can be reused.

When the data comes from JSON instead of HTML

Many modern sites load data from API endpoints that return JSON. If the endpoint is available, the crawler becomes simpler: request the URL, call response.json(), and work with a Python dictionary or list. This is often faster and more stable than parsing HTML because the response is already structured.

A practical crawler uses both paths: prefer an API when one exists, and fall back to HTML parsing when it does not. The decision is not about which method is more advanced; it is about which source exposes the data reliably.

Where a basic crawler breaks down

A small script is fine for a demo, but real sites expose its limits. The first problem is speed: one request at a time is slow, and a failed request can stop the run. Production crawlers need retries with backoff, rate limiting, and some form of parallelism.

The second problem is link quality. Simple crawlers revisit pages, follow repeated links, and crash on broken URLs or 404 responses. A visited set, duplicate skipping, and a log of failed links make large crawls more reliable.

The third problem is permission. A basic crawler may ignore robots.txt and crawl-delay rules. Some sites specify how often and how deeply they allow crawling. Ignoring those rules can lead to blocked IPs, CAPTCHAs, or rate limits. Check the site's robots policy before running a crawl.

How Scrapy changes the crawler

Scrapy is a Python framework built for crawling at scale. It handles concurrency, retries, deduplication, and proxy management, so the crawler code can focus on what to crawl and what to extract.

The main parts are:

  • Spiders define what to crawl and how to parse it.
  • Schedulers decide which URLs to visit next.
  • Downloaders fetch pages, optionally through proxies or headless browsers.
  • Pipelines clean, transform, and save scraped data.
  • Middlewares handle retries, headers, user agents, and delays.

Those parts work together so a crawl can cover thousands of pages without low-level async code.

Creating a Scrapy project and spider

Scrapy's command-line tool creates the project structure, settings, and sample spider. A typical setup uses uv to keep the environment isolated, then runs scrapy startproject and scrapy genspider.

A spider needs three basics: a name, allowed_domains, and start_urls. The parse() method receives a response for each downloaded page. CSS selectors such as .titleline a:first-child find the title and link. The spider yields a dictionary for each story, and Scrapy collects those items for export.

To follow links, the spider can use response.follow() with a callback. A listing spider can walk the "More" pagination link and open a limited number of comment threads. Safety limits such as max_listing_pages and max_threads_to_follow keep the crawl from running indefinitely.

Selectors, exports, and pipelines

Scrapy supports CSS selectors and XPath. CSS is shorter for simple layouts; XPath is more precise when the crawler needs to move up or sideways in the HTML tree. When testing a selector, limit the output to a few items first.

Exporting is built in. Running scrapy crawl with -O writes a fresh JSON or CSV file each time. Lowercase -o appends instead of overwriting. Scrapy converts each yielded dictionary into a structured row.

Pipelines handle cleanup before the data is written. A pipeline can trim whitespace from titles, convert relative links to absolute URLs with urljoin, coerce comment counts to integers, and drop items with missing titles. Pipelines are enabled in settings.py with ITEM_PIPELINES, where the number is the pipeline priority: lower numbers run first.

Responsible crawling at scale

Scaling a crawl is not only about speed. A site may not be built to handle hundreds of requests per second from one script. Responsible crawling balances efficiency with politeness.

A practical checklist:

  • Check robots.txt before crawling.
  • Add delays between requests.
  • Identify the crawler clearly in the User-Agent.
  • Limit concurrent requests per domain.
  • Distribute large crawls across multiple machines instead of one large bot.

Scrapy exposes these controls directly. ROBOTSTXT_OBEY is enabled by default in settings.py. DOWNLOAD_DELAY sets the wait between requests, CONCURRENT_REQUESTS_PER_DOMAIN limits parallel requests to the same site, and AUTOTHROTTLE_ENABLED lets Scrapy adjust speed based on server response times. A clear User-Agent string helps site owners identify the crawler.

For distributed crawls, Scrapy can run across multiple instances with tools such as Scrapyd for remote spider management and Scrapy-Redis for shared queues. More bots do not mean more permission; site limits and available APIs still apply.

When JavaScript and anti-bot systems enter the picture

Static pages are the easy case. JavaScript rendering, rate limits, browser fingerprints, and anti-bot systems turn a simple script into an infrastructure project. A rendering API or a managed scraping service can handle the browser, proxy rotation, and retry logic so the crawler code stays focused on extraction.

That tradeoff is useful when the crawl targets product pages, news sites, search results, or any site that loads data dynamically. A managed service can be used as a full replacement for browser-based scraping or mixed into an existing pipeline only for difficult pages.

Decision checklist: which crawler setup fits

Use this checklist before choosing a stack.

  • If the site is static and the crawl is small, start with requests and BeautifulSoup.
  • If the crawl needs concurrency, retries, deduplication, and structured exports, move to Scrapy.
  • If the data is available from a JSON endpoint, prefer the endpoint over HTML parsing.
  • If the page requires JavaScript to show the data, add a rendering tool or a managed API.
  • If the site publishes robots.txt or crawl-delay rules, configure the crawler to respect them.
  • If the crawl spans many domains, set per-domain limits and consider separate crawlers for large sites.
  • If the output will feed analytics or machine learning, add a pipeline to clean and validate fields before export.

Failure distinctions that change the fix

Not every failed crawl has the same cause. The fix depends on which failure appears.

  • Duplicate URLs: the crawler revisits pages because it lacks a visited set. Fix: track seen URLs and skip duplicates.
  • Broken links or 404s: the crawler crashes or stops. Fix: catch request errors, log failed URLs, and continue.
  • Blocked requests: the site returns CAPTCHAs or rate limits. Fix: check robots.txt, slow the crawl, use a clear User-Agent, and consider a proxy or rendering service.
  • Empty data on a JavaScript page: the HTML does not contain the values. Fix: render the page with a browser tool or a managed API.
  • Messy exported fields: titles have extra whitespace, links are relative, or counts are strings. Fix: add a Scrapy pipeline to normalize and validate each field.

Field map for the Scrapy item and pipeline

The Hacker News spider yields a dictionary with these fields:

  • story_id: the story identifier from the row.
  • title: the story title text.
  • url: the story link.
  • discussion_url: the absolute URL of the comment thread.
  • comment_count: the number of comments, coerced to an integer.

The pipeline then:

  • Drops items with no title.
  • Trims whitespace from the title.
  • Converts url and discussion_url to absolute URLs with urljoin.
  • Converts comment_count to int, defaulting to 0 on failure.

That field map is the contract between the spider and the export. If a field is missing or malformed, the pipeline either fixes it or drops the item before it reaches the JSON or CSV file.

Crawler stack comparison
SetupBest forHandles concurrencyHandles retriesHandles pipelinesHandles JavaScript
requests + BeautifulSoupSmall static crawls and learningNoNoNoNo
ScrapyLarger crawls with scheduling and exportsYesYesYesVia middleware or rendering service
Managed rendering APIJavaScript-heavy or anti-bot pagesService-sideService-sideNoYes

Capability summary based on the described behavior of each setup; it is not a benchmark.

Scrapy settings for polite crawling
SettingPurpose
ROBOTSTXT_OBEYRespect the site's robots.txt rules
DOWNLOAD_DELAYWait between requests
CONCURRENT_REQUESTS_PER_DOMAINLimit parallel requests to the same site
AUTOTHROTTLE_ENABLEDAdjust speed based on server response times
USER_AGENTIdentify the crawler clearly

Settings are shown as described in the Scrapy configuration examples; defaults and behavior can vary by Scrapy version.

Frequently asked questions

What is the difference between web crawling and web scraping?

Crawling finds pages by following links automatically. Scraping extracts data from those pages, such as titles, prices, or reviews.

What are common use cases for Python web crawlers?

Common uses include indexing pages, monitoring prices, tracking SEO changes, collecting research data, and building datasets for analytics or AI.

How do I build a basic web crawler in Python?

Start with requests and BeautifulSoup: fetch a page, extract links, and loop through them. For larger crawls, Scrapy is the usual next step.

What advantages does Scrapy offer for web crawling?

Scrapy is fast, scalable, and built for serious crawling. It handles async requests, retries, deduplication, throttling, and data pipelines out of the box.

Do I need permission to crawl a website?

Not always, but check robots.txt, respect rate limits, and avoid crawling private or login-protected content without permission.

How do I handle JavaScript-heavy websites?

For JavaScript-heavy sites, use browser tools like Playwright or a rendering API to get fully rendered HTML.

Can I use AI or machine learning with crawled data?

Yes. Once the data is cleaned and structured, it can be used for analytics, machine learning, or LLM workflows.

How do I avoid getting blocked while crawling?

Use polite request rates, realistic headers, and delays between requests. For tougher sites, proxies or a managed rendering service can help.

{"type": "story", "story_id": "45706792", "title": "Show HN: Diagram as code tool with draggable customizations", "url": "https://github.com/RohanAdwankar/oxdraw", "source": "hackernews"},
{"type": "discussion", "story_id": "45710065", "story_title": "Asbestosis", "story_url": "https://diamondgeezer.blogspot.com/2025/10/asbestosis.html", "discussion_url": "https://news.ycombinator.com/item?id=45710065", "comment_count": 93}
{"story_id": "45711279", "title": "Let's Help NetBSD Cross the Finish Line Before 2025 Ends", "url": "https://mail-index.netbsd.org/netbsd-users/2025/10/26/msg033327.html", "discussion_url": "https://news.ycombinator.com/item?id=45711279", "comment_count": 55}