Skip to content

Async Web Scraping in Python with asyncio and aiohttp

Sequential scraping wastes time: each request waits for the last to finish before the next begins. Python's asyncio lets you keep the readable, top-to-bottom shape of ordinary code while the runtime interleaves many network waits at once. This guide walks through the core asyncio building blocks, then applies them to a two-stage pipeline that first collects article links and then fetches and parses each page concurrently with aiohttp and BeautifulSoup.

Why asyncio fits scraping work

You do not need to add a package to get concurrency: asyncio already lives inside Python's standard library. It gives you the same async and await vocabulary that JavaScript developers use, and it sidesteps the bookkeeping that comes with hand-managed threads. The mental model is simple — write ordinary-looking functions, flag the parts that wait on the network with await, and let the runtime decide what runs while something else is pending. Three pieces make this work: a coroutine (any function declared with async def), a task (a coroutine you hand to asyncio.create_task() so it gets scheduled), and a future (the placeholder for a result that has not arrived yet). The event loop is the engine that drives all three.

How async and await change execution order

Before async/await, slow operations were often wired together with nested callbacks — the pyramid of doom — which grew hard to read in both JavaScript and Python. The async/await style replaces that nesting with a linear function body. The example below defines a coroutine that pauses for one second and then prints, and schedules it several times over. Called one after another, the pauses would add up to roughly ten seconds; scheduled as tasks and awaited together, the script finishes in about one second because the pauses overlap instead of stacking. The top-level code sits inside an async main() that run() starts, which is what brings the event loop to life.

Choosing the right asyncio primitive

The asyncio toolkit is small but easy to mix up. Use the table below as a quick reference when you are deciding which call to reach for in a scraping pipeline: gather when you want all results together, as_completed when you want them as they finish, wait_for when a hung request should be cancelled, and Semaphore when you need to cap how many requests are in flight.

Stage one: crawl the link list

The worked example compiles a list of programming-language creators. Stage one fetches a Wikipedia index page and collects the article links; stage two visits each article and reads its infobox. Install the two libraries first, then save the crawler into a file such as scraper.py. The crawler sets a BASE_URL constant, wraps an aiohttp ClientSession in a fetch() helper that hands back the page text, and defines a crawl() coroutine. That coroutine parses the index with BeautifulSoup, gathers links through the CSS selector div.div-col a, and resolves each href against the base URL. main() awaits crawl() and pretty-prints the collected links.

Command example

pip install aiohttp
pip install BeautifulSoup

Stage two: scrape each page concurrently

The scrape() coroutine receives a single link, awaits fetch() for its HTML, and parses that HTML with BeautifulSoup. It reads the language name from caption.infobox-title, then hunts for the creator inside the infobox table with a :-soup-contains pseudo-selector that matches a header link whose text is Developer or Designed by. The result is a two-item list of name and creator. When no infobox title turns up, the function returns an empty list instead. In main(), every link is turned into a scheduled scrape() call collected in a task list, and asyncio.gather executes the whole batch in parallel and returns the combined results.

Field map and failure distinction

Two selectors drive the extraction, and they fail differently. The name selector caption.infobox-title is the gate: if it returns nothing, scrape() returns an empty list and no creator lookup happens. The creator selector table.infobox tr:has(th a:-soup-contains("Developer", "Designed by")) td only runs after a name is found; if it returns nothing, the creator value stays None but the name is still returned. That distinction matters when you audit results: an empty list means the page had no recognizable infobox, while a name paired with None means the infobox existed but the developer row did not match.

Before you run it: readiness checklist

The pipeline works, but a production scraper needs more than concurrency. Check each of these before pointing it at a real site:

  • User-agent: pass a headers dictionary to session.get so the request identifies itself as you intend.
  • Throttling: add reasonable delays so you do not overwhelm the target server.
  • IP limits: some sites restrict by geography or cap concurrent or total requests from a single IP.
  • JavaScript: single-page apps and JS-heavy sites need a real JavaScript engine, which a plain aiohttp fetch will not provide.
  • Error handling: gather propagates exceptions, so wrap individual scrape() calls if you want one bad page to not sink the batch.

What the exercise demonstrates

The lesson is that you can run many tasks at once in Python without committing to full multi-threading. Your scraping logic stays linear and easy to follow thanks to async/await, yet the pipeline still makes efficient use of the machine's CPU cores. The two-stage shape — collect URLs first, then fan out with gather — carries over to any site where you need a list before you can fetch per-page detail.

asyncio features and functions reference
FeatureDescription
as_completed()Executes a list of coroutines and returns an iterator object for their results
create_task()Executes the given coroutine concurrently in the context of a task
ensure_future()Accepts a list of different parameters types and verifies they are all Future-like objects
gather()Concurrently executes the passed awaitables using tasks and returns their combined results
get_event_loop()Provides access to the currently active event loop instance
run()Executes the given coroutine - typically used for the main function
sleep()Suspends the current task for the indicated number of seconds - akin to the standard time.sleep() function
wait()Execute a list of awaitables and waits until the condition specified in the when parameter is met
wait_for()Similar to wait , but cancels the future when a timeout occurs
Lock()Provides access to a mutex object
Semaphore()Provides access to a semaphore object
Task()The task object, as returned by create_task()

Reference table of asyncio primitives as described in the guide; consult the official Python asyncio documentation for current signatures and behavior.

Frequently asked questions

Do I need to install anything for asyncio itself?

No. asyncio is part of Python's standard library. The example does require aiohttp for asynchronous HTTP calls and BeautifulSoup for parsing, both installed via pip.

Why does the example finish in about a second instead of ten?

Each scheduled function still pauses for one second, but the tasks run concurrently, so their waits overlap rather than stacking.

What is the difference between create_task and gather?

create_task schedules a single coroutine for concurrent execution and returns a task. gather takes a collection of awaitables, runs them concurrently, and returns their combined results.

Why does scrape() sometimes return an empty list?

When the caption.infobox-title selector finds nothing, there is no name to report, so the function returns an empty list and skips the creator lookup.

Can this pipeline scrape JavaScript-heavy sites?

Not as written. aiohttp fetches the raw HTML; sites that render content with JavaScript need a JavaScript-capable environment to produce the data you want.

$ python3 scraper.py
['https://en.wikipedia.org/wiki/A_Sharp_(.NET)',
 'https://en.wikipedia.org/wiki/A-0_System',
 'https://en.wikipedia.org/wiki/A%2B_(programming_language)',
 'https://en.wikipedia.org/wiki/ABAP',
 'https://en.wikipedia.org/wiki/ABC_(programming_language)',
 'https://en.wikipedia.org/wiki/ABC_ALGOL',
 'https://en.wikipedia.org/wiki/ACC_(programming_language)',
 'https://en.wikipedia.org/wiki/Accent_(programming_language)',

 LOTS MORE