BeautifulSoup Web Scraping Tutorial: Parse HTML with Python
This guide walks through a complete Python scraping workflow using Requests to fetch a page and BeautifulSoup to parse it. You will select elements by tag, class, ID and CSS selector, handle pagination, and export the results to CSV or JSON. The running example is Hacker News, a simple site that keeps the focus on parsing rather than anti-bot defenses.
Getting the HTML
The first step is to retrieve a page's contents with an HTTP GET request. The Python Requests library is a common choice for this. Install it with the exact command in code0001, then use requests.get() to fetch the target URL. The response object exposes status_code; a 200 means success, while other codes such as 404 indicate an error. Checking the status before parsing prevents crashes when a request fails due to a temporary outage, a wrong URL or a blocked IP. The example below uses Hacker News as the target and stores the raw bytes in html_content.
Command example
pip install requests==2.32.3Parsing the HTML with BeautifulSoup
BeautifulSoup turns a raw HTML string into a searchable tree. Install it with the exact command in code0002. The BeautifulSoup() constructor takes the HTML content and a parser name; "html.parser" selects the built-in parser. Pass response.content rather than response.text because the raw bytes avoid character-encoding problems. Once parsed, the soup object exposes useful shortcuts: soup.title returns the title tag, soup.title.string returns its text, len(soup.find_all("a")) counts links, and soup.get_text() returns visible text.
Command example
pip install beautifulsoup4==4.12.3Python example
import requests
from bs4 import BeautifulSoup
response = requests.get("https://news.ycombinator.com/")
html_content = response.content
# Use Beautiful Soup to parse the HTML
soup = BeautifulSoup(html_content, "html.parser")Targeting DOM elements
The DOM is the tree of HTML nodes that makes up a page. BeautifulSoup offers several ways to pick nodes from that tree. The right method depends on whether you need the first match or every match, and whether you want to express the target as a tag, an attribute or a CSS selector.
Finding by tag
find() walks the document from top to bottom and returns the first matching tag. find_all() returns a list of every matching tag, which you can iterate over. For example, soup.find("a") returns the first anchor, while soup.find_all("a") returns every anchor. The same pattern applies to span: soup.find("span") returns the first span, and soup.find_all("span") returns all spans so you can read each one's content or attributes.
Finding by class or ID
To select by class, pass class_ rather than class, because class is a reserved Python keyword. soup.find(class_="athing") returns the first element with that class, and soup.find_all(class_="athing") returns every element with it. To select by ID, pass id="pagespace" to find(). These attribute-based lookups are concise when the target markup has stable class or ID values.
Python example
soup.find(id="pagespace")CSS selectors
CSS selectors become valuable when elements lack reliable IDs or class names. A manual loop over table rows and their children can extract a score, but it is verbose and fragile. The selector td:nth-child(2) > span:nth-child(1) expresses the same target in one line: the first span that is an immediate child of a td, where that td is the second child of its parent tr. soup.select() returns every match, and soup.select_one() returns the first match or None.
Python example
soup.select("tr.athing")Python example
soup.select_one("tr.athing")Python example
soup.select_one("#pagespace")Debugging selectors in the browser
Browser developer tools make selectors easy to test. Open DevTools with F12, switch to the document tab, and press Ctrl or ⌘ + F to open the search bar. Enter any CSS expression such as html body and the browser highlights the first match; pressing Enter cycles through the rest. The reverse also works: right-click an element, choose Copy, then Copy Selector to place a selector on your clipboard. Treat copied selectors as a starting point because they are often absolute and brittle; verify them before relying on them in a script.
Advanced selector expressions
CSS selectors cover child and descendant combinators, attribute selectors and pseudo-classes. a > p matches paragraphs that are direct children of an anchor, while a p matches any paragraph descendant. Sibling combinators such as h2 ~ p select all paragraphs after an h2 at the same level, and h2 + p selects only the paragraph immediately following the h2. Attribute selectors such as p[data-test="foo"] match elements with a specific attribute value. Pseudo-classes such as :first-child, :last-child and :nth-child select elements by position; for example, section > p:nth-child(2) selects the second paragraph in a section.
Keeping selectors maintainable
Storing selectors in named constants at the top of a script makes DOM changes easier to fix. For example, a title selector and a score selector can be defined once and reused. This pattern keeps selector updates localized instead of scattered through the code.
Python example
TITLE_SELECTOR = "title"
SCORE_SELECTOR = "td:nth-child(2) > span:nth-child(1)"
...Traversing the DOM to collect articles
On Hacker News, each article sits inside a tr with the class athing. Selecting those rows with soup.find_all(class_="athing") gives a list of article containers. Within each row, the rank lives in a span with class rank, and the title and URL live in a span with class titleline. The URL is the href of the anchor inside titleline. A common mistake is forgetting that find() and select_one() return None when no match exists; always check for None before reading attributes. The example below builds a list of dictionaries with URL, title and rank for each article.
Python example
import requests
from bs4 import BeautifulSoup
# Send a GET request to the website
response = requests.get("https://news.ycombinator.com/")
html_content = response.content
# Use BeautifulSoup to parse the HTML content
soup = BeautifulSoup(html_content, "html.parser")
# Find all articles with the class 'athing'
articles = soup.find_all(class_="athing")Handling pagination
Pagination means repeating the scraping logic for each page until no more pages remain. Track progress with two variables: is_scraping, a boolean that stays True while pages remain, and current_page, an integer for the page being fetched. The Hacker News URL accepts a p parameter, so the loop requests https://news.ycombinator.com/?p={current_page}. A More link with class morelink appears on every page except the last. If morelink is present, increment current_page and continue; otherwise set is_scraping to False and report the totals.
Python example
is_scraping = True
current_page = 1Exporting the scraped data
Once the records are in a list, convert them to a format that is easy to read and analyze. CSV suits tabular data and spreadsheets; JSON suits structured data and web applications. The examples below write the scraped_data list to hn_articles.csv and hn_articles.json.
Export to CSV
The csv module provides the tools for CSV output. Open the file in append mode with newline="" and encoding="utf-8-sig". A csv.DictWriter maps each dictionary key to a column header. Check whether the file is empty before writing headers so repeated runs do not duplicate them, then call writerows() to write every record. The exact implementation is in code0035.
Export to JSON
JSON is a human-readable format for structured data. The json.dump() function takes the Python object to convert and a file object created with open(). Writing with indent=4 produces readable output. The exact implementation is in code0036.
Overcoming web scraping challenges
Real sites often deploy CAPTCHAs, fingerprinting and other anti-bot systems. Scraping also consumes bandwidth and requires ongoing maintenance. Hacker News is deliberately simple, which is why the example works without special handling. Harder targets need a combination of technical and ethical measures.
Rotate User-Agents
The User-Agent header identifies the client's operating system, vendor and version. A server that sees many requests from one User-Agent may block them. Rotating the User-Agent for each request reduces that risk.
Implement rate limiting
Anti-scraping systems watch request frequency. A single IP that sends too many requests too quickly can be flagged. Adding delays between requests gives the server time to process each one before the next arrives.
Use a proxy server
A rotating proxy sits between your script and the target site, routing requests through a pool of IP addresses that change periodically or after a set number of requests. This spreads traffic across addresses instead of concentrating it on one.
Add retry logic
Retry logic improves reliability by handling temporary failures such as server downtime or dropped connections. Retrying a failed request a limited number of times before giving up increases the chance of successful retrieval.
Respect robots.txt and terms
Check the target site's terms of service and follow the rules in its robots.txt file. Avoid collecting personal information without consent, which can violate privacy regulations. Ethical scraping combines technical measures with respect for these boundaries.
Final code
The complete script combines the pieces above: pagination, article extraction, retry logic and JSON export. It retries failed requests up to max_retries attempts with a delay between attempts, and it always pauses between page requests. The exact implementation is in code0037.
Conclusion
You have seen how Requests fetches HTML, how BeautifulSoup parses it, how find/find_all/select/select_one target elements, how pagination loops through pages, and how to export records to CSV or JSON. The same workflow applies to other sites, though harder targets may require the anti-bot measures described above.
| Method | Returns | Use when | None risk |
|---|---|---|---|
| find(tag) | First matching tag or None | You need one element and want the first in document order | Yes — check before reading attributes |
| find_all(tag) | List of all matching tags | You need every match to iterate over | No — empty list if none |
| find(class_=...) | First matching element or None | The target has a stable class | Yes — check before reading attributes |
| find(id=...) | First matching element or None | The target has a stable ID | Yes — check before reading attributes |
| select(css) | List of all matching elements | You need a CSS expression covering multiple matches | No — empty list if none |
| select_one(css) | First matching element or None | You need one element matched by a CSS expression | Yes — check before reading attributes |
Return shapes and None behavior reflect the BeautifulSoup methods described in the tutorial; verify against your installed BeautifulSoup version.
| Field | Location in markup | Extraction | Example value |
|---|---|---|---|
| rank | span with class rank inside tr.athing | article.find(class_="rank").getText().replace(".", "") | 1 |
| title | span with class titleline inside tr.athing | article.find(class_="titleline").getText() | Article headline |
| URL | anchor inside span.titleline | article.find(class_="titleline").find("a").get("href") | https://example.com/story |
Example values are illustrative; the actual values depend on the page at scrape time.
Frequently asked questions
Why pass response.content instead of response.text to BeautifulSoup?
response.content holds the raw HTML as bytes, which BeautifulSoup can decode more reliably than the text representation in response.text. Using bytes avoids character-encoding problems that can garble non-ASCII text.
Why does BeautifulSoup use class_ instead of class?
class is a reserved keyword in Python, so BeautifulSoup accepts class_ as the keyword argument for class-based lookups.
What happens when find() or select_one() finds nothing?
Both return None. Reading an attribute from None raises an error, so check the result before using it. find_all() and select() return an empty list instead, which is safe to iterate over.
When should I use select() instead of find_all()?
Use select() when a CSS expression captures the target more precisely than a tag or class lookup, especially when the element has no stable ID or class. find_all() is simpler when a tag or class alone identifies the target.
How does the pagination loop know when to stop?
The loop looks for a More link with class morelink. If it exists, the script increments current_page and continues. If it is absent, the script sets is_scraping to False and exits.