Parsing HTML with Regex in Python: When It Works and Where It Breaks
Regular expressions are a fast, dependency-free way to pull a known string pattern out of a page you have already downloaded. They suit simple, well-formed markup and narrow targets. They are not a general HTML parser, and this guide shows both sides using the same captured examples: a title extraction, a statistics search, link extraction, and two filtering tasks, followed by a case where the approach misses data.
What regex parsing actually does
A regular expression describes a sequence to search for inside text. Most general-purpose languages provide regex support natively or through a library, which is why it is attractive for a quick script: there is nothing extra to install. The Python module used throughout the examples is re. Its findall function returns every match for a pattern in a string, while re.compile turns a pattern string into a reusable pattern object you can search against many lines. The tutorial was run on Python 3.8.10 and is expected to work on Python 3 and later. See protected module code0001 for the minimal findall example.
Example 1: extracting a title from a simple file
The first example starts from a small local file, simple.html, shown in protected module code0002. The goal is to print only the text inside the title element. The script in protected module code0003 compiles a pattern for the opening title tag, reads the file line by line, and when a line matches, strips the opening and closing tags before printing. Running it, as shown in protected module code0004, produces the page title on its own. This works because the title sits on a single line in a predictable position. Use this shape of solution only when you can state, in advance, exactly what the surrounding markup looks like.
Example 2: downloading a page and finding one statistic
Before searching, the tutorial downloads a page to a local file. Protected module code0005 uses urllib.request.urlretrieve to fetch the PyPI homepage and write it to file.html in the same directory as the script. The captured page displays several statistics at once, including projects, releases, files, and users. In the downloaded markup, those figures appear as paragraph elements inside a statistics container, shown in protected module code0006. The captured values were 410,645 projects, 3,892,920 releases, 6,961,696 files, and 633,650 users. Treat those numbers as a snapshot from the capture, not as current figures.
Reading the statistics pattern
Because the word projects occurs elsewhere on the page, searching for the word alone would return too much. The script in protected module code0007 instead fetches the page as bytes and applies a pattern that requires a digit sequence, allows comma separators, requires whitespace, and ends with the word projects. Reading the pattern piece by piece: the leading b marks a bytes pattern, the digit token means one or more numbers, a comma is literal, the question mark makes the preceding comma optional, the whitespace token means one or more spaces, and projects is a literal word. findall returns a list, so the script iterates and decodes each match for printing. The captured run in protected module code0008 printed the projects figure from the snapshot. The lesson is to anchor your pattern to the full shape of the value, not to a single keyword.
Example 3: extracting http and https links
The same fetch-and-search structure can collect links. Protected module code0009 searches the downloaded bytes for href values beginning with http or https and prints each one, also returning them as a list. The pattern captures only the URL inside the quotes, so the output is a clean list of referenced addresses rather than whole tags. This is a good fit for regex because the target has a fixed shape: the attribute name, an equals sign, a quoted value, and a scheme prefix.
Example 4: filtering out empty tags
Some HTML elements are void elements: they cannot contain child nodes, they have a start tag but no end tag, and a browser will not complain if an end tag is present anyway. The filtering script in protected module code0010 downloads the page, copies it to an altered file, and for each tag name in its fixed list removes any line containing that tag's start tag, using a temporary file before replacing the output. Running the command in protected module code0011 leaves the original download in file.html and the filtered version in altered-file.html. Note the constraint: removal is line-based, so it assumes each void tag occupies its own line.
Example 5: filtering out comments
HTML comments begin with an opening marker and end with a closing marker, as shown in protected module code0012. The script in protected module code0013 applies the same line-based idea: it compiles a pattern for the comment opener, keeps only lines that do not match, and writes them to the altered file. Running it with the command given in the source removes comment lines from the downloaded content. As with void tags, this only works cleanly when comments occupy whole lines of their own.
Where regex fails: nested and irregular markup
The failure case uses a hand-written file, arbitrary-html-file.html in protected module code0014, with three sections. The first two sections each contain a simple second-level heading on one line. The third section's heading is different: it wraps a paragraph and a line-break element across several lines. The script in protected module code0015 searches for a heading that opens and closes on the same line. Its output, shown in protected module code0017, contains only the first two headings. The third heading is silently missed, not because the file is invalid, but because its structure differs from the pattern. Adding special-case rules for every variant is possible, yet there is no standard set of rules, so each new irregular page can break the parser again. For arbitrary or nested markup, a real HTML parser or browser-based tool is the safer choice.
Decision checklist
Use regex when all of these are true: the markup is simple and well-formed, the target has a fixed shape, and you can verify the output against the page. Reach for a dedicated HTML parser when any of these appear: nested elements at several levels, the same element written in different shapes, content split across lines, or attributes you need to read reliably. Before shipping a regex parser, run this checklist: confirm the target value is unique in context, test against a page where the same element appears in a different shape, check for missed matches rather than only wrong ones, and keep the downloaded HTML so failures can be reproduced. Regex remains useful for quick extraction, filtering, and pattern searches, but it should not be the only line of defence for unpredictable pages.
Frequently asked questions
When is regex enough for HTML, and when should I switch tools?
Regex is a text pattern, not a document model. It works when the markup you need is predictable and flat, such as a single-line title or a repeated number-plus-word pattern. It becomes unreliable when the same element can contain nested elements, line breaks, attributes in different orders, or other irregular structure, because a pattern written for one shape will not describe the others.
Why did the projects pattern need more than the word projects?
Check that the value is unique in context, not just present somewhere on the page. In the captured PyPI example, the word projects appears more than once, so the pattern also requires digits, optional comma separators, and whitespace before the word. Even then, treat the result as captured evidence from that page at that time, not as a live count.
What counts as an empty tag, and how was it filtered?
Void elements have a start tag but no end tag and cannot contain child nodes. The tutorial's filtering example uses a fixed list of such tag names and removes any line containing a matching start tag, writing the remaining lines to a new file. That line-based approach is only safe when each tag sits on its own line.
How were HTML comments removed in the example?
The example searches each line for the comment opener and keeps only lines that do not match. That removes whole lines containing comments, but it will not cleanly remove a comment that shares a line with real content or spans several lines.