Ignore Non-HTML URLs When Web Crawling
A crawler that follows every link will waste time on files it cannot parse as pages.
Filtering those URLs before a full download keeps the crawl focused on HTML, saves bandwidth, and prevents image, document, and other binary responses from entering a page-processing pipeline.
Two ways to filter before you download
Two practical approaches cover most cases. The first looks at the URL itself. The second asks the server what it will return, without fetching the full body. Choosing between them depends on how much you trust the URL naming and how costly a mistaken download would be.
Check the URL suffix first
Inspect the ending of the URL and compare it against a list of extensions you do not want to crawl. The example implementation in protected module code0001 builds an image-extension list and uses Python's os.path.splitext to extract the suffix from a sample logo.png URL, printing an abort message for a match and a continue message otherwise. This works well when sites use clear extensions, but it can miss files served without an extension or misjudge a URL whose extension does not match its actual content.
Confirm with a HEAD request and Content-Type
Send a HEAD request and read the Content-Type response header. Because a HEAD request retrieves metadata rather than the whole response, it is a lighter probe than a GET. In protected module code0002, a HEAD request to the site base returns text/html; charset=utf-8, so the code proceeds to a full GET, while a PNG example returns image/png and would be aborted. Treat a non-HTML Content-Type as a signal to skip the complete request.
A practical decision checklist
Run the suffix check as a fast first pass, then use the HEAD check for URLs that pass or have ambiguous endings. Log the URL, the suffix decision, and the Content-Type you observed so you can audit why a link was skipped. Revisit your extension list when a site introduces new file types, and do not assume every server returns an accurate Content-Type.
Frequently asked questions
Which check should I run first?
Use the suffix check first for obvious file extensions, then confirm with a HEAD request when the URL gives no clear extension or the server may serve a different type than the name suggests. The suffix check is cheap and local; the HEAD check relies on the server's declared Content-Type.
Why use a HEAD request instead of downloading the page?
A HEAD request asks for headers only, so it can reveal Content-Type without downloading the full body. If that header does not indicate HTML, you can skip the full GET request for that URL.