How to get file type of an URL in Python?
When crawling, you often need to know whether a link points to a page, an image, a document, or something else before fetching it.
Python gives you two practical routes: infer the type locally from the URL itself, or ask the server with a lightweight request. This guide explains when each route is reliable, what each one returns, and how to combine them in a scraping workflow.
Two ways to identify a URL's file type
Two approaches cover most crawling decisions. The first reads the URL string alone. The second contacts the destination and reads its declared type. Choosing between them depends on whether the URL is trustworthy and whether you can afford a network request for every link.
Method 1: Guess from the extension with mimetypes
Option 1: Use the mimetypes module
Python ships with mimetypes, so no extra installation is needed. Pass a URL to its type-guessing function and it maps the file extension to a MIME type. A URL ending in .pdf is recognised as application/pdf, which is enough to route a PDF to a document handler or skip it during HTML crawling.
The limitation is obvious once you see the failure case: a URL with no extension, such as http://example.com/file, yields (None, None). Extension-less routes, query-string downloads, and misleading extensions can all defeat this method. Treat its answer as a guess based on naming, not as proof of what the server holds.
The complete worked example for this method is provided in protected module code0001, including both the successful PDF case and the extension-less failure case.
Method 2: Ask the server with a HEAD request
Option 2: Send a HEAD request and read Content-Type
A HEAD request asks the server for headers without downloading the full response body. The Content-Type header then states what the server says it is sending. That makes this method useful for extension-less URLs and for checking that a link really serves the type its name suggests.
In the worked examples, one response reports text/html; charset=utf-8, identifying a web page, while another reports image/png, identifying a PNG image. Note that the header may include extra parameters such as a charset, so compare the main type before the semicolon when you classify resources.
This method costs a network request per URL, and some servers respond differently to HEAD than to GET. Use it when accuracy matters more than speed, or as a fallback after the local guess fails. The complete worked example is provided in protected module code0002.
Decision checklist: which method to use
Use this checklist before fetching or crawling a URL:
- Does the URL end in a clear extension? If yes, try mimetypes first for a free, instant guess.
- Did mimetypes return (None, None)? Fall back to a HEAD request and read Content-Type.
- Does the header include parameters such as charset? Strip everything after the semicolon before comparing types.
- Do you need to guarantee the content before downloading? Prefer the server's header over the URL's spelling.
- Are you processing thousands of links? Batch the cheap local check first and reserve HEAD requests for ambiguous cases.
A simple rule of thumb: extension first, server second. The local guess handles the common case at no cost, and the header check resolves the cases where naming alone cannot be trusted.
Related questions and tutorial
Related Web Crawling web scraping questions:
- Does WebCrawler still exist?
- Is Google a web crawler?
- What is a web crawler used for?
- How to ignore non-HTML URLs when web crawling?
Web Crawling web scraping tutorial: Learn web scraping with Web Crawling
Frequently asked questions
Which approach works without contacting the server?
Use the built-in mimetypes module when the URL already carries a recognizable extension. It needs no network request, but returns (None, None) when there is no extension to inspect, as with a bare path such as http://example.com/file.
How can I detect a type when the URL has no extension?
Send a HEAD request and read the Content-Type response header. The request fetches metadata rather than the full body, so it can identify types even when the URL has no extension, for example image/png for a PNG resource or text/html; charset=utf-8 for a web page.
Which method should I choose first?
Start with mimetypes for a fast, offline guess from the extension. If that returns no result, or you need confirmation of what the server will actually return, use the HEAD request and its Content-Type header before deciding whether to download or crawl the resource.