How to Find All Pages on a Website: Methods, Limits, and a Practical Workflow
A complete URL inventory is the starting point for scraping, SEO audits, migrations, and monitoring. No single method reveals every page, so the practical answer is to combine discovery sources and understand what each one can miss. This guide walks through search operators, sitemaps, robots.txt, desktop crawlers, custom scripts, no-code flows, and large-scale API processing.
Start with a coverage plan, not a single tool
Pages become visible in different ways: search engine indexes, sitemap files, internal links, server logs, analytics, and dynamically generated routes. A quick search check is useful for a first look, but it will not produce a complete inventory. Treat each source as one layer and compare the results.
A useful sequence is:
- Run a site: query to see what is indexed.
- Check /sitemap.xml and /robots.txt for officially exposed URLs and sitemap locations.
- Crawl internal links with an SEO spider or a small script.
- Compare the crawl against analytics, Search Console, server logs, and sitemap entries to spot orphan candidates.
- Export and deduplicate the combined list.
If the site is large, JavaScript-heavy, or protected by bot checks, the later stages may need rendering, proxy rotation, or concurrency controls.
Use Google search operators for a fast first look
The site: operator shows pages that Google has chosen to include in its results. It is fast and free, but it is not a complete crawl. Combine it with inurl:, intitle:, filetype:, and exclusion operators to narrow results by folder, title, file type, or section.
The operator reference table below preserves the original examples and descriptions. Use it as a quick lookup, then verify important URLs directly on the site.
A practical example: if you want blog pages, search site:example.com inurl:blog. If you want PDFs, search site:example.com filetype:pdf. If a large section is noisy, exclude it with site:example.com -inurl:blog.
Search operators are best for discovery and spot checks. They should not be the only source in a URL inventory.
Check sitemaps and robots.txt before crawling
A sitemap is usually an XML file that lists URLs the site wants search engines to discover. The important element is loc, which contains a URL. Other fields such as lastmod, changefreq, and priority provide metadata; lastmod is often the most useful because it can indicate when a page was updated.
Large sites often use a sitemap index. The main file points to child sitemaps, and each child must be opened to extract its page URLs. The supplied sitemap-index example shows separate English and French sitemaps, which is a useful reminder that one sitemap file may not contain the whole site.
Common sitemap locations include /sitemap.xml, /sitemap.xml.gz, /sitemap_index.xml, /sitemap.php, and /sitemapindex.xml. If the default path does not work, a search such as site:example.com filetype:xml can reveal indexed XML files, feeds, or sitemap files. Filter the results for names like sitemap, post-sitemap, page-sitemap, product-sitemap, category-sitemap, or sitemap_index.
robots.txt is a plain text file at the site root. It can point to sitemaps and can list paths that crawlers are asked not to access. A Disallow rule is a crawling instruction, not an indexing removal. It also does not prove that a URL exists or that the page is appropriate to scrape. Respect the rule unless you have explicit permission.
Use an SEO spider when you need a visual crawl
Desktop crawlers follow internal links and collect URLs, status codes, titles, meta descriptions, canonicals, redirects, and broken links. They are useful for small and medium sites, migration checks, and technical SEO reviews.
The comparison table below preserves the original tool rows and free-tier descriptions. Treat the free-tier limits and feature descriptions as a dated reference and check current vendor terms before relying on them.
In Screaming Frog, start a crawl in Crawl mode, enter the site URL, and let the tool follow internal links. To focus on actual pages, filter the results to HTML. The Internal tab shows URLs found on the target site; the External tab shows links pointing away. Response Codes helps find redirects, 404s, blocked URLs, and server errors. Page Titles and Meta Description support SEO checks. Images, Canonicals, Hreflang, and other tabs support deeper reviews.
Before assuming a site is broken, review crawl speed, user agent, HTTP headers, and JavaScript rendering settings. Screaming Frog follows robots.txt by default, so blocked sections may not be fetched. For a clean URL list, keep the export focused on internal HTML pages and avoid mixing in external links or static assets.
Build a custom script when you need control
A custom script lets you decide how URLs are discovered, filtered, stored, and processed. It is useful when you need to find all URLs on a domain at scale, repeat the process regularly, combine sitemap URLs with crawled URLs, extract data from each page, or feed a database or pipeline.
The trade-off is handling edge cases: redirects, duplicate URLs, canonical tags, rate limits, JavaScript-rendered links, broken pages, and blocked requests. Test on a small batch before running a full list.
The supplied Python workflow uses a sitemap as the starting point, parses urlset and sitemapindex files, removes duplicates, saves a URL list, extracts product fields with AI extraction rules, runs concurrent scraping tasks, combines JSON files into a CSV, and saves logs. The code modules below preserve the exact inputs, outputs, field names, and constants. The workflow is a demonstration, not a promise that a sitemap reveals every possible URL on a large ecommerce site.
Crawl internal links when there is no usable sitemap
If a site does not expose a sitemap, or the sitemap is incomplete, you can start from the homepage and follow internal links. The basic loop is: extract links from a page, add new same-domain URLs to a queue, visit each queued URL, and repeat until no new internal URLs remain.
This method finds pages reachable through links. It can still miss orphan pages, private pages, search-result pages, dynamically generated routes, and JavaScript-only links. A production crawler usually adds robots.txt handling, crawl delays, retries, better URL normalization, canonical handling, JavaScript rendering, persistent storage, logging, and duplicate content detection.
Find orphan pages by comparing sources
An orphan page exists on a site but has no internal links pointing to it. Crawlers that follow links cannot reach it, so a crawl alone will not reveal it.
Orphan pages can appear when old landing pages are removed from navigation but not deleted, product or category pages are replaced during a migration, campaign pages are created for ads or email traffic only, pages exist in the CMS but were never linked internally, or URLs still receive traffic from external links or search engines.
Compare several sources:
- Server logs show requested URLs, including pages that may not be linked internally.
- Analytics can reveal pages receiving traffic even when navigation does not expose them.
- Search Console can show indexed or clicked URLs that a crawler did not find.
- XML sitemaps may contain URLs that are not internally linked.
- Backlink tools can uncover pages that external sites still link to.
A practical workflow is to crawl the site, export landing page URLs from analytics, export indexed or clicked URLs from Search Console, export server log URLs if available, and compare the lists. URLs that appear in analytics, logs, Search Console, or sitemaps but not in the crawl are orphan candidates. Decide whether each one should be linked, redirected, updated, noindexed, or removed.
Target specific file types
Sometimes you only need PDFs, images, spreadsheets, presentations, or downloadable documents. Use Google's filetype: operator with site: for a quick check of indexed files. For example, site:example.com filetype:pdf finds indexed PDFs. The same pattern works for xlsx, docx, and pptx.
This only shows files Google has indexed. It will not find files blocked from indexing, hidden behind forms, generated dynamically, or unavailable to Googlebot.
If you already have a URL list, filter it by file extension. This works when the extension appears in the URL path, such as /files/report.pdf or /images/product-photo.webp. It may miss files served through dynamic URLs like /download?id=123; for those, inspect the Content-Type response header instead of relying only on the URL extension.
In Screaming Frog, use the Images tab for image files, the Internal tab filtered by extension such as .pdf, and the Response Codes tab to check whether files return 200, 301, 404, or other statuses. Export the filtered results to CSV. For better coverage, combine search, crawl, sitemap, analytics, and server log checks.
Automate discovery with no-code tools
Make and n8n can run a repeatable URL discovery flow without code. A typical scenario triggers a site: search, fetches sitemap.xml, parses loc nodes, filters out unwanted URLs, removes duplicates, and writes the results to Google Sheets. Scheduling the scenario daily or weekly keeps the list fresh.
No-code tools work well when the site is publicly accessible, the sitemap is clean and unprotected, and the goal is lightweight monitoring. They can struggle when the sitemap requires JavaScript rendering, the site uses dynamic routing, anti-bot protections are present, or you need to crawl thousands of pages efficiently. In those cases, a programmatic API or a dedicated scraping service may be more appropriate.
Process URLs at scale
Large URL inventories often need proxy rotation, CAPTCHA handling, JavaScript rendering, and consistent HTML output. A scraping API can handle those parts so your workflow can focus on extracting titles, meta descriptions, product fields, or other structured data. Whether a given provider actually delivers each of those capabilities depends on its current plan and documentation, so verify before you commit a pipeline to it. The supplied Python and Node.js examples below call an API endpoint with an api_key, url, and render_js parameter; the exact request shape is preserved in the code modules.
A common workflow is to fetch all URLs from a sitemap, visit each page, extract metadata such as the page title and meta description, and export the results for SEO audits, content analysis, or scraping pipelines. The supplied Python and Node.js examples show this pattern with sitemap parsing, concurrent requests, and CSV output. The exact code modules below preserve the original inputs and outputs.
Programmatic crawlers often break down when IPs get blocked, CAPTCHAs appear, JavaScript-heavy pages load partial content, the site rate-limits aggressively, or you need stable performance over thousands of pages. Test on a small batch first and make sure your use case follows the site's terms.
Choose the method that matches your goal
Use this checklist to pick a starting point:
- Quick check: Google search operators. Fast and free, but incomplete.
- SEO analysis: Screaming Frog or another SEO spider. Good for metadata, link structure, canonicals, redirects, and crawl limits.
- Full site mapping: Python plus a scraping API. More flexible than GUI tools and better for large or dynamic sites.
- Scalable data pipeline: a scraping API with proxy rotation, CAPTCHA handling, JavaScript rendering, and consistent HTML responses.
- No-code monitoring: Make or n8n with a sitemap fetch and Google Sheets output.
Each method has strengths and limits. Pick the one that matches your goals, scale, and technical comfort level, then combine sources where coverage matters.
What to do after you have the URL list
A full URL list opens several workflows: scraping structured data such as products, articles, reviews, or FAQs; auditing titles, meta descriptions, headings, and internal linking; checking status codes for broken pages; generating screenshots for QA or archiving; monitoring content changes; performing accessibility or performance audits in bulk; building datasets for machine learning or analytics; detecting duplicate or thin content; preparing for a migration or redesign; and feeding URLs into automation tools for alerts or reports.
Once you have a repeatable workflow, URL discovery becomes easier to automate. From there you can move on to metadata extraction, structured data collection, site architecture audits, change monitoring, or larger data pipelines.
| Operator | Example | What it helps you do |
|---|---|---|
| site: | site:example.com | Find Google-indexed pages from a specific domain. Good for a quick overview, but not a complete crawl. |
| inurl: | site:example.com inurl:blog | Find pages where the URL contains a specific word, folder, or pattern. Useful for sections like blogs, docs, products, or categories. |
| intitle: | site:example.com intitle:pricing | Find pages with specific words in the title. Helpful for locating landing pages, reports, product pages, or content hubs. |
| filetype: | site:example.com filetype:pdf | Find indexed files such as PDFs, spreadsheets, presentations, or documents. |
| - | site:example.com -inurl:blog | Exclude sections you do not want in the results. Useful when a large section is polluting your search. |
| Combined operators | site:example.com inurl:2026 intitle:report filetype:pdf | Stack operators to narrow results to a very specific set of URLs. |
Search results reflect what Google has indexed and chosen to show. They are not a complete URL inventory.
| Tool | Free tier | Strengths | Limitations |
|---|---|---|---|
| Screaming Frog SEO Spider | Crawls up to 500 URLs for free | Very detailed desktop crawler; great for technical SEO audits, redirects, metadata, canonicals, status codes, and custom extraction | Free version is capped at 500 URLs; JavaScript rendering, saving crawls, advanced configuration, and API integrations require a paid license |
| XML-Sitemaps.com | Generates sitemaps for up to 500 URLs for free | Simple browser-based crawler; useful when you quickly need a basic XML sitemap or URL export | Limited for large sites; fewer technical audit features than dedicated SEO crawlers |
| Ahrefs Webmaster Tools / Site Audit | Free for verified websites, with crawl credit limits | Strong SEO auditing features, backlink context, and issue reporting for sites you own or can verify | Mainly useful for verified properties; broader Ahrefs data and higher limits require paid plans |
| Semrush Site Audit | Limited free auditing; higher crawl limits on paid SEO Toolkit plans | Clean UI, strong SEO diagnostics, crawl configuration, and reporting | Free usage is limited; larger crawls and advanced workflows require a paid plan |
Free-tier limits and feature descriptions are a dated reference. Check current vendor terms before relying on them.
Frequently asked questions
What is the easiest way to find all pages on a website?
For a quick, no-setup check, use Google's site: operator. It shows indexed pages instantly, though not always comprehensively. For fuller coverage, combine it with sitemap checks and a crawler such as Screaming Frog or a scraping API.
Can I see all pages on a website without coding?
Yes. Tools such as Screaming Frog, Ahrefs Site Audit, Semrush, Make, and n8n let you view pages visually without writing code. Enter the domain, start a crawl or workflow, and export the results.
How do I use Google to find all links on a domain?
Use operators such as site:example.com, site:example.com inurl:blog, or site:example.com intitle:guide. Combine operators to narrow results by URL pattern, title, or file type. Google's Search Operators page provides official documentation.
What's the best tool to crawl all pages of a website?
For SEO work, Screaming Frog is a common choice. For cloud-scale crawling or JavaScript-heavy sites, a scraping API may be more suitable. For full automation or pipelines, Python plus a scraping API gives more control. Open-source users can also explore crawlers such as scrapy/scrapy.
How can I extract all website URLs into Excel or CSV?
Most crawlers and scraping tools can export URLs to CSV. Open the CSV in Excel or Google Sheets. Scrapy and Python's pandas library make automated exports easy.
What if a website blocks crawlers - how can I still get URLs?
Use a scraping service with proxy rotation, CAPTCHA handling, and JavaScript rendering. Test on a small batch first and make sure your use case follows the site's terms. For dynamic rendering, consult the service's documentation.
"organic_results": [
{
"url": "https://bodrovis.tech/",
"displayed_url": "https://bodrovis.tech"
},
{
"url": "https://bodrovis.tech/en/teaching",
"displayed_url": "https://bodrovis.tech › teaching"
},
{
"url": "https://bodrovis.tech/ru/blog/",
"displayed_url": "https://bodrovis.tech › blog"
}
]