Build a Recursive Sitemap Link Extractor in n8n
If you need a reliable list of a website's pages before scraping or auditing it, the sitemap is usually the fastest starting point. This guide walks through an n8n workflow that asks a domain for its robots.txt file, looks for sitemap declarations, falls back to /sitemap.xml when none are found, and then follows nested sitemap files until every page link has been collected into a Google Sheet.
What the workflow does
The workflow accepts a domain through a webhook, fetches robots.txt first, and checks whether the response contains a Sitemap: line. If it does, the workflow extracts XML sitemap URLs and begins fetching them. If it does not, the workflow requests /sitemap.xml directly. Each sitemap response is classified as binary or text. Binary responses are checked for a .gz extension; gzipped files are decompressed and the binary key is renamed so the XML parser can read a single property. XML content is converted to JSON, then two Code nodes split the results: one emits non-XML page links, and the other emits child sitemap URLs. Page links are appended to Google Sheets, while child sitemap URLs are fed back into the fetch step to continue the recursion.
Before you start
You need an API key for the web scraping integration, an n8n instance (cloud or self-hosted), and a Google Sheet with a single column named links. The workflow uses the scraping integration node to fetch robots.txt and sitemap files, so the API key must be available to those nodes. The Google Sheet is the final destination for page URLs; sitemap URLs are not written there.
Step 1: Accept the domain through a webhook
Create a Webhook node named "Domain to scrape". Trigger it with a URL that includes a domain query parameter, for example the webhook URL followed by ?domain=n8n.io. Downstream nodes read that value with {{ $('Domain to scrape').item.json.query.domain }}. Keeping the domain in a query parameter makes it easy to reuse the same workflow for different sites without editing node settings.
Step 2: Check robots.txt first
Add a web scraping integration node named "Scrape robots.txt file". Set its URL to https://{{ $('Domain to scrape').item.json.query.domain }}/robots.txt. Turn Render JS off because robots.txt is static text. Enable Retry On Fail and set On Error to Continue using the error output so the workflow can fall back to sitemap.xml if the request fails. Then add an IF node named "If sitemap links are found" with the left value {{ $json.data }}, the operator contains, and the right value Sitemap:. If the condition is true, the workflow goes to the XML link extraction step. If it is false, the workflow fetches /sitemap.xml.
Step 3: Fall back to /sitemap.xml
Add a web scraping integration node named "Scrape sitemap.xml file" with the URL https://{{ $('Domain to scrape').item.json.query.domain }}/sitemap.xml. Keep Render JS off and Retry On Fail on. This node feeds the binary/text detection step. The fallback matters because not every site lists its sitemap in robots.txt, and some sites publish only the default /sitemap.xml path.
Step 4: Distinguish binary and text responses
Add an IF node named "If it's a binary file" with the left value {{ $binary }} and the operator is not empty. If the result is true, the workflow checks whether the file extension is gz. If the result is false, the response is treated as text JSON and sent directly to link extraction. This branch is important because sitemap responses can arrive either as plain text or as binary files, and the downstream parser needs to know which path to take.
Step 4.2: Decompress gzipped sitemaps
Add an IF node named "If it's a .gz file" with the left value {{ $binary.data.fileExtension }}, the operator equals, and the right value gz. When true, add a Compression node named "Decompress .gz file". Connect it to a Code node named "Rename key" that moves the decompressed binary from file_0 to data so the XML parser can consume a single property. The Code node runs once per item and returns the modified item. After that, forward the item to the Extract from File node with the XML parser. If the binary is not gzipped, send it directly to Extract from File. If the earlier binary check was false, skip to the link extraction step because the extractors can parse URLs from text as well.
Step 5: Parse XML into JSON
Add an Extract from File node with the operation set to XML and the binary property set to data. This converts binary XML into structured JSON so the URL extraction logic can work on predictable fields. From this point, the workflow forks into two Code nodes: one for non-XML page links and one for XML child sitemap links.
Step 6: Extract page links and child sitemap links
Create two Code nodes and connect both to the Extract from File node and to the non-binary branch of the binary check. That way the same extraction logic works whether the sitemap arrived as XML or as text. The first Code node returns only non-XML links as separate items in the shape { "link": "..." }. It scans every nested string field, normalizes escaped control characters, extracts URLs, decodes & to &, trims trailing punctuation and brackets, ignores query strings and hashes when checking extensions, and skips links ending in .xml or .xml.gz as well as links to www.sitemaps.org and www.w3.org. It also deduplicates results with a Set. The second Code node does the inverse: it extracts only URLs ending in .xml or .xml.gz, normalizes text, deduplicates, and outputs each match as { json: { xml: "https://..." } }. Connect the non-XML node to Google Sheets Append. Connect the XML node to a web scraping integration node named "Scrape xml file" so child sitemaps are fetched and fed back into the binary/text, decompress, and parse loop.
Step 7: Append page links to Google Sheets
Add a Google Sheets node with the Append operation named "Append links to sheet". Set the document to your Sheet URL and the sheet to the worksheet name, for example Sheet1. Map the columns as { "links": "{{ $json.link }}" } and set useAppend to true. This node should receive items only from the non-XML extraction node so that the sheet stores page URLs rather than sitemap URLs.
Step 8: Close the recursion loop
Wire the "Scrape xml file" web scraping integration node back to the "If it's a binary file" node. Each pass through the loop fetches one more sitemap file, splits its contents into page URLs and child sitemap URLs, and sends the page URLs to the sheet. The loop reaches its natural end when the XML extraction node stops producing new .xml or .xml.gz URLs, since there is then nothing left to fetch. Until that point, ordinary page links keep arriving in the sheet alongside the sitemap traversal.
Testing the workflow
Trigger the webhook with a request such as curl "https://<your-webhook-url>?domain=n8n.io". If the workflow is wired correctly, page links should begin appearing in the Google Sheet you configured. If nothing appears, check the binary/text branch, the gz check, and whether the Google Sheets node is receiving items from the non-XML extraction node.
Memory caution for very large sitemaps
Very large sitemaps, or sitemaps with many nested children, can consume a lot of memory and crash the workflow depending on your n8n plan or server configuration. If that happens, consider upgrading your n8n plan for more memory or self-hosting n8n on a machine with more RAM. It also helps to test with a small domain first so you can confirm the recursion and sheet append behavior before running against a large site.
Decision checklist before you run it
Use this checklist to confirm the workflow is ready:
- Does the webhook URL include the domain query parameter?
- Is Render JS off for both robots.txt and sitemap.xml fetches?
- Is Retry On Fail enabled on both web scraping integration fetch nodes?
- Does the robots.txt IF node check for the exact string Sitemap:?
- Does the binary check use {{ $binary }} with is not empty?
- Does the gz check compare {{ $binary.data.fileExtension }} to gz?
- Does the Rename key Code node move file_0 to data?
- Is the Google Sheets node connected only to the non-XML extraction node?
- Is the XML extraction node wired back to the binary check for recursion?
Failure distinctions
A few failure modes look similar but need different fixes. If robots.txt returns no Sitemap: line, the workflow should take the false path to /sitemap.xml; that is expected behavior, not an error. If robots.txt itself fails, the On Error Continue setting should route the workflow to the fallback. If a sitemap returns binary data, the binary check should be true; if the extension is gz, the decompression branch should run. If the XML parser receives no data, check whether the Rename key node actually moved file_0 to data. If page links never reach the sheet, confirm the Google Sheets node is connected to the non-XML extraction node rather than the XML extraction node.
Frequently asked questions
Why check robots.txt before sitemap.xml?
robots.txt often declares sitemap locations, so reading it first can reveal the correct sitemap URLs without guessing. If no Sitemap: line is present, the workflow falls back to /sitemap.xml.
How does the workflow handle gzipped sitemaps?
When a binary response has the gz extension, the workflow decompresses it with a Compression node, then a Code node renames the binary key from file_0 to data so the XML parser can read it. Non-gzipped binary files go straight to the XML parser.
Why are there two Code nodes for link extraction?
One Code node emits non-XML page links for the Google Sheet, and the other emits XML or XML.gz child sitemap links that feed back into the fetch step. Keeping them separate prevents sitemap URLs from being written to the sheet and lets the recursion continue.
What should I do if the workflow crashes on a large sitemap?
Very large sitemaps can use a lot of memory. Consider upgrading your n8n plan for more memory or self-hosting n8n on a machine with more RAM. Testing with a smaller domain first can also help confirm the workflow before running it at scale.
{ "links": "{{ $json.link }}" }
curl "https://<your-webhook-url>?domain=n8n.io"