Reading TDMRep, ai.txt and RSL: A Purpose Gate for Scrapers
A robots.txt file can tell a crawler whether it may fetch a URL. It cannot say that search indexing is welcome while model training is refused, or that mining is available only under a license. Several newer formats try to carry that second answer. This guide shows how to read them, where they disagree, and how to turn them into one verdict per purpose. It is educational, not legal advice; the law in this area is still moving, so check your own situation with a lawyer in your jurisdiction.
Why one robots.txt bit is not enough
The Robots Exclusion Protocol matches a crawler's product token to Allow and Disallow rules. That vocabulary has no field for purpose. Site owners work around the gap by listing AI crawler names by hand, which approximates an intent rather than stating it. A list of vendor tokens also ages badly: a new crawler name defeats it, and a suffixed token can inherit the rules of its shorter parent. Blocking a training crawler does not necessarily stop the same content reaching a model through a shared crawl corpus unless that corpus crawler is also blocked.
The four purposes a gate should distinguish
Purpose-based controls separate the use from the fetch. The same small set of purposes recurs across the newer formats, even where each format spells them differently and carries only some of them:
- search: index the page and link back to it.
- ai-train: train or fine-tune a generative model.
- ai-input: feed a model at answer time, also called retrieval-augmented generation (RAG).
- tdm: text and data mining, the broadest category in EU law and the purpose a normal pipeline passes to the gate.
Most scraping work is none of the first three. A recital of the EU copyright directive treats price monitoring and market research as text and data mining. Use ai-train or ai-input only when a model touches what you pulled.
Where the rules come from, and what gives them force
Two EU instruments matter. Under the Copyright in the Digital Single Market Directive, mining of lawfully accessed content is permitted by default, and that default survives only until the rightsholder signals a reservation in a recognised machine-readable way. Absent such a signal, the permission stands; once a valid reservation is published, the exception no longer covers that content. The EU AI Act then obliges providers of general-purpose AI models to honour those reservations through a copyright policy. A code of practice published in July 2025 and signed by 23 providers as of July 2026 commits signatories to crawl with robots.txt-aware crawlers and honor machine-readable reservations. From 2 August 2026, breaching these duties risks a fine of up to 3% of global turnover or €15 million, whichever is higher.
The files themselves declare a preference; they do not enforce it. In December 2025 a US court rejected an argument that a crawler ignoring robots.txt commits illegal circumvention, comparing the file's authority to a lawn sign asking visitors to stay off the grass rather than a technical access barrier. The binding force therefore comes from copyright law and the AI Act, which makes honouring the file a deliberate compliance and reputational choice rather than something the protocol enforces by itself.
The signal landscape
Several formats compete to carry the same intent. They overlap and cover different points between access control and licensing offer. The legal directory table below maps each signal to what it expresses, where it lives, and the values it uses. Read it as a dated reference from July 2026 and check current conditions before relying on any single format.
Parsing TDMRep
TDMRep is the reservation format designed around Article 4 of the DSM Directive. It emerged from a W3C Community Group and was finalised in May 2024; it is a community report rather than a formal web standard, yet it is the reservation mechanism that working publishers actually deploy. Two properties carry the meaning: tdm-reservation, where 1 marks the content reserved and 0 marks it unreserved, and an optional tdm-policy URL that points at licensing terms.
For a web page, a site can publish those properties through three channels that stack in a fixed order. The base layer is a /.well-known/tdmrep.json file mapping URL paths to reservations. A tdm-reservation HTTP header for a specific resource sits above that file, and an HTML <meta name="tdm-reservation"> tag sits above both. EPUB and PDF metadata rank higher still, which matters when you mine documents rather than landing pages. Because a resource-level signal beats a site-level one, a reader that consults only the well-known file will miss per-page overrides. A value that is absent at one level does not clear the value inherited from a lower level.
Three rules decide whether a reader is correct. Path matching in the well-known file is first-match-wins in document order, not longest-match. Only 0 and 1 are valid, and anything else falls back to unset, which means not reserved, so a malformed reservation silently becomes permission. The third rule concerns the file shape: the spec says a JSON array, with brackets mandatory even for one rule, but in July 2026 Cambridge University Press served a bare object. Looping over that object walks the keys rather than the rules, losing a real reservation with its policy link. A correct reader coerces a lone object into a list, skips anything that is not a dict, and honors all three levels.
Reading the policy behind a reservation
When a reservation includes a tdm-policy URL, that URL usually resolves to an ODRL document describing which mining uses are permitted and what conditions accompany each one. A reservation with a policy is not the same as a flat refusal: the policy tells you whom to approach and on what terms. For each tdm:mine permission in the document, pull out the purpose constraint and any duty attached to it, and accept operands whether they appear bare or with the odrl: prefix that deployed files use. Read the operands leniently, because the deployed files prefix them and use a purpose value (stm:eu-dsm-article3) that the base spec never defined. A reader that insists on the bare form will miss the constraint entirely and misreport the permission as unconditional or absent. The same standard can produce opposite answers. Elsevier grants research mining with no duty attached and requires consent for every other purpose. Springer Nature requires consent even for research, and each policy lists the email address to contact. A worked contrast makes the difference concrete. Elsevier's policy permits mining where the purpose equals stm:eu-dsm-article3 with no duty, and attaches an obtainConsent duty to every other purpose. Springer Nature's policy attaches obtainConsent to both branches, so even the research purpose needs a request. Both documents name a contact address in the assigner block, which is where a reader should look before assuming a reservation is a flat refusal.
Parsing ai.txt
Spawning's ai.txt takes the media-side view of the same question. The file sits at /ai.txt and declares which file types a site permits for training. Two design choices separate it from TDMRep and matter when you write a parser. First, it scopes by file type rather than by URL path, grouping types into five media categories: images, audio, video, text, and code. Second, it defaults to opted out: with no file, or with every category switched off, nothing may be trained on. In TDMRep silence means permission, but in ai.txt silence means refusal. Take the syntax from Spawning's generator. Descriptions found elsewhere show a Disallow: ai_training: images form that the generator never emits. What the generator actually writes is robots.txt-style directives whose units are file extensions. The parser is short because the format is short. The catch is adoption. A July 2026 scan of 42 domains found no ai.txt at all, and a second pass over 60 sites picked for this format found not one valid file, with 46 answering 404. Those 60 were the artist platforms and stock libraries the format was written for, and Spawning's own domains served none. The name also covers several unrelated proposals, so the name alone does not tell you which grammar you are reading. Parse ai.txt if you meet one, because it is cheap, but do not expect it to carry your decision.
Parsing RSL: when the answer is a price, not a no
Really Simple Licensing sits above the opt-out formats and reframes no as not for free. It extends robots.txt with a License: line pointing to an XML file, and that file states, per purpose, what it permits, what it prohibits, and at what price. Medium's file shows the whole argument in one document: it permits search, indexing, and retrieval under attribution for free, and prohibits training under a subscription payment that carries a form to ask. Search and training get different answers on the same content, which one robots.txt line cannot say. The Guardian takes a stricter line: it permits ai-train and ai-input under subscription and prohibits every other use, so search is refused rather than sold. Both files were reachable through a License: line and were live in July 2026. The RSL spec advertises its license files as application/rsl+xml in the Link header, but Medium serves plain application/xml. A reader that trusts the advertised type finds nothing, so the parser checks the body for an <rsl root instead.
Combining signals into one purpose-based decision
Some sites publish more than one signal, and the signals can disagree. In the July 2026 scan 19% carried two or more: robots.txt AI blocks plus a TDMRep reservation, or robots.txt AI blocks plus an RSL license. One more format matters even at zero adoption. The IETF's AI Preferences working group is standardizing a Content-Usage field, carried as an HTTP header or a robots.txt rule, with two categories (train-ai, search) and values y or n. In July 2026 it was a draft, not a published standard, and the working group was targeting August 2026 to send it to the IESG. No live deployment appeared in the scan, but its resolution rule is the part reused here. The job of a purpose gate is to take a URL and a purpose, read every signal, and return one verdict: allow, deny, or needs-license. When signals combine, any n wins, then any y, otherwise unknown. One added rule: a paid license path outranks a plain block, because a License: line means the block is negotiable. The verdicts line up with what each site intends. Medium allows search and retrieval and makes training negotiable. Nature licenses training but denies real-time retrieval, and blocks the RAG crawlers with no license path. Stack Overflow denies search and training in its Content-Signal, and its blanket Disallow: / catches retrieval too. The Guardian prices training and retrieval and refuses the rest. None of that fits a yes-or-no flag, so the gate keeps each reason attached for a human to audit.
What sites actually publish
The gate has nothing to read unless sites publish signals. In July 2026 a scan of 42 domains recorded which signals each one serves. The list covered academic publishers, European and US news, the RSL backers, reference and developer sites, and ecommerce. It also included edrlab.org and cloudflare.com, the two authors of TDMRep and Content-Signal, because how an author uses its own standard is worth knowing. The scan ran on plain requests. If a site timed out, or answered a bare client with a challenge page, it stayed in the denominator and counted as publishing nothing, so read the numbers as minimums.
The scan output below is a dated reference from July 2026, not a current measurement. Re-run the companion script before relying on any figure.
What the numbers show
robots.txt AI blocking is mainstream, and everything purpose-based is early. The one Content-Signal is Stack Overflow. Cloudflare serves one too, but it belongs to the Cohere-ai group, so the scan reads the * group where a generic crawler looks, and counts one. The TDMRep signals came almost entirely from academic and European publishers, the sites the standard was written for. Eight of the nine carry a reservation. The ninth is edrlab.org, the lab that wrote the protocol, and it publishes an explicit 0 to say mining is fine. That European skew shows up outside the sample too. EDRLab reported that 143 of the top 250 French sites had implemented TDMRep as of 2024, more than half the top sites in a single country, against the 21% the global scan found across a mixed sample. Of the 42, three carried a training verdict that was deny from robots.txt and needs-license from a TDM policy or RSL file at once. robots.txt blocks the use, and the license offers to sell it. A gate that reads only one of them gets the wrong answer. Independent studies read the same way. The Data Provenance Initiative's Consent in Crisis measured this early: in a single year, 2023 to 2024, 28%+ of the most actively maintained sources in C4 went fully restricted through robots.txt. Cloudflare measured 52% of crawler requests going to AI training by June 2026, up from 22% a year earlier. So more sites block AI crawlers every year, but almost none of them say why in a way a scan can read.
The limits of these signals
Reading these files well still leaves three gaps to design around.
The first is who carries the burden. Silence means permission under TDMRep and the DSM Directive, so a rightsholder who publishes nothing has reserved nothing, and most publish nothing. You cannot fix that from the scraper side: you read what is there, you record what you read, and your default decides the rest. Whether you may fetch a page at all is a separate question with its own case law.
The second is fragmentation. A single purpose, do not train, can arrive as a robots.txt rule, a TDMRep reservation, an ai.txt line, a Content-Signal, or a Content-Usage header, and two of those spell the category differently. The European Commission ran a consultation into early 2026 on which machine-readable opt-out to endorse and plans to publish an agreed list. Until that list arrives, a compliant reader supports several formats, not one.
The third is the standard for machine-readable itself. A German court held in December 2025 that a reservation counts only if a machine can both perceive and interpret it, not merely if a person can read it. No EU-wide answer exists yet, so favor a structured signal like TDMRep over buried prose, and record which signal you read and when. The EU's Court of Justice may change all of this. It heard its first generative-AI copyright case in March 2026, and one question referred to it asks whether training a model falls under the Article 4 exception at all.
Building purpose-awareness into a pipeline
You still have to read these signals, so the gate has to work at scale. The naive version does not. It fetches three files per URL before it even looks at the page: robots.txt, the well-known reservation, and the license. A million URLs means millions of extra requests, and almost all are redundant. ai.txt does not vote here, because it states permission per file type and cannot answer a URL-level question.
Those three files are per-origin, not per-page, so fetch them once per host and cache them. But cache the well-known file's rules, not its answer, because it matches paths and every URL needs its own match. The tdm-reservation header and <meta> tag are per-page, and they arrive on the response you already fetched, so they cost nothing extra.
The gate forces two operational choices. The first is the default: with no signal, UNKNOWN becomes ALLOW (fail-open, matching silence-is-permission) or DENY (fail-closed). Apply the same default when a signal fetch fails, so a 503 on robots.txt does not silently flip you to permissive. The second is the record: a lineage line is what an auditor or a buyer asks for, so store it beside the row it gated.
One caveat applies to the readers. The token scan asks whether a fixed list of AI crawlers may fetch the root, and it is a survey, not a crawl-decision engine. For the real may-my-bot-fetch-this-path question, run the same protego rules against your own product token and the exact URL. Do not write your own robots.txt parsing either. A naive reader misses sites that allowlist named bots and deny the rest with one User-agent: *. Verified-bot schemes, where a crawler signs its requests, are starting to gate access, so a 2026 crawler must also prove who it is.
Putting it into practice
Most of what you meet is still a block with no stated purpose, as the scan showed. So the purpose you act on is usually one you inferred, from a crawler name or from silence, and the site never declared it. You own the gate. The fetch under it is the part you can buy.
Reading the control files needs only plain requests. Fetching the content they govern means getting past anti-bot and rendering, and a scraping API can do both. A fetch through one can also return the origin's headers prefixed with spb-, so the reservation arrives as spb-tdm-reservation and the gate reads it off the same response. You read the origin signals before the fetch, so origin_verdict can deny a URL before you spend a fetch on it. You read the page reservation after it, off the response you already have. The gate does not stop the fetch. It decides what you may do with the page once you have it.
| Signal | Expresses | Where it lives | Values |
|---|---|---|---|
| robots.txt AI tokens | Fetch, per user-agent group | /robots.txt | Allow / Disallow |
| Content-Signal (Cloudflare) | search / ai-input / ai-train | /robots.txt | yes / no |
| Content-Usage (IETF AIPREF) | train-ai / search | HTTP header or /robots.txt | y / n |
| TDMRep | TDM reservation, plus a policy link | header, <meta>, /.well-known/tdmrep.json | 1 / 0 |
| ai.txt (Spawning) | AI-training, per file type | /ai.txt | Allow / Disallow |
| RSL | Per-purpose license and price | /robots.txt link to a license file | XML permits/prohibits |
Dated reference from July 2026. Check current conditions before relying on any single format.
Frequently asked questions
Is TDMRep legally binding?
TDMRep is a format, not a law. Its force comes from EU rules: the DSM Directive removes the TDM exception once a machine-readable reservation exists, and the AI Act adds a duty on GPAI providers. Article 4 covers commercial mining too, so a reservation removes your exception even if you train nothing.
What is the difference between robots.txt and TDMRep?
robots.txt controls fetching: whether a crawler, named or caught by the * group, may request a URL. TDMRep controls use: whether already-fetched content may be mined, and under what license. A page can be allowed in robots.txt yet reserved in TDMRep, so the two are complementary layers, not substitutes.
Does ai.txt actually stop AI training?
No file stops training on its own. Each one declares a preference. ai.txt states, per file type, that training is not permitted, and a compliant crawler honors it. Adoption is low, and most sites express training opt-outs through robots.txt AI tokens instead, so ai.txt is one signal among several.
Is llms.txt an AI scraping control?
No. llms.txt tells a model which of your pages to read at answer time. It does not say who may use those pages, or for what, so a purpose gate has nothing to read there. For control, read robots.txt, TDMRep, Content-Signal, or RSL.
How do I handle conflicting AI-usage signals?
Resolve them per purpose, not per site. For each purpose, collect every signal's vote and apply a most-restrictive rule: a deny outweighs an allow, and a paid license path outranks a plain block. Keep the reasons attached, so a denied or licensable URL is auditable later.
Do I need to parse these signals if I scrape outside the EU?
Training outside the EU does not exempt you. The AI Act's copyright duty applies to any general-purpose model placed on the EU market, wherever it was trained. Beyond the law, a site can block a crawler that ignores its file, wherever that crawler runs.
What is the fastest way to check one URL right now?
Fetch three things: /robots.txt, /.well-known/tdmrep.json, and the page's own tdm-reservation header. Those three cover the most-deployed signals in the July 2026 scan. Add RSL and Content-Signal parsing as you meet sites that publish them.