Also called: robots.txt, Robots Exclusion Protocol, REP, RFC 9309, robots exclusion standard
DocumentedAn IETF Proposed Standard; each operator documents how its own crawlers follow it.
robots.txt is the plain-text file at a site's root where owners state which paths each crawler may fetch, standardized as the Robots Exclusion Protocol in IETF RFC 9309 (September 2022). AI companies map their opt-outs to user-agent tokens in it, but it governs crawling only, and compliance is voluntary.
RFC 9309, by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, is an IETF Proposed Standard that formalized decades of practice. The file must sit at /robots.txt, encoded in UTF-8. It is organized in groups: one or more User-agent lines, matched case-insensitively against a crawler's product token, followed by Allow and Disallow rules. When several rules match a path, the most specific match (the one with the most octets) wins, and an equivalent Allow beats a Disallow. A crawler with no matching group falls back to the * group; with no * group, everything is allowed.
The RFC also sets operational rules. Crawlers must parse at least 500 KiB, should not use a cached copy for more than 24 hours unless the file is unreachable, may treat a 4xx response as permission to crawl everything, and must treat a 5xx or network error as a complete disallow. In the RFC's own words, "These rules are not a form of access authorization."
How it works
AI operators publish tokens with distinct purposes, which lets a site separate uses. OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for training, and says a robots.txt change can take about 24 hours to reach its search systems. Google points to Google-Extended for limiting training and grounding in some of its AI systems outside Search, and Apple's Applebot-Extended controls whether Applebot's crawl may train Apple's foundation models. User-initiated fetchers are a separate case: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
Why it matters
robots.txt is the main control surface between a site and AI crawlers, so one line can make a site ineligible for an engine's answers or withhold it from training. Two limits matter. It governs fetching, and has no effect on content already collected or obtained elsewhere, which is why usage-level signals such as Content Signals, TDMRep and RSL exist. And it is a request that a crawler can ignore. Teams should audit robots.txt and CDN bot rules together, decide token by token, and confirm the effect in server logs.
Example
A group User-agent: OAI-SearchBot with Allow: /, followed by a group User-agent: GPTBot with Disallow: /, keeps a site eligible to be fetched for ChatGPT search while withholding it from OpenAI's training crawler. It removes nothing collected before the change, and it does not govern user-initiated ChatGPT-User fetches.
Also called: X-Robots-Tag header, robots HTTP header
DocumentedDocumented by Google, Apple and Bing for their own crawlers.
X-Robots-Tag is an HTTP response header that carries the same indexing and snippet rules as the robots meta tag, such as noindex or nosnippet, for any file a server returns. It is the way to apply those rules to PDFs, images and other non-HTML resources, and it can target a single crawler by name.
Google documents that any rule usable in a robots meta tag can be sent as X-Robots-Tag, as several headers or as a comma-separated list, for example X-Robots-Tag: noindex. A rule can be scoped to one crawler by prefixing its name, as in X-Robots-Tag: googlebot: nofollow; rules without a name apply to all crawlers. Header names, crawler names and values are case-insensitive. When rules conflict, Google applies the more restrictive one, and it combines negative rules from general and crawler-specific instructions.
Apple says Applebot honors the header as well and recommends it for non-HTML resources such as PDFs or images. Bing documents its own directive set, including nocache and noarchive, which it tied in 2023 to content use in what was then Bing Chat. The header only works on a URL a crawler may fetch: Google states that if robots.txt blocks a page, its indexing and serving rules are never seen.
Why it matters
Many cited resources are not HTML: whitepapers, research PDFs, slide decks, images. X-Robots-Tag is their only page-level control, so it decides whether a PDF can appear in search features and, through nosnippet or max-snippet, how much of it Google may use in AI Overviews and AI Mode. Teams should set it at the server or CDN for the file types that need rules, test responses with a header checker, and avoid sending conflicting rules from different layers.
Example
A server sends X-Robots-Tag: noindex on every response under /downloads/ that ends in .pdf. The whitepapers stay out of Google's index and search features, while the HTML landing pages linking to them remain indexable.
Also called: preview controls, snippet controls, nosnippet, max-snippet, data-nosnippet, noindex
DocumentedDocumented by Google, Apple and Bing for their own products; each engine's scope differs.
Preview controls are robots directives that limit how much of a page a search engine may show, and Google documents them as the page-level way to limit use in AI Overviews and AI Mode: nosnippet, max-snippet, the data-nosnippet attribute and noindex. Since 2026, Search Console also offers a site-wide opt-out from Google's generative AI features.
Google's specifications define each control. nosnippet removes text snippets and video previews and, per Google, keeps the content from being used as direct input for AI Overviews and AI Mode. max-snippet:[number] caps snippet length in characters and also limits use in AI Overviews and AI Mode, except where the publisher has granted separate permission, for example through structured data or a license; max-snippet:0 equals nosnippet. The data-nosnippet attribute excludes parts of a page and is valid only on span, div and section elements. noindex removes the page from results altogether. When rules conflict, the most restrictive applies.
Other engines differ. Apple says Applebot will not use content marked nosnippet as context for AI-generated output, and keeps pages marked isAccessibleForFree: false out of that context. Bing announced in September 2023 that noarchive excludes content from its chat answers and from training Microsoft's generative models, while nocache limits that use to URL, title and snippet; with both present, Bing applies nocache.
How it works
Announced in June 2026 and rolled out to all websites as of 2026-08-31, Google's Search generative AI control sits in Search Console settings, per property. Excluding a property removes its links and content from AI Overviews, AI Mode and generative AI features in Discover, with no traffic or impressions from them. Google says the control is not used as a ranking signal elsewhere in Search and does not affect training, for which it points to Google-Extended. Google's AI features documentation page did not yet mention the control when checked on 2026-11-07.
Why it matters
These controls decide how much of a page Google's AI features can use while the page stays indexed for classic results, something Google-Extended does not do for Search. The trade-off is real: limiting snippets limits classic snippets too, and the Search Console control removes a site from AI features entirely, along with the traffic they bring. Teams should decide per page type (pricing, paywalled articles, documentation), prefer data-nosnippet for narrow exclusions, check how Apple and Bing read the same tags, and verify changes with the URL Inspection tool.
Example
A publisher adds <meta name="robots" content="max-snippet:120"> to its news articles and wraps subscriber-only paragraphs in <div data-nosnippet>. The pages stay indexed and ranked, while Google may show at most 120 characters in snippets and AI features, and none of the protected paragraphs.
Also called: llms.txt, /llms.txt, llms.txt proposal, LLMs.txt
DocumentedClaimedDocumented as a proposal with visible adoption; claimed as a visibility mechanism, since Google says its AI features need no such file and none of the engine documentation reviewed here says it is read for answers.
llms.txt is a proposed Markdown file at a site's root, /llms.txt, that gives AI agents a description of the site and curated links to Markdown versions of its pages. Proposed by Jeremy Howard in September 2024 and revised as v2 in August 2026, it is widely published, while Google says its AI features need no such file.
The proposal at llmstxt.org specifies a file whose only required element is an H1 with the site or project name. It can be followed by a blockquote summary, free-form detail sections without headings, and H2 sections that list Markdown links, each optionally followed by a colon and notes. By convention, an H2 named Optional holds secondary links an agent can skip. Pages are expected to offer Markdown versions at the same URL, with .md appended (page.html.md) or replacing the extension (page.md).
Version 2, dated 2026-08-10, added discovery through standard link relations: rel="alternate" type="text/markdown" points from a page to its Markdown version, and rel="describedby" points to the llms.txt covering it, either as an HTML link element or an HTTP Link header. It also defined how llms.txt files in subpaths apply (the most specific one wins), removed the llms_txt2ctx tool from the proposal, and gave the Optional section no mechanical meaning.
How it works
An agent fetches /llms.txt, reads the summary, follows the links relevant to its task, and retrieves clean Markdown instead of rendering HTML. The proposal reports its own adoption: thousands of sites, automatic generation by documentation platforms (it lists Mintlify, GitBook, Yoast SEO and others as integrations), audits in Chrome's Lighthouse, and llms.txt files for the developer documentation of OpenAI, Anthropic and Gemini. These are the proponent's statements, not an independent audit.
Why it matters
For agents that read documentation, such as coding assistants and research agents, llms.txt and Markdown page versions cut the cost of reading a site and avoid rendering failures. For visibility in AI answers, the evidence is thin: Google states that sites need no new machine-readable files or AI text files to appear in its AI features. Teams can publish llms.txt cheaply as agent infrastructure, keep it in sync with the sitemap, and avoid promising citation gains from it.
Example
A payments company publishes /llms.txt with an H1, a one-paragraph summary, and H2 sections for its main documentation areas, each line linking to a .md version of a page with a short note. Its HTML pages carry rel="alternate" type="text/markdown" links, so an agent landing on any page can find the Markdown.
Also called: llms full text file, full-documentation text file
ObservedA convention visible on many documentation sites; absent from the llms.txt proposal itself.
llms-full.txt is a community convention that complements llms.txt: a single plain-text file containing a site's entire documentation in Markdown, so an agent can ingest everything in one request. It is not part of the llms.txt proposal, but documentation platforms such as Mintlify generate it automatically, and some files run to tens of megabytes.
Where llms.txt is a map with links, llms-full.txt is the territory: the full text of every documentation page concatenated into one file at the site root. Neither version of the llms.txt proposal mentions it, including the page describing what changed in v2; it spread through documentation tooling. Mintlify, for example, hosts both files automatically for every project and describes llms-full.txt as combining the entire documentation site into a single file, with each page's title, source URL, description and full Markdown content.
Live examples checked on 2026-11-07 show the range of sizes: Mintlify's own documentation file is about 1.7 MB, the Model Context Protocol site's about 2.4 MB, and the Claude developer platform's about 50 MB. The scope is documentation-style content meant for bulk ingestion; the convention defines no ordering, size limit or update rule.
Why it matters
A single file is convenient for agents and for developers loading documentation into a model's context or a retrieval index, and it replaces dozens of separate fetches. It also has costs: large files exceed most context windows, go stale if not regenerated, and duplicate content already on HTML pages. None of the engine documentation reviewed here mentions using it for citations. Teams with documentation can publish it when their platform generates it, keep it current, and treat it as developer infrastructure rather than a visibility lever.
Example
Instead of fetching 40 separate .md pages, a coding agent downloads one llms-full.txt holding the whole API documentation. At 2 MB it fits a retrieval index easily; at 50 MB it has to be chunked before any model can use it.
DocumentedObservedDesign and generated syntax are documented in Spawning's announcement and plugin code; honoring by readers is the vendor's claim; the specification page serves a maintenance notice.
ai.txt is a file format proposed by Spawning in 2023 for declaring, at a site's root, whether its text, images, audio, video and code may be used to train AI models. Spawning designed it to be checked when media is downloaded rather than when a site is crawled. Its specification page now shows only a maintenance notice.
Spawning announced ai.txt on 2023-05-30 as a robots.txt-like file for AI permissions. Its rationale: robots.txt is read at crawl time, so it cannot affect links already collected in datasets such as LAION-5B, while a file checked at download time lets owners change permissions for media hosted on their own domains. Spawning framed it as a machine-readable opt-out under the text and data mining exception in Article 4 of the EU copyright directive; that is the vendor's framing, and whether the file is honored depends entirely on its readers. Spawning's API relayed the permissions to partners, naming Hugging Face and Stability AI.
The exact syntax survives in Spawning's open-source WordPress plugin (Apache-2.0, last pushed 2024-07-09). Its generator writes a # Spawning AI header and User-agent: *, then per-extension Allow: or Disallow: lines built from glob lists per media type (for example Disallow: *.mp3), ending with Disallow: /, or Allow: / when text is allowed. When every type is blocked, it writes Disallow: * and Disallow: /.
Why it matters
ai.txt was an early attempt at usage-level signaling, separating permission to train from permission to crawl, and the idea returns in later signals such as Cloudflare's Content Signals, TDMRep and RSL. Today it is a weak lever: the specification page is in maintenance, current adoption is undocumented, and the only readers Spawning named were its own API partners. Teams wanting to state training preferences should use robots.txt tokens and the newer signals, and can leave an existing ai.txt in place as a harmless legacy file.
Example
A photographer's site serves a plugin-generated ai.txt that disallows image and video extensions while allowing text, ending with Allow: /. A dataset tool that checks ai.txt before downloading would skip the photos; a crawler that never reads the file is unaffected.
Spawning-Inc/spawning-wp-plugin, WordPress plugin generating ai.txt (form-handler.php and ai_txt_options.json, main branch), published n/d, read 2026-11-07: https://github.com/Spawning-Inc/spawning-wp-plugin
Spawning, ai.txt page (serves a Source.Plus maintenance notice), published n/d, read 2026-11-07: https://spawning.ai/ai-txt
Also called: schema.org vocabulary, schema markup, structured data vocabulary
DocumentedClaimedThe vocabulary is documented; claims that particular types raise AI citations are vendor assertions.
Schema.org is the shared vocabulary of types and properties, such as Article, Organization, Product, FAQPage or DefinedTerm, that websites use to describe their content to machines, most often embedded as JSON-LD. Google requires no special schema for its AI features, but says structured data should match the text visible on the page.
Schema.org is a collaborative vocabulary of hierarchical types, all descending from Thing, and their properties, published in numbered releases; the release shown on its pages on 2026-11-07 is 30.1, dated 2026-09-16. Google supports three serializations (JSON-LD, Microdata and RDFa), recommends JSON-LD, and notes that most Search structured data uses the schema.org vocabulary. Search engines use selected types to power rich results: Google's search gallery, last updated 2026-10-16, lists features including articles, breadcrumbs, events, FAQs, organizations, products, recipes and reviews.
For AI features, Google states that "there's also no special schema.org structured data that you need to add", while listing as good practice making sure structured data matches the visible text. Vendor claims that particular types raise AI citations remain claims. The vocabulary's clearest value for AI search is stating entities, facts and relationships unambiguously.
Why it matters
Structured data is one of the few ways a site can state facts in machine-readable form: who the organization is, what a product costs, who wrote an article, what a term means. Engines use it for rich results and entity understanding, and Google's guidance treats mismatches with visible content as a problem. Teams should mark up what each page genuinely shows, use the types search engines document, connect entities with properties such as sameAs, validate the markup, and treat any AI citation lift from markup as unproven.
Example
A glossary page marks itself up as a DefinedTermSet and each entry as a DefinedTerm with name, description and inDefinedTermSet, so its definitions are machine-readable even though Google offers no glossary rich result.
Also called: DefinedTerm, DefinedTermSet, schema.org DefinedTerm, glossary markup
Documented
DefinedTerm and DefinedTermSet are schema.org types for publishing glossaries and controlled vocabularies in machine-readable form. A DefinedTerm is a word, name, acronym or phrase with a formal definition; a DefinedTermSet is the glossary, dictionary or classification scheme that contains it. Google offers no rich result for either type.
On schema.org (release 30.1, 2026-09-16), DefinedTerm sits under Intangible and is described as "A word, name, acronym, phrase, etc. with a formal definition", used for subject classification, glossaries, dictionaries and product or creative-work types. Its own properties are termCode (a code identifying the term within its set) and inDefinedTermSet (the containing set, as a DefinedTermSet or a URL), and it inherits general properties such as name and description from Thing. DefinedTermSet sits under CreativeWork, so it can carry authorship, dates and licensing, and it lists its terms with hasDefinedTerm. CategoryCode and CategoryCodeSet are their more specific subtypes for coded classifications.
The types describe the meaning of terms and trigger no Google rich result: DefinedTerm does not appear in Google's structured data search gallery, last updated 2026-10-16.
Why it matters
Glossaries are a natural source for definitional answers, and DefinedTerm markup makes each definition an addressable, machine-readable unit with a stable URL and a declared parent set. That helps any system that parses structured data identify the term, its definition and its publisher without guessing from layout. None of the engine documentation reviewed here claims a citation benefit, so the case rests on clarity: teams publishing glossaries can add the markup cheaply, keep it identical to the visible text, and give each term a permanent anchor.
Example
A glossary page carries one DefinedTermSet with a name, publisher and license, and each entry is a DefinedTerm with name, description, a url pointing to its anchor, and inDefinedTermSet pointing back to the set.
DocumentedAn open protocol documented at indexnow.org; how quickly each engine acts on a submission is not documented.
IndexNow is an open protocol that lets a website notify participating search engines the moment a URL is added, updated or deleted, instead of waiting for a recrawl. A submission to one participating engine is shared with the others, which include Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Google is not among them.
A site proves control of its host with a key file, a UTF-8 text file containing the key, placed at the root and named after the key, or at a path declared with the keyLocation parameter; a key has 8 to 128 characters drawn from letters, digits and dashes. A single URL is submitted with an HTTP GET to an engine's /indexnow endpoint carrying url and key. A batch uses an HTTP POST with a JSON body (host, key, optional keyLocation, urlList) of up to 10,000 URLs. Responses include 200 (received), 202 (received, key validation pending), 403 (invalid key), 422 (URLs outside the host) and 429 (too many requests); a 200 confirms receipt only.
Participating engines agree that submitted URLs are shared automatically with all the others. The registry file on indexnow.org lists Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Google is not on that list.
Why it matters
Bing's index feeds Microsoft Copilot and Bing's AI summaries, and Bing presents IndexNow as a way to help AI systems reference the current version of a page. Bing's own guidance pairs a complete sitemap with accurate lastmod dates for coverage and IndexNow for real-time changes, while stating that no tool can guarantee when or how content appears in AI-generated results. Teams can enable IndexNow through their CMS or CDN, submit on every publish, update and deletion, and watch Bing Webmaster Tools for errors.
Example
A retailer changes the price of 3,000 products overnight. Its platform posts the changed URLs to IndexNow in one batch, so Bing and the other participating engines learn of the changes immediately instead of at their next recrawl. Google still finds them through crawling and the sitemap.
Also called: XML sitemap, sitemap.xml, Sitemaps protocol, sitemap index
Documented
Sitemaps are files, usually XML, that list a site's URLs for search engines, optionally with the date each page last changed. Defined by the Sitemaps protocol 0.9, they help crawlers discover pages and recrawl changed ones. Google and Bing both ignore their priority and change-frequency fields and rely on accurate lastmod values.
The protocol at sitemaps.org defines a urlset of url entries, each with a required loc (under 2,048 characters) and optional lastmod (W3C Datetime, for example 2026-11-07), changefreq and priority. One file holds at most 50,000 URLs and 50 MB uncompressed and may be gzipped; larger sites split into several files listed in a sitemap index. Plain-text sitemaps with one URL per line, and RSS or Atom feeds, are also accepted. Sites announce sitemaps with a Sitemap: line in robots.txt or through an engine's webmaster tools.
The engines say how they read them. Google uses lastmod only if it is "consistently and verifiably" accurate, ignores priority and changefreq, and treats a sitemap as a hint that does not guarantee crawling. Bing calls lastmod a key signal for recrawl priority, ignores the same two fields, and rechecks a sitemap at least once a day. Google says no new machine-readable files are needed for its AI features, and Bing's 2025 guidance applies the standard sitemap to AI-powered search.
Why it matters
AI search systems built on search indexes inherit their discovery: a page an engine never finds cannot be retrieved for an answer. A sitemap with complete coverage and truthful lastmod dates is the cheapest freshness signal a site controls, and Bing explicitly frames it, paired with IndexNow, as the base for visibility in AI-powered search. Teams should generate sitemaps automatically, change lastmod only on significant content updates, keep redirected and noindexed URLs out, and monitor processing errors in Search Console and Bing Webmaster Tools.
Example
A documentation site regenerates its sitemap on every deploy and changes lastmod only when a page's main content changes, never on template edits. Bing and Google can then prioritize the 40 pages that changed this week over the 4,000 that did not.
Also called: Content Signals Policy, Content-Signal, content signals, ai-train signal, ai-input signal
DocumentedEmergingThe policy is published under CC0; honoring it depends on each crawler, and the 2026 `use` extension was announced as a test.
Cloudflare Content Signals is a policy, published in September 2025, that adds machine-readable usage preferences to robots.txt through a Content-Signal line with three signals: search, ai-input and ai-train. It separates permission to appear in classic search from permission to feed AI answers or train models. Signals express preferences and do not technically block crawlers.
Cloudflare introduced the policy on 2025-09-24 and released it under a CC0 license so anyone can implement it. search covers building a search index and providing results such as links and short excerpts, and explicitly excludes AI-generated search summaries. ai-input covers inputting content into one or more AI models, as in retrieval-augmented generation or grounding. ai-train covers training or fine-tuning AI models. A line such as Content-Signal: search=yes, ai-train=no grants one use and restricts another; an omitted signal neither grants nor restricts permission. The policy text states that restrictions are express reservations of rights under Article 4 of EU Directive 2019/790.
At the policy's launch, Cloudflare said it would serve search=yes, ai-train=no to the more than 3.8 million domains then using its managed robots.txt, leaving ai-input unset. On 2026-07-01 it announced, in testing, an optional use field with the values immediate (interact, store nothing), reference (index, excerpt and link back; the default) and full (summarize and reproduce), and began adding use=reference for managed robots.txt customers.
How it works
The signals sit inside robots.txt, under a user-agent group, so any crawler that reads robots.txt can read them without fetching a new file. Cloudflare says they "are not technical countermeasures against scraping" and recommends pairing them with its WAF and bot management; its July 2026 announcement adds that it tracks content use per bot and that bots abusing the signals lose verified status. A broader vocabulary is being drafted at the IETF: draft-ietf-aipref-vocab-08 (2026-09-14) defines train-ai, ai-use and search categories, and states that its text does not yet reflect working-group consensus.
Why it matters
The distinction between AI answering (ai-input) and AI training (ai-train) is the one most publishers care about, and Content Signals is the first widely deployed syntax for it, carried by a major edge network. It remains a preference: enforcement depends on crawler cooperation and on Cloudflare's own bot policies. Teams should set each signal deliberately, since a blank ai-input states no preference on AI answers, check what Cloudflare's managed robots.txt already serves for their domain, and follow the IETF work that may supersede it.
Example
User-Agent: *, then Content-Signal: search=yes, ai-input=yes, ai-train=no, then Allow: / says: index this site for search, use it to ground AI answers, do not train models on it. Dropping ai-input=yes would leave AI answering with no stated preference.
Also called: TDMRep, TDM Reservation Protocol, tdm-reservation, tdmrep.json
DocumentedObservedA W3C Community Group final report, not a W3C standard; adoption is observed, not measured.
TDMRep, the TDM Reservation Protocol, is a W3C Community Group specification that lets rightsholders declare in machine-readable form whether text and data mining rights are reserved for their web content, and where a licensing policy can be found. It was designed as a technical answer to the opt-out in Article 4 of the EU's 2019 copyright directive.
The latest final report, version 3 (the document is dated 2024-05-10, and the group lists it under 2024-08-09), is edited by Laurent Le Meur of EDRLab and states that it is "not a W3C Standard nor is it on the W3C Standards Track". It defines two properties. tdm-reservation takes 1 (rights reserved) or 0 (not reserved); any other value counts as unset. tdm-policy points to a policy, machine-readable (JSON or JSON-LD using a profile of ODRL 2.2) or human-readable, that can give contacts and conditions such as consent or compensation.
The properties can be expressed in a site-wide /.well-known/tdmrep.json file with path rules, in HTTP response headers (the report's currently preferred technique), in HTML meta elements, in EPUB package metadata, or in PDF XMP metadata. HTTP headers override the site-wide file, and metadata inside a document overrides both.
How it works
The report presents itself as a technical response to Article 4 of Directive 2019/790, under which mining for purposes other than scientific research depends on whether rightsholders have expressly reserved their rights in an appropriate manner, such as machine-readable means. It also mentions Article 3, covering research organizations and cultural heritage institutions, and notes that some of its beneficiaries are not bound by certain policy requirements. The Community Group has since discussed finer-grained signals and said it would follow the IETF AI Preferences work before choosing between its own vocabulary and an IETF consensus.
Why it matters
For publishers with EU audiences, the question is whether a reservation was expressed appropriately, and TDMRep offers a precise, documented form that crawlers can parse, unlike free-text terms of service. It works alongside robots.txt: robots.txt asks crawlers not to fetch, while TDMRep states a rights position about mining whatever is fetched. Teams can add the HTTP headers or the well-known file at low cost, but should not assume that every AI crawler reads them, since adoption has not been measured.
Example
A French news publisher serves tdm-reservation: 1 and a tdm-policy URL in the HTTP headers of every article, pointing to a JSON-LD policy with a licensing contact. A crawler that implements TDMRep knows that mining is reserved and where to ask for a license.
Also called: RSL, Really Simple Licensing, RSL 1.0, Open License Protocol
DocumentedEmergingA published 1.0 specification; its own site names publisher and CDN supporters and no AI company committed to honoring it.
RSL, Really Simple Licensing, is an open specification for publishing machine-readable licensing terms for content, including terms for AI training and AI answers, and for acquiring licenses at scale. RSL 1.0 was published on 2025-12-10 at Recommendation status. It adds a middle path between blocking crawlers and free access: access under stated terms.
The specification (document RSL-SPEC-1.0) is published by the RSL Technical Steering Committee, with editors from the RSL Collective, Condé Nast, Ziff Davis, Schema.org, Yahoo, Automattic, O'Reilly Media and Fastly. An RSL document describes content (assets identified by URL) and one or more license elements. Each license has permits and prohibits lists, where prohibitions win, a payment element whose type can be free, attribution, crawl, use, training, purchase, subscription or contribution, and optional reporting and legal terms. Usage categories include all, ai-all, ai-train, ai-input (grounding and generating AI responses), ai-index and search, the last excluding AI summaries.
Sites attach licenses through a License: directive in robots.txt, an HTTP Link header with rel="license", HTML link or script elements, RSS items, or metadata embedded in files.
How it works
A crawler reads robots.txt, finds the License: URL, fetches the RSL document and checks whether its intended use is permitted and on what terms. Under the optional Open License Protocol, built on OAuth 2.0, it requests a license token from the publisher's license server and presents it with its requests, which a server can verify through the optional Crawler Authorization Protocol and its License authentication scheme. The RSL site lists featured supporters including Akamai, Cloudflare, Creative Commons, Fastly, Reddit, Vox Media, Yahoo and Ziff Davis.
Why it matters
RSL moves the control stack from crawl-or-block to crawl under terms, which matters to publishers who want AI answers to use their content on conditions instead of for free or not at all. Its practical weight depends on adoption by AI crawlers and enforcement by infrastructure providers, and the specification's own site documents supporters among publishers and CDNs, with no AI developer named. Teams can publish RSL terms alongside robots.txt at low cost, but should not expect payments or compliance until crawlers commit publicly.
Example
A publisher's robots.txt adds a License: line pointing to an RSL file that permits search for free and requires payment for ai-train and ai-input. A crawler that implements RSL obtains a token from the publisher's license server before using the articles to ground answers.
Also called: pay per crawl, Pay Per Crawl, HTTP 402 for crawlers, Pay Per Use (Cloudflare)
DocumentedEmergingThe mechanism is documented by Cloudflare; its documentation still calls it a closed beta, and Cloudflare is reshaping it as Pay Per Use.
Pay-per-crawl is Cloudflare's mechanism, introduced in private beta on 2025-07-01, that lets site owners charge AI crawlers per request: a crawler without payment intent receives HTTP 402 Payment Required with a price, and Cloudflare bills and settles as merchant of record. In July 2026 Cloudflare said it is evolving the program into Pay Per Use.
Cloudflare's 2025-07-01 announcement describes a third option between blocking and free access. For each crawler, a site owner chooses allow, charge (a flat per-request price for the domain) or block. A crawler can discover the price reactively, receiving a 402 with a crawler-price header and retrying with crawler-exact-price, or declare a ceiling up front with crawler-max-price, receiving 200 and a crawler-charged header when the price fits. Crawlers register and sign requests with Web Bot Auth (HTTP Message Signatures) so they cannot be impersonated, and existing WAF and bot-management rules run first.
The program has kept changing. On 2025-08-27 AI Crawl Control became generally available and let paid plans answer blocked crawlers with a 402. In December 2025 the beta added a discovery API for verified crawlers, and a changelog entry dated 2026-06-16 added URI-pattern exemptions and dynamic pricing. Cloudflare's documentation still described pay per crawl as a closed beta on 2026-11-07.
How it works
On 2026-07-01 Cloudflare announced it is "evolving Pay Per Crawl into Pay Per Use", so publishers are paid when content creates value. With Ceramic.ai, opted-in publishers are paid each time their content appears in Ceramic's AI search results; with You.com, an agent can pay on demand for a specific piece of premium content. The same announcement set new defaults from 2026-09-15: for new domains, training and agent crawlers are blocked on pages that show ads, while search crawlers stay allowed.
Why it matters
Pay-per-crawl turns access into a price, which changes the economics between publishers and AI companies if crawlers agree to pay. For AI search visibility the trade-off is direct: charging or blocking a crawler that feeds an answer engine can remove a site from that engine's answers, while allowing it gives the content away. Teams on Cloudflare should review their AI Crawl Control settings, especially the defaults applied since 2026-09-15, decide per crawler whether visibility or payment matters more, and check verified crawler activity in their logs.
Example
A recipe site sets a per-request price for training crawlers and allows search crawlers for free. A training crawler without a billing relationship receives 402 Payment Required with a crawler-price header, while a search crawler fetches normally and the site stays eligible for that engine's answers.
Also called: Model Context Protocol, MCP server, MCP client
DocumentedOfficial specification and documentation, versioned by date.
MCP, the Model Context Protocol, is an open-source standard for connecting AI applications to external systems such as data sources, tools and workflows through one common protocol. Introduced by Anthropic on 2024-11-25 and hosted since December 2025 by the Linux Foundation's Agentic AI Foundation, its current specification revision is dated 2026-07-28.
The official documentation compares MCP to "a USB-C port for AI applications": one standard way for applications such as Claude or ChatGPT to reach files, databases, search engines, calendars or design tools, in place of a custom integration for each pairing. The specification uses JSON-RPC 2.0 messages between hosts (the AI applications), clients (connectors inside a host) and servers (services that provide context and capabilities). Servers can offer resources (data), prompts (templated messages) and tools (functions the model can call).
The 2026-07-28 revision, which follows 2025-11-25, made the protocol stateless: it removed the initialization handshake and protocol-level sessions, so every request carries its protocol version and client capabilities. It introduced a multi-round-trip pattern for requests that need more input, moved long-running tasks into an official extension, and deprecated the Roots, Sampling and Logging features. Anthropic donated MCP to the Agentic AI Foundation on 2025-12-09, and its maintainers keep the technical decisions.
How it works
A host such as a chat app or coding agent connects, through a client, to one or more MCP servers over a transport such as stdio or Streamable HTTP. Each server describes the tools, resources and prompts it offers; the model decides when to call a tool, and the specification's security principles require the host to obtain the user's consent before exposing data or invoking tools. Extensions such as MCP Apps add interactive interfaces rendered inside conversations.
Why it matters
MCP is how agents reach live systems instead of crawled pages: inventory, prices, bookings, account data, internal documents. For a business, an MCP server lets assistants answer from current, authoritative data in place of whatever was last crawled. It complements the publisher control surface: robots.txt and llms.txt govern what agents read on the open web, AGENTS.md tells coding agents how to work in a project, and MCP governs how they call tools. Teams exposing an MCP server should treat it as an API, with authentication, rate limits and precise tool descriptions.
Example
A retailer runs an MCP server with a product-search tool and a stock-check tool. An assistant connected to it answers "is this jacket in stock in medium near me?" with live inventory and price, without relying on a product page crawled last week.
Also called: Search generative AI control, AI Overviews opt-out, AI Mode opt-out, Search Console generative AI setting
Documented
Search generative AI control is a Search Console setting, tested from 2026-06-03 and available to all sites since 2026-08-31, that lets a site owner exclude the site from Google's generative AI features (AI Overviews, AI Mode and generative AI in Discover) without affecting classic search ranking. Excluded sites receive no impressions or traffic from those features.
The control sits under Settings, then Search generative AI, in Search Console. A property can be set to Include, the default, under which links and content can appear in and help ground AI responses, or to Exclude, under which links and content are hidden from those features and cannot be used to generate AI responses or previews. Child properties inherit the parent's setting unless their owners override it, and changes generally apply within a few days.
Google states that the setting is not a ranking or inclusion signal for other parts of Search, though crawled content may still help Google understand queries and page language. It does not govern model training, for which Google points owners to Google-Extended, and it does not override Merchant Center or Google Ads choices. Removing content from Search entirely still requires noindex. Before this control existed, limiting appearance in AI Overviews meant preview controls such as nosnippet, which also shrink the classic result.
Why it matters
It is the first Google switch that separates appearing in classic results from appearing in AI answers, so publishers no longer face an all-or-nothing trade with Search. For most brands seeking visibility the right setting is Include, since exclusion removes them from answers buyers increasingly read; the option matters most for publishers whose business depends on clicks, subscriptions or licensing. Google suggests reviewing the generative AI performance report first to see what visibility would be lost. Teams should also check that no child property has overridden the domain-level choice by accident.
Example
A subscription news publisher sets its domain property to Exclude after finding that AI Overview impressions rarely turn into subscribers. Its classic search listings are unaffected, while AI Overviews and AI Mode stop showing its articles within a few days.
Also called: Bing AI preferences, ai-grounding directive, ai-training directive, DisallowExtended, Bing grounding opt-out
DocumentedEmergingAnnounced by Microsoft on 2026-10-30; Bing starts honoring them on 2027-02-01, replacing NoArchive and NoCache for AI uses.
Bing AI Training and AI Grounding preferences are two independent publisher controls, announced on 2026-10-30, that let site owners decide separately whether Microsoft may train generative models on their content and whether it may use the content to ground AI answers. They replace the older NoArchive and NoCache signals for these uses and take effect from 2027-02-01.
Microsoft defines training as using content to modify the learned parameters of a generative AI model, following the IETF AI Preferences draft, and grounding as using content as input to inform AI-generated responses; indexing and ranking for classic results are neither. The training preference can be set site-wide or per path in robots.txt through a shared __usage_ai-training user agent with Disallow, or per page with <meta name="Bingbot" content="ai-training:disallow"> or the matching X-Robots-Tag header. The grounding preference is page-level only, with three values: Allow (the default), DisallowExtended, which keeps grounding inside Microsoft-operated experiences such as Copilot and Microsoft 365 while excluding third-party grounding services such as grounding API access in Foundry, and Disallow.
Sites using NoArchive or NoCache can record temporary grounding preferences in Bing Webmaster Tools from 2026-12-15, and new preferences must be published by 2027-03-01. Neither setting changes eligibility for Bing search results.
Why it matters
Bing's index feeds Copilot and, through grounding services, AI products built by others, so a page's grounding setting decides whether it can appear in a wide set of AI answers. Microsoft warns that Disallow or DisallowExtended will significantly reduce visibility in AI experiences and that no choice guarantees inclusion, traffic or payment. Brands seeking AI visibility should leave grounding allowed and audit legacy NoArchive or NoCache tags before the 2027 deadlines, since those tags stop being the AI signal after the transition. Publishers protecting content can now block training without disappearing from AI answers.
Example
A medical publisher adds a Disallow: / rule under the __usage_ai-training user agent in robots.txt, keeps grounding allowed on its public guides so Copilot can cite them, and marks licensed research pages with ai-grounding:disallowextended to keep them out of third-party grounding APIs.
Also called: ACP, OpenAI and Stripe commerce protocol, Agentic Checkout Spec
DocumentedEmergingSpecification and positioning per its site, Stripe's launch post and OpenAI's commerce documentation; adoption beyond ChatGPT is stated by proponents, and ChatGPT's own in-chat checkout was scaled back in 2026.
Agentic Commerce Protocol (ACP) is an open standard, developed by OpenAI and Stripe and released under Apache 2.0 on 2025-09-29, that defines how AI agents, buyers and businesses complete purchases programmatically: product feeds for discovery, a checkout interface merchants implement, and delegated payment through scoped tokens. ChatGPT was its first implementation.
ACP describes itself as an open standard for programmatic commerce flows between buyers, AI agents and businesses, giving merchants a blueprint for making their checkout usable by agents. As documented in OpenAI's commerce docs, its pieces are product feed specifications (feeds, products and promotions, delivered by API or SFTP), an Agentic Checkout Spec that merchants implement so an agent can create and update a checkout, and a Delegated Payment Spec that payment providers implement so credentials pass through a programmatically controlled, permissioned token. Stripe was the first compatible payment provider. Endpoints can be exposed over REST or MCP and connect to existing commerce backends, with the merchant keeping the customer relationship.
OpenAI launched Instant Checkout in ChatGPT on ACP the day the protocol was released, starting with US Etsy sellers. In March 2026 OpenAI said Instant Checkout was moving to merchants' ChatGPT apps and that ACP would serve as infrastructure connecting users to merchants, so the protocol outlives the feature that introduced it. Google's Universal Commerce Protocol addresses the same problem with a different design.
Why it matters
For merchants, ACP is the integration path into ChatGPT's commerce stack: its product feed specification governs how products appear in ChatGPT shopping, and its checkout and payment specifications govern any agent-completed purchase. Even with in-chat checkout scaled back, a clean feed in OpenAI's format is how a merchant controls price, availability and promotions in ChatGPT's product results. Teams that sell through ChatGPT should keep feeds in that format, follow the specification's versioned changes, and hold off on deep checkout integrations until the surfaces that use them settle.
Example
A home-goods retailer exports its catalog nightly as a product feed in OpenAI's format with current prices and stock, so ChatGPT shopping shows accurate availability. A buyer who picks a lamp is sent to the retailer's site to pay.
DocumentedEmergingSpecification and partner lists per ucp.dev and Google's announcements; usage beyond Google's surfaces depends on other platforms adopting it.
Universal Commerce Protocol (UCP) is an open-source standard for agentic commerce, launched by Google on 2026-01-11 with Shopify, Etsy, Wayfair, Target and Walmart, that lets AI agents discover what a business offers and complete catalog search, cart, checkout and order steps through one shared interface. Google built it to power checkout in AI Mode and the Gemini app.
UCP gives agents and commerce platforms a common language so an agent needs no custom integration for each store. A business publishes its supported services and capabilities in a JSON manifest at /.well-known/ucp, which agents read to find endpoints and payment options. The specification at ucp.dev lists capabilities for catalog search and lookup, cart, checkout, identity linking through OAuth 2.0 and orders, plus a draft for lodging bookings; transports include REST and JSON-RPC, with MCP and A2A bindings. Payments build on the Agent Payments Protocol (AP2), which uses payment mandates and verifiable credentials as proof of user consent.
Retailers remain the merchant of record. Google launched UCP to power checkout on eligible product listings in AI Mode and the Gemini app, starting with US retailers, and said in January 2026 that Chrome will support it. The specification is published under Apache 2.0, and ucp.dev now lists co-developers beyond the launch partners, including Amazon, Microsoft, Meta, Salesforce and Stripe.
Why it matters
UCP is the route by which a product listing in Google's AI surfaces becomes a completed purchase without a site visit, so for retailers it decides whether they can sell inside AI Mode and Gemini or only be mentioned there. It also shows commerce data turning into agent-facing infrastructure: capability manifests and checkout endpoints join product feeds as things a store maintains. Teams on Shopify or similar platforms should check whether their platform handles UCP for them; larger retailers should decide how loyalty, discounts and returns carry over when the purchase happens inside an agent.
Example
A sporting-goods retailer publishes a /.well-known/ucp manifest declaring checkout, discount and fulfillment capabilities. When a shopper in AI Mode chooses its tent, the purchase completes with Google Pay inside the conversation and the order lands in the retailer's own system.
Also called: Web Model Context Protocol, document.modelContext, WebMCP origin trial
DocumentedEmergingDraft Community Group Report of the W3C Web Machine Learning Community Group (2026-11-02), not on the W3C Standards Track; Chrome origin trial from Chrome 149.
WebMCP is a proposed browser API that lets a web page expose its own functions to AI agents as structured tools, such as 'search flights' or 'add to cart', so an agent can call them instead of clicking through the interface. It is a W3C community draft, edited by Microsoft and Google engineers, in origin trial in Chrome.
WebMCP brings the tool model of the Model Context Protocol into the browser. The draft specification, published by the W3C Web Machine Learning Community Group and edited by engineers from Microsoft and Google, defines a document.modelContext object through which a page registers tools with registerTool(), each with a name, a description and an input schema that an agent can discover and execute. The tools run in the page's own JavaScript, inside the user's session, so they reuse the site's existing logic and login state. Chrome describes two forms: a declarative API for routine actions defined in HTML forms, and an imperative API for dynamic, JavaScript-driven interactions.
The draft states that it is neither a W3C Standard nor on the Standards Track. Chrome opened an early preview program on 2026-02-10 and an origin trial starting in Chrome 149, announced on 2026-06-09. WebMCP differs from an MCP server, which runs on a backend and is reached over the network: WebMCP tools exist only while the page is open in a browser an agent controls.
Why it matters
Browser agents today guess at buttons and forms, and they stumble on custom widgets, multi-step flows and pages that change state. WebMCP lets a site declare which actions exist and what inputs they need, which could make agent tasks on that site faster and more reliable. It is early: the API is a draft, support is limited to a Chrome trial, and use by mainstream agents is unproven. Product teams with transactional flows can prototype a few tools now; most marketing sites can wait while keeping their forms semantic and well labeled, which also helps the declarative form.
Example
An airline registers a searchFlights tool with origin, destination and date fields. A user's browser agent calls it with structured inputs and reads the results, instead of typing into the date picker and scraping the results grid.
Also called: Agent Skills, SKILL.md, skills, Claude Skills
DocumentedEmergingFormat per agentskills.io; the adoption count ('more than 70 agent platforms') is Anthropic's own claim.
Agent Skills is an open format for packaging instructions, scripts and reference files that teach an AI agent a specific task. Each skill is a folder with a SKILL.md file whose name and description tell the agent when to load it. Anthropic launched it in October 2025 and donated it to the Agentic AI Foundation in October 2026.
A skill is a folder whose required SKILL.md file holds metadata (at minimum a name and a description) and instructions, optionally with scripts/, references/ and assets/ folders. Agents load skills through progressive disclosure: at startup they read only names and descriptions, they read the full instructions when a task matches a description, and they run bundled code or open referenced files only as needed. That keeps many skills available at a small context cost.
Anthropic introduced Agent Skills in October 2025, published the format as an open standard in December 2025, and on 2026-10-22 donated the standard, its name, the GitHub organization and the agentskills.io domain to the Agentic AI Foundation under the Linux Foundation, stating that more than 70 agent platforms, including ChatGPT, Cursor, GitHub Copilot and VS Code, had integrated it. Skills differ from AGENTS.md, which gives an agent standing instructions for a project, and from MCP, which connects agents to live tools and data.
Why it matters
Skills are becoming a distribution channel into agents: a company can publish a skill that teaches agents to use its product, API or method correctly, the way a developer guide teaches people. When an agent picks a tool to complete a task, a ready skill lowers the cost of choosing that product and cuts misuse. Cloudflare's Agent Readiness score already checks whether a site publishes skills. Teams with developer-facing products can write skills for their main workflows; for most content sites, clear machine-readable documentation matters more than the format itself.
Example
A payments company publishes a skill named 'create-subscription' whose description says when to use it and whose instructions walk an agent through idempotency keys and webhooks, with a script that validates the request. Coding agents that load it stop making the mistakes the company kept seeing in support tickets.
Also called: Markdown for Agents, Accept: text/markdown, Markdown negotiation, agent-friendly Markdown
DocumentedEmergingHTTP content negotiation is long established; its use for agents is recent and documented chiefly by Cloudflare, whose scan found 3.9% of top sites passing.
Markdown content negotiation is serving a Markdown version of a web page to clients that request it with the HTTP header Accept: text/markdown, while browsers keep receiving HTML at the same URL. Cloudflare popularized it as Markdown for Agents on 2026-02-12, converting pages at its edge so AI agents read lighter, cleaner text.
HTTP has long let a client state which formats it prefers through the Accept header and let a server choose a representation accordingly. Applied to AI agents, a server returns text/markdown for the same URL when asked and adds Vary: Accept so caches store HTML and Markdown separately. This differs from the llms.txt approach of publishing separate .md files and listing them in an index: negotiation needs no new URLs.
Cloudflare's Markdown for Agents performs the conversion at the edge for enabled zones on paid plans. It strips headers, footers, navigation, scripts and styles, adds YAML frontmatter from meta tags, keeps JSON-LD in a fenced block, and returns an x-markdown-tokens estimate and a Content-Signal header that by default allows training, search and AI input unless the origin sets its own. Cloudflare measured its announcement post at 16,180 tokens as HTML and 3,150 as Markdown, and said coding agents such as Claude Code and OpenCode already send the header.
Why it matters
Agents pay for every token they read, and page chrome can outweigh the content, so a clean Markdown representation makes a page cheaper and less error-prone to use, which may make it likelier to be read in full. It also removes guesswork from extraction: headings, lists and tables arrive as structure. The cost is governance: both representations must say the same thing, since serving agents different content than people would look like cloaking. Teams on Cloudflare can switch it on; others can add negotiation in their framework. Answer engines have not documented using it for retrieval or citation.
Example
A developer-docs site enables Markdown negotiation. When a coding agent fetches the authentication guide with Accept: text/markdown, it receives about 2,000 tokens of headings and code blocks instead of 11,000 tokens of HTML, and the browser version of the page is unchanged.
Also called: HTTP Message Signatures for bots, signed agents, bot request signing, Signature-Agent
DocumentedEmergingIETF Web Bot Auth working group draft (revision -01, 2026-10-29), not yet an RFC; supported by Cloudflare and in experiment at Google.
Web Bot Auth is an emerging IETF protocol that lets bots and AI agents prove who they are by cryptographically signing their HTTP requests, using HTTP Message Signatures and a published key directory. Sites can verify the signature instead of trusting a user agent string, which is easy to spoof, or an IP list, which is hard to maintain.
In Web Bot Auth, an operator holds private keys and publishes the matching public keys in a directory at /.well-known/http-message-signatures-directory. Each request carries a Signature header, a Signature-Input header describing the signed components with the tag web-bot-auth, and a Signature-Agent header pointing to the key directory, so the receiving site or its CDN can fetch the key and check that the request came from that operator. The protocol proves control of a key; it says nothing about whether the bot is welcome or what it may do.
The working-group draft, titled 'HTTP Message Signatures for automated traffic', is authored by engineers from Cloudflare and Google; revision -01 is dated 2026-10-29. Cloudflare accepts Web Bot Auth as a verification method in its verified bots and agents program, and Google's crawler documentation says Google-Agent is experimenting with it under the https://agent.bot.goog identity.
Why it matters
As agents fetch pages for individual users and often ignore robots.txt, sites need a reliable way to tell a real assistant from a scraper using its name. Signed requests make it practical to allow, rate-limit or charge specific operators, and they already underpin verified-bot programs such as Cloudflare's. For a brand, the practical step is to make sure its CDN or bot manager verifies signatures instead of blocking unfamiliar automated traffic, so legitimate agents can complete visits and purchases. Adoption by major AI operators is partial and the specification can still change.
Example
A ticketing site's bot manager sees two requests claiming to be the same AI agent. One carries a valid signature tied to the operator's key directory and reaches the event page; the unsigned one is challenged.
Also called: AIPREF, aipref, Content-Usage header, AI usage preferences, train-ai
DocumentedEmergingTwo IETF working-group drafts on the Standards Track (vocabulary -08, 2026-09-14; attachment -05, 2026-08-19); the vocabulary draft states its text does not yet reflect working-group consensus.
AI Preferences (AIPREF) is an IETF effort to standardize how publishers state whether their content may be used for AI training, AI use and search. It defines a shared vocabulary with labels such as train-ai and search, and ways to attach those preferences through a Content-Usage HTTP header or a new robots.txt rule.
The IETF AI Preferences working group splits the problem in two. The vocabulary draft defines categories of use: AI Training (train-ai), using an asset to modify a generative model's learned parameters; AI Use (ai-use), using an asset as input to a generative model when the user did not directly provide it; and Search (search), using an asset in an application that selects assets and links users to their original location. The attachment draft, by Gary Illyes of Google and Martin Thomson of Mozilla, carries those preferences in a Content-Usage HTTP response header, for example Content-Usage: train-ai=n, and in a Content-Usage rule inside robots.txt groups, updating RFC 9309.
Both documents are drafts, and the vocabulary text warns that it does not yet reflect consensus, so labels may change. Microsoft's October 2026 Bing preferences already adopt the draft's definition of training, and Cloudflare's Content Signals express similar choices with different labels.
Why it matters
Today a site that wants to block AI training but stay visible in AI search must learn each operator's crawler names and opt-out rules. A shared vocabulary would let one statement cover every compliant operator and separate training from answer-time use, the distinction that decides whether a brand stays citable. Marketers should not wait on it: preferences are signals, compliance is voluntary, and the labels are unsettled. The practical move is to follow the drafts, keep crawler-specific rules for the operators that matter, and avoid blanket AI blocks that also remove the site from AI answers.
Example
A publisher adds Content-Usage: train-ai=n to its robots.txt group for all crawlers, signaling that its articles may be indexed and cited but not used to train models, and keeps specific Disallow rules for training crawlers that do not yet read the new rule.