---
title: "Standards, files and protocols | The AI Search Dictionary"
description: "The files and protocols that tell crawlers and agents what they may do with a site: robots.txt, llms.txt, schema.org, MCP and more."
canonical: "https://hireobi.com/glossary/standards-and-protocols"
human_url: "https://hireobi.com/glossary/standards-and-protocols"
markdown_url: "https://hireobi.com/glossary/standards-and-protocols.md"
updated: "2026-11-08"
---

# Standards, files and protocols

The files and protocols that tell crawlers and agents what they may do with a site: robots.txt, llms.txt, schema.org, MCP and more. A part of The AI Search Dictionary (https://hireobi.com/glossary), version 1.6, last verified 2026-11-08. License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/), with attribution to Obi and a link to https://hireobi.com/glossary.

### robots.txt (Robots Exclusion Protocol, RFC 9309)

Link: https://hireobi.com/glossary/standards-and-protocols#robots-txt
Also called: robots.txt, Robots Exclusion Protocol, REP, RFC 9309, robots exclusion standard
Label: Documented (An IETF Proposed Standard; each operator documents how its own crawlers follow it.)

**robots.txt is the plain-text file at a site's root where owners state which paths each crawler may fetch, standardized as the Robots Exclusion Protocol in IETF RFC 9309 (September 2022). AI companies map their opt-outs to user-agent tokens in it, but it governs crawling only, and compliance is voluntary.**

RFC 9309, by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, is an IETF Proposed Standard that formalized decades of practice. The file must sit at `/robots.txt`, encoded in UTF-8. It is organized in groups: one or more `User-agent` lines, matched case-insensitively against a crawler's product token, followed by `Allow` and `Disallow` rules. When several rules match a path, the most specific match (the one with the most octets) wins, and an equivalent `Allow` beats a `Disallow`. A crawler with no matching group falls back to the `*` group; with no `*` group, everything is allowed.

The RFC also sets operational rules. Crawlers must parse at least 500 KiB, should not use a cached copy for more than 24 hours unless the file is unreachable, may treat a 4xx response as permission to crawl everything, and must treat a 5xx or network error as a complete disallow. In the RFC's own words, "These rules are not a form of access authorization."

**How it works.** AI operators publish tokens with distinct purposes, which lets a site separate uses. OpenAI documents `OAI-SearchBot` for ChatGPT search and `GPTBot` for training, and says a robots.txt change can take about 24 hours to reach its search systems. Google points to `Google-Extended` for limiting training and grounding in some of its AI systems outside Search, and Apple's `Applebot-Extended` controls whether Applebot's crawl may train Apple's foundation models. User-initiated fetchers are a separate case: OpenAI says robots.txt rules may not apply to `ChatGPT-User`, and Perplexity says `Perplexity-User` generally ignores them.

**Why it matters.** robots.txt is the main control surface between a site and AI crawlers, so one line can make a site ineligible for an engine's answers or withhold it from training. Two limits matter. It governs fetching, and has no effect on content already collected or obtained elsewhere, which is why usage-level signals such as Content Signals, TDMRep and RSL exist. And it is a request that a crawler can ignore. Teams should audit robots.txt and CDN bot rules together, decide token by token, and confirm the effect in server logs.

**Example.** A group `User-agent: OAI-SearchBot` with `Allow: /`, followed by a group `User-agent: GPTBot` with `Disallow: /`, keeps a site eligible to be fetched for ChatGPT search while withholding it from OpenAI's training crawler. It removes nothing collected before the change, and it does not govern user-initiated `ChatGPT-User` fetches.

Related terms: [X-Robots-Tag](https://hireobi.com/glossary/standards-and-protocols#x-robots-tag), [AI crawler](https://hireobi.com/glossary/how-ai-search-works#ai-crawler), [Crawl propagation delay](https://hireobi.com/glossary/how-ai-search-works#crawl-propagation-delay), [GPTBot](https://hireobi.com/glossary/how-ai-search-works#gptbot), [OAI-SearchBot](https://hireobi.com/glossary/how-ai-search-works#oai-searchbot), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals)

Sources:

- IETF RFC 9309, Robots Exclusion Protocol (Koster, Illyes, Zeller, Sassman), Proposed Standard, published 2022-09, read 2026-11-07: https://www.rfc-editor.org/rfc/rfc9309.html
- IETF Datatracker, RFC 9309 status page, published 2022-09, read 2026-11-07: https://datatracker.ietf.org/doc/rfc9309/
- OpenAI, Overview of OpenAI Crawlers, published n/d, read 2026-11-07: https://developers.openai.com/api/docs/bots
- Perplexity, Perplexity Crawlers, published n/d, read 2026-11-07: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Google Search Central, AI features and your website, published n/d, read 2026-11-07: https://developers.google.com/search/docs/appearance/ai-features
- Apple Support, About Applebot, published 2026-09-04, read 2026-11-07: https://support.apple.com/en-us/119829

Checked: 2026-11-07

### X-Robots-Tag

Link: https://hireobi.com/glossary/standards-and-protocols#x-robots-tag
Also called: X-Robots-Tag header, robots HTTP header
Label: Documented (Documented by Google, Apple and Bing for their own crawlers.)

**X-Robots-Tag is an HTTP response header that carries the same indexing and snippet rules as the robots meta tag, such as `noindex` or `nosnippet`, for any file a server returns. It is the way to apply those rules to PDFs, images and other non-HTML resources, and it can target a single crawler by name.**

Google documents that any rule usable in a robots meta tag can be sent as `X-Robots-Tag`, as several headers or as a comma-separated list, for example `X-Robots-Tag: noindex`. A rule can be scoped to one crawler by prefixing its name, as in `X-Robots-Tag: googlebot: nofollow`; rules without a name apply to all crawlers. Header names, crawler names and values are case-insensitive. When rules conflict, Google applies the more restrictive one, and it combines negative rules from general and crawler-specific instructions.

Apple says Applebot honors the header as well and recommends it for non-HTML resources such as PDFs or images. Bing documents its own directive set, including `nocache` and `noarchive`, which it tied in 2023 to content use in what was then Bing Chat. The header only works on a URL a crawler may fetch: Google states that if robots.txt blocks a page, its indexing and serving rules are never seen.

**Why it matters.** Many cited resources are not HTML: whitepapers, research PDFs, slide decks, images. X-Robots-Tag is their only page-level control, so it decides whether a PDF can appear in search features and, through `nosnippet` or `max-snippet`, how much of it Google may use in AI Overviews and AI Mode. Teams should set it at the server or CDN for the file types that need rules, test responses with a header checker, and avoid sending conflicting rules from different layers.

**Example.** A server sends `X-Robots-Tag: noindex` on every response under /downloads/ that ends in .pdf. The whitepapers stay out of Google's index and search features, while the HTML landing pages linking to them remain indexable.

Related terms: [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [nosnippet, max-snippet, data-nosnippet, noindex (preview controls)](https://hireobi.com/glossary/standards-and-protocols#nosnippet-max-snippet-data-nosnippet-noindex), [Googlebot](https://hireobi.com/glossary/how-ai-search-works#googlebot), [Applebot and Applebot-Extended](https://hireobi.com/glossary/how-ai-search-works#applebot-and-applebot-extended), [Bingbot](https://hireobi.com/glossary/how-ai-search-works#bingbot)

Sources:

- Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, published n/d, read 2026-11-07: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
- Apple Support, About Applebot, published 2026-09-04, read 2026-11-07: https://support.apple.com/en-us/119829
- Bing Webmaster Blog, Announcing new options for webmasters to control usage of their content in Bing Chat, published 2023-09-22, read 2026-11-07: https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat

Checked: 2026-11-07

### nosnippet, max-snippet, data-nosnippet, noindex (preview controls)

Link: https://hireobi.com/glossary/standards-and-protocols#nosnippet-max-snippet-data-nosnippet-noindex
Also called: preview controls, snippet controls, nosnippet, max-snippet, data-nosnippet, noindex
Label: Documented (Documented by Google, Apple and Bing for their own products; each engine's scope differs.)

**Preview controls are robots directives that limit how much of a page a search engine may show, and Google documents them as the page-level way to limit use in AI Overviews and AI Mode: `nosnippet`, `max-snippet`, the `data-nosnippet` attribute and `noindex`. Since 2026, Search Console also offers a site-wide opt-out from Google's generative AI features.**

Google's specifications define each control. `nosnippet` removes text snippets and video previews and, per Google, keeps the content from being used as direct input for AI Overviews and AI Mode. `max-snippet:[number]` caps snippet length in characters and also limits use in AI Overviews and AI Mode, except where the publisher has granted separate permission, for example through structured data or a license; `max-snippet:0` equals `nosnippet`. The `data-nosnippet` attribute excludes parts of a page and is valid only on `span`, `div` and `section` elements. `noindex` removes the page from results altogether. When rules conflict, the most restrictive applies.

Other engines differ. Apple says Applebot will not use content marked `nosnippet` as context for AI-generated output, and keeps pages marked `isAccessibleForFree: false` out of that context. Bing announced in September 2023 that `noarchive` excludes content from its chat answers and from training Microsoft's generative models, while `nocache` limits that use to URL, title and snippet; with both present, Bing applies `nocache`.

**How it works.** Announced in June 2026 and rolled out to all websites as of 2026-08-31, Google's Search generative AI control sits in Search Console settings, per property. Excluding a property removes its links and content from AI Overviews, AI Mode and generative AI features in Discover, with no traffic or impressions from them. Google says the control is not used as a ranking signal elsewhere in Search and does not affect training, for which it points to `Google-Extended`. Google's AI features documentation page did not yet mention the control when checked on 2026-11-07.

**Why it matters.** These controls decide how much of a page Google's AI features can use while the page stays indexed for classic results, something `Google-Extended` does not do for Search. The trade-off is real: limiting snippets limits classic snippets too, and the Search Console control removes a site from AI features entirely, along with the traffic they bring. Teams should decide per page type (pricing, paywalled articles, documentation), prefer `data-nosnippet` for narrow exclusions, check how Apple and Bing read the same tags, and verify changes with the URL Inspection tool.

**Example.** A publisher adds `<meta name="robots" content="max-snippet:120">` to its news articles and wraps subscriber-only paragraphs in `<div data-nosnippet>`. The pages stay indexed and ranked, while Google may show at most 120 characters in snippets and AI features, and none of the protected paragraphs.

Related terms: [Google-Extended](https://hireobi.com/glossary/how-ai-search-works#google-extended), [X-Robots-Tag](https://hireobi.com/glossary/standards-and-protocols#x-robots-tag), [Google AI Overviews](https://hireobi.com/glossary/platforms#google-ai-overviews), [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [Search generative AI control (Search Console)](https://hireobi.com/glossary/standards-and-protocols#search-generative-ai-control), [Bing AI Training and AI Grounding preferences](https://hireobi.com/glossary/standards-and-protocols#bing-ai-training-and-grounding-preferences)

Sources:

- Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, published n/d, read 2026-11-07: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
- Google Search Central, AI features and your website, published n/d, read 2026-11-07: https://developers.google.com/search/docs/appearance/ai-features
- Search Console Help, Search generative AI control, published n/d, read 2026-11-07: https://support.google.com/webmasters/answer/16908024?hl=en
- Google Keyword blog, New opportunities, control and insights for website owners (Mrinalini Loew; updated 2026-08-31), published 2026-06-03, read 2026-11-07: https://blog.google/products-and-platforms/products/search/new-controls-website-owners/
- Apple Support, About Applebot, published 2026-09-04, read 2026-11-07: https://support.apple.com/en-us/119829
- Bing Webmaster Blog, Announcing new options for webmasters to control usage of their content in Bing Chat, published 2023-09-22, read 2026-11-07: https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat

Checked: 2026-11-07

### llms.txt (the protocol entry)

Link: https://hireobi.com/glossary/standards-and-protocols#llms-txt-2
Also called: llms.txt, /llms.txt, llms.txt proposal, LLMs.txt
Label: Documented, Claimed (Documented as a proposal with visible adoption; claimed as a visibility mechanism, since Google says its AI features need no such file and none of the engine documentation reviewed here says it is read for answers.)

**llms.txt is a proposed Markdown file at a site's root, `/llms.txt`, that gives AI agents a description of the site and curated links to Markdown versions of its pages. Proposed by Jeremy Howard in September 2024 and revised as v2 in August 2026, it is widely published, while Google says its AI features need no such file.**

The proposal at llmstxt.org specifies a file whose only required element is an H1 with the site or project name. It can be followed by a blockquote summary, free-form detail sections without headings, and H2 sections that list Markdown links, each optionally followed by a colon and notes. By convention, an H2 named Optional holds secondary links an agent can skip. Pages are expected to offer Markdown versions at the same URL, with `.md` appended (`page.html.md`) or replacing the extension (`page.md`).

Version 2, dated 2026-08-10, added discovery through standard link relations: `rel="alternate" type="text/markdown"` points from a page to its Markdown version, and `rel="describedby"` points to the llms.txt covering it, either as an HTML link element or an HTTP `Link` header. It also defined how llms.txt files in subpaths apply (the most specific one wins), removed the `llms_txt2ctx` tool from the proposal, and gave the Optional section no mechanical meaning.

**How it works.** An agent fetches `/llms.txt`, reads the summary, follows the links relevant to its task, and retrieves clean Markdown instead of rendering HTML. The proposal reports its own adoption: thousands of sites, automatic generation by documentation platforms (it lists Mintlify, GitBook, Yoast SEO and others as integrations), audits in Chrome's Lighthouse, and llms.txt files for the developer documentation of OpenAI, Anthropic and Gemini. These are the proponent's statements, not an independent audit.

**Why it matters.** For agents that read documentation, such as coding assistants and research agents, llms.txt and Markdown page versions cut the cost of reading a site and avoid rendering failures. For visibility in AI answers, the evidence is thin: Google states that sites need no new machine-readable files or AI text files to appear in its AI features. Teams can publish llms.txt cheaply as agent infrastructure, keep it in sync with the sitemap, and avoid promising citation gains from it.

**Example.** A payments company publishes `/llms.txt` with an H1, a one-paragraph summary, and H2 sections for its main documentation areas, each line linking to a `.md` version of a page with a short note. Its HTML pages carry `rel="alternate" type="text/markdown"` links, so an agent landing on any page can find the Markdown.

Related terms: [llms.txt](https://hireobi.com/glossary/visibility-levers#llms-txt), [llms-full.txt](https://hireobi.com/glossary/standards-and-protocols#llms-full-txt), [The llms.txt proposal](https://hireobi.com/glossary/studies-and-sources#the-llms-txt-proposal), [AGENTS.md](https://hireobi.com/glossary/visibility-levers#agents-md), [Content extractability](https://hireobi.com/glossary/visibility-levers#content-extractability), [Sitemaps](https://hireobi.com/glossary/standards-and-protocols#sitemaps)

Sources:

- Jeremy Howard, The /llms.txt file (proposal v2; first published 2024-09-03), published 2026-08-10, read 2026-11-07: https://llmstxt.org/
- llmstxt.org, Changes from v1 to v2, published n/d, read 2026-11-07: https://llmstxt.org/changes.html
- Google Search Central, AI features and your website, published n/d, read 2026-11-07: https://developers.google.com/search/docs/appearance/ai-features

Checked: 2026-11-07

### llms-full.txt

Link: https://hireobi.com/glossary/standards-and-protocols#llms-full-txt
Also called: llms full text file, full-documentation text file
Label: Observed (A convention visible on many documentation sites; absent from the llms.txt proposal itself.)

**llms-full.txt is a community convention that complements llms.txt: a single plain-text file containing a site's entire documentation in Markdown, so an agent can ingest everything in one request. It is not part of the llms.txt proposal, but documentation platforms such as Mintlify generate it automatically, and some files run to tens of megabytes.**

Where llms.txt is a map with links, llms-full.txt is the territory: the full text of every documentation page concatenated into one file at the site root. Neither version of the llms.txt proposal mentions it, including the page describing what changed in v2; it spread through documentation tooling. Mintlify, for example, hosts both files automatically for every project and describes llms-full.txt as combining the entire documentation site into a single file, with each page's title, source URL, description and full Markdown content.

Live examples checked on 2026-11-07 show the range of sizes: Mintlify's own documentation file is about 1.7 MB, the Model Context Protocol site's about 2.4 MB, and the Claude developer platform's about 50 MB. The scope is documentation-style content meant for bulk ingestion; the convention defines no ordering, size limit or update rule.

**Why it matters.** A single file is convenient for agents and for developers loading documentation into a model's context or a retrieval index, and it replaces dozens of separate fetches. It also has costs: large files exceed most context windows, go stale if not regenerated, and duplicate content already on HTML pages. None of the engine documentation reviewed here mentions using it for citations. Teams with documentation can publish it when their platform generates it, keep it current, and treat it as developer infrastructure rather than a visibility lever.

**Example.** Instead of fetching 40 separate `.md` pages, a coding agent downloads one `llms-full.txt` holding the whole API documentation. At 2 MB it fits a retrieval index easily; at 50 MB it has to be chunked before any model can use it.

Related terms: [llms.txt (the protocol entry)](https://hireobi.com/glossary/standards-and-protocols#llms-txt-2), [llms.txt](https://hireobi.com/glossary/visibility-levers#llms-txt), [Context window](https://hireobi.com/glossary/how-ai-search-works#context-window), [Chunking](https://hireobi.com/glossary/how-ai-search-works#chunking), [AGENTS.md](https://hireobi.com/glossary/visibility-levers#agents-md)

Sources:

- Mintlify documentation, llms.txt, published n/d, read 2026-11-07: https://www.mintlify.com/docs/ai/llmstxt
- Jeremy Howard, The /llms.txt file (proposal v2), published 2026-08-10, read 2026-11-07: https://llmstxt.org/
- llmstxt.org, Changes from v1 to v2, published n/d, read 2026-11-07: https://llmstxt.org/changes.html
- Mintlify, llms-full.txt (live file), published n/d, read 2026-11-07: https://www.mintlify.com/docs/llms-full.txt
- Model Context Protocol, llms-full.txt (live file), published n/d, read 2026-11-07: https://modelcontextprotocol.io/llms-full.txt
- Anthropic, Claude developer platform llms-full.txt (live file), published n/d, read 2026-11-07: https://platform.claude.com/llms-full.txt

Checked: 2026-11-07

### ai.txt (Spawning)

Link: https://hireobi.com/glossary/standards-and-protocols#ai-txt
Also called: ai.txt, Spawning ai.txt
Label: Documented, Observed (Design and generated syntax are documented in Spawning's announcement and plugin code; honoring by readers is the vendor's claim; the specification page serves a maintenance notice.)

**ai.txt is a file format proposed by Spawning in 2023 for declaring, at a site's root, whether its text, images, audio, video and code may be used to train AI models. Spawning designed it to be checked when media is downloaded rather than when a site is crawled. Its specification page now shows only a maintenance notice.**

Spawning announced ai.txt on 2023-05-30 as a robots.txt-like file for AI permissions. Its rationale: robots.txt is read at crawl time, so it cannot affect links already collected in datasets such as LAION-5B, while a file checked at download time lets owners change permissions for media hosted on their own domains. Spawning framed it as a machine-readable opt-out under the text and data mining exception in Article 4 of the EU copyright directive; that is the vendor's framing, and whether the file is honored depends entirely on its readers. Spawning's API relayed the permissions to partners, naming Hugging Face and Stability AI.

The exact syntax survives in Spawning's open-source WordPress plugin (Apache-2.0, last pushed 2024-07-09). Its generator writes a `# Spawning AI` header and `User-agent: *`, then per-extension `Allow:` or `Disallow:` lines built from glob lists per media type (for example `Disallow: *.mp3`), ending with `Disallow: /`, or `Allow: /` when text is allowed. When every type is blocked, it writes `Disallow: *` and `Disallow: /`.

**Why it matters.** ai.txt was an early attempt at usage-level signaling, separating permission to train from permission to crawl, and the idea returns in later signals such as Cloudflare's Content Signals, TDMRep and RSL. Today it is a weak lever: the specification page is in maintenance, current adoption is undocumented, and the only readers Spawning named were its own API partners. Teams wanting to state training preferences should use robots.txt tokens and the newer signals, and can leave an existing ai.txt in place as a harmless legacy file.

**Example.** A photographer's site serves a plugin-generated `ai.txt` that disallows image and video extensions while allowing text, ending with `Allow: /`. A dataset tool that checks ai.txt before downloading would skip the photos; a crawler that never reads the file is unaffected.

Related terms: [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals), [TDMRep (TDM Reservation Protocol)](https://hireobi.com/glossary/standards-and-protocols#tdmrep-tdm-reservation-protocol), [RSL (Really Simple Licensing)](https://hireobi.com/glossary/standards-and-protocols#rsl-really-simple-licensing), [Training data vs retrieval data](https://hireobi.com/glossary/how-ai-search-works#training-data-vs-retrieval-data)

Sources:

- Spawning, ai.txt: A new way for websites to set permissions for AI (announcement), published 2023-05-30, read 2026-11-07: https://spawning.substack.com/p/aitxt-a-new-way-for-websites-to-set
- Spawning-Inc/spawning-wp-plugin, WordPress plugin generating ai.txt (form-handler.php and ai_txt_options.json, main branch), published n/d, read 2026-11-07: https://github.com/Spawning-Inc/spawning-wp-plugin
- Spawning, ai.txt page (serves a Source.Plus maintenance notice), published n/d, read 2026-11-07: https://spawning.ai/ai-txt

Checked: 2026-11-07

### Schema.org

Link: https://hireobi.com/glossary/standards-and-protocols#schema-org
Also called: schema.org vocabulary, schema markup, structured data vocabulary
Label: Documented, Claimed (The vocabulary is documented; claims that particular types raise AI citations are vendor assertions.)

**Schema.org is the shared vocabulary of types and properties, such as Article, Organization, Product, FAQPage or DefinedTerm, that websites use to describe their content to machines, most often embedded as JSON-LD. Google requires no special schema for its AI features, but says structured data should match the text visible on the page.**

Schema.org is a collaborative vocabulary of hierarchical types, all descending from Thing, and their properties, published in numbered releases; the release shown on its pages on 2026-11-07 is 30.1, dated 2026-09-16. Google supports three serializations (JSON-LD, Microdata and RDFa), recommends JSON-LD, and notes that most Search structured data uses the schema.org vocabulary. Search engines use selected types to power rich results: Google's search gallery, last updated 2026-10-16, lists features including articles, breadcrumbs, events, FAQs, organizations, products, recipes and reviews.

For AI features, Google states that "there's also no special schema.org structured data that you need to add", while listing as good practice making sure structured data matches the visible text. Vendor claims that particular types raise AI citations remain claims. The vocabulary's clearest value for AI search is stating entities, facts and relationships unambiguously.

**Why it matters.** Structured data is one of the few ways a site can state facts in machine-readable form: who the organization is, what a product costs, who wrote an article, what a term means. Engines use it for rich results and entity understanding, and Google's guidance treats mismatches with visible content as a problem. Teams should mark up what each page genuinely shows, use the types search engines document, connect entities with properties such as `sameAs`, validate the markup, and treat any AI citation lift from markup as unproven.

**Example.** A glossary page marks itself up as a `DefinedTermSet` and each entry as a `DefinedTerm` with `name`, `description` and `inDefinedTermSet`, so its definitions are machine-readable even though Google offers no glossary rich result.

Related terms: [Structured data (schema.org) for AI features](https://hireobi.com/glossary/visibility-levers#structured-data-for-ai-features), [DefinedTerm / DefinedTermSet](https://hireobi.com/glossary/standards-and-protocols#definedterm-definedtermset), [sameAs](https://hireobi.com/glossary/visibility-levers#sameas), [Entity SEO](https://hireobi.com/glossary/visibility-levers#entity-seo), [Google Knowledge Graph](https://hireobi.com/glossary/visibility-levers#google-knowledge-graph)

Sources:

- Schema.org, DefinedTerm (release 30.1), published 2026-09-16, read 2026-11-07: https://schema.org/DefinedTerm
- Google Search Central, Introduction to structured data markup in Google Search, published n/d, read 2026-11-07: https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data
- Google Search Central, Structured data markup that Google Search supports (search gallery), published 2026-10-16, read 2026-11-07: https://developers.google.com/search/docs/appearance/structured-data/search-gallery
- Google Search Central, AI features and your website, published n/d, read 2026-11-07: https://developers.google.com/search/docs/appearance/ai-features

Checked: 2026-11-07

### DefinedTerm / DefinedTermSet

Link: https://hireobi.com/glossary/standards-and-protocols#definedterm-definedtermset
Also called: DefinedTerm, DefinedTermSet, schema.org DefinedTerm, glossary markup
Label: Documented

**DefinedTerm and DefinedTermSet are schema.org types for publishing glossaries and controlled vocabularies in machine-readable form. A DefinedTerm is a word, name, acronym or phrase with a formal definition; a DefinedTermSet is the glossary, dictionary or classification scheme that contains it. Google offers no rich result for either type.**

On schema.org (release 30.1, 2026-09-16), DefinedTerm sits under Intangible and is described as "A word, name, acronym, phrase, etc. with a formal definition", used for subject classification, glossaries, dictionaries and product or creative-work types. Its own properties are `termCode` (a code identifying the term within its set) and `inDefinedTermSet` (the containing set, as a DefinedTermSet or a URL), and it inherits general properties such as `name` and `description` from Thing. DefinedTermSet sits under CreativeWork, so it can carry authorship, dates and licensing, and it lists its terms with `hasDefinedTerm`. CategoryCode and CategoryCodeSet are their more specific subtypes for coded classifications.

The types describe the meaning of terms and trigger no Google rich result: DefinedTerm does not appear in Google's structured data search gallery, last updated 2026-10-16.

**Why it matters.** Glossaries are a natural source for definitional answers, and DefinedTerm markup makes each definition an addressable, machine-readable unit with a stable URL and a declared parent set. That helps any system that parses structured data identify the term, its definition and its publisher without guessing from layout. None of the engine documentation reviewed here claims a citation benefit, so the case rests on clarity: teams publishing glossaries can add the markup cheaply, keep it identical to the visible text, and give each term a permanent anchor.

**Example.** A glossary page carries one `DefinedTermSet` with a name, publisher and license, and each entry is a `DefinedTerm` with `name`, `description`, a `url` pointing to its anchor, and `inDefinedTermSet` pointing back to the set.

Related terms: [Schema.org](https://hireobi.com/glossary/standards-and-protocols#schema-org), [Structured data (schema.org) for AI features](https://hireobi.com/glossary/visibility-levers#structured-data-for-ai-features), [sameAs](https://hireobi.com/glossary/visibility-levers#sameas), [Entity SEO](https://hireobi.com/glossary/visibility-levers#entity-seo), [Answer capsule (answer block)](https://hireobi.com/glossary/visibility-levers#answer-capsule-answer-block)

Sources:

- Schema.org, DefinedTerm (release 30.1), published 2026-09-16, read 2026-11-07: https://schema.org/DefinedTerm
- Schema.org, DefinedTermSet (release 30.1), published 2026-09-16, read 2026-11-07: https://schema.org/DefinedTermSet
- Google Search Central, Structured data markup that Google Search supports (search gallery), published 2026-10-16, read 2026-11-07: https://developers.google.com/search/docs/appearance/structured-data/search-gallery

Checked: 2026-11-07

### IndexNow

Link: https://hireobi.com/glossary/standards-and-protocols#indexnow
Also called: IndexNow protocol, IndexNow API
Label: Documented (An open protocol documented at indexnow.org; how quickly each engine acts on a submission is not documented.)

**IndexNow is an open protocol that lets a website notify participating search engines the moment a URL is added, updated or deleted, instead of waiting for a recrawl. A submission to one participating engine is shared with the others, which include Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Google is not among them.**

A site proves control of its host with a key file, a UTF-8 text file containing the key, placed at the root and named after the key, or at a path declared with the `keyLocation` parameter; a key has 8 to 128 characters drawn from letters, digits and dashes. A single URL is submitted with an HTTP GET to an engine's `/indexnow` endpoint carrying `url` and `key`. A batch uses an HTTP POST with a JSON body (`host`, `key`, optional `keyLocation`, `urlList`) of up to 10,000 URLs. Responses include 200 (received), 202 (received, key validation pending), 403 (invalid key), 422 (URLs outside the host) and 429 (too many requests); a 200 confirms receipt only.

Participating engines agree that submitted URLs are shared automatically with all the others. The registry file on indexnow.org lists Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Google is not on that list.

**Why it matters.** Bing's index feeds Microsoft Copilot and Bing's AI summaries, and Bing presents IndexNow as a way to help AI systems reference the current version of a page. Bing's own guidance pairs a complete sitemap with accurate `lastmod` dates for coverage and IndexNow for real-time changes, while stating that no tool can guarantee when or how content appears in AI-generated results. Teams can enable IndexNow through their CMS or CDN, submit on every publish, update and deletion, and watch Bing Webmaster Tools for errors.

**Example.** A retailer changes the price of 3,000 products overnight. Its platform posts the changed URLs to IndexNow in one batch, so Bing and the other participating engines learn of the changes immediately instead of at their next recrawl. Google still finds them through crawling and the sitemap.

Related terms: [Bingbot](https://hireobi.com/glossary/how-ai-search-works#bingbot), [Microsoft Copilot (Bing and consumer app)](https://hireobi.com/glossary/platforms#microsoft-copilot), [Sitemaps](https://hireobi.com/glossary/standards-and-protocols#sitemaps), [Crawl propagation delay](https://hireobi.com/glossary/how-ai-search-works#crawl-propagation-delay), [Content freshness and dating](https://hireobi.com/glossary/visibility-levers#content-freshness-and-dating)

Sources:

- IndexNow, Documentation, published n/d, read 2026-11-07: https://www.indexnow.org/documentation
- IndexNow, participating search engines registry (searchengines.json), published n/d, read 2026-11-07: https://www.indexnow.org/searchengines.json
- Bing Webmaster Blog, Introducing AI Performance in Bing Webmaster Tools Public Preview, published 2026-02-10, read 2026-11-07: https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
- Bing Webmaster Blog, Keeping Content Discoverable with Sitemaps in AI Powered Search (Fabrice Canel and Krishna Madhavan), published 2025-07-31, read 2026-11-07: https://blogs.bing.com/webmaster/2025/7/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search/

Checked: 2026-11-07

### Sitemaps

Link: https://hireobi.com/glossary/standards-and-protocols#sitemaps
Also called: XML sitemap, sitemap.xml, Sitemaps protocol, sitemap index
Label: Documented

**Sitemaps are files, usually XML, that list a site's URLs for search engines, optionally with the date each page last changed. Defined by the Sitemaps protocol 0.9, they help crawlers discover pages and recrawl changed ones. Google and Bing both ignore their priority and change-frequency fields and rely on accurate `lastmod` values.**

The protocol at sitemaps.org defines a `urlset` of `url` entries, each with a required `loc` (under 2,048 characters) and optional `lastmod` (W3C Datetime, for example 2026-11-07), `changefreq` and `priority`. One file holds at most 50,000 URLs and 50 MB uncompressed and may be gzipped; larger sites split into several files listed in a sitemap index. Plain-text sitemaps with one URL per line, and RSS or Atom feeds, are also accepted. Sites announce sitemaps with a `Sitemap:` line in robots.txt or through an engine's webmaster tools.

The engines say how they read them. Google uses `lastmod` only if it is "consistently and verifiably" accurate, ignores `priority` and `changefreq`, and treats a sitemap as a hint that does not guarantee crawling. Bing calls `lastmod` a key signal for recrawl priority, ignores the same two fields, and rechecks a sitemap at least once a day. Google says no new machine-readable files are needed for its AI features, and Bing's 2025 guidance applies the standard sitemap to AI-powered search.

**Why it matters.** AI search systems built on search indexes inherit their discovery: a page an engine never finds cannot be retrieved for an answer. A sitemap with complete coverage and truthful `lastmod` dates is the cheapest freshness signal a site controls, and Bing explicitly frames it, paired with IndexNow, as the base for visibility in AI-powered search. Teams should generate sitemaps automatically, change `lastmod` only on significant content updates, keep redirected and noindexed URLs out, and monitor processing errors in Search Console and Bing Webmaster Tools.

**Example.** A documentation site regenerates its sitemap on every deploy and changes `lastmod` only when a page's main content changes, never on template edits. Bing and Google can then prioritize the 40 pages that changed this week over the 4,000 that did not.

Related terms: [IndexNow](https://hireobi.com/glossary/standards-and-protocols#indexnow), [Content freshness and dating](https://hireobi.com/glossary/visibility-levers#content-freshness-and-dating), [Internal linking and discoverability](https://hireobi.com/glossary/visibility-levers#internal-linking-and-discoverability), [Crawl budget](https://hireobi.com/glossary/how-ai-search-works#crawl-budget), [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt)

Sources:

- sitemaps.org, Sitemaps XML format (protocol 0.9), published n/d, read 2026-11-07: https://www.sitemaps.org/protocol.html
- Google Search Central, Build and submit a sitemap, published n/d, read 2026-11-07: https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- Bing Webmaster Blog, Keeping Content Discoverable with Sitemaps in AI Powered Search (Fabrice Canel and Krishna Madhavan), published 2025-07-31, read 2026-11-07: https://blogs.bing.com/webmaster/2025/7/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search/
- Google Search Central, AI features and your website, published n/d, read 2026-11-07: https://developers.google.com/search/docs/appearance/ai-features

Checked: 2026-11-07

### Cloudflare Content Signals

Link: https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals
Also called: Content Signals Policy, Content-Signal, content signals, ai-train signal, ai-input signal
Label: Documented, Emerging (The policy is published under CC0; honoring it depends on each crawler, and the 2026 `use` extension was announced as a test.)

**Cloudflare Content Signals is a policy, published in September 2025, that adds machine-readable usage preferences to robots.txt through a `Content-Signal` line with three signals: `search`, `ai-input` and `ai-train`. It separates permission to appear in classic search from permission to feed AI answers or train models. Signals express preferences and do not technically block crawlers.**

Cloudflare introduced the policy on 2025-09-24 and released it under a CC0 license so anyone can implement it. `search` covers building a search index and providing results such as links and short excerpts, and explicitly excludes AI-generated search summaries. `ai-input` covers inputting content into one or more AI models, as in retrieval-augmented generation or grounding. `ai-train` covers training or fine-tuning AI models. A line such as `Content-Signal: search=yes, ai-train=no` grants one use and restricts another; an omitted signal neither grants nor restricts permission. The policy text states that restrictions are express reservations of rights under Article 4 of EU Directive 2019/790.

At the policy's launch, Cloudflare said it would serve `search=yes, ai-train=no` to the more than 3.8 million domains then using its managed robots.txt, leaving `ai-input` unset. On 2026-07-01 it announced, in testing, an optional `use` field with the values `immediate` (interact, store nothing), `reference` (index, excerpt and link back; the default) and `full` (summarize and reproduce), and began adding `use=reference` for managed robots.txt customers.

**How it works.** The signals sit inside robots.txt, under a user-agent group, so any crawler that reads robots.txt can read them without fetching a new file. Cloudflare says they "are not technical countermeasures against scraping" and recommends pairing them with its WAF and bot management; its July 2026 announcement adds that it tracks content use per bot and that bots abusing the signals lose verified status. A broader vocabulary is being drafted at the IETF: draft-ietf-aipref-vocab-08 (2026-09-14) defines `train-ai`, `ai-use` and `search` categories, and states that its text does not yet reflect working-group consensus.

**Why it matters.** The distinction between AI answering (`ai-input`) and AI training (`ai-train`) is the one most publishers care about, and Content Signals is the first widely deployed syntax for it, carried by a major edge network. It remains a preference: enforcement depends on crawler cooperation and on Cloudflare's own bot policies. Teams should set each signal deliberately, since a blank `ai-input` states no preference on AI answers, check what Cloudflare's managed robots.txt already serves for their domain, and follow the IETF work that may supersede it.

**Example.** `User-Agent: *`, then `Content-Signal: search=yes, ai-input=yes, ai-train=no`, then `Allow: /` says: index this site for search, use it to ground AI answers, do not train models on it. Dropping `ai-input=yes` would leave AI answering with no stated preference.

Related terms: [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [TDMRep (TDM Reservation Protocol)](https://hireobi.com/glossary/standards-and-protocols#tdmrep-tdm-reservation-protocol), [RSL (Really Simple Licensing)](https://hireobi.com/glossary/standards-and-protocols#rsl-really-simple-licensing), [Pay-per-crawl (Cloudflare experiment)](https://hireobi.com/glossary/standards-and-protocols#pay-per-crawl), [Training data vs retrieval data](https://hireobi.com/glossary/how-ai-search-works#training-data-vs-retrieval-data), [ai.txt (Spawning)](https://hireobi.com/glossary/standards-and-protocols#ai-txt)

Sources:

- Cloudflare Blog, Giving users choice with Cloudflare's new Content Signals Policy (Will Allen), published 2025-09-24, read 2026-11-07: https://blog.cloudflare.com/content-signals-policy/
- Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, published 2026-07-01, read 2026-11-07: https://blog.cloudflare.com/content-independence-day-ai-options/
- IETF Datatracker, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), published 2026-09-14, read 2026-11-07: https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/

Checked: 2026-11-07

### TDMRep (TDM Reservation Protocol)

Link: https://hireobi.com/glossary/standards-and-protocols#tdmrep-tdm-reservation-protocol
Also called: TDMRep, TDM Reservation Protocol, tdm-reservation, tdmrep.json
Label: Documented, Observed (A W3C Community Group final report, not a W3C standard; adoption is observed, not measured.)

**TDMRep, the TDM Reservation Protocol, is a W3C Community Group specification that lets rightsholders declare in machine-readable form whether text and data mining rights are reserved for their web content, and where a licensing policy can be found. It was designed as a technical answer to the opt-out in Article 4 of the EU's 2019 copyright directive.**

The latest final report, version 3 (the document is dated 2024-05-10, and the group lists it under 2024-08-09), is edited by Laurent Le Meur of EDRLab and states that it is "not a W3C Standard nor is it on the W3C Standards Track". It defines two properties. `tdm-reservation` takes 1 (rights reserved) or 0 (not reserved); any other value counts as unset. `tdm-policy` points to a policy, machine-readable (JSON or JSON-LD using a profile of ODRL 2.2) or human-readable, that can give contacts and conditions such as consent or compensation.

The properties can be expressed in a site-wide `/.well-known/tdmrep.json` file with path rules, in HTTP response headers (the report's currently preferred technique), in HTML `meta` elements, in EPUB package metadata, or in PDF XMP metadata. HTTP headers override the site-wide file, and metadata inside a document overrides both.

**How it works.** The report presents itself as a technical response to Article 4 of Directive 2019/790, under which mining for purposes other than scientific research depends on whether rightsholders have expressly reserved their rights in an appropriate manner, such as machine-readable means. It also mentions Article 3, covering research organizations and cultural heritage institutions, and notes that some of its beneficiaries are not bound by certain policy requirements. The Community Group has since discussed finer-grained signals and said it would follow the IETF AI Preferences work before choosing between its own vocabulary and an IETF consensus.

**Why it matters.** For publishers with EU audiences, the question is whether a reservation was expressed appropriately, and TDMRep offers a precise, documented form that crawlers can parse, unlike free-text terms of service. It works alongside robots.txt: robots.txt asks crawlers not to fetch, while TDMRep states a rights position about mining whatever is fetched. Teams can add the HTTP headers or the well-known file at low cost, but should not assume that every AI crawler reads them, since adoption has not been measured.

**Example.** A French news publisher serves `tdm-reservation: 1` and a `tdm-policy` URL in the HTTP headers of every article, pointing to a JSON-LD policy with a licensing contact. A crawler that implements TDMRep knows that mining is reserved and where to ask for a license.

Related terms: [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals), [RSL (Really Simple Licensing)](https://hireobi.com/glossary/standards-and-protocols#rsl-really-simple-licensing), [ai.txt (Spawning)](https://hireobi.com/glossary/standards-and-protocols#ai-txt), [Training data vs retrieval data](https://hireobi.com/glossary/how-ai-search-works#training-data-vs-retrieval-data)

Sources:

- W3C TDM Reservation Protocol Community Group, TDM Reservation Protocol (TDMRep), Final Community Group Report, published 2024-05-10, read 2026-11-07: https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/
- W3C, TDM Reservation Protocol Community Group page (reports and news), published n/d, read 2026-11-07: https://www.w3.org/community/tdmrep/

Checked: 2026-11-07

### RSL (Really Simple Licensing)

Link: https://hireobi.com/glossary/standards-and-protocols#rsl-really-simple-licensing
Also called: RSL, Really Simple Licensing, RSL 1.0, Open License Protocol
Label: Documented, Emerging (A published 1.0 specification; its own site names publisher and CDN supporters and no AI company committed to honoring it.)

**RSL, Really Simple Licensing, is an open specification for publishing machine-readable licensing terms for content, including terms for AI training and AI answers, and for acquiring licenses at scale. RSL 1.0 was published on 2025-12-10 at Recommendation status. It adds a middle path between blocking crawlers and free access: access under stated terms.**

The specification (document RSL-SPEC-1.0) is published by the RSL Technical Steering Committee, with editors from the RSL Collective, Condé Nast, Ziff Davis, Schema.org, Yahoo, Automattic, O'Reilly Media and Fastly. An RSL document describes `content` (assets identified by URL) and one or more `license` elements. Each license has `permits` and `prohibits` lists, where prohibitions win, a `payment` element whose type can be free, attribution, crawl, use, training, purchase, subscription or contribution, and optional reporting and legal terms. Usage categories include `all`, `ai-all`, `ai-train`, `ai-input` (grounding and generating AI responses), `ai-index` and `search`, the last excluding AI summaries.

Sites attach licenses through a `License:` directive in robots.txt, an HTTP `Link` header with `rel="license"`, HTML link or script elements, RSS items, or metadata embedded in files.

**How it works.** A crawler reads robots.txt, finds the `License:` URL, fetches the RSL document and checks whether its intended use is permitted and on what terms. Under the optional Open License Protocol, built on OAuth 2.0, it requests a license token from the publisher's license server and presents it with its requests, which a server can verify through the optional Crawler Authorization Protocol and its `License` authentication scheme. The RSL site lists featured supporters including Akamai, Cloudflare, Creative Commons, Fastly, Reddit, Vox Media, Yahoo and Ziff Davis.

**Why it matters.** RSL moves the control stack from crawl-or-block to crawl under terms, which matters to publishers who want AI answers to use their content on conditions instead of for free or not at all. Its practical weight depends on adoption by AI crawlers and enforcement by infrastructure providers, and the specification's own site documents supporters among publishers and CDNs, with no AI developer named. Teams can publish RSL terms alongside robots.txt at low cost, but should not expect payments or compliance until crawlers commit publicly.

**Example.** A publisher's robots.txt adds a `License:` line pointing to an RSL file that permits `search` for free and requires payment for `ai-train` and `ai-input`. A crawler that implements RSL obtains a token from the publisher's license server before using the articles to ground answers.

Related terms: [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [Pay-per-crawl (Cloudflare experiment)](https://hireobi.com/glossary/standards-and-protocols#pay-per-crawl), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals), [TDMRep (TDM Reservation Protocol)](https://hireobi.com/glossary/standards-and-protocols#tdmrep-tdm-reservation-protocol), [ai.txt (Spawning)](https://hireobi.com/glossary/standards-and-protocols#ai-txt)

Sources:

- RSL Technical Steering Committee, Really Simple Licensing (RSL) 1.0 Specification (RSL-SPEC-1.0), published 2025-12-10, read 2026-11-07: https://rslstandard.org/rsl
- RSL Standard, homepage (featured supporters), published n/d, read 2026-11-07: https://rslstandard.org/

Checked: 2026-11-07

### Pay-per-crawl (Cloudflare experiment)

Link: https://hireobi.com/glossary/standards-and-protocols#pay-per-crawl
Also called: pay per crawl, Pay Per Crawl, HTTP 402 for crawlers, Pay Per Use (Cloudflare)
Label: Documented, Emerging (The mechanism is documented by Cloudflare; its documentation still calls it a closed beta, and Cloudflare is reshaping it as Pay Per Use.)

**Pay-per-crawl is Cloudflare's mechanism, introduced in private beta on 2025-07-01, that lets site owners charge AI crawlers per request: a crawler without payment intent receives HTTP 402 Payment Required with a price, and Cloudflare bills and settles as merchant of record. In July 2026 Cloudflare said it is evolving the program into Pay Per Use.**

Cloudflare's 2025-07-01 announcement describes a third option between blocking and free access. For each crawler, a site owner chooses allow, charge (a flat per-request price for the domain) or block. A crawler can discover the price reactively, receiving a 402 with a `crawler-price` header and retrying with `crawler-exact-price`, or declare a ceiling up front with `crawler-max-price`, receiving 200 and a `crawler-charged` header when the price fits. Crawlers register and sign requests with Web Bot Auth (HTTP Message Signatures) so they cannot be impersonated, and existing WAF and bot-management rules run first.

The program has kept changing. On 2025-08-27 AI Crawl Control became generally available and let paid plans answer blocked crawlers with a 402. In December 2025 the beta added a discovery API for verified crawlers, and a changelog entry dated 2026-06-16 added URI-pattern exemptions and dynamic pricing. Cloudflare's documentation still described pay per crawl as a closed beta on 2026-11-07.

**How it works.** On 2026-07-01 Cloudflare announced it is "evolving Pay Per Crawl into Pay Per Use", so publishers are paid when content creates value. With Ceramic.ai, opted-in publishers are paid each time their content appears in Ceramic's AI search results; with You.com, an agent can pay on demand for a specific piece of premium content. The same announcement set new defaults from 2026-09-15: for new domains, training and agent crawlers are blocked on pages that show ads, while search crawlers stay allowed.

**Why it matters.** Pay-per-crawl turns access into a price, which changes the economics between publishers and AI companies if crawlers agree to pay. For AI search visibility the trade-off is direct: charging or blocking a crawler that feeds an answer engine can remove a site from that engine's answers, while allowing it gives the content away. Teams on Cloudflare should review their AI Crawl Control settings, especially the defaults applied since 2026-09-15, decide per crawler whether visibility or payment matters more, and check verified crawler activity in their logs.

**Example.** A recipe site sets a per-request price for training crawlers and allows search crawlers for free. A training crawler without a billing relationship receives `402 Payment Required` with a `crawler-price` header, while a search crawler fetches normally and the site stays eligible for that engine's answers.

Related terms: [RSL (Really Simple Licensing)](https://hireobi.com/glossary/standards-and-protocols#rsl-really-simple-licensing), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals), [AI crawler](https://hireobi.com/glossary/how-ai-search-works#ai-crawler), [IP-verified crawlers](https://hireobi.com/glossary/how-ai-search-works#ip-verified-crawlers), [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt)

Sources:

- Cloudflare Blog, Introducing pay per crawl: Enabling content owners to charge AI crawlers for access (Will Allen), published 2025-07-01, read 2026-11-07: https://blog.cloudflare.com/introducing-pay-per-crawl/
- Cloudflare Docs, What is pay per crawl?, published n/d, read 2026-11-07: https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/
- Cloudflare Docs, AI Crawl Control changelog, published n/d, read 2026-11-07: https://developers.cloudflare.com/ai-crawl-control/changelog/
- Cloudflare press release, Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules, published 2026-07-01, read 2026-11-07: https://www.cloudflare.com/en-au/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/
- Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, published 2026-07-01, read 2026-11-07: https://blog.cloudflare.com/content-independence-day-ai-options/

Checked: 2026-11-07

### MCP (Model Context Protocol)

Link: https://hireobi.com/glossary/standards-and-protocols#mcp
Also called: Model Context Protocol, MCP server, MCP client
Label: Documented (Official specification and documentation, versioned by date.)

**MCP, the Model Context Protocol, is an open-source standard for connecting AI applications to external systems such as data sources, tools and workflows through one common protocol. Introduced by Anthropic on 2024-11-25 and hosted since December 2025 by the Linux Foundation's Agentic AI Foundation, its current specification revision is dated 2026-07-28.**

The official documentation compares MCP to "a USB-C port for AI applications": one standard way for applications such as Claude or ChatGPT to reach files, databases, search engines, calendars or design tools, in place of a custom integration for each pairing. The specification uses JSON-RPC 2.0 messages between hosts (the AI applications), clients (connectors inside a host) and servers (services that provide context and capabilities). Servers can offer resources (data), prompts (templated messages) and tools (functions the model can call).

The 2026-07-28 revision, which follows 2025-11-25, made the protocol stateless: it removed the initialization handshake and protocol-level sessions, so every request carries its protocol version and client capabilities. It introduced a multi-round-trip pattern for requests that need more input, moved long-running tasks into an official extension, and deprecated the Roots, Sampling and Logging features. Anthropic donated MCP to the Agentic AI Foundation on 2025-12-09, and its maintainers keep the technical decisions.

**How it works.** A host such as a chat app or coding agent connects, through a client, to one or more MCP servers over a transport such as stdio or Streamable HTTP. Each server describes the tools, resources and prompts it offers; the model decides when to call a tool, and the specification's security principles require the host to obtain the user's consent before exposing data or invoking tools. Extensions such as MCP Apps add interactive interfaces rendered inside conversations.

**Why it matters.** MCP is how agents reach live systems instead of crawled pages: inventory, prices, bookings, account data, internal documents. For a business, an MCP server lets assistants answer from current, authoritative data in place of whatever was last crawled. It complements the publisher control surface: robots.txt and llms.txt govern what agents read on the open web, AGENTS.md tells coding agents how to work in a project, and MCP governs how they call tools. Teams exposing an MCP server should treat it as an API, with authentication, rate limits and precise tool descriptions.

**Example.** A retailer runs an MCP server with a product-search tool and a stock-check tool. An assistant connected to it answers "is this jacket in stock in medium near me?" with live inventory and price, without relying on a product page crawled last week.

Related terms: [AGENTS.md](https://hireobi.com/glossary/visibility-levers#agents-md), [NLWeb](https://hireobi.com/glossary/visibility-levers#nlweb), [Retrieval-augmented generation (RAG)](https://hireobi.com/glossary/how-ai-search-works#retrieval-augmented-generation), [Apps in ChatGPT (Apps SDK)](https://hireobi.com/glossary/platforms#apps-in-chatgpt), [Agentic search](https://hireobi.com/glossary/how-ai-search-works#agentic-search), [Computer-use and browsing agents](https://hireobi.com/glossary/platforms#computer-use-and-browsing-agents)

Sources:

- Model Context Protocol, What is the Model Context Protocol (MCP)?, published n/d, read 2026-11-07: https://modelcontextprotocol.io/docs/getting-started/intro
- Model Context Protocol, Specification (revision 2026-07-28), published 2026-07-28, read 2026-11-07: https://modelcontextprotocol.io/specification/latest
- Model Context Protocol, Key Changes in revision 2026-07-28, published 2026-07-28, read 2026-11-07: https://modelcontextprotocol.io/specification/2026-07-28/changelog
- Model Context Protocol Blog, MCP joins the Agentic AI Foundation, published 2025-12-09, read 2026-11-07: https://blog.modelcontextprotocol.io/posts/2025-12-09-mcp-joins-agentic-ai-foundation/
- Anthropic, Introducing the Model Context Protocol, published 2024-11-25, read 2026-11-07: https://www.anthropic.com/news/model-context-protocol

Checked: 2026-11-07

### Search generative AI control (Search Console)

Link: https://hireobi.com/glossary/standards-and-protocols#search-generative-ai-control
Also called: Search generative AI control, AI Overviews opt-out, AI Mode opt-out, Search Console generative AI setting
Label: Documented

**Search generative AI control is a Search Console setting, tested from 2026-06-03 and available to all sites since 2026-08-31, that lets a site owner exclude the site from Google's generative AI features (AI Overviews, AI Mode and generative AI in Discover) without affecting classic search ranking. Excluded sites receive no impressions or traffic from those features.**

The control sits under Settings, then Search generative AI, in Search Console. A property can be set to Include, the default, under which links and content can appear in and help ground AI responses, or to Exclude, under which links and content are hidden from those features and cannot be used to generate AI responses or previews. Child properties inherit the parent's setting unless their owners override it, and changes generally apply within a few days.

Google states that the setting is not a ranking or inclusion signal for other parts of Search, though crawled content may still help Google understand queries and page language. It does not govern model training, for which Google points owners to Google-Extended, and it does not override Merchant Center or Google Ads choices. Removing content from Search entirely still requires `noindex`. Before this control existed, limiting appearance in AI Overviews meant preview controls such as `nosnippet`, which also shrink the classic result.

**Why it matters.** It is the first Google switch that separates appearing in classic results from appearing in AI answers, so publishers no longer face an all-or-nothing trade with Search. For most brands seeking visibility the right setting is Include, since exclusion removes them from answers buyers increasingly read; the option matters most for publishers whose business depends on clicks, subscriptions or licensing. Google suggests reviewing the generative AI performance report first to see what visibility would be lost. Teams should also check that no child property has overridden the domain-level choice by accident.

**Example.** A subscription news publisher sets its domain property to Exclude after finding that AI Overview impressions rarely turn into subscribers. Its classic search listings are unaffected, while AI Overviews and AI Mode stop showing its articles within a few days.

Related terms: [Generative AI performance report (Search Console)](https://hireobi.com/glossary/measurement#generative-ai-performance-report), [nosnippet, max-snippet, data-nosnippet, noindex (preview controls)](https://hireobi.com/glossary/standards-and-protocols#nosnippet-max-snippet-data-nosnippet-noindex), [Google-Extended](https://hireobi.com/glossary/how-ai-search-works#google-extended), [Google AI Overviews](https://hireobi.com/glossary/platforms#google-ai-overviews), [Google AI Mode](https://hireobi.com/glossary/platforms#google-ai-mode), [Google Search Central AI features documentation](https://hireobi.com/glossary/studies-and-sources#google-search-central-ai-features-documentation)

Sources:

- Google Search Console Help, Search generative AI control, published n/d, read 2026-11-07: https://support.google.com/webmasters/answer/16908024?hl=en
- Google (Mrinalini Loew), New opportunities, control and insights for website owners (updated 2026-08-31), published 2026-06-03, read 2026-11-07: https://blog.google/products-and-platforms/products/search/new-controls-website-owners/

Checked: 2026-11-07

### Bing AI Training and AI Grounding preferences

Link: https://hireobi.com/glossary/standards-and-protocols#bing-ai-training-and-grounding-preferences
Also called: Bing AI preferences, ai-grounding directive, ai-training directive, DisallowExtended, Bing grounding opt-out
Label: Documented, Emerging (Announced by Microsoft on 2026-10-30; Bing starts honoring them on 2027-02-01, replacing NoArchive and NoCache for AI uses.)

**Bing AI Training and AI Grounding preferences are two independent publisher controls, announced on 2026-10-30, that let site owners decide separately whether Microsoft may train generative models on their content and whether it may use the content to ground AI answers. They replace the older NoArchive and NoCache signals for these uses and take effect from 2027-02-01.**

Microsoft defines training as using content to modify the learned parameters of a generative AI model, following the IETF AI Preferences draft, and grounding as using content as input to inform AI-generated responses; indexing and ranking for classic results are neither. The training preference can be set site-wide or per path in robots.txt through a shared `__usage_ai-training` user agent with `Disallow`, or per page with `<meta name="Bingbot" content="ai-training:disallow">` or the matching `X-Robots-Tag` header. The grounding preference is page-level only, with three values: Allow (the default), DisallowExtended, which keeps grounding inside Microsoft-operated experiences such as Copilot and Microsoft 365 while excluding third-party grounding services such as grounding API access in Foundry, and Disallow.

Sites using NoArchive or NoCache can record temporary grounding preferences in Bing Webmaster Tools from 2026-12-15, and new preferences must be published by 2027-03-01. Neither setting changes eligibility for Bing search results.

**Why it matters.** Bing's index feeds Copilot and, through grounding services, AI products built by others, so a page's grounding setting decides whether it can appear in a wide set of AI answers. Microsoft warns that Disallow or DisallowExtended will significantly reduce visibility in AI experiences and that no choice guarantees inclusion, traffic or payment. Brands seeking AI visibility should leave grounding allowed and audit legacy NoArchive or NoCache tags before the 2027 deadlines, since those tags stop being the AI signal after the transition. Publishers protecting content can now block training without disappearing from AI answers.

**Example.** A medical publisher adds a `Disallow: /` rule under the `__usage_ai-training` user agent in robots.txt, keeps grounding allowed on its public guides so Copilot can cite them, and marks licensed research pages with `ai-grounding:disallowextended` to keep them out of third-party grounding APIs.

Related terms: [Bingbot](https://hireobi.com/glossary/how-ai-search-works#bingbot), [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [X-Robots-Tag](https://hireobi.com/glossary/standards-and-protocols#x-robots-tag), [Microsoft Copilot (Bing and consumer app)](https://hireobi.com/glossary/platforms#microsoft-copilot), [Google-Extended](https://hireobi.com/glossary/how-ai-search-works#google-extended), [Training data vs retrieval data](https://hireobi.com/glossary/how-ai-search-works#training-data-vs-retrieval-data)

Sources:

- Bing Search Blog (Krishna Madhavan), Your content, your choice: clearer preferences for training and grounding, published 2026-10-30, read 2026-11-07: https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/

Checked: 2026-11-07

### Agentic Commerce Protocol (ACP)

Link: https://hireobi.com/glossary/standards-and-protocols#agentic-commerce-protocol
Also called: ACP, OpenAI and Stripe commerce protocol, Agentic Checkout Spec
Label: Documented, Emerging (Specification and positioning per its site, Stripe's launch post and OpenAI's commerce documentation; adoption beyond ChatGPT is stated by proponents, and ChatGPT's own in-chat checkout was scaled back in 2026.)

**Agentic Commerce Protocol (ACP) is an open standard, developed by OpenAI and Stripe and released under Apache 2.0 on 2025-09-29, that defines how AI agents, buyers and businesses complete purchases programmatically: product feeds for discovery, a checkout interface merchants implement, and delegated payment through scoped tokens. ChatGPT was its first implementation.**

ACP describes itself as an open standard for programmatic commerce flows between buyers, AI agents and businesses, giving merchants a blueprint for making their checkout usable by agents. As documented in OpenAI's commerce docs, its pieces are product feed specifications (feeds, products and promotions, delivered by API or SFTP), an Agentic Checkout Spec that merchants implement so an agent can create and update a checkout, and a Delegated Payment Spec that payment providers implement so credentials pass through a programmatically controlled, permissioned token. Stripe was the first compatible payment provider. Endpoints can be exposed over REST or MCP and connect to existing commerce backends, with the merchant keeping the customer relationship.

OpenAI launched Instant Checkout in ChatGPT on ACP the day the protocol was released, starting with US Etsy sellers. In March 2026 OpenAI said Instant Checkout was moving to merchants' ChatGPT apps and that ACP would serve as infrastructure connecting users to merchants, so the protocol outlives the feature that introduced it. Google's Universal Commerce Protocol addresses the same problem with a different design.

**Why it matters.** For merchants, ACP is the integration path into ChatGPT's commerce stack: its product feed specification governs how products appear in ChatGPT shopping, and its checkout and payment specifications govern any agent-completed purchase. Even with in-chat checkout scaled back, a clean feed in OpenAI's format is how a merchant controls price, availability and promotions in ChatGPT's product results. Teams that sell through ChatGPT should keep feeds in that format, follow the specification's versioned changes, and hold off on deep checkout integrations until the surfaces that use them settle.

**Example.** A home-goods retailer exports its catalog nightly as a product feed in OpenAI's format with current prices and stock, so ChatGPT shopping shows accurate availability. A buyer who picks a lamp is sent to the retailer's site to pay.

Related terms: [Agentic commerce](https://hireobi.com/glossary/field#agentic-commerce), [Universal Commerce Protocol (UCP)](https://hireobi.com/glossary/standards-and-protocols#universal-commerce-protocol), [Product feeds and AI shopping visibility](https://hireobi.com/glossary/visibility-levers#product-feeds-and-ai-shopping-visibility), [Apps in ChatGPT (Apps SDK)](https://hireobi.com/glossary/platforms#apps-in-chatgpt), [ChatGPT search](https://hireobi.com/glossary/platforms#chatgpt-search), [MCP (Model Context Protocol)](https://hireobi.com/glossary/standards-and-protocols#mcp)

Sources:

- Agentic Commerce Protocol, official site, published n/d, read 2026-11-07: https://www.agenticcommerce.dev/
- Stripe (Jeff Weinstein, Steve Kaliski), Developing an open standard for agentic commerce, published 2025-09-29, read 2026-11-07: https://stripe.com/blog/developing-an-open-standard-for-agentic-commerce
- OpenAI Developers, Commerce documentation (product feeds, Agentic Checkout Spec, Delegated Payment Spec), published n/d, read 2026-11-07: https://developers.openai.com/commerce
- Digital Commerce 360 (Brian Warmoth), OpenAI shifts checkout plans in its agentic commerce strategy, published 2026-03-06, read 2026-11-07: https://digitalcommerce360.com/2026/03/06/openai-shifts-checkout-plans-agentic-commerce-strategy/

Checked: 2026-11-07

### Universal Commerce Protocol (UCP)

Link: https://hireobi.com/glossary/standards-and-protocols#universal-commerce-protocol
Also called: UCP, Google UCP, /.well-known/ucp
Label: Documented, Emerging (Specification and partner lists per ucp.dev and Google's announcements; usage beyond Google's surfaces depends on other platforms adopting it.)

**Universal Commerce Protocol (UCP) is an open-source standard for agentic commerce, launched by Google on 2026-01-11 with Shopify, Etsy, Wayfair, Target and Walmart, that lets AI agents discover what a business offers and complete catalog search, cart, checkout and order steps through one shared interface. Google built it to power checkout in AI Mode and the Gemini app.**

UCP gives agents and commerce platforms a common language so an agent needs no custom integration for each store. A business publishes its supported services and capabilities in a JSON manifest at `/.well-known/ucp`, which agents read to find endpoints and payment options. The specification at ucp.dev lists capabilities for catalog search and lookup, cart, checkout, identity linking through OAuth 2.0 and orders, plus a draft for lodging bookings; transports include REST and JSON-RPC, with MCP and A2A bindings. Payments build on the Agent Payments Protocol (AP2), which uses payment mandates and verifiable credentials as proof of user consent.

Retailers remain the merchant of record. Google launched UCP to power checkout on eligible product listings in AI Mode and the Gemini app, starting with US retailers, and said in January 2026 that Chrome will support it. The specification is published under Apache 2.0, and ucp.dev now lists co-developers beyond the launch partners, including Amazon, Microsoft, Meta, Salesforce and Stripe.

**Why it matters.** UCP is the route by which a product listing in Google's AI surfaces becomes a completed purchase without a site visit, so for retailers it decides whether they can sell inside AI Mode and Gemini or only be mentioned there. It also shows commerce data turning into agent-facing infrastructure: capability manifests and checkout endpoints join product feeds as things a store maintains. Teams on Shopify or similar platforms should check whether their platform handles UCP for them; larger retailers should decide how loyalty, discounts and returns carry over when the purchase happens inside an agent.

**Example.** A sporting-goods retailer publishes a `/.well-known/ucp` manifest declaring checkout, discount and fulfillment capabilities. When a shopper in AI Mode chooses its tent, the purchase completes with Google Pay inside the conversation and the order lands in the retailer's own system.

Related terms: [Agentic Commerce Protocol (ACP)](https://hireobi.com/glossary/standards-and-protocols#agentic-commerce-protocol), [Product feeds and AI shopping visibility](https://hireobi.com/glossary/visibility-levers#product-feeds-and-ai-shopping-visibility), [Google AI Mode](https://hireobi.com/glossary/platforms#google-ai-mode), [Gemini (app)](https://hireobi.com/glossary/platforms#gemini), [MCP (Model Context Protocol)](https://hireobi.com/glossary/standards-and-protocols#mcp), [Agentic search](https://hireobi.com/glossary/how-ai-search-works#agentic-search)

Sources:

- Universal Commerce Protocol, official site and specification, published n/d, read 2026-11-07: https://ucp.dev/
- Google Developers Blog (Amit Handa, Ashish Gupta), Under the Hood: Universal Commerce Protocol (UCP), published 2026-01-11, read 2026-11-07: https://developers.googleblog.com/under-the-hood-universal-commerce-protocol-ucp/
- Google (Vidhya Srinivasan), New tech and tools for retailers to succeed in an agentic shopping era, published 2026-01-11, read 2026-11-07: https://blog.google/products/ads-commerce/agentic-commerce-ai-tools-protocol-retailers-platforms/
- Google (Parisa Tabriz), The new era of browsing: Putting Gemini to work in Chrome, published 2026-01-28, read 2026-11-07: https://blog.google/products-and-platforms/products/chrome/gemini-3-auto-browse/

Checked: 2026-11-07

### WebMCP

Link: https://hireobi.com/glossary/standards-and-protocols#webmcp
Also called: Web Model Context Protocol, document.modelContext, WebMCP origin trial
Label: Documented, Emerging (Draft Community Group Report of the W3C Web Machine Learning Community Group (2026-11-02), not on the W3C Standards Track; Chrome origin trial from Chrome 149.)

**WebMCP is a proposed browser API that lets a web page expose its own functions to AI agents as structured tools, such as 'search flights' or 'add to cart', so an agent can call them instead of clicking through the interface. It is a W3C community draft, edited by Microsoft and Google engineers, in origin trial in Chrome.**

WebMCP brings the tool model of the Model Context Protocol into the browser. The draft specification, published by the W3C Web Machine Learning Community Group and edited by engineers from Microsoft and Google, defines a `document.modelContext` object through which a page registers tools with `registerTool()`, each with a name, a description and an input schema that an agent can discover and execute. The tools run in the page's own JavaScript, inside the user's session, so they reuse the site's existing logic and login state. Chrome describes two forms: a declarative API for routine actions defined in HTML forms, and an imperative API for dynamic, JavaScript-driven interactions.

The draft states that it is neither a W3C Standard nor on the Standards Track. Chrome opened an early preview program on 2026-02-10 and an origin trial starting in Chrome 149, announced on 2026-06-09. WebMCP differs from an MCP server, which runs on a backend and is reached over the network: WebMCP tools exist only while the page is open in a browser an agent controls.

**Why it matters.** Browser agents today guess at buttons and forms, and they stumble on custom widgets, multi-step flows and pages that change state. WebMCP lets a site declare which actions exist and what inputs they need, which could make agent tasks on that site faster and more reliable. It is early: the API is a draft, support is limited to a Chrome trial, and use by mainstream agents is unproven. Product teams with transactional flows can prototype a few tools now; most marketing sites can wait while keeping their forms semantic and well labeled, which also helps the declarative form.

**Example.** An airline registers a `searchFlights` tool with origin, destination and date fields. A user's browser agent calls it with structured inputs and reads the results, instead of typing into the date picker and scraping the results grid.

Related terms: [MCP (Model Context Protocol)](https://hireobi.com/glossary/standards-and-protocols#mcp), [Computer-use and browsing agents](https://hireobi.com/glossary/platforms#computer-use-and-browsing-agents), [AX (Agent Experience)](https://hireobi.com/glossary/field#ax-agent-experience-optimization), [NLWeb](https://hireobi.com/glossary/visibility-levers#nlweb), [Semantic HTML](https://hireobi.com/glossary/visibility-levers#semantic-html)

Sources:

- W3C Web Machine Learning Community Group, WebMCP (Draft Community Group Report), published 2026-11-02, read 2026-11-07: https://webmachinelearning.github.io/webmcp/
- Chrome for Developers (Andre Cipriani Bandarra), WebMCP is available for early preview, published 2026-02-10, read 2026-11-07: https://developer.chrome.com/blog/webmcp-epp
- Chrome for Developers (Alexandra Klepper), Join the WebMCP origin trial, published 2026-06-09, read 2026-11-07: https://developer.chrome.com/blog/ai-webmcp-origin-trial

Checked: 2026-11-07

### Agent Skills (SKILL.md)

Link: https://hireobi.com/glossary/standards-and-protocols#agent-skills
Also called: Agent Skills, SKILL.md, skills, Claude Skills
Label: Documented, Emerging (Format per agentskills.io; the adoption count ('more than 70 agent platforms') is Anthropic's own claim.)

**Agent Skills is an open format for packaging instructions, scripts and reference files that teach an AI agent a specific task. Each skill is a folder with a `SKILL.md` file whose name and description tell the agent when to load it. Anthropic launched it in October 2025 and donated it to the Agentic AI Foundation in October 2026.**

A skill is a folder whose required `SKILL.md` file holds metadata (at minimum a name and a description) and instructions, optionally with `scripts/`, `references/` and `assets/` folders. Agents load skills through progressive disclosure: at startup they read only names and descriptions, they read the full instructions when a task matches a description, and they run bundled code or open referenced files only as needed. That keeps many skills available at a small context cost.

Anthropic introduced Agent Skills in October 2025, published the format as an open standard in December 2025, and on 2026-10-22 donated the standard, its name, the GitHub organization and the agentskills.io domain to the Agentic AI Foundation under the Linux Foundation, stating that more than 70 agent platforms, including ChatGPT, Cursor, GitHub Copilot and VS Code, had integrated it. Skills differ from AGENTS.md, which gives an agent standing instructions for a project, and from MCP, which connects agents to live tools and data.

**Why it matters.** Skills are becoming a distribution channel into agents: a company can publish a skill that teaches agents to use its product, API or method correctly, the way a developer guide teaches people. When an agent picks a tool to complete a task, a ready skill lowers the cost of choosing that product and cuts misuse. Cloudflare's Agent Readiness score already checks whether a site publishes skills. Teams with developer-facing products can write skills for their main workflows; for most content sites, clear machine-readable documentation matters more than the format itself.

**Example.** A payments company publishes a skill named 'create-subscription' whose description says when to use it and whose instructions walk an agent through idempotency keys and webhooks, with a script that validates the request. Coding agents that load it stop making the mistakes the company kept seeing in support tickets.

Related terms: [AGENTS.md](https://hireobi.com/glossary/visibility-levers#agents-md), [MCP (Model Context Protocol)](https://hireobi.com/glossary/standards-and-protocols#mcp), [llms.txt (the protocol entry)](https://hireobi.com/glossary/standards-and-protocols#llms-txt-2), [AX (Agent Experience)](https://hireobi.com/glossary/field#ax-agent-experience-optimization), [Agent Readiness score (Cloudflare)](https://hireobi.com/glossary/measurement#agent-readiness-score)

Sources:

- Agent Skills, Overview (agentskills.io), published n/d, read 2026-11-07: https://agentskills.io/
- Anthropic, Donating Agent Skills to the Agentic AI Foundation, published 2026-10-22, read 2026-11-07: https://www.anthropic.com/news/donating-agent-skills-to-the-agentic-ai-foundation
- Cloudflare Blog, Introducing the Agent Readiness score. Is your site agent-ready?, published 2026-04-17, read 2026-11-07: https://blog.cloudflare.com/agent-readiness/

Checked: 2026-11-07

### Markdown content negotiation (Markdown for Agents)

Link: https://hireobi.com/glossary/standards-and-protocols#markdown-content-negotiation
Also called: Markdown for Agents, Accept: text/markdown, Markdown negotiation, agent-friendly Markdown
Label: Documented, Emerging (HTTP content negotiation is long established; its use for agents is recent and documented chiefly by Cloudflare, whose scan found 3.9% of top sites passing.)

**Markdown content negotiation is serving a Markdown version of a web page to clients that request it with the HTTP header `Accept: text/markdown`, while browsers keep receiving HTML at the same URL. Cloudflare popularized it as Markdown for Agents on 2026-02-12, converting pages at its edge so AI agents read lighter, cleaner text.**

HTTP has long let a client state which formats it prefers through the `Accept` header and let a server choose a representation accordingly. Applied to AI agents, a server returns `text/markdown` for the same URL when asked and adds `Vary: Accept` so caches store HTML and Markdown separately. This differs from the llms.txt approach of publishing separate `.md` files and listing them in an index: negotiation needs no new URLs.

Cloudflare's Markdown for Agents performs the conversion at the edge for enabled zones on paid plans. It strips headers, footers, navigation, scripts and styles, adds YAML frontmatter from meta tags, keeps JSON-LD in a fenced block, and returns an `x-markdown-tokens` estimate and a `Content-Signal` header that by default allows training, search and AI input unless the origin sets its own. Cloudflare measured its announcement post at 16,180 tokens as HTML and 3,150 as Markdown, and said coding agents such as Claude Code and OpenCode already send the header.

**Why it matters.** Agents pay for every token they read, and page chrome can outweigh the content, so a clean Markdown representation makes a page cheaper and less error-prone to use, which may make it likelier to be read in full. It also removes guesswork from extraction: headings, lists and tables arrive as structure. The cost is governance: both representations must say the same thing, since serving agents different content than people would look like cloaking. Teams on Cloudflare can switch it on; others can add negotiation in their framework. Answer engines have not documented using it for retrieval or citation.

**Example.** A developer-docs site enables Markdown negotiation. When a coding agent fetches the authentication guide with `Accept: text/markdown`, it receives about 2,000 tokens of headings and code blocks instead of 11,000 tokens of HTML, and the browser version of the page is unchanged.

Related terms: [llms.txt (the protocol entry)](https://hireobi.com/glossary/standards-and-protocols#llms-txt-2), [llms-full.txt](https://hireobi.com/glossary/standards-and-protocols#llms-full-txt), [Content extractability](https://hireobi.com/glossary/visibility-levers#content-extractability), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals), [Semantic HTML](https://hireobi.com/glossary/visibility-levers#semantic-html), [Agent Readiness score (Cloudflare)](https://hireobi.com/glossary/measurement#agent-readiness-score)

Sources:

- Cloudflare Docs, Markdown for Agents, published n/d, read 2026-11-07: https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
- Cloudflare Blog (Celso Martinho, Will Allen), Markdown for Agents, published 2026-02-12, read 2026-11-07: https://blog.cloudflare.com/markdown-for-agents/
- Cloudflare Blog, Introducing the Agent Readiness score. Is your site agent-ready?, published 2026-04-17, read 2026-11-07: https://blog.cloudflare.com/agent-readiness/

Checked: 2026-11-07

### Web Bot Auth

Link: https://hireobi.com/glossary/standards-and-protocols#web-bot-auth
Also called: HTTP Message Signatures for bots, signed agents, bot request signing, Signature-Agent
Label: Documented, Emerging (IETF Web Bot Auth working group draft (revision -01, 2026-10-29), not yet an RFC; supported by Cloudflare and in experiment at Google.)

**Web Bot Auth is an emerging IETF protocol that lets bots and AI agents prove who they are by cryptographically signing their HTTP requests, using HTTP Message Signatures and a published key directory. Sites can verify the signature instead of trusting a user agent string, which is easy to spoof, or an IP list, which is hard to maintain.**

In Web Bot Auth, an operator holds private keys and publishes the matching public keys in a directory at `/.well-known/http-message-signatures-directory`. Each request carries a `Signature` header, a `Signature-Input` header describing the signed components with the tag `web-bot-auth`, and a `Signature-Agent` header pointing to the key directory, so the receiving site or its CDN can fetch the key and check that the request came from that operator. The protocol proves control of a key; it says nothing about whether the bot is welcome or what it may do.

The working-group draft, titled 'HTTP Message Signatures for automated traffic', is authored by engineers from Cloudflare and Google; revision -01 is dated 2026-10-29. Cloudflare accepts Web Bot Auth as a verification method in its verified bots and agents program, and Google's crawler documentation says Google-Agent is experimenting with it under the `https://agent.bot.goog` identity.

**Why it matters.** As agents fetch pages for individual users and often ignore robots.txt, sites need a reliable way to tell a real assistant from a scraper using its name. Signed requests make it practical to allow, rate-limit or charge specific operators, and they already underpin verified-bot programs such as Cloudflare's. For a brand, the practical step is to make sure its CDN or bot manager verifies signatures instead of blocking unfamiliar automated traffic, so legitimate agents can complete visits and purchases. Adoption by major AI operators is partial and the specification can still change.

**Example.** A ticketing site's bot manager sees two requests claiming to be the same AI agent. One carries a valid signature tied to the operator's key directory and reaches the event page; the unsigned one is challenged.

Related terms: [Google-Agent](https://hireobi.com/glossary/how-ai-search-works#google-agent), [IP-verified crawlers](https://hireobi.com/glossary/how-ai-search-works#ip-verified-crawlers), [AI crawler](https://hireobi.com/glossary/how-ai-search-works#ai-crawler), [Pay-per-crawl (Cloudflare experiment)](https://hireobi.com/glossary/standards-and-protocols#pay-per-crawl), [Computer-use and browsing agents](https://hireobi.com/glossary/platforms#computer-use-and-browsing-agents), [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt)

Sources:

- IETF Web Bot Auth WG, HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-01), published 2026-10-29, read 2026-11-07: https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/
- Cloudflare Docs, Web Bot Auth (bot verification), published n/d, read 2026-11-07: https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/
- Google Crawling Infrastructure, List of Google user-triggered fetchers (Google-Agent), published n/d, read 2026-11-07: https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent

Checked: 2026-11-07

### AI Preferences (IETF AIPREF)

Link: https://hireobi.com/glossary/standards-and-protocols#ai-preferences-aipref
Also called: AIPREF, aipref, Content-Usage header, AI usage preferences, train-ai
Label: Documented, Emerging (Two IETF working-group drafts on the Standards Track (vocabulary -08, 2026-09-14; attachment -05, 2026-08-19); the vocabulary draft states its text does not yet reflect working-group consensus.)

**AI Preferences (AIPREF) is an IETF effort to standardize how publishers state whether their content may be used for AI training, AI use and search. It defines a shared vocabulary with labels such as `train-ai` and `search`, and ways to attach those preferences through a `Content-Usage` HTTP header or a new robots.txt rule.**

The IETF AI Preferences working group splits the problem in two. The vocabulary draft defines categories of use: AI Training (`train-ai`), using an asset to modify a generative model's learned parameters; AI Use (`ai-use`), using an asset as input to a generative model when the user did not directly provide it; and Search (`search`), using an asset in an application that selects assets and links users to their original location. The attachment draft, by Gary Illyes of Google and Martin Thomson of Mozilla, carries those preferences in a `Content-Usage` HTTP response header, for example `Content-Usage: train-ai=n`, and in a `Content-Usage` rule inside robots.txt groups, updating RFC 9309.

Both documents are drafts, and the vocabulary text warns that it does not yet reflect consensus, so labels may change. Microsoft's October 2026 Bing preferences already adopt the draft's definition of training, and Cloudflare's Content Signals express similar choices with different labels.

**Why it matters.** Today a site that wants to block AI training but stay visible in AI search must learn each operator's crawler names and opt-out rules. A shared vocabulary would let one statement cover every compliant operator and separate training from answer-time use, the distinction that decides whether a brand stays citable. Marketers should not wait on it: preferences are signals, compliance is voluntary, and the labels are unsettled. The practical move is to follow the drafts, keep crawler-specific rules for the operators that matter, and avoid blanket AI blocks that also remove the site from AI answers.

**Example.** A publisher adds `Content-Usage: train-ai=n` to its robots.txt group for all crawlers, signaling that its articles may be indexed and cited but not used to train models, and keeps specific Disallow rules for training crawlers that do not yet read the new rule.

Related terms: [robots.txt (Robots Exclusion Protocol, RFC 9309)](https://hireobi.com/glossary/standards-and-protocols#robots-txt), [Cloudflare Content Signals](https://hireobi.com/glossary/standards-and-protocols#cloudflare-content-signals), [TDMRep (TDM Reservation Protocol)](https://hireobi.com/glossary/standards-and-protocols#tdmrep-tdm-reservation-protocol), [Google-Extended](https://hireobi.com/glossary/how-ai-search-works#google-extended), [RSL (Really Simple Licensing)](https://hireobi.com/glossary/standards-and-protocols#rsl-really-simple-licensing), [Training data vs retrieval data](https://hireobi.com/glossary/how-ai-search-works#training-data-vs-retrieval-data)

Sources:

- IETF AIPREF WG, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), published 2026-09-14, read 2026-11-07: https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/
- IETF AIPREF WG, Associating AI Usage Preferences with Content in HTTP (draft-ietf-aipref-attach-05), published 2026-08-19, read 2026-11-07: https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/
- Bing Search Blog (Krishna Madhavan), Your content, your choice: clearer preferences for training and grounding, published 2026-10-30, read 2026-11-07: https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/

Checked: 2026-11-07

## Next actions

- See every term of the dictionary: https://hireobi.com/glossary.md
- Read the facts about Obi: https://hireobi.com/ai-instructions.md