--- published: true layout: post title: Learning About Standards via Cloudflare Radar AI Insights image: https://kinlane-productions2.s3.amazonaws.com/algorotoscope-master/copper-circuit-train-tracks-bending.jpg author: name: kinlane tags: - Standards - Cloudflare - AI - Bots - Agents - Crawling - APIs.json - Discovery --- I am learning from the [AI Insights updates on Cloudflare Radar](https://developers.cloudflare.com/changelog/post/2026-04-17-radar-ai-insights-updates/). I have long been a champion of how we'll be automating the onboarding of clients, bots, agents, and other non-human users of the web. I consider my work on data.json for the Obama administration and my ongoing investment into apis.json a PHD in this area. Cloudflare has a unique position as the DNS edge for much of the web when it comes to understanding this realm, and I find their breakdown of standards used to produce their AI insights as part of Cloudflare radar to be educational. - **robots.txt** — [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html) The Robots Exclusion Protocol, originally from 1994 and formally standardized as RFC 9309 in September 2022 IETF. A plain-text file at a site's root that tells crawlers which paths they may or may not access. - **Sitemap** — [https://www.sitemaps.org/protocol.html](https://www.sitemaps.org/protocol.html) An XML protocol introduced by Google in 2005 (with Yahoo and Microsoft adopting in 2006) Wikipedia that lists URLs on a site along with metadata like last-modified date and change frequency to help search engines crawl more efficiently. - **AI rules in robots.txt** — [https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/) The IETF AIPREF working group's draft extension to robots.txt that adds a Content-Usage directive GitHub so sites can express AI-specific preferences (training, search, inference) alongside traditional Allow/Disallow rules. - **Link headers** — [https://www.rfc-editor.org/rfc/rfc8288.html](https://www.rfc-editor.org/rfc/rfc8288.html) RFC 8288 (Web Linking) defines a model for relationships between web resources and serializes those links in HTTP headers via the Link header field IETF. Used for pagination, alternate representations, and resource relationships. - **OAuth discovery** — [https://datatracker.ietf.org/doc/html/rfc8414](https://datatracker.ietf.org/doc/html/rfc8414) RFC 8414 defines a metadata format at /.well-known/oauth-authorization-server that OAuth 2.0 clients use to discover an authorization server's endpoints and capabilities IETF. - **Markdown negotiation** — [https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation) Using the standard HTTP Accept: text/markdown header so servers can return clean markdown to agents and HTML to browsers from the same URL. RFC 7763 registered the text/markdown MIME type in March 2016 Ekamoira. - **Universal Commerce Protocol (UCP)** — [https://ucp.dev/](https://ucp.dev/) An open-source standard developed by Google with Shopify, Etsy, Wayfair, Target, Walmart, and 20+ partners Medium that lets AI agents discover merchant capabilities and complete commerce flows (checkout, identity linking, order management) through a standardized manifest at /.well-known/ucp. - **Content signals** — [https://blog.cloudflare.com/content-signals-policy/](https://blog.cloudflare.com/content-signals-policy/) Cloudflare's Content Signals Policy extends robots.txt with three categories — search, ai-input, and ai-train — so operators can express how crawlers may use content after it's been fetched Cloudflare. Released under CC0. - **OAuth Protected Resource** — [https://www.rfc-editor.org/rfc/rfc9728](https://www.rfc-editor.org/rfc/rfc9728) RFC 9728, published April 2025, defines a metadata format at /.well-known/oauth-protected-resource that OAuth 2.0 clients use to learn what scopes, token types, and authorization servers an API requires RFC Editor. Widely used for MCP authorization discovery. - **Agent Skills** — [https://agentskills.io/home](https://agentskills.io/home) An open standard originally developed by Anthropic for filesystem-based SKILL.md files — folders of instructions, scripts, and resources that agents load on demand Agent Skills to specialize for domain tasks. - **Web Bot Auth** — [https://blog.cloudflare.com/web-bot-auth/](https://blog.cloudflare.com/web-bot-auth/) A Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture) that uses HTTP Message Signatures (RFC 9421) Anchor Browser Docs so bots can cryptographically sign requests and sites can verify their identity without relying on user-agent strings or IP allowlists. - **MCP Server Card** — [https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1649](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1649) SEP-1649, a draft MCP enhancement proposal for a .well-known/mcp.json (or server-card.json) endpoint that lets clients discover server capabilities, transports, auth requirements, and protocol version before connecting GitHub. - **API catalog** — [https://datatracker.ietf.org/doc/rfc9727/](https://datatracker.ietf.org/doc/rfc9727/) RFC 9727 defines the "api-catalog" well-known URI and link relation to facilitate automated discovery and usage of published APIs IETF — a machine-readable index of a publisher's APIs, versions, and documentation links. - **A2A Agent Card** — [https://a2a-protocol.org/latest/specification/](https://a2a-protocol.org/latest/specification/) In the Agent2Agent protocol, the Agent Card is a JSON metadata document published by an A2A server describing its identity, capabilities, skills, service endpoint, and authentication requirements A2a-protocol, typically hosted at /.well-known/agent-card.json. - **WebMCP** — [https://developer.chrome.com/blog/webmcp-mcp-usage](https://developer.chrome.com/blog/webmcp-mcp-usage) A proposed browser standard, co-developed by Google Chrome and Microsoft Edge, with two APIs under navigator.modelContext that let a web page register callable tools for an in-browser AI agent Chrome Developers. Shipping behind a flag in Chrome 146 Canary. - **x402 Payment** — [https://www.x402.org/](https://www.x402.org/) An open protocol from Coinbase that revives the HTTP 402 "Payment Required" status code to embed stablecoin payments directly into HTTP interactions Coinbase. In December 2025, Coinbase and Cloudflare launched the x402 Foundation to steward the protocol. - **AP2 (Agent Payments Protocol)** — [https://ap2-protocol.org/specification/](https://ap2-protocol.org/specification/) A protocol introduced by Google with Mastercard, PayPal, Adyen, and 60+ organizations that uses Verifiable Credentials ("Mandates") to create cryptographically signed proof of user intent for agent-driven transactions Vellum — covering Intent Mandates, Cart Mandates, and Payment Mandates. There are a couple items on here that are old hat for me, things like robots.txt, sitemap, and OAuth. There are some things I've been wrestling with for a short while, like Agent Cards, WebMCP, and Agent Skills. Then there are entirely new things like x402 Payments, AP2, Web Bot Auth, and others I know absolutely nothing about. There are also a lot of awareness expanding concepts regarding how AI rules are used in robots.txt, how ubiquitous link headers have become, and nuanced crawling has become with content signals. There is a lot to reconcile here with the properties I've been cultivating as part of APIs.json indexing and onboarding. I see a lot we are taking for granted in the diff between this list and apis.json properties. I am coming at it from the human side, and the very smart people behind these standards are coming at it from the bot perspective. I am beginning to to see the business side of things reflected in Universal Commerce Protocol (UCP), x402 Payment, and Agents Payments Protocol. But it is not enough. I'll need to dive in deeper, but I don't see enough of the things business and regulatory folks will care about. I see a lot of the human bits left on the table, taking humans for granted in the cracks of this system. Regardless, there is a lot of learning to be had here when it comes to building on the original vision of the API economy I had back in 2010. ![Cloudflare Adoption of AI Agent Standards](https://kinlane-productions2.s3.us-east-1.amazonaws.com/cloud-flare-adoption-of-ai-agent-standards.png)