---
published: true
layout: post
title: 'The AI Licensing Market Is Visible in HTTP Status Codes'
image: https://kinlane-images.s3.amazonaws.com/apievangelist/api-evangelist-images/the-ai-licensing-market-is-visible-in-http-status-codes.png
date: 2026-08-04
author: Kin Lane
tags:
  - AI
  - Agents
  - Discovery
  - Machine Readability
  - Monetization
  - Provenance
  - Research
  - Strategy
---
Yesterday I [pulled apart how Time serves ads to AI agents](https://apievangelist.com/2026/08/03/how-time-serves-ads-to-ai-agents/), and I said at the end that I wanted to find out how many other publishers were doing the same thing. So I built the measurement, pointed it at 436 sites, and got an answer I did not expect: almost nobody. Differential serving — showing agents a different document than you show humans — turns up on 1.8% of the panel, and **Time is the only site in it serving advertising payload to agents. Zero others.** The thing the trade press has been treating as the beginning of a wave is, as of today, one publisher with two advertisers. If you were about to write a strategy deck about agent-facing ad inventory, the honest data says you have time.

But while I was looking for ads I kept tripping over something else, and it turns out to be the real story. A quarter of the major publishers I probed were answering my requests with **HTTP 402 Payment Required**. Not 403, not a block — a bill. Forbes and USA Today and the San Francisco Chronicle were telling my crawler, in a machine-readable response body, that it was "not authorized to access this content without a valid TollBit Token." Investopedia was answering with an email address for content licensing. The commercialization of the agentic web is not happening through advertising. It is happening through tolls, it is already deployed at real scale, and it is completely invisible unless you go looking for it with the right user agent.

So I widened the probe to 185 media properties across eight categories and asked a more specific question: not *is this site gated*, but *which AI companies is it gated against*. That distinction is everything, because the gate is selective. The Atlantic answers OpenAI's crawler with a 200 and Anthropic's with a 402. The Associated Press does the same. The San Francisco Chronicle lets OpenAI and Perplexity walk in and bills Anthropic. Forbes bills everybody. **A publisher's HTTP status codes tell you which AI companies have signed a content deal with them and which have not** — and that is a market which otherwise exists entirely inside private contracts that nobody publishes.

Here is what 185 outlets look like. Of the 153 that gave me a clean baseline, 17.0% gate at least one AI vendor behind a 402, and 11.8% do it *selectively* — some vendors in, others billed. Another 30.7% block agents outright without offering a price, and 52.3% are still wide open to everyone. TollBit is collecting for fourteen of the gated outlets; twelve run their own first-party toll rather than joining a marketplace. That split matters more than it looks: joining TollBit is buying into a market, while building your own toll is a bet that you have enough leverage to negotiate directly.

The vendor-by-vendor numbers are where it gets pointed. Google reaches 79.1% of these outlets without paying and is billed by only 3.9%; Apple is at 78.4% and 5.9%. OpenAI reaches 66.7% and is billed by 7.8%. **Anthropic reaches 64.7% and is billed by 15.0% — roughly double OpenAI's exposure.** Head to head, six outlets let OpenAI in free while billing Anthropic, and only two do the reverse. I want to be careful about what that does and does not prove, because a 402 means *no current paid access*, not refusal — some of those are live negotiations, and some are renewals that lapsed last month. But as a snapshot of who has papered their deals, it is the clearest picture I have seen anybody produce, and it did not come from a leak or a source. It came from status codes.

The category breakdown produced the line I cannot stop thinking about. Entertainment and lifestyle publishers are the most aggressively gated at 45.8%. Science and health sit at 5.9%. And the tech trade press — the outlets writing all the coverage about AI companies scraping the web — gate AI companies at **0.0%**. Not one. The people explaining this fight to everyone else are giving their work away to every model that asks, while the celebrity and recipe sites have already built the toll booth. Make of that what you will.

I went in expecting to find that individual newsrooms were making these calls, and that is not what the data shows. The postures cluster by owner. Every People Inc property I probed — People, Investopedia, Allrecipes, Serious Eats, Travel+Leisure, Verywell Health — runs an identical first-party toll, blocks OpenAI and Google and Amazon outright, and bills the other eight vendors. Six mastheads, one decision, confirmed by Investopedia's own 402 response naming People Inc's content-licensing address. Hearst splits by *division*: its newspapers bill Anthropic and no one else, while Esquire on the magazine side bills only CommonCrawl. And Penske Media, which acquired Vox Media in June, is running two completely different policies at once — Deadline and Variety and Rolling Stone bill nine vendors through TollBit, while the newly-acquired Verge and Vox and Polygon run no toll at all. The acquisition has not propagated. That is a natural experiment sitting in my panel, and the weekly run will catch the exact week it changes, or prove that it never does.

I want to be straight about how much I got wrong on the way here, because the errors are more instructive than the findings and because this whole exercise exists to be a level head in a space full of confident nonsense. My first pass at the Time teardown was probed entirely with `curl`, and it told me a tidy story about which crawlers were blocked that turned out to be an artifact of curl's TLS fingerprint rather than anything about Time's policy — the identical user agent from Python got a completely different answer. Then, building this, I initially called the 402 responses "cloaking," which would have been a false accusation against nine publishers doing something entirely defensible; charging for access is not concealment. And I first reported People Inc as "open to nobody" because I had collapsed *blocked* and *billed* into a single bucket, which inverted the actual finding — refusing an AI company and charging one are opposite commercial signals. Every one of those mistakes produced a confident, quotable, wrong sentence. The tooling now runs two independent HTTP clients and refuses to report a result when they disagree, and it throws out any site that does not give a clean human baseline, which cost me 17.3% of the cohort.

That last number is the one I would attack first if someone else published this, so let me put it up front rather than in a footnote: seventeen percent of the outlets I probed could not be measured at all, because they bot-defend every client including a plain browser. Reuters answers a desktop Chrome user agent with a 401. You cannot learn anything about a site's *agent* policy when it refuses *everyone*, and counting those as "blocking AI" would have inflated my headline by a third. They are excluded, and the exclusion rate is published next to every rate that depends on it.

None of this is in `robots.txt`. Not one of these publishers announces any of it. The entire commercial layer — who is billed, who walks in free, which intermediary is collecting, what the price signal even is — lives in edge configuration and response bodies, discoverable only by impersonating a crawler and reading the status code. We spent twenty years building a discovery layer out of files at well-known paths that describe a site's intentions honestly, and the most consequential thing happening to the web right now is described in none of them. It is the same argument I keep making about [provenance over enforcement](https://apievangelist.com/2026/07/04/api-reviews-and-provenance-over-enforcement/) and about [monetizing without walling off the map](https://apievangelist.com/2026/07/15/monetizing-my-apis-and-mcp-without-walling-off-the-map/): the artifacts only mean something if the behavior at the edge matches what they say, and right now nobody is checking.

So I am going to keep checking. The panel runs every Sunday, the deal map gets a new snapshot each week, and the interesting output is not the level but the movement — the week Penske pushes TollBit onto The Verge, the week somebody's Anthropic 402 turns into a 200. Publishers are building the toll booth. Nobody is building the ad product. And the whole market is legible to anyone willing to send a few hundred HTTP requests and be honest about what the answers do not prove.