I Built an AI Telemetry System Because My Memory Is Too Forgiving

My memory is generous to me. It keeps the clean result and quietly misplaces the retries, the abandoned approaches, and the stretches when I was busy without being effective. I built Agent Telemetry because I wanted a record with less sympathy for the story I tell myself about my own work.

Revision note, October 9, 2026: This article now dates its subscription and initial outcome-coverage statements and clarifies that the governed loop it describes is historical.

The dashboard counts tokens, sessions, and dollars. They are easy to see and hard to interpret. A large token total can mean that I completed hard work, repeated weak instructions, chose an expensive model, or simply left several agents running. The total cannot tell those stories apart.

I want evidence that accumulates slowly enough to answer four uncomfortable questions.

Are my engineering standards helping? Am I becoming more effective, rather than merely generating more activity? Were my August subscriptions a good economic choice compared with the same observed usage at API list prices? Which projects are taking the most from me in time and attention?

A productive afternoon cannot answer any of them. A productive month may not be enough either. The system preserves comparable history so I can eventually stop guessing.

A standard needs something to answer to

I like explicit standards: acceptance criteria, review passes, tests, evidence, and a clear definition of done. I also know how easily a standard can become ceremony. A process can feel rigorous because it creates a lot of motion. More rounds, findings, and generated artifacts may show care. They may also show that the first pass was poor or that the process has learned to feed itself.

Usage data needs an outcome layer. When I first published this article, the governed feature-cycle loop supplied that context. Its telemetry recorded represented specifications, review rounds, findings, accepted outcomes, duration, tokens, and API-equivalent cost. An accepted feature cycle counted once even if several retained records contained terminal evidence. Acceptance efficiency asked what share of the specifications represented in a selected period reached acceptance. The metric catalog publishes those definitions and their caveats instead of leaving the labels to intuition.

That loop was retired on September 8, 2026. Its datasets remain published as frozen history, while the current system publishes successor outcome receipts separately. Historical governed-loop records can still support comparisons within their defined cohort, but they do not represent the whole current outcome system.

Within a comparable range, I can examine what happened around a change in standards: whether accepted cycles became more common, the number of rounds fell, or findings moved earlier instead of merely multiplying. I can also see whether cost and elapsed time rose for a result I considered worth the extra scrutiny. Those observations can tell me whether a standard deserves a closer test, a simpler form, or no change at all.

What happened around it is different from what the standard caused. These are observations from real projects, not a controlled trial. Project difficulty, model changes, my own skill, and plain luck can all move at once. A trend can make a cherished rule look suspicious. It cannot convict the rule by itself.

Recurring friction without a corresponding outcome signal is enough to identify what deserves investigation. It is not enough to supply a causal verdict.

I want to measure effectiveness without defining it as volume

Tokens are wonderfully easy to count. Engineering effectiveness refuses that convenience.

Fewer tokens may mean I learned to frame a problem well. More may mean I delegated a difficult investigation and caught a serious error. Rising output alongside stalled accepted work would leave me with a busier machine, not better engineering.

I care about the relationship among activity, outcomes, and rework. Activity shows that the tools ran. Outcome records show only what an explicit adapter observed for its defined cohort. Rework indicates where a result required more rounds or findings. None of those categories can stand in for the others.

Collection health belongs beside them. A missing provider root, a late snapshot, or an unobserved field is part of the answer because a clean-looking trend built over a gap is not clean. Before I compare two periods, models, or projects, I need to know whether both sides were observed on comparable terms. If they were not, the decision is to repair the evidence or narrow the question, not admire the chart.

The system is strict about a small distinction I value: null means unobserved or inapplicable; zero means an observed zero. Coverage also begins at different times for different hosts, providers, projects, and outcome adapters. That makes the charts less tidy, which is preferable to a smooth line manufactured from assumptions.

The project’s operating contract says usage volume cannot establish code quality, developer productivity, causal model superiority, or an invoice. The collector does not ingest prompts, messages, or code. The public tier separately excludes private paths, machine or account identity, private project names, and the private mapping. That preserves enough metadata to ask better questions without building a surveillance transcript of my working day.

The subscription comparison is a counterfactual

When I first published this article on August 21, 2026, I was paying $200 per month to OpenAI and $200 per month to Anthropic, for $400 total. Those amounts describe that publication point. They are not a claim about my current subscriptions.

I wanted to know what I received for that fixed spend, but calling the difference “savings” would overstate the evidence. The telemetry prices observed usage against the providers’ published API rate cards. OpenAI and Anthropic publish model-specific rates for the applicable token classes. The project produces an exact API-equivalent dollar amount only when an observed model has an exact configured price match. Usage without that match stays unpriced. Any best-effort range remains separate from the exact amount.

The result is a dated list-price counterfactual: what the observed token mix would cost at the applicable published rates. It is neither an API invoice nor proof that I saved the difference. Behavior under a flat subscription may differ from behavior under metered billing. Some requests might never be made if each one carried a visible marginal price.

Configured subscription amortization is a separate lens. It can divide a configured fixed amount across accepted cycles, but it must not be added to API-equivalent cost. A configured amount is also not independent evidence of a current payment.

Keeping those quantities apart makes the economic question harder and more useful. I can ask whether a fixed plan appears well utilized, whether an apparently cheap model still looks cheap after rework, or whether an attractive comparison depends on a large unpriced bucket. A high API-equivalent amount shows utilization under the counterfactual rates, not realized value. A low amount may justify reconsidering a fixed plan, although availability, predictable billing, and the option to explore remain outside the token ledger.

Projects spend attention differently

The cost I feel most acutely is rarely the token bill. It is the project that keeps reopening itself in my head.

Agent Telemetry attributes sessions, tokens, cost, host environment, and UTC activity to privacy-safe project identities. The public view keeps private names and paths out of the repository. The machine-readable manifest catalogs the exhaustive public datasets, their coverage, schemas, semantics, and hashes. I can compare allocation without turning the public dashboard into a map of my machine.

Compute cost and attention cost are not interchangeable. One project may consume a large model budget while requiring little beyond review. Another may use fewer tokens while demanding repeated interventions, context rebuilding, and decisions that linger after the terminal closes. Sessions and tokens tell me where compute went. They do not measure the human attention that accompanied it.

The current system also provides a separate, content-free local timer for recorded human attention. Its daily aggregates are published only when opted in. The observations depend on explicit timer use: session spans and agent runtime never fill the gaps, and an absent attention row does not mean zero attention. That gives the allocation question a second input where it has actually been recorded.

The historical governed-loop duration metric helps only as a rough friction signal. It runs from dispatch to verdict and includes queue idle. It is not a timesheet, and I will not convert it into labor dollars. Paired with review rounds, findings, and outcome records, it can point to friction worth investigating.

Where comparable attention and outcome records exist, I can ask whether a project’s demand on me is matched by its recorded progress. If the observations are missing, I need a narrower question or better evidence before deciding what to change.

What a longer record can answer

The public page offers fixed 7-, 30-, 90-day, and all-time UTC views. Complete daily rows support exact custom UTC ranges. I care about those ranges because a snapshot is mostly biography: what I happened to be doing, with the models and constraints available that week.

A longer, coverage-aware record can establish a baseline before I change a standard and support a comparison afterward. It can help separate an unusual project from a broader shift. It can show whether an apparent efficiency survives a model change, whether a lower-cost workflow is associated with more review rounds, or whether a project continues consuming attention after its strategic value has faded. Those patterns would justify investigation, not a causal verdict.

The record has to earn its upkeep by informing a decision. A standard associated with recurring friction and no corresponding outcome signal should be tested or simplified. A supposedly cheap model associated with expensive rework should lose some of its price advantage in my reasoning, even if the API column stays small. A project that dominates attention while producing little observed progress may deserve to be narrowed or paused. Before acting, I still have to check the cohort, coverage dates, missing values, and alternative explanations.

I expect useful evidence to be annoying. A standard I am proud of may become the right subject for a closer test. A project I call strategic may demand a clearer account of its attention cost. An economic comparison may look impressive until unpriced usage or mismatched coverage changes the denominator. Any of those possibilities is more valuable than a large number that agrees with me.

My memory will continue keeping the flattering edit. When a decision is due, it will have to argue with timestamps, denominators, coverage gaps, and a record that remembers the retries.