# tracelint

> **A linter for agent runs** — it reads the execution trace of a tool-calling agent (what it
> *actually did*) and flags structural bugs deterministically, with the exact evidence and a CI
> exit code. It runs *after* the run, on the trace — not on your code — and no second model ever
> judges it.

`tracelint` reads a tool-calling agent's trace and reports structural defects — schema-violating
tool calls, ignored tool errors, hallucinated arguments, loops, and redundant calls — each with the
exact trace lines as evidence, and returns a CI exit code. It also ships a fault injector and a
per-fault recovery scorecard.

Model-as-judge detection of these defects is unreliable (published trace-error benchmarks show low
localization accuracy). Many of these defects are *structurally decidable* and need no judge — that
is the entire premise of this tool. No second model ever judges the trace.

**[View the live demo report](https://ashwinugale.github.io/tracelint/)** — the constructed
validation suite (one planted instance of every defect, clean controls, and legitimate-but-suspicious
cases) plus the robust-vs-buggy recovery scorecard, generated by `tracelint demo`.

## Limitations (read first)

1. Deterministic rules catch **structural** defects, not whether the final answer was correct.
2. Hallucinated-argument, loop, and redundant-call findings are **candidates** unless structurally
   proven — legitimate value transforms and intentional retries can trip them; each is shown with
   its evidence for human review, never asserted as a verdict. High-confidence hallucination
   detection requires the tool schema to declare field origins (`x-value-origin`).
3. The recovery scorecard needs labeled task outcomes (success oracles); without them it measures
   **behavioral** recovery only ("did not crash"), a weaker claim than correctness.
4. A trace is only as complete as its instrumentation. A rule whose required field is missing is
   **suppressed with a stated reason** — `tracelint` never lints a partial trace as if complete.

## Quick start

The demo runs a keyless validation suite and a recovery scorecard end to end — no API key, no
model download:

```bash
pip install tracelint
tracelint demo --html demo.html
```

Lint a trace in CI:

```bash
tracelint check ./trace.json --tools ./tools.json     # exit 2 on a hard_defect
```

Exit codes: `0` clean · `2` a structurally-provable defect (`hard_defect`) · `3` an input error.
Heuristic candidates never fail CI on their own; suppressions are disclosed but are not defects.

## The rules

| Rule | Finding | Tiers |
|------|---------|-------|
| R1 | schema violation — args fail the tool's JSON Schema | `hard_defect` |
| R2a | tool returned an error | `hard_event` (structured signal) / `candidate` (heuristic) |
| R2b | an errored result's value reused by a later side-effecting call | `hard_defect` / `candidate` |
| R3 | hallucinated argument — value not derivable from provenance | `candidate`; `hard_defect` if the field is annotated `provided` |
| R4 | loop — N identical no-progress calls (polls/retries excluded) | `candidate` |
| R5 | redundant call — identical call + identical result, no mutation between | `candidate` |
| R6 | malformed arguments — the emitted tool-call arguments are not valid JSON | `hard_defect` |
| R7 | unknown tool — a call to a tool absent from the declared toolset (possible hallucinated tool) | `candidate` |

`hard_event` and `hard_defect` are orthogonal to the finding kind: a tool-error event is a
`hard_event` from a structured status field but a `candidate` from an exception-like string in
free-form content.

## Input format

A trace is a JSON object (`.json`, or `.jsonl` for many):

```json
{
  "run_id": "run-1",
  "steps": [
    {"type": "message", "role": "user", "content": "cancel order 4521 if it hasn't shipped"},
    {"type": "tool_call", "call_id": "c1", "name": "get_order_status", "args": {"order_id": "4521"}},
    {"type": "tool_result", "call_id": "c1", "content": {"status": "processing"}, "status": "ok"},
    {"type": "tool_call", "call_id": "c2", "name": "cancel_order",
     "args": {"order_id": "4521", "reason": "not_shipped"}}
  ],
  "final": "Order 4521 has been cancelled."
}
```

`tools.json` supplies the ground truth the rules check against:

```json
{
  "tools": {
    "cancel_order": {
      "schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                 "required": ["order_id"]},
      "metadata": {"side_effecting": true}
    }
  }
}
```

A tool can also declare **what failure looks like** in its result, so a domain failure returned as
a transport success (HTTP 200 carrying `{"status": "declined"}`) is caught structurally instead of
slipping through:

```json
{
  "tools": {
    "charge_card": {
      "metadata": {
        "side_effecting": true,
        "failure_when": {"pointer": "/status", "in": ["declined", "failed"]}
      }
    }
  }
}
```

`failure_when` is a JSON Pointer into the result plus a match (`in` / `equals` / `exists`); a match
is a structured error for R2 (feeding R2a and, on reuse into a side-effecting call, R2b). A
side-effecting tool with **no** `failure_when` and an unclassifiable result is *suppressed with a
reason* — never counted as a clean pass.

The rules run against **one canonical trace schema**; a thin **adapter** translates each source's
format into it, so the rules never change. Built in: `from_openai_messages` (OpenAI chat message
lists), `from_langfuse_trace` (a [Langfuse](https://langfuse.com) trace's observations), and
`from_otel_spans` (**OpenTelemetry / [OpenInference](https://github.com/Arize-ai/openinference)** —
the universal standard, so it reaches Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, and datasets
like TRAIL, not just one vendor). See `examples/langfuse_cookbook.py` to lint the traces you
already collect in Langfuse and write findings back as scores.

**On real traces:** the adapters are validated against live data, not just the spec —
`from_langfuse_trace` on real Langfuse v4 runs, and `from_otel_spans` on real
[TRAIL](https://huggingface.co/datasets/PatronusAI/TRAIL) benchmark traces, where tracelint
deterministically localized real tool errors, a malformed tool call, and excessive-retry loops
with no model in the loop. Real exports vary, so a new source may need a small adapter tweak — and
when a field a rule needs is absent, that rule **suppresses** (says so) rather than guessing, so an
unhandled quirk degrades safely instead of producing a wrong result. More adapters are future work.

## Lint the traces you already collect

`check` reads native tracelint JSON by default, but `--format` points it straight at the traces
your stack already emits — no manual schema conversion:

```bash
tracelint check spans.json    --format openinference   # OTel/OpenInference: Phoenix, OTLP, TRAIL
tracelint check messages.json --format openai          # an OpenAI chat message list
tracelint check trace.json    --format langfuse        # a Langfuse trace export
```

Most rules need no tool schemas, so this works keyless; add `--tools tools.json` to light up the
schema-dependent rules (R1, and R3's high-confidence tier). A multi-trace input (a `.jsonl` file, a
JSON array, or an OTLP export carrying several `trace_id`s) fans out to one report each. From the
library, the same one-liner:

```python
from tracelint import lint_otel_trace

report = lint_otel_trace(spans)   # spans: your OpenInference span export (a list of dicts)
print(report.exit_code)           # 0 or 2
```

See `examples/lint_openinference_phoenix.py` for an offline, keyless end-to-end run (Phoenix-shaped
spans → findings, with and without a tool registry).

Straight from a running [Arize Phoenix](https://phoenix.arize.com) instance:

```python
import phoenix as px
from tracelint import lint_otel_trace

spans = px.Client().get_spans_dataframe().to_dict("records")
print(lint_otel_trace(spans).exit_code)
```

Both Phoenix shapes are handled: the span-export JSON (top-level `span_kind`) and the
`get_spans_dataframe()` records (attributes as `attributes.*` columns).

## Add to CI

`tracelint check` returns exit `2` on a structurally-provable defect, so it gates a build directly.
Point it at the traces your agent test job already produces — a defect fails the job; heuristic
candidates never do.

**GitHub Actions** — the ready-made action:

```yaml
name: lint-agent-traces
on: [push, pull_request]
jobs:
  tracelint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... your step that runs the agent and writes traces to ./traces ...
      - uses: AshwinUgale/tracelint@v0.4.1
        with:
          traces: "traces/*.jsonl"
          format: "openinference"     # or native / openai / langfuse
          tools: "tools.json"          # optional — lights up R1, R3, R2 predicates
```

**Any CI, without the action** — it's one pip install and one command:

```bash
pip install tracelint
tracelint check traces/*.jsonl --format openinference --tools tools.json
```

**pre-commit** — lint only the trace files a commit touches:

```yaml
repos:
  - repo: https://github.com/AshwinUgale/tracelint
    rev: v0.4.1
    hooks:
      - id: tracelint
        files: ^traces/.*\.jsonl$
        args: ["--format", "openinference", "--tools", "tools.json"]
```

Traces have to come from somewhere: tracelint lints artifacts, it doesn't run your agent. The usual
shape is a test job that exercises the agent, captures its trace (OpenInference/OTel, OpenAI, or
Langfuse), and then runs `tracelint check` on that file.

## Recovery scorecard

Measure how an agent behaves under injected faults, scored against deterministic success oracles:

```bash
tracelint scorecard --demo --faults timeout,error,rate_limit --runs 5
```

The baseline must satisfy the oracle first (else recovery is not measured). Each fault type reports
a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to
behavioral recovery, labeled as weaker.

## Library

```python
from tracelint import lint_trace, default_rules, Trace, ToolRegistry

trace = Trace.load("trace.json")
registry = ToolRegistry.load("tools.json")
report = lint_trace(trace, default_rules(), registry)
print(report.exit_code)          # 0 or 2
for f in report.active_findings:
    print(f.rule, f.tier.value, f.summary)
```

## Development

```bash
python -m pytest
ruff check src tests
```

The core is dependency-light (`jsonschema` + stdlib) and the whole test suite is deterministic and
offline. A real OpenAI trace-generating agent lives behind the opt-in `[real-agent]` extra and is
never part of the linter. Python 3.10–3.12.