---
name: agent-observability
description: strategies for agent observability (logging, tracing, metrics). Use this to instrument agents for debugging, performance tracking, and quality assurance.
---

# Agent Observability Strategies

## Goal
Move beyond simple monitoring ("Is it running?") to deep observability ("How is it thinking?"), enabling the diagnosis of complex failures in non-deterministic systems.

## The Three Pillars of Observability

### 1. Structured Logging (The Diary)
* **Definition:** Immutable, timestamped records of discrete events.
* **Best Practice:** Use structured JSON logs to capture the full context: prompt/response pairs, intermediate reasoning (Chain of Thought), and tool inputs/outputs.
* **Pattern:** Record the *intent* before an action and the *outcome* after to distinguish between decision failures and execution failures.

### 2. Distributed Tracing (The Narrative)
* **Definition:** A visual "yarn" connecting individual log entries (spans) into a single end-to-end task execution.
* **Usage:** Essential for root cause analysis. It reveals if a bad final answer was caused by a retrieval failure (RAG), a tool error, or an LLM hallucination.
* **Standard:** Use OpenTelemetry to link spans across services.



### 3. Metrics (The Scorecard)
Aggregated data points for tracking health over time. Separate these into two dashboards:

#### System Metrics (Operational Health)
* **Audience:** SREs / DevOps.
* **Key Metrics:** P99 Latency, Error Rate (traces with `error=true`), Token Consumption, and API Cost per Run.

#### Quality Metrics (Decision Health)
* **Audience:** Product / Data Science.
* **Key Metrics:**
    * **Trajectory Adherence:** Did the agent follow the ideal path?
    * **Hallucination Rate:** Frequency of ungrounded statements.
    * **Task Completion Rate:** Percentage of traces reaching a "success" state.

## Operational Best Practices
* **Dynamic Sampling:** To save costs, log 100% of errors but only sample 10% of successful traces in production.
* **PII Redaction:** Integrate PII scrubbing directly into the logging pipeline to sanitize user inputs before storage.