# TraceVerde
**The most comprehensive OpenTelemetry auto-instrumentation library for LLM/GenAI applications**
*Trace from OpenTelemetry traces. Verde meaning green - for sustainable, transparent AI observability.*
[Documentation](https://mandark-droid.github.io/genai_otel_instrument/) | [Examples](https://github.com/Mandark-droid/genai_otel_instrument/tree/main/examples) | [Discord](https://discord.gg/6SVz6VKK) | [PyPI](https://pypi.org/project/genai-otel-instrument/)
[](https://badge.fury.io/py/genai-otel-instrument)
[](https://pypi.org/project/genai-otel-instrument/)
[](https://opensource.org/licenses/Apache-2.0)
[](https://pepy.tech/project/genai-otel-instrument)
[](https://pepy.tech/project/genai-otel-instrument)
[](https://github.com/Mandark-droid/genai_otel_instrument)
[](https://opentelemetry.io/)
[](https://github.com/Mandark-droid/genai_otel_instrument/actions)
[](https://bonigarcia.dev/context-engineering/ai-ecosystem.html)
---
## Get Started in 30 Seconds
```bash
pip install genai-otel-instrument
```
```python
import genai_otel
genai_otel.instrument()
# Your existing code works unchanged - traces, metrics, and costs are captured automatically
import openai
client = openai.OpenAI()
response = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello!"}])
```
That's it. No wrappers, no decorators, no config files. Every LLM call, database query, and agent interaction is automatically traced with full cost breakdown.
### Let your coding agent do it
Using an AI coding agent or assistant? Paste this prompt into it, in the repository you want instrumented. It adds the two lines above in the right place, installs the right extras, configures through environment variables, and verifies the result with a real run. The same prompt is in the [docs](https://mandark-droid.github.io/genai_otel_instrument/getting-started/coding-agents/).
Copy the prompt
```text
Add OpenTelemetry tracing for LLM calls to this Python application using TraceVerde.
Package on PyPI: genai-otel-instrument Import name: genai_otel
Docs: https://mandark-droid.github.io/genai_otel_instrument/
Steps
1. Find the entry point of every process this application starts: the module that
runs first (CLI main, ASGI/WSGI app module, worker start-up script, the first cell
of a notebook).
2. Find which LLM SDKs and agent frameworks the code imports, for example openai,
anthropic, langchain, langgraph, crewai, pydantic_ai, autogen_agentchat. Install
the package with the matching extras and add it to the project's dependency file
(requirements.txt, pyproject.toml, or whatever this project uses):
pip install "genai-otel-instrument[openai,langgraph]"
Extras are named after the library: openai, databricks, anthropic, google, langchain,
langgraph, crewai, pydantic-ai, autogen-agentchat, llamaindex, and others listed
in the docs. Plain "pip install genai-otel-instrument" is enough when the SDK is
already a dependency of this project.
3. At the very top of each entry point, add:
import genai_otel
genai_otel.instrument()
This must run before any LLM SDK or agent framework is imported, and before any
client, LLM or agent object is created. A client created earlier is not traced.
Call it once per process. Do not wrap, decorate or otherwise change existing LLM
calls: instrumentation is automatic.
4. Configure through environment variables, not code. Add them wherever this project
keeps its configuration (.env.example, deployment manifests, the README):
OTEL_SERVICE_NAME=
OTEL_EXPORTER_OTLP_ENDPOINT=
OTEL_EXPORTER_OTLP_HEADERS=
Never hard-code endpoints, API keys or headers. Leave content capture off, which
is the default: GENAI_ENABLE_CONTENT_CAPTURE=true records prompts and completions
on spans, so only turn it on if I ask for it.
5. Only if I ask for it:
- limit what is instrumented: GENAI_ENABLED_INSTRUMENTORS=openai,langgraph
- sample traces: GENAI_SAMPLING_RATE=0.5
6. Verify with a real run. Run the smallest path of the application that makes an
LLM call, with OTEL_EXPORTER_OTLP_ENDPOINT set to an empty string for that run.
Spans are then printed to the console instead of exported. Check that:
- there is one span per LLM call, carrying gen_ai.request.model,
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and
gen_ai.usage.cost.total when the model has a known price;
- if an agent framework is used, its spans are the parents of the LLM spans, in
the same trace.
Then restore the real endpoint.
7. Report back: the files you changed, the install line, the environment variables to
set, and the span names you saw in step 6. If you could not verify something, say
so instead of assuming it works.
Rules
- Make the smallest change that works. Do not refactor unrelated code.
- Do not add a second LLM tracing library next to this one. If one is already
present, or the application already sets up its own OpenTelemetry TracerProvider,
stop and ask me how to proceed.
- If an agent framework ships its own tracing, leave it off unless I ask; running
both reports every model request twice.
- Do not commit secrets, and do not print or log API keys.
```
## Why TraceVerde?
| Feature | TraceVerde | OpenLIT | Traceloop/OpenLLMetry | Langfuse | Galileo | Arize (Phoenix) | Opik (Comet) |
|---------|-----------|---------|----------------------|----------|---------|-----------------|--------------|
| Zero-code setup | Yes | Yes | Yes | SDK required | SDK required | SDK / auto (OpenInference) | SDK required |
| LLM providers | 28 | 25+ | 15+ | Via integrations | Via integrations | Via integrations | Via integrations |
| Multi-agent frameworks | 13 (CrewAI, LangGraph, ADK, AutoGen, Claude Agent SDK, Strands Agents, Microsoft Agent Framework, Agno, etc.) | Limited | Limited | Limited | Limited | Limited | Limited |
| Cost tracking | Automatic (1,700+ models) | Manual config | Manual config | Manual config | Yes | Yes | Yes |
| GPU metrics (NVIDIA + AMD) | Yes | No | No | No | No | No | No |
| MCP tool instrumentation | Yes (databases, caches, vector DBs, queues) | Limited | Limited | No | No | No | No |
| Evaluation (PII, toxicity, bias, hallucination, prompt injection) | Built-in (6 detectors) | No | No | Separate service | Extensive (core focus) | Built-in (Phoenix evals) | Built-in (core focus) |
| OpenTelemetry native | Yes | Yes | Yes | Partial | Partial | Yes (OpenInference) | Partial |
| Self-hosted / on-prem | Yes (fully local) | Yes | Yes | Yes | Enterprise tier | Yes (Phoenix) | Yes |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 | MIT | Proprietary | Open-source + Commercial | Apache-2.0 |
## What Gets Instrumented?
### LLM Providers (28+)
OpenAI, Databricks Model Serving, OpenRouter, CometAPI, Anthropic, Google AI, Google GenAI, AWS Bedrock, Azure OpenAI, Cohere, Mistral AI, Together AI, Groq, Ollama, vLLM, llama.cpp, Vertex AI, xAI SDK, Replicate, HuggingFace, SambaNova, Sarvam AI, ElevenLabs, Hyperbolic, LiteLLM, TypeSafe AI / Jev
Databricks instrumentation covers OpenAI-compatible serving and AI Gateway clients,
the native Workspace SDK, and MLflow deployment clients (including completed streaming
responses). Install `genai-otel-instrument[databricks]`; see the [Databricks setup and
examples](https://mandark-droid.github.io/genai_otel_instrument/guides/llm-providers/#databricks-model-serving)
for OpenAI, SDK, and MLflow code.
**Decision models** (OpenInference `DECISION` spans, priced separately from chat): OpenAI Decisions (`gpt-6-luna`, also via OpenRouter and Azure), SystemOne servers (TypeSafe Jev, LiquidAI D1, Cloudflare Clef, self-hosted Hub decision models on vLLM / SGLang), Granite Switch's decision adapter, plain-HTTP calls to those endpoints, and in-process models through `genai_otel.decision_span`
[See all providers with examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/llm-providers/)
### Multi-Agent Frameworks (13)
CrewAI, LangGraph, Google ADK, AutoGen, AutoGen AgentChat, OpenAI Agents SDK, Pydantic AI, AWS Bedrock Agents, Anthropic Claude Agent SDK, Strands Agents, Microsoft Agent Framework, Agno, and Antigravity
[See all frameworks with examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/multi-agent-frameworks/)
### MCP Tools (20+)
**Databases:** PostgreSQL, MySQL, MongoDB, SQLAlchemy, TimescaleDB, OpenSearch, Elasticsearch, FalkorDB
**Caching:** Redis | **Queues:** Kafka, RabbitMQ | **Storage:** MinIO, SeaweedFS (S3 API)
**Vector DBs:** Pinecone, Weaviate, Qdrant, ChromaDB, Milvus, FAISS, LanceDB
[See all MCP tools >>](https://mandark-droid.github.io/genai_otel_instrument/guides/mcp-tools/)
### Built-in Evaluation (6 Detectors)
PII Detection (GDPR/HIPAA/PCI-DSS), Toxicity Detection, Bias Detection, Prompt Injection Detection, Restricted Topics, Hallucination Detection
[See all evaluation features with examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/evaluation/)
## Screenshots
OpenAI traces with token usage, costs, and latency
More screenshots
### Ollama (Local LLM)
### SmolAgents with Tool Calls
### GPU Metrics
### OpenSearch Dashboard
## Key Features
### Automatic Cost Tracking
1,700+ models across 40+ providers with per-request cost breakdown. Supports differential pricing (prompt vs completion), reasoning tokens, cache pricing, and custom model pricing.
```python
# Cost tracking is enabled by default - just instrument and go
genai_otel.instrument()
# Or add custom pricing for proprietary models
export GENAI_CUSTOM_PRICING_JSON='{"chat":{"my-model":{"promptPrice":0.001,"completionPrice":0.002}}}'
```
[Cost tracking guide >>](https://mandark-droid.github.io/genai_otel_instrument/guides/cost-tracking/)
### Token Usage Metrics
Token usage is recorded under the current OpenTelemetry GenAI metric conventions: five counters
(`gen_ai.client.inference.usage.input_tokens`, `.output_tokens`, `.cache_read.input_tokens`,
`.cache_write.input_tokens`, `.reasoning.output_tokens`) broken down by `gen_ai.token.modality`,
and two per-call histograms (`gen_ai.client.inference.operation.input_tokens` / `.output_tokens`).
The `gen_ai.client.token.usage` counter, which the conventions no longer define, is still recorded
alongside them until 2.0. [Metric reference >>](https://mandark-droid.github.io/genai_otel_instrument/reference/semantic-conventions/#token-usage-metrics)
### GPU Metrics (NVIDIA + AMD)
Real-time monitoring of utilization, memory, temperature, power, PCIe throughput, throttling, and ECC errors. Multi-GPU aggregate metrics included.
```bash
pip install genai-otel-instrument[gpu] # NVIDIA
pip install genai-otel-instrument[amd-gpu] # AMD
```
[GPU metrics guide >>](https://mandark-droid.github.io/genai_otel_instrument/guides/gpu-metrics/)
### Multi-Agent Tracing
Complete span hierarchy for agent frameworks with automatic context propagation:
```
Crew Execution
+-- Agent: Senior Researcher (gpt-4)
| +-- Task: Research OpenTelemetry
| +-- openai.chat.completions (tokens: 1250, cost: $0.03)
+-- Agent: Technical Writer (ollama:llama2)
+-- Task: Write blog post
+-- ollama.chat (tokens: 890, cost: $0.00)
```
### Multimodal Observability (v1.1.0)
First-class capture of image, audio, video, and document content parts on
OpenAI, Anthropic, Google Gemini, and Groq spans, and of the audio on Sarvam and
ElevenLabs speech calls (the audio sent for transcription, the audio text-to-speech
returns; the transcript and spoken text follow content capture). Bytes are offloaded
to your configured object store (MinIO / SeaweedFS / S3 / filesystem / HTTP) and referenced from
spans by URI — they never appear inline in span attributes.
```bash
# Opt in (default is off — text-only behaviour is byte-identical to 1.0.x)
export GENAI_OTEL_MEDIA_CAPTURE_MODE=full
export GENAI_OTEL_MEDIA_STORE=minio
export GENAI_OTEL_MEDIA_STORE_ENDPOINT=http://localhost:9000
export GENAI_OTEL_MEDIA_STORE_ACCESS_KEY=...
export GENAI_OTEL_MEDIA_STORE_SECRET_KEY=...
# Optional: plug in a redactor before upload
export GENAI_OTEL_MEDIA_REDACTOR=genai_otel.media.redactors.face_blur
```
For SeaweedFS, set `GENAI_OTEL_MEDIA_STORE=seaweed` and point
`GENAI_OTEL_MEDIA_STORE_ENDPOINT` at its S3 gateway (commonly port 8333). The SDK
uses path-style addressing; create the bucket before running the application.
Spans get two co-emitted representations of the same multimodal content:
- A **flat, queryable attribute namespace** — `gen_ai.prompt.{n}.content.{m}.{type, media_uri, media_mime_type, media_byte_size, media_source}` plus a `gen_ai.completion.*` mirror — for backends that index on flat attributes.
- The **upstream-canonical `gen_ai.input.messages` / `gen_ai.output.messages` JSON** conforming to the [gen-ai message schemas](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-input-messages.json) in the dedicated `semantic-conventions-genai` repo, including the `document` modality standardised by our upstream PR #142, plus the optional `byte_size` and `stripped_reason` shapes proposed in #143 / #144 (see [Standards Contributions](#opentelemetry-standards-contributions) below).
[Multimodal guide >>](https://mandark-droid.github.io/genai_otel_instrument/guides/multimodal/)
### Safety & Evaluation
```python
genai_otel.instrument(
enable_pii_detection=True, # GDPR/HIPAA/PCI-DSS compliance
enable_toxicity_detection=True, # Perspective API + Detoxify
enable_bias_detection=True, # 8 bias categories
enable_prompt_injection_detection=True,
enable_hallucination_detection=True,
enable_restricted_topics=True,
)
```
[Evaluation guide with 50+ examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/evaluation/)
## Configuration
```bash
# Required
export OTEL_SERVICE_NAME=my-llm-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# Optional
export GENAI_ENABLE_GPU_METRICS=true
export GENAI_ENABLE_COST_TRACKING=true
export GENAI_SAMPLING_RATE=0.5 # Reduce volume in production
export GENAI_ENABLED_INSTRUMENTORS=openai,crewai # Select specific instrumentors
```
[Full configuration reference >>](https://mandark-droid.github.io/genai_otel_instrument/getting-started/configuration/)
## Backend Integration
Works with any OpenTelemetry-compatible backend:
Jaeger, Zipkin, Prometheus, Grafana, Datadog, New Relic, Honeycomb, AWS X-Ray, Google Cloud Trace, Elastic APM, Splunk, SigNoz, self-hosted OTel Collector
Pre-built [Grafana dashboard templates](https://github.com/Mandark-droid/genai_otel_instrument/tree/main/dashboards) included.
## Examples
90+ ready-to-run examples covering every provider, framework, and evaluation feature:
```
examples/
+-- openai/ # OpenAI chat, embeddings
+-- anthropic/ # Anthropic + PII/toxicity detection
+-- ollama/ # Local models + all evaluation features
+-- crewai_example.py # Multi-agent crew orchestration
+-- langgraph_example.py # Stateful graph workflows
+-- google_adk_example.py # Google Agent Development Kit
+-- antigravity/ # Google Antigravity SDK agent turns
+-- autogen_example.py # Microsoft AutoGen agents
+-- pii_detection/ # 10 PII examples (GDPR, HIPAA, PCI-DSS)
+-- toxicity_detection/ # 8 toxicity examples
+-- bias_detection/ # 8 bias examples (hiring compliance, etc.)
+-- prompt_injection/ # 6 injection defense examples
+-- hallucination/ # 4 hallucination detection examples
+-- ... # And many more
```
[Browse all examples >>](https://github.com/Mandark-droid/genai_otel_instrument/tree/main/examples)
## OpenTelemetry Standards Contributions
TraceVerde isn't only a consumer of OpenTelemetry GenAI semantic conventions — production gaps surfaced by the library are being upstreamed back into the spec, and one is now part of the standard. Proposals on [`open-telemetry/semantic-conventions-genai`](https://github.com/open-telemetry/semantic-conventions-genai):
| PR | Proposal | Status |
|---|---|---|
| [**#142**](https://github.com/open-telemetry/semantic-conventions-genai/pull/142) | Add `document` to the `Modality` enum on `BlobPart` / `FilePart` / `UriPart` — PDFs, DOCX, and other non-image/video/audio payloads currently fall through to the free-form `string` branch. BFSI KYC extraction is a high-volume real example. | **Merged** 2026-05-16 |
| [**#143**](https://github.com/open-telemetry/semantic-conventions-genai/pull/143) | Add optional `byte_size` on the three media-part types so consumers get a uniform handle on payload size whether the content was carried inline, by URI, or by provider file id — useful for cost-of-capture telemetry and storage planning. Pydantic `ge=0` → JSON schema `"minimum": 0`. | Open — reference scenarios for `FilePart` / `UriPart` requested |
| [**#144**](https://github.com/open-telemetry/semantic-conventions-genai/pull/144) | Make `content` / `file_id` / `uri` optional and add a free-form `stripped_reason` (`size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`) so an instrumentation can fail-closed — record that it observed a media part but intentionally did not capture its bytes — while preserving the original part `type` and `modality`. Enforced via a top-level `anyOf` so structurally-empty parts cannot validate. | Open — with reviewers |
All three were migrated from the closed [`open-telemetry/semantic-conventions#3673`](https://github.com/open-telemetry/semantic-conventions/pull/3673) after the GenAI conventions split into the dedicated repo on 2026-05-05. Each PR ships under the new repo's V2 Weaver schema with the corresponding `models.ipynb` updates and `make check-policies` / `make generate-all` validation.
TraceVerde already emits this shape on the wire, providing the reference implementation for these conventions. Since v1.10.0 the GenAI semconv tier is opted into by default (`OTEL_SEMCONV_STABILITY_OPT_IN` defaults to `gen_ai/dup`), so spans carry both the current and the superseded attribute spellings.
## Ecosystem & Framework Contributions
Beyond the spec, `genai_otel` is shipping **inside agent frameworks** as their OpenTelemetry observability layer — in-tree where the project accepts it, or as a standalone plugin where the project's policy keeps integrations out of the core tree — demonstrating the library powering real third-party agents, not just first-party services:
| Integration | Framework | Contribution | Status |
|---|---|---|---|
| [**hermes-otel-plugin**](https://github.com/Mandark-droid/hermes-otel-plugin) | [NousResearch Hermes](https://github.com/NousResearch/hermes-agent) | A standalone `otel` plugin (`hermes plugins install Mandark-droid/hermes-otel-plugin`) exporting Hermes turns / LLM calls / tool calls as OTel GenAI spans **and** dashboard log records, with no changes to Hermes core. When `genai-otel-instrument` is installed it additionally unlocks on-prem **GPU / energy / CO2 metrics**, **local-model cost** (parameter-size pricing for Ollama / HF / vLLM), and an inline **eval / guardrail suite** (PII, toxicity, bias, prompt-injection, restricted-topics, hallucination) scored on prompt **and** response — signals no vanilla OTel SDK or other GenAI instrumentor emits. Originally upstream [PR #48184](https://github.com/NousResearch/hermes-agent/pull/48184) — approved on code review, then republished standalone per Hermes's policy that observability backends ship as standalone plugin repos. | **Shipped** (standalone plugin, MIT) |
## Featured In
- [**Context Engineering - AI ecosystem**](https://bonigarcia.dev/context-engineering/ai-ecosystem.html) - TraceVerde is listed under *Assurance & governance > Observability* in the AI ecosystem catalog that accompanies Boni García's book [*Context Engineering: Build Consistent, Accurate, Predictable AI Systems*](https://www.manning.com/books/context-engineering) (Manning).
## Who Uses TraceVerde?
TraceVerde is used by developers and teams building production GenAI applications. If you're using TraceVerde, we'd love to hear from you!
[Add your company](https://github.com/Mandark-droid/genai_otel_instrument/issues/new?title=Add+my+company+to+users+list&labels=users) | [Join Discord](https://discord.gg/6SVz6VKK)
## Community
- [Documentation](https://mandark-droid.github.io/genai_otel_instrument/) - Full guides, API reference, and tutorials
- [Discord](https://discord.gg/6SVz6VKK) - Chat with the community
- [GitHub Issues](https://github.com/Mandark-droid/genai_otel_instrument/issues) - Bug reports and feature requests
- [Contributing](https://mandark-droid.github.io/genai_otel_instrument/community/contributing/) - How to contribute
- [Human and LLM agent contributions](https://mandark-droid.github.io/genai_otel_instrument/community/agent-contributions/) - How agents can report gaps or submit focused fixes
## License
Copyright 2025 Kshitij Thakkar. Licensed under the [Apache License 2.0](LICENSE).
