## The Continuous-Improvement Stack for Agents
Detect failures, triage root causes, and ship fixes backed by production data.
[](https://pypi.org/project/judgeval/)
[](https://docs.judgmentlabs.ai/documentation)
[](https://x.com/JudgmentLabs)
[](https://www.linkedin.com/company/judgmentlabs)
## Overview
Judgeval is an open-source Python SDK for agent improvement. It provides tracing and Agent judge evaluation for LLM-powered applications — so you can detect failures, understand what went wrong, and validate fixes against real production cases before shipping.
To get started, dive into the [docs](https://docs.judgmentlabs.ai/documentation).
## Why Judgeval
**OpenTelemetry-based tracing** -- Instrument any function with `@Tracer.observe()`. Automatically captures inputs, outputs, and LLM token usage. Built on OpenTelemetry for full compatibility with existing observability stacks.
**Agent judges** -- Define prompt-based scorers to evaluate traces at scale. Judges apply labels to traces — a binary judge applies its label when true, and a classification judge applies the chosen option — and those labels accumulate into a searchable record of how your agent acted over time. Run judges against live production traffic or replay them on historical traces to validate fixes before shipping.
**Online monitoring** -- Automatically score live production traffic server-side with no latency impact. When a judge applies a label, that label surfaces as a structured signal — configure Slack alerts so regressions and recurrences never go unnoticed.
**Broad integrations** -- Auto-instrumentation for OpenAI, Anthropic, Google GenAI, and Together AI. Framework support for LangGraph, and Claude Agent SDK.
## Quickstart
Install the SDK:
```bash
pip install judgeval
```
Set your credentials:
```bash
export JUDGMENT_API_KEY=...
export JUDGMENT_ORG_ID=...
```
Add observability to your agent with two lines of setup:
```python
from judgeval import Tracer, wrap
from openai import OpenAI
Tracer.init(project_name="my-project")
client = wrap(OpenAI())
@Tracer.observe(span_type="tool")
def search(query: str) -> str:
results = vector_db.search(query)
return results
@Tracer.observe(span_type="agent")
def run_agent(question: str) -> str:
context = search(question)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"{context}\n\n{question}"}],
)
return response.choices[0].message.content
run_agent("What is the capital of the United States?")
```
### SQL
Use `Judgeval.sql(sql_text)` for read-only queries against Judgment's virtual
schema, which abstracts the underlying storage. The server validates incoming
queries, rejects writes, and enforces organization and project scope. The client
uses its existing API key, organization membership, and resolved project.
Viewer access and the public query rate limit apply.
```python
from judgeval import Judgeval
client = Judgeval(project_name="my-project")
print(client.discover_schema())
result = client.sql("SELECT count() AS run_count FROM telemetry.traces")
print(result["rows"])
```
`client.discover_schema()` returns a Markdown string with the server's generated
tables, column types and descriptions, row semantics, examples, and query limits,
using the same reference as MCP `discover_schema`. It contains no project data
and requires organization viewer access, but no resolved project. The HTTP
equivalent is `GET /v1/sql/schema`, which returns
`{"schema": "...Markdown reference..."}`.
For query execution, use `POST /v1/projects/{projectId}/sql` with
`Authorization: Bearer `, `X-Organization-Id: `, and
JSON body `{"sql": "SELECT count() AS run_count FROM telemetry.traces"}`.
Organization and project scope are derived by the server. Use SQL predicates on
supported catalog columns and `LIMIT` to narrow results. Physical tables,
writes, multiple statements, and caller-specified execution limits are
unsupported. DAL catalog allowlists, tenant isolation, and result limits of
1,000 rows and 5 MiB apply; over-limit results return an error. SQL text must
contain a non-whitespace character and cannot exceed 50,000 characters.
The response contains `catalog_version`, `columns` (name, type, nullable),
`rows`, `row_count`, and `elapsed_ms`.
SQL integers outside JavaScript's safe range (`-(2**53 - 1)` to `2**53 - 1`)
arrive as exact decimal strings, including inside nested arrays and objects.
For example, `9007199254740993` arrives as `"9007199254740993"`; use `int(value)`
when you need a Python integer. Small integers and floating-point values remain
numbers, and column types retain their original SQL types.
Validation and execution errors use the SDK's existing exception mapping.
## Integrations
Supports OpenAI, Anthropic, Google GenAI, Together AI, LangGraph, and Claude Agent SDK. See the full [integrations docs](https://docs.judgmentlabs.ai/documentation/integrations/introduction).
## CLI
Manage agents, traces, judges, labels, and evaluations from the terminal. Query trace history, deploy judges, review the labels judges applied, and run evals against production data — all without leaving your shell. See the [CLI repo](https://github.com/JudgmentLabs/cli/) and [docs](https://docs.judgmentlabs.ai/documentation/cli).
## MCP Server
Connect Judgment to any MCP-compatible AI tool. Query agent traces, invoke judges, browse the labels judges applied, and surface failures directly inside your AI assistant or IDE. See the [docs](https://docs.judgmentlabs.ai/documentation/mcp-server).
## Links
- [Documentation](https://docs.judgmentlabs.ai/documentation)
---
Judgeval is created and maintained by [Judgment Labs](https://judgmentlabs.ai/).
