---
id: changelog-2025
title: 🐍 2025
sidebar_label: 🐍 2025
---

2025 was all about making LLM evaluation production-ready:

- **Tracing & observability** matured with deep integrations across LangChain, LlamaIndex, CrewAI, PydanticAI, and OpenAI Agents—plus first-class OpenTelemetry support
- **Agent evaluation** took center stage with new metrics for task completion, tool correctness, and MCP interactions
- **Multimodal capabilities** expanded across test cases and metrics
- **Provider support** broadened to include Anthropic, Gemini, Amazon Bedrock, and improved Ollama/Azure setups
- **Safety coverage** grew with guardrails, red-teaming, and compliance metrics
- **Reliability** improved with better async handling, timeouts, and retries
- **Documentation** expanded with comprehensive tutorials to help teams ship confidently

## December

December strengthened evaluation, multimodal support, and prompt optimization. Multimodal test cases now flow through standard evaluation paths with better placeholder detection, Azure OpenAI support, and clearer validation errors. Prompt optimization expanded with GEPA plus new algorithms, alongside more consistent schema-based outputs and broader provider configuration via typed `Settings`.

### New Feature

#### v3.7.6

- Add support for multimodal conversational test cases and goldens by automatically detecting `[DEEPEVAL:IMG:...]` placeholders across fields and attaching an `imagesMapping` so referenced images can be resolved during dataset loading. ([#2373](https://github.com/confident-ai/deepeval/pull/2373)) <!-- pr:2373 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.7.5

- Add an example script showing how to run prompt optimization with a model callback, a small golden dataset, and relevancy metrics to print original vs optimized prompts. ([#2347](https://github.com/confident-ai/deepeval/pull/2347)) <!-- pr:2347 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v3.7.4

- Add GEPA (Genetic-Pareto) prompt optimization to automatically improve prompt templates against goldens and metrics. Provide `GEPARunner.optimize(...)` with reusable runner state, sync/async execution, configurable tie-breaking, and an `OptimizationReport` attached to the returned prompt. ([#2293](https://github.com/confident-ai/deepeval/pull/2293)) <!-- pr:2293 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add MIPROv2, COPRO, and SIMBA prompt-optimization algorithms with new configuration options and runner support, enabling additional search strategies and cooperative candidate proposals during optimization. ([#2341](https://github.com/confident-ai/deepeval/pull/2341)) <!-- pr:2341 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add support for a Portkey-backed model configured via settings. Introduce Portkey-specific options (API key, model name, base URL, provider) and validate required values early to reduce misconfiguration errors. ([#2342](https://github.com/confident-ai/deepeval/pull/2342)) <!-- pr:2342 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.3

- Add Azure OpenAI support for multimodal models, including image+text prompts and optional structured/JSON outputs. Multimodal model initialization can now select Azure based on configuration, using your deployment settings and tracking token-based cost. ([#2319](https://github.com/confident-ai/deepeval/pull/2319)) <!-- pr:2319 --> ([dhinkris](https://github.com/dhinkris))

### Experimental Feature

#### v3.7.5

- Add a proof-of-concept multimodal path by auto-detecting image placeholders in dataset inputs/turns and routing supported RAG-style metrics accordingly, without requiring a separate test case type. ([#2346](https://github.com/confident-ai/deepeval/pull/2346)) <!-- pr:2346 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

### Improvement

#### v3.7.6

- Refactor evaluation to treat multimodal LLM test cases like standard LLM cases, simplifying metric execution and removing special multimodal-only handling paths. ([#2369](https://github.com/confident-ai/deepeval/pull/2369)) <!-- pr:2369 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add a dedicated CI workflow and pytest coverage for metrics, including multimodal conversational cases. Improve multimodal detection and propagate the `multimodal` flag through evaluation step generation and scoring. Prevent invalid model usage for multimodal metrics by raising an error. ([#2375](https://github.com/confident-ai/deepeval/pull/2375)) <!-- pr:2375 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve LLM metric output consistency by standardizing schema-based generation and fallback parsing. Add configuration options for more model providers (including token pricing and Bedrock settings) and align defaults for Ollama and OpenAI model selection. ([#2378](https://github.com/confident-ai/deepeval/pull/2378)) <!-- pr:2378 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.5

- Make the Ollama, Anthropic, and Gemini integrations optional at runtime. If an integration isn’t installed, raise a clear error explaining the missing dependency and how to install it. ([#2345](https://github.com/confident-ai/deepeval/pull/2345)) <!-- pr:2345 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve CI reliability by including optional model provider dependencies (`ollama`, `anthropic`, `google-genai`) in the development dependency set, reducing failures when running tests that require these integrations. ([#2357](https://github.com/confident-ai/deepeval/pull/2357)) <!-- pr:2357 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Prevent `multimodal` from being serialized in golden records by excluding it from model output. This reduces noisy fields in exported datasets and API payloads. ([#2368](https://github.com/confident-ai/deepeval/pull/2368)) <!-- pr:2368 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.7.4

- Improve API key management across LLM providers by standardizing on typed `Settings` for model name, endpoint/base URL, and secrets. Constructor arguments still take precedence, and secret values are only unwrapped when building the client. ([#2330](https://github.com/confident-ai/deepeval/pull/2330)) <!-- pr:2330 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve the staleness policy docs by pointing reopen requests to a new `MAINTAINERS.md` file. This clarifies who to mention when reviving inactive issues and what details to include. ([#2331](https://github.com/confident-ai/deepeval/pull/2331)) <!-- pr:2331 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.3

- Rename the pytest plugin entry point from `plugins` to `deepeval` so the plugin is registered under a clearer name. ([#2308](https://github.com/confident-ai/deepeval/pull/2308)) <!-- pr:2308 --> ([Gavin Morgan](https://github.com/gavmor))
- Improve agentic metric docs with corrected code samples and clearer guidance that PlanAdherence, PlanQuality, and StepEfficiency are trace-only metrics that must run via `evals_iterator` or the `observe` decorator. ([#2316](https://github.com/confident-ai/deepeval/pull/2316)) <!-- pr:2316 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve dataset conversions to carry `additional_metadata` from test cases into generated goldens, preserving metadata through CSV/JSON imports. Also prevent mixing single-turn and multi-turn items in the same dataset with clearer type errors. ([#2336](https://github.com/confident-ai/deepeval/pull/2336)) <!-- pr:2336 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Support per-trace API keys when sending and flushing traces, so background flush uses the correct credentials. This prevents traces from being uploaded with the wrong API key when multiple keys are used in the same process. ([#2337](https://github.com/confident-ai/deepeval/pull/2337)) <!-- pr:2337 --> ([Kritin Vongthongsri](https://github.com/kritinv))

### Bug Fix

#### v3.7.6

- Fix arena test case parameter validation by passing the correct arguments when checking each case, preventing incorrect validation failures for arena-based evaluations. ([#2372](https://github.com/confident-ai/deepeval/pull/2372)) <!-- pr:2372 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix multi-turn Arena G-Eval comparisons when some turns have no retrieval context, and correctly apply multimodal evaluation rules when images are present. ([#2376](https://github.com/confident-ai/deepeval/pull/2376)) <!-- pr:2376 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix MCP metrics to generate a single unified `reason` from all interaction reasons, with consistent sync/async behavior and correct cost tracking for native models. Also relax PlanAdherenceMetric required inputs and update tests to use a valid default model name. ([#2381](https://github.com/confident-ai/deepeval/pull/2381)) <!-- pr:2381 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix multimodal model validation by resolving callable model metadata factories and improving prompt concatenation for image inputs, preventing errors when checking supported multimodal models. ([#2382](https://github.com/confident-ai/deepeval/pull/2382)) <!-- pr:2382 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.5

- Fix `pydantic_ai` integration imports so the package no longer crashes when optional `pydantic-ai` and OpenTelemetry dependencies are missing, using safe fallbacks and clearer optional-dependency errors. ([#2354](https://github.com/confident-ai/deepeval/pull/2354)) <!-- pr:2354 --> ([trevor-cai](https://github.com/trevor-cai))
- Fix dependency lockfile to match `pyproject.toml`, preventing CI failures and inconsistent installs caused by mismatched dependency groups and markers. ([#2358](https://github.com/confident-ai/deepeval/pull/2358)) <!-- pr:2358 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix CLI test runs to avoid finalizing the same test run twice. This prevents duplicate uploads or local saves and reduces temp file race issues when `deepeval test run` hands off finalization to the CLI. ([#2360](https://github.com/confident-ai/deepeval/pull/2360)) <!-- pr:2360 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix binary verdict JSON examples to use lowercase booleans (`true`/`false`) instead of Python-style `True`/`False`, reducing invalid JSON output from metric templates. ([#2365](https://github.com/confident-ai/deepeval/pull/2365)) <!-- pr:2365 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.4

- Fix Anthropic client initialization to unwrap `SecretStr` API keys and consistently prefer an explicit constructor key over settings. Raise a clear error when the key is missing or empty, and add tests to prevent regressions. ([#2329](https://github.com/confident-ai/deepeval/pull/2329)) <!-- pr:2329 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix `execute` to avoid raising on async gather timeouts when errors are configured to be ignored, allowing timed-out metrics to be marked and execution to continue. ([#2335](https://github.com/confident-ai/deepeval/pull/2335)) <!-- pr:2335 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix JSON corruption on NFS by flushing and fsyncing lock-protected writes for test runs and the prompt cache. This prevents truncated or partially written files during parallel runs on network storage, with added tests to verify the behavior. ([#2338](https://github.com/confident-ai/deepeval/pull/2338)) <!-- pr:2338 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix parsing of provider-prefixed model names so inputs like `provider/model` correctly resolve to the underlying model name. ([#2343](https://github.com/confident-ai/deepeval/pull/2343)) <!-- pr:2343 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix URL and endpoint fallback resolution for local, Ollama, and Azure models so configured settings are used correctly instead of boolean values, preventing invalid base URLs during initialization. ([#2344](https://github.com/confident-ai/deepeval/pull/2344)) <!-- pr:2344 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix CLI test runs by loading the correct pytest plugin. Update the plugin argument to `deepeval` so the updated entry point is used and tests run with the intended plugin enabled. ([#2348](https://github.com/confident-ai/deepeval/pull/2348)) <!-- pr:2348 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix test discovery by adding a missing `__init__.py`, ensuring the test suite is treated as a module and runs reliably across environments. ([#2349](https://github.com/confident-ai/deepeval/pull/2349)) <!-- pr:2349 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.3

- Fix `HumanEval` so `verbose_mode` is respected and not always treated as enabled. Also fix predictions DataFrame creation by aligning the collected row fields with the DataFrame columns, preventing a column mismatch `ValueError` during evaluation. ([#2323](https://github.com/confident-ai/deepeval/pull/2323)) <!-- pr:2323 --> ([Levent K. (M.Sc.)](https://github.com/dermodmaster))

## November

November improved observability and evaluation workflows. Tracing expanded with Anthropic `messages.create` capture, richer tool-call visibility for LangChain and LlamaIndex, and clearer CrewAI spans. Evaluation grew with experiment support for `compare()` runs, new `ExactMatchMetric` and `PatternMatchMetric`, and a conversational golden synthesizer plus updated agent evaluation docs.

### New Feature

#### v3.7.1

- Add support for sending `compare()` runs as experiments, including test run summaries, hyperparameters, and run duration, and optionally opening the results in a browser. ([#2287](https://github.com/confident-ai/deepeval/pull/2287)) <!-- pr:2287 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for passing a Google service account key when using Gemini via Vertex AI, including a new CLI option to save it in config. This enables authenticated Vertex AI access without relying on default credentials. ([#2291](https://github.com/confident-ai/deepeval/pull/2291)) <!-- pr:2291 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for overriding the Confident API base URL via `CONFIDENT_BASE_URL`, allowing use of custom or self-hosted endpoints. Also align the API key header name to `CONFIDENT-API-KEY` for better compatibility. ([#2305](https://github.com/confident-ai/deepeval/pull/2305)) <!-- pr:2305 --> ([Tanay](https://github.com/tanayvaswani))
- Support creating `MLLMImage` from Base64 data by providing `dataBase64` and `mimeType`, and prevent invalid combinations like setting both `url` and `dataBase64`. Add `as_data_uri()` to return a data URI when Base64 data is available. ([#2306](https://github.com/confident-ai/deepeval/pull/2306)) <!-- pr:2306 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.7.2

- Add a conversational golden synthesizer to generate multi-turn scenarios from docs, contexts, or from scratch, with sync/async APIs and optional expected outcomes. Include new conversational styling options to control scenario context, roles, and task. ([#2310](https://github.com/confident-ai/deepeval/pull/2310)) <!-- pr:2310 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.7.0

- Add Anthropic integration that automatically captures `messages.create` (sync and async) calls for tracing, including model, inputs/outputs, token usage, and tool calls when available. ([#2224](https://github.com/confident-ai/deepeval/pull/2224)) <!-- pr:2224 --> ([Tanay](https://github.com/tanayvaswani))
- Add tracing for CrewAI knowledge retrieval events, recording the query as span input and the retrieved knowledge as span output for clearer observability. ([#2261](https://github.com/confident-ai/deepeval/pull/2261)) <!-- pr:2261 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add non-LLM metrics for exact equality and regex full matching. Use `ExactMatchMetric` to compare `actual_output` vs `expected_output`, and `PatternMatchMetric` to validate `actual_output` against a pattern with optional case-insensitive matching and verbose logs. ([#2274](https://github.com/confident-ai/deepeval/pull/2274)) <!-- pr:2274 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

### Improvement

#### v3.7.1

- Relax the dependency pin for `pytest-rerunfailures` to allow newer versions, improving compatibility with modern pytest releases and reducing dependency conflicts during installation. ([#2304](https://github.com/confident-ai/deepeval/pull/2304)) <!-- pr:2304 --> ([Konstantin Kutsy](https://github.com/bostadynamics))
- Remove unused temporary scripts from the repository to keep the codebase cleaner and reduce clutter. ([#2309](https://github.com/confident-ai/deepeval/pull/2309)) <!-- pr:2309 --> ([Bowen Liang](https://github.com/bowenliang123))

#### v3.7.2

- Fix README code block formatting so the `.env.local` setup snippet renders correctly and is easier to copy and follow. ([#2312](https://github.com/confident-ai/deepeval/pull/2312)) <!-- pr:2312 --> ([Bhuvnesh](https://github.com/DevilsAutumn))

#### v3.7.0

- Add `tools_called` tracking for LangChain and LlamaIndex traces, capturing tool name, inputs, and outputs on both the parent span and trace. This makes tool usage visible in recorded runs and improves debugging of agent workflows. ([#2251](https://github.com/confident-ai/deepeval/pull/2251)) <!-- pr:2251 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a documented issue lifecycle policy: inactive issues may be closed after 12 months, with guidance on how to request reopening and which issues are excluded. ([#2273](https://github.com/confident-ai/deepeval/pull/2273)) <!-- pr:2273 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add documentation for running end-to-end evaluations with OpenAI Agents using `evals_iterator()`, including synchronous and asynchronous examples and automatic trace generation per golden. ([#2275](https://github.com/confident-ai/deepeval/pull/2275)) <!-- pr:2275 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve non-LLM metric documentation with clearer wording, corrected references, and more consistent parameter and calculation descriptions for `ExactMatchMetric` and `PatternMatchMetric`. ([#2276](https://github.com/confident-ai/deepeval/pull/2276)) <!-- pr:2276 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add telemetry logging around OpenAI and Anthropic integrations to capture tracing when their client classes are patched. This improves observability of provider integration behavior during runtime. ([#2279](https://github.com/confident-ai/deepeval/pull/2279)) <!-- pr:2279 --> ([Tanay](https://github.com/tanayvaswani))

### Bug Fix

#### v3.7.1

- Fix tracing masking to return the value from a custom mask function in `TaskManager.mask`, so masked data is actually propagated instead of being discarded. ([#2289](https://github.com/confident-ai/deepeval/pull/2289)) <!-- pr:2289 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix runtime crashes in the OpenAI Agents callback handler by adding missing explicit imports and replacing wildcard imports. This prevents `NameError` issues and cleans up linting problems around undefined names. ([#2290](https://github.com/confident-ai/deepeval/pull/2290)) <!-- pr:2290 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix prompt template handling by catching `JSONDecodeError` and `TypeError` during parsing, and prevent crashes by wrapping `os.makedirs` in a try/except. Remove stray debug output and avoid overly broad exception handling for clearer failures. ([#2295](https://github.com/confident-ai/deepeval/pull/2295)) <!-- pr:2295 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix cache reads by creating a fresh temp cache when the existing cache file can’t be parsed or loaded. This prevents failures and keeps test runs moving forward even if the cache is corrupted. ([#2296](https://github.com/confident-ai/deepeval/pull/2296)) <!-- pr:2296 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix prompt and test-run workflows on read-only filesystems by gating disk I/O and optional `portalocker` usage. Skip local caching when the environment is read-only while continuing to upload results. ([#2297](https://github.com/confident-ai/deepeval/pull/2297)) <!-- pr:2297 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix the simulator’s example JSON output to use valid JSON booleans (`false` instead of `False`), preventing JSON parse errors. Add an `AlwaysJsonModel` stub and a regression test to ensure JSON mode output stays parseable. ([#2301](https://github.com/confident-ai/deepeval/pull/2301)) <!-- pr:2301 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.7.0

- Fix Anthropic and OpenAI integration tests to use `LlmSpanContext` for prompt and metric collection, with `thread_id` passed separately. This aligns tracing usage with the current API and prevents test failures. ([#2256](https://github.com/confident-ai/deepeval/pull/2256)) <!-- pr:2256 --> ([Tanay](https://github.com/tanayvaswani))
- Fix Anthropic async integration tests by switching to the tool’s Anthropic client, updating prompt version handling, and adding a new trace fixture for `messages.create`. ([#2258](https://github.com/confident-ai/deepeval/pull/2258)) <!-- pr:2258 --> ([Tanay](https://github.com/tanayvaswani))
- Fix Anthropic integration tests to use the official `anthropic` client and updated tracing expectations, keeping async/sync trace fixtures in sync with current outputs. ([#2259](https://github.com/confident-ai/deepeval/pull/2259)) <!-- pr:2259 --> ([Tanay](https://github.com/tanayvaswani))
- Fix TaskCompletionMetric task handling so extracted tasks only replace `task` when it wasn’t provided at initialization. Prevents a provided task from being overwritten during repeated `measure`/`a_measure` calls. ([#2260](https://github.com/confident-ai/deepeval/pull/2260)) <!-- pr:2260 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix OpenTelemetry token counting by falling back to `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` when provider-specific attributes are missing, ensuring input/output token counts are captured consistently. ([#2263](https://github.com/confident-ai/deepeval/pull/2263)) <!-- pr:2263 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix Python 3.9 compatibility by replacing `bool | None` type hints with `Optional[bool]`, preventing syntax errors when using the package on py39. ([#2264](https://github.com/confident-ai/deepeval/pull/2264)) <!-- pr:2264 --> ([OwenKephart](https://github.com/OwenKephart))
- Fix settings and dotenv test behavior by restoring auto-refresh when environment variables change and using the correct telemetry opt-out variable (`DEEPEVAL_TELEMETRY_OPT_OUT`). Add an `enable_dotenv` test marker and environment sandboxing, and improve boolean coercion coverage. ([#2266](https://github.com/confident-ai/deepeval/pull/2266)) <!-- pr:2266 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix TestRun loading and updates to preserve the in-memory state when disk reads or writes fail. Only replace the current data on a successful load, warn on errors, and fall back to in-memory updates. Ensure the parent directory exists before saving. ([#2267](https://github.com/confident-ai/deepeval/pull/2267)) <!-- pr:2267 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix integration tests by centralizing URL/JSON formatting helpers and ensuring OpenAI tracing updates span and trace attributes consistently. ([#2269](https://github.com/confident-ai/deepeval/pull/2269)) <!-- pr:2269 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix Pydantic v2 deprecation warnings by migrating all models from class-based `Config` to `ConfigDict`. Imports and common workflows no longer emit `DeprecationWarning`s. ([#2272](https://github.com/confident-ai/deepeval/pull/2272)) <!-- pr:2272 --> ([Andres Soto](https://github.com/andres-ito-traversal))
- Fix DROP batching by requiring schema-aware `batch_generate(prompts, schemas)` and failing fast with clearer errors when unsupported. Remove the obsolete `type=` argument from `batch_predict()` to match `predict()`, and make the base `batch_generate` raise `NotImplementedError` for clearer behavior. ([#2278](https://github.com/confident-ai/deepeval/pull/2278)) <!-- pr:2278 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix LangChain integration tests by importing `create_tool_calling_agent` from a stable module path, reducing breakage across LangChain versions. ([#2281](https://github.com/confident-ai/deepeval/pull/2281)) <!-- pr:2281 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix PostHog dependency constraints to allow versions from 5.4.0 up to (but not including) 7.0.0, improving compatibility with supported PostHog releases. ([#2283](https://github.com/confident-ai/deepeval/pull/2283)) <!-- pr:2283 --> ([Trevor Wilson](https://github.com/BloggerBust))

## October

October made tracing and evaluation more robust with `gen_ai.*.messages` normalization, structured message types, JSON-safe metadata, and better agent output capture across OpenAI, PydanticAI, and CrewAI. Async reliability improved with per-task timeouts and cooperative timeout budgeting so stalled work fails fast while runs finalize. Metrics gained async-by-default Hallucination evaluation, new agent-focused metrics, and configurable logging.

### Backward Incompatible Change

#### v3.6.9

- Add cooperative timeout budgeting across retries and tasks, and always persist test cases and metrics when runs are cancelled or time out. Introduce `*_OVERRIDE` env settings for per-attempt and per-task timeouts, gather buffer, and stack-trace logging, and default the OpenAI client timeout from settings. ([#2247](https://github.com/confident-ai/deepeval/pull/2247)) <!-- pr:2247 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Revert settings auto-refresh based on environment changes, restoring the previous cached Settings behavior. Telemetry and error reporting now read `DEEPEVAL_TELEMETRY_OPT_OUT` and `ERROR_REPORTING` directly from environment variables again. ([#2253](https://github.com/confident-ai/deepeval/pull/2253)) <!-- pr:2253 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v3.6.8

- Remove patched LlamaIndex agent wrappers and attach metrics/metric collections via tracing context instead. This simplifies the integration and keeps LlamaIndex agents unmodified while still enriching agent and LLM spans with the expected metadata. ([#2233](https://github.com/confident-ai/deepeval/pull/2233)) <!-- pr:2233 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.6.6

- Update the CrewAI integration to use the latest event APIs and simplify setup. Remove the custom `Agent` wrapper so you can use CrewAI’s built-in `Agent` directly while still enabling tracing via `instrument_crewai()`. ([#2152](https://github.com/confident-ai/deepeval/pull/2152)) <!-- pr:2152 --> ([Mayank](https://github.com/spike-spiegel-21))

### New Feature

#### v3.6.8

- Add per-task timeouts to semaphore-guarded async evaluation work, so individual stalled tasks fail fast instead of hanging the whole run. When exceeded, the task raises `asyncio.TimeoutError`. ([#2134](https://github.com/confident-ai/deepeval/pull/2134)) <!-- pr:2134 --> ([Harsh S](https://github.com/yujiiroo))
- Add a `tool` decorator for the CrewAI integration that propagates `metric` and `metric_collection` onto tool spans while staying compatible with existing CrewAI decorator usage patterns. ([#2206](https://github.com/confident-ai/deepeval/pull/2206)) <!-- pr:2206 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add new agent evaluation metrics (Goal Accuracy, Topic Adherence, Plan Adherence, Plan Quality, Tool Use, and Step Efficiency), and improve trace handling by relying on a metric’s `requires_trace` flag. Also prevent duplicate trace results from being reported in test output. ([#2238](https://github.com/confident-ai/deepeval/pull/2238)) <!-- pr:2238 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add async-friendly eval iteration for the PydanticAI integration so `evals_iterator()` can collect and await tasks while finalizing and serializing traces, with optional agent-level metrics during runs. ([#2241](https://github.com/confident-ai/deepeval/pull/2241)) <!-- pr:2241 --> ([trevor-cai](https://github.com/trevor-cai))

#### v3.6.7

- Add OpenAI integration support with clearer dependency errors, and update evaluation flow to avoid relying on OpenAI-specific test case queues. CI now runs integration tests when API keys are available and safely skips them otherwise. ([#2173](https://github.com/confident-ai/deepeval/pull/2173)) <!-- pr:2173 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add CrewAI wrappers `Crew`, `Agent`, and `LLM` that accept `metrics` and `metric_collection` and pass them into tracing spans. This lets you capture per-run metrics automatically when using `with trace(metrics=...)`. ([#2189](https://github.com/confident-ai/deepeval/pull/2189)) <!-- pr:2189 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.6.6

- Add display of conversational turns in multi-turn evaluations, showing role, truncated content, and any tools used. Turns are now included in test results and appear in CLI output and log/file reports. ([#2113](https://github.com/confident-ai/deepeval/pull/2113)) <!-- pr:2113 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add saving of the trace ID in the Pydantic AI instrumentator so it can be accessed later from the same `run` context. This makes it possible to reference past traces for follow-up actions like annotation. ([#2140](https://github.com/confident-ai/deepeval/pull/2140)) <!-- pr:2140 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add `test_run_id` to the `EvaluationResult` returned by `evaluate`, so you can reference the created test run programmatically. The existing `confident_link` is still returned when available. ([#2156](https://github.com/confident-ai/deepeval/pull/2156)) <!-- pr:2156 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.6.3

- Add support for pulling prompts by `label` and cache them separately from version-based pulls. Improve prompt cache reliability by using file locking and falling back to the API when the cache is missing, locked, or unreadable. ([#2154](https://github.com/confident-ai/deepeval/pull/2154)) <!-- pr:2154 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Improvement

#### v3.6.9

- Add automatic settings refresh when environment variables change and expand dotenv-related tests using the `enable_dotenv` marker to validate boolean coercion. Update telemetry env handling to use `DEEPEVAL_TELEMETRY_OPT_OUT` for clearer opt-out behavior. ([#2249](https://github.com/confident-ai/deepeval/pull/2249)) <!-- pr:2249 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.6.8

- Add timeouts around async task orchestration to prevent `asyncio.gather` from hanging indefinitely. On timeout, pending tasks are cancelled and drained before the error is raised, improving reliability of async evaluations. ([#2136](https://github.com/confident-ai/deepeval/pull/2136)) <!-- pr:2136 --> ([S3lc0uth](https://github.com/S3lc0uth))
- Improve test run metrics aggregation and results table output by refactoring into clearer helper functions. The results table formatting is now more consistent, easier to extend, and handles separators and empty rows more cleanly. ([#2153](https://github.com/confident-ai/deepeval/pull/2153)) <!-- pr:2153 --> ([Ayesha Shafique](https://github.com/Aisha630))
- Add support for passing arguments to embedding models and for customizing ConversationalGEval prompts via an `evaluation_template`. Fix MCP scoring to avoid division-by-zero when no scores are produced, and expand quickstart/docs with a template customization example. ([#2203](https://github.com/confident-ai/deepeval/pull/2203)) <!-- pr:2203 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve error surfacing during evaluation and tracing with a clearer error taxonomy and typed messages. When required inputs are missing or async tasks fail, affected spans are marked ERRORED while evaluation continues. Skip metric collection for failed nodes and keep progress reporting accurate when work is skipped. ([#2207](https://github.com/confident-ai/deepeval/pull/2207)) <!-- pr:2207 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add model request parameters (like `temperature` and `max_tokens`) to the traced LLM input messages when available, making it easier to see the exact settings used for a call. ([#2210](https://github.com/confident-ai/deepeval/pull/2210)) <!-- pr:2210 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve OpenAI integration tracing to better handle legacy and Responses API calls. Input/output extraction is now guarded to prevent crashes, messages are rendered consistently, and tool-only outputs are captured so traces still show what happened. ([#2211](https://github.com/confident-ai/deepeval/pull/2211)) <!-- pr:2211 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve the Hallucination metric by moving the required parameter list from module scope to a class-level attribute for consistency with other metrics. This makes required inputs easier to inspect and validate when integrating with custom observability tooling. ([#2215](https://github.com/confident-ai/deepeval/pull/2215)) <!-- pr:2215 --> ([Anurag Gowda](https://github.com/AnuragGowda))
- Add an OpenAI integration cookbook with a ready-to-run Colab notebook showing how to trace OpenAI SDK calls and run evaluations for standalone requests and full LLM apps. ([#2237](https://github.com/confident-ai/deepeval/pull/2237)) <!-- pr:2237 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.6.7

- Add structured prompt metadata and improved `Prompt.load()` parsing, including safer fallbacks when JSON is invalid or malformed. Test runs now capture and persist prompts seen during LLM spans for easier tracking and reproducibility. ([#2102](https://github.com/confident-ai/deepeval/pull/2102)) <!-- pr:2102 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add structured message types for LLM spans, including text, tool call, and tool output payloads. This improves typing and serialization for `input` and `output` when tracing multi-part model interactions. ([#2116](https://github.com/confident-ai/deepeval/pull/2116)) <!-- pr:2116 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve code formatting and lint compliance in OpenAI integration and trace test helpers, reducing lint noise and keeping patching logic easier to maintain. ([#2166](https://github.com/confident-ai/deepeval/pull/2166)) <!-- pr:2166 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add configurable metric logging controls, including enable/disable, verbosity, flush, and sampling rate, separate from trace sampling. This also renames `CONFIDENT_SAMPLE_RATE` to `CONFIDENT_TRACE_SAMPLE_RATE` for clarity. ([#2174](https://github.com/confident-ai/deepeval/pull/2174)) <!-- pr:2174 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve tracing so parent spans automatically include `tools_called` when tool spans run underneath them, even if the parent didn’t record tool calls directly. ([#2175](https://github.com/confident-ai/deepeval/pull/2175)) <!-- pr:2175 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve LangChain and LangGraph integration docs with clearer metric usage examples and new guidance for component-level evals. Update snippets to pass metrics inline and document how to attach metrics to LLMs and tools. Hide the PydanticAI integration page from the sidebar. ([#2177](https://github.com/confident-ai/deepeval/pull/2177)) <!-- pr:2177 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve dataset turn serialization by using `json.dumps(..., ensure_ascii=False)` so non-ASCII characters are preserved instead of being escaped in the output JSON. ([#2186](https://github.com/confident-ai/deepeval/pull/2186)) <!-- pr:2186 --> ([danerlt](https://github.com/danerlt))
- Improve multimodal metric evaluation by adding a `_log_metric_to_confident` flag and propagating it through sync and async `measure` calls, making it easier to control metric logging behavior in different execution modes. ([#2191](https://github.com/confident-ai/deepeval/pull/2191)) <!-- pr:2191 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve docs by adding tabbed examples for model integrations (OpenAI, Anthropic, Gemini, Ollama, Grok, Azure OpenAI, Amazon Bedrock, Vertex AI), making it easier to copy the right setup for each provider. ([#2196](https://github.com/confident-ai/deepeval/pull/2196)) <!-- pr:2196 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix typos and wording in the metrics DAG documentation to improve clarity and readability. ([#2198](https://github.com/confident-ai/deepeval/pull/2198)) <!-- pr:2198 --> ([Simone Busoli](https://github.com/simoneb))

#### v3.6.6

- Add a test mode for tracing integrations so spans can be captured in-memory instead of exported over OTLP. This makes integration CI tests more reliable by avoiding network calls and letting tests assert on collected trace data. ([#2131](https://github.com/confident-ai/deepeval/pull/2131)) <!-- pr:2131 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve optional CrewAI integration imports by handling missing dependencies cleanly and logging details in verbose mode, while also applying consistent formatting and lint fixes to keep CI passing. ([#2158](https://github.com/confident-ai/deepeval/pull/2158)) <!-- pr:2158 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve verbose logging for missing optional dependencies by emitting warnings instead of errors. Logs now show the missing module name when available and avoid tracebacks while pointing to the caller for easier debugging. Messages are only shown when `DEEPEVAL_VERBOSE_MODE` is enabled. ([#2159](https://github.com/confident-ai/deepeval/pull/2159)) <!-- pr:2159 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve PydanticAI tracing by including `gen_ai.system_instructions` in the captured input and flattening agent outputs to the final non-thinking text when `final_result` is missing. ([#2160](https://github.com/confident-ai/deepeval/pull/2160)) <!-- pr:2160 --> ([Mayank](https://github.com/spike-spiegel-21))
- Prevent sync HTTP calls from hanging indefinitely by enforcing per-attempt timeouts and retrying failures with a configurable Tenacity backoff policy. ([#2162](https://github.com/confident-ai/deepeval/pull/2162)) <!-- pr:2162 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.6.3

- Improve Amazon Bedrock request building by passing `generation_kwargs` through as-is, removing automatic snake_case-to-camelCase parameter translation. This makes parameter names consistent with what Bedrock expects and avoids unexpected remapping. ([#2106](https://github.com/confident-ai/deepeval/pull/2106)) <!-- pr:2106 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.6.2

- Improve OpenTelemetry tracing by normalizing `gen_ai.*.messages` that use `parts` into plain role/content messages and by forcing trace/span metadata into JSON-safe strings, including circular-reference handling, to prevent export/serialization failures. ([#2114](https://github.com/confident-ai/deepeval/pull/2114)) <!-- pr:2114 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve trace and agent input/output flattening by normalizing message parts and making non-text content JSON-serializable. This reduces errors when traces include structured or non-text payloads. ([#2115](https://github.com/confident-ai/deepeval/pull/2115)) <!-- pr:2115 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve the Hallucination metric by enabling `async_mode=True` by default, so evaluations run asynchronously unless you opt out. This can reduce blocking during metric execution in async-capable workflows. ([#2117](https://github.com/confident-ai/deepeval/pull/2117)) <!-- pr:2117 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))
- Improve code formatting and lint compliance by cleaning up imports and exception handling in tracing utilities, reducing ruff/black warnings without changing behavior. ([#2119](https://github.com/confident-ai/deepeval/pull/2119)) <!-- pr:2119 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve readability of cards and expandable sections in dark mode by refining background, borders, and text contrast. Adjust hover and focus states to keep interactive elements clear and accessible. ([#2122](https://github.com/confident-ai/deepeval/pull/2122)) <!-- pr:2122 --> ([Debangshu](https://github.com/debangshu919))
- Add per-task timeouts for async `observed_callback` execution so slow callbacks don’t block evaluation indefinitely, raising `asyncio.TimeoutError` after the configured limit. Synchronous callbacks are unaffected. ([#2127](https://github.com/confident-ai/deepeval/pull/2127)) <!-- pr:2127 --> ([Tharun K](https://github.com/tharun634))

### Bug Fix

#### v3.6.9

- Fix `EvaluationDataset.save_as` serialization so critical fields (like `tools_called`, `expected_tools`, metadata, and custom columns) are preserved across JSON, JSONL, and CSV. Multi-turn datasets now save turns as structured objects in JSON/JSONL, and CSV embeds full turn data as a JSON string while extending headers accordingly. ([#2227](https://github.com/confident-ai/deepeval/pull/2227)) <!-- pr:2227 --> ([Wang Junwei](https://github.com/wjunwei2001))
- Fix unclosed `aiohttp` client sessions when using `AmazonBedrockModel` with `aiobotocore`, preventing post-evaluation warnings about unclosed sessions and connectors. ([#2250](https://github.com/confident-ai/deepeval/pull/2250)) <!-- pr:2250 --> ([m.tsukada](https://github.com/licux))

#### v3.6.8

- Fix embedding model initialization so `generation_kwargs` is passed as a dict and client options are provided via `**client_kwargs`. Also add explicit parameters for required connection settings (like API keys, endpoints, and host) to reduce confusion when configuring clients. ([#2209](https://github.com/confident-ai/deepeval/pull/2209)) <!-- pr:2209 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix the CrewAI example notebook by adding tracing around `crew.kickoff()` and reusing the answer relevancy metric, so execution traces and metric reporting work more reliably in the walkthrough. ([#2212](https://github.com/confident-ai/deepeval/pull/2212)) <!-- pr:2212 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix `a_generate_goldens_from_contexts` so generated goldens use the correct `source_file` for each context instead of mismatching indices, and keep progress/scores aligned with the right input. ([#2213](https://github.com/confident-ai/deepeval/pull/2213)) <!-- pr:2213 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix span result extraction to treat `TraceSpanApiStatus.SUCCESS` as a successful span status, so enum-based statuses are handled correctly. Adds a regression test to prevent status comparisons from incorrectly marking spans as failed. ([#2214](https://github.com/confident-ai/deepeval/pull/2214)) <!-- pr:2214 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix `ToolCall.__repr__` to serialize `input_parameters` and dict `output` with `ensure_ascii=False`, so non-ASCII characters are shown correctly instead of being escaped in the printed representation. ([#2230](https://github.com/confident-ai/deepeval/pull/2230)) <!-- pr:2230 --> ([danerlt](https://github.com/danerlt))
- Fix Contextual Precision verdict payloads to use a singular `reason` field instead of `reasons`, improving compatibility with schema-based generation and JSON parsing. ([#2234](https://github.com/confident-ai/deepeval/pull/2234)) <!-- pr:2234 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix multimodal contextual precision verdict parsing by using the singular `reason` field to match the expected template and schema. Prevents missing reasons and related TypeErrors when generating or reading verdicts. ([#2235](https://github.com/confident-ai/deepeval/pull/2235)) <!-- pr:2235 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.6.7

- Prevent core tests from unintentionally calling the Confident backend by clearing Confident API keys from the environment and in-memory settings, and disabling dotenv autoload for these tests. This keeps `tests/test_core` isolated and avoids accidental external network use. ([#2165](https://github.com/confident-ai/deepeval/pull/2165)) <!-- pr:2165 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix test isolation by sandboxing `os.environ` per test and resetting settings before and after each run. This prevents `settings.edit(persist=False)` from leaking environment changes across tests and altering timeouts, retry policies, and other settings. ([#2168](https://github.com/confident-ai/deepeval/pull/2168)) <!-- pr:2168 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix multimodal metric parameter validation by using `check_mllm_test_case_params` instead of the LLM-only checker. This ensures multimodal test cases are validated with the correct rules and avoids incorrect parameter errors. ([#2170](https://github.com/confident-ai/deepeval/pull/2170)) <!-- pr:2170 --> ([Ayesha Shafique](https://github.com/Aisha630))
- Fix synthesizer generation so all evolved prompts are saved as Goldens instead of only the last one. Improve JSON turn serialization to preserve non-ASCII characters. Update docs to clarify when `expected_output` is produced and how to use a custom embedder for context construction. ([#2171](https://github.com/confident-ai/deepeval/pull/2171)) <!-- pr:2171 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix trace evaluation to always run even when there are no leftover tasks, and handle `_snapshot_tasks()` failures by treating them as empty. Trace evaluation is only skipped when the event loop is closed. ([#2178](https://github.com/confident-ai/deepeval/pull/2178)) <!-- pr:2178 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix G-Eval metric evaluations failing with OpenAI `o4-mini` by treating it as a model without logprobs support. The evaluator now automatically falls back to standard scoring when `o4-mini` (including `o4-mini-2025-04-16`) is used, avoiding 403 errors and completing with valid results. ([#2184](https://github.com/confident-ai/deepeval/pull/2184)) <!-- pr:2184 --> ([Niyas Hameed](https://github.com/niyasrad))
- Fix `is_successful` to correctly set and return `success` on the happy path based on the score threshold, avoiding false results when checking metric outcomes. ([#2188](https://github.com/confident-ai/deepeval/pull/2188)) <!-- pr:2188 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix evaluation tracing by mapping traces to goldens and skipping any that can’t be mapped. Prevent DFS from failing agentic test execution by finalizing runs even when spans are missing. Add async regression coverage and reset per-test state to avoid cross-test leakage. ([#2190](https://github.com/confident-ai/deepeval/pull/2190)) <!-- pr:2190 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix `assert_test` validation by rejecting mismatched metric types for LLM, conversational, and multimodal test cases. Update MultimodalToolCorrectnessMetric to use `BaseMultimodalMetric` and report the correct metric name. ([#2193](https://github.com/confident-ai/deepeval/pull/2193)) <!-- pr:2193 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix OpenAI multimodal user messages by stringifying mixed content to avoid Pydantic validation errors. Preserve the original list payload in `messages` for Responses, and add tests to prevent import-time side effects from SDK patching. ([#2199](https://github.com/confident-ai/deepeval/pull/2199)) <!-- pr:2199 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.6.6

- Fix broken tracing integration tests by moving the trace test manager into the package and updating imports so tests no longer depend on a `tests.*` module path. ([#2167](https://github.com/confident-ai/deepeval/pull/2167)) <!-- pr:2167 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.6.3

- Fix `gpt-5-chat-latest` being treated as a reasoning model that forces `temperature=1`. This restores support for `temperature=0.0` and lets users control output determinism as expected. ([#2121](https://github.com/confident-ai/deepeval/pull/2121)) <!-- pr:2121 --> ([himanushi](https://github.com/himanushi))
- Fix Google Colab buttons in the framework integration docs by pointing them to the correct example notebook paths, so the notebooks open properly from the documentation. ([#2130](https://github.com/confident-ai/deepeval/pull/2130)) <!-- pr:2130 --> ([Mayank](https://github.com/spike-spiegel-21))
- Revert the previous handling for empty `expected_tools` in the tool correctness metric, restoring the earlier scoring behavior when no expected tools are provided. ([#2139](https://github.com/confident-ai/deepeval/pull/2139)) <!-- pr:2139 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix G_Eval score normalization when the score range does not start at 0. Scores now subtract the lower bound before dividing by the range span, so values like 1–5 correctly map to 0.0–1.0. Adds test coverage for the corrected behavior. ([#2142](https://github.com/confident-ai/deepeval/pull/2142)) <!-- pr:2142 --> ([Priyank Bansal](https://github.com/Stu-ops))
- Fix PydanticAI agent tracing to capture input and output messages more reliably. If `final_result` is missing, the output now falls back to the last recorded message, improving completeness of recorded spans. ([#2149](https://github.com/confident-ai/deepeval/pull/2149)) <!-- pr:2149 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix Amazon Bedrock requests to stop forcing a default `temperature` value. `temperature` is now only sent when provided via `generation_kwargs`, letting Bedrock apply its own defaults. ([#2151](https://github.com/confident-ai/deepeval/pull/2151)) <!-- pr:2151 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.6.2

- Fix OpenAI Agents span handling so LLM span properties update only for spans marked as `llm`. This prevents spans from being skipped due to an incorrect early return and restores expected agent behavior. ([#2123](https://github.com/confident-ai/deepeval/pull/2123)) <!-- pr:2123 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix documentation code examples to correctly iterate over datasets, preventing `TypeError: 'EvaluationDataset' object is not iterable` when following the testing snippets. ([#2132](https://github.com/confident-ai/deepeval/pull/2132)) <!-- pr:2132 --> ([Denis](https://github.com/denis-snyk))
- Fix ToolCorrectnessMetric crashing with ZeroDivisionError when `expected_tools` is empty. It now returns 1.0 when both `tools_called` and `expected_tools` are empty, and 0.0 when tools are called but none are expected. Added tests for these edge cases. ([#2135](https://github.com/confident-ai/deepeval/pull/2135)) <!-- pr:2135 --> ([Priyank Bansal](https://github.com/Stu-ops))

## September

September made agent evaluation and tracing easier to adopt with expanded quickstarts and guides across LangChain, LangGraph, CrewAI, PydanticAI, and OpenAI Agents. Tracing improved with better input/output capture, OpenTelemetry/OTLP export behavior, and new APIs like `update_current_span` and `update_current_trace()`. Evaluation added G-Eval templating updates, MCP and conversational/DAG capabilities, and better dataset round-tripping.

### Backward Incompatible Change

#### v2.4.8

- Remove span feedback from the OpenTelemetry exporter so traces no longer parse or emit the `confident.span.feedback` attribute, reducing exporter dependencies and payload. ([#1942](https://github.com/confident-ai/deepeval/pull/1942)) <!-- pr:1942 --> ([Mayank](https://github.com/spike-spiegel-21))
- Change benchmark `evaluate` results to return strongly typed Pydantic models instead of untyped dicts or floats, with a consistent `overall_accuracy` interface and optional benchmark-specific fields. This is a breaking change for code expecting raw primitives. Also pin `datasets` to &lt;4.0.0 to avoid failures from deprecated loader scripts. ([#1975](https://github.com/confident-ai/deepeval/pull/1975)) <!-- pr:1975 --> ([trevor-inflection](https://github.com/trevor-inflection))

### New Feature

#### v3.5.9

- Add `evaluation_template` support to MultimodalGEval so you can customize how evaluation steps and results are generated, including strict results. Also tighten exception handling and imports to satisfy lint rules. ([#2090](https://github.com/confident-ai/deepeval/pull/2090)) <!-- pr:2090 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add Jinja template interpolation for prompt rendering, with `template` and `messages_template` now validated to be mutually exclusive to prevent ambiguous prompt types. ([#2100](https://github.com/confident-ai/deepeval/pull/2100)) <!-- pr:2100 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v3.5.5

- Add a PydanticAI `Agent` wrapper that automatically captures traces and metrics and patches the underlying model. Also export an OpenTelemetry instrumentation helper so you can instrument PydanticAI more easily without manual setup each run. ([#2071](https://github.com/confident-ai/deepeval/pull/2071)) <!-- pr:2071 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.6

- Add `set-debug` and `unset-debug` CLI commands to configure verbose logging, tracing, gRPC verbosity, and error reporting. Settings can be applied immediately and optionally persisted to a dotenv file, with a no-op guard to avoid output when nothing changes. ([#2082](https://github.com/confident-ai/deepeval/pull/2082)) <!-- pr:2082 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add support for capturing OpenAI Agents `trace` context into tool tracing, including workflow name, group/thread id, and metadata. Improve input/output handling so traced runs keep the initial input and select the correct output when running inside a trace. ([#2087](https://github.com/confident-ai/deepeval/pull/2087)) <!-- pr:2087 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.3

- Add a unified, configurable retry policy across all supported model providers. Improve transient error detection and provider-specific handling, with opt-in delegation to provider SDK retries. Allow runtime-tunable retry logging levels and env-driven backoff settings. ([#2047](https://github.com/confident-ai/deepeval/pull/2047)) <!-- pr:2047 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add tracing support for sync and async generator functions, ensuring observer spans stay open while items are yielded and close cleanly on completion or errors. ([#2074](https://github.com/confident-ai/deepeval/pull/2074)) <!-- pr:2074 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v3.5.0

- Add optional OpenTelemetry (OTLP) tracing for dataset evaluation runs via `run_otel`, generating a per-run ID and emitting start/stop spans plus per-item dummy spans. This enables exporting evaluation traces to an OTLP endpoint for run-level observability. ([#2008](https://github.com/confident-ai/deepeval/pull/2008)) <!-- pr:2008 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.1

- Add token-level streaming timestamps to LLM tracing spans, recording each emitted token with a precise ISO time to help analyze generation latency and pacing. ([#2048](https://github.com/confident-ai/deepeval/pull/2048)) <!-- pr:2048 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add prompt version listing and update prompt pulling to use version IDs, with optional background refresh that keeps the local cache up to date. ([#2057](https://github.com/confident-ai/deepeval/pull/2057)) <!-- pr:2057 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.4.8

- Add a PydanticAI integration that instruments `Agent.run` with OpenTelemetry spans and exports agent input/output and optional custom trace attributes. Provide `setup_instrumentation()` to patch the agent safely and configure span exporting when the OpenTelemetry SDK is available. ([#1851](https://github.com/confident-ai/deepeval/pull/1851)) <!-- pr:1851 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add MCP metrics for conversational evaluations, including args correctness, task completion, and tool correctness. These metrics support async execution, strict scoring, and verbose reasoning to help debug tool-using interactions. ([#1894](https://github.com/confident-ai/deepeval/pull/1894)) <!-- pr:1894 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add support for setting trace name, tags, metadata, thread ID, and user ID via `confident.trace.*` span attributes. Existing `confident.trace.attributes` is still read for compatibility but is planned for deprecation. ([#1897](https://github.com/confident-ai/deepeval/pull/1897)) <!-- pr:1897 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a configurable `language` parameter to `ConversationSimulator` so prompts can be generated in any language. Default behavior remains English, so existing usage continues to work without changes. ([#1899](https://github.com/confident-ai/deepeval/pull/1899)) <!-- pr:1899 --> ([Johan Cifuentes](https://github.com/JohanCifuentes03))
- Add MCP evaluation support for single-turn test cases with the new `MCPUseMetric`, and introduce `MultiTurnMCPUseMetric` for multi-turn conversations. This updates the MCP metrics set to better score whether the right MCP primitives and arguments are used for a task. ([#1908](https://github.com/confident-ai/deepeval/pull/1908)) <!-- pr:1908 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add a new tracing update interface that sets span data directly and introduces `update_llm_span` for token counts. This simplifies instrumenting LLM and retriever steps and makes metric evaluation work from span inputs/outputs without requiring a prebuilt test case. ([#1909](https://github.com/confident-ai/deepeval/pull/1909)) <!-- pr:1909 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for passing trace `environment`, `metric_collection`, and an optional LLM test case through OpenTelemetry attributes, so these fields are attached to exported traces and can override the default environment when provided. ([#1919](https://github.com/confident-ai/deepeval/pull/1919)) <!-- pr:1919 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add automatic loading of `.env.local` then `.env` at import time so configuration works out of the box, while keeping existing process env vars highest priority. Allow opting out via `DEEPEVAL_DISABLE_DOTENV=1`. Include a `.env.example` and expand docs on environment setup and provider keys. ([#1938](https://github.com/confident-ai/deepeval/pull/1938)) <!-- pr:1938 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add support for trace-level metrics in end-to-end evaluations, so you can attach metrics to a whole trace via `update_current_trace()` and have them run and reported alongside span-level metrics. ([#1949](https://github.com/confident-ai/deepeval/pull/1949)) <!-- pr:1949 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add an option to run conversation simulation remotely via the API with `run_remote=True`. This allows generating user turns without a local simulator model, and raises a clear error when the API key is missing. ([#1959](https://github.com/confident-ai/deepeval/pull/1959)) <!-- pr:1959 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for GPT-5 completion parameters such as `reasoning_effort`. You can now pass new model-specific options via a dedicated params dict, avoiding code changes when new parameters are introduced. ([#1965](https://github.com/confident-ai/deepeval/pull/1965)) <!-- pr:1965 --> ([John Lemmon](https://github.com/john-lemmon-lime))
- Add `--save=dotenv[:path]` to provider set/unset so credentials can be stored in a `.env` file instead of the JSON store, reducing the chance of leaking secrets. Expand set/unset tests across providers and prepare for future secure storage backends. ([#1967](https://github.com/confident-ai/deepeval/pull/1967)) <!-- pr:1967 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add MCP evaluation examples for single-turn and multi-turn conversations, showing how to connect to MCP servers, invoke tools, and build test cases from tool calls and model outputs. ([#1979](https://github.com/confident-ai/deepeval/pull/1979)) <!-- pr:1979 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add support for customizing GEval prompts via an injectable `evaluation_template`, and export `GEvalTemplate` for easier reuse. Improve evaluation docs with expanded component-level guidance, unit testing in CI/CD coverage, and updated custom embedding model configuration examples. ([#1986](https://github.com/confident-ai/deepeval/pull/1986)) <!-- pr:1986 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add `save_as` support for conversational goldens so multi-turn datasets can be exported to JSON or CSV. Turns are serialized into a single field for portable round-tripping, and `save_as` now errors clearly when called on an empty dataset. ([#1991](https://github.com/confident-ai/deepeval/pull/1991)) <!-- pr:1991 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add a `public` option when pulling datasets so you can fetch publicly shared cookbook datasets without requiring private access. ([#1995](https://github.com/confident-ai/deepeval/pull/1995)) <!-- pr:1995 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add component-level evals for LangGraph by propagating `metrics` and `metric_collection` metadata through LLM and tool spans. Include a patched `tool` decorator so tools can carry metric settings without custom wiring. ([#2000](https://github.com/confident-ai/deepeval/pull/2000)) <!-- pr:2000 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add prompt metadata to LLM tracing spans, including `alias` and `version`. This lets traces record which prompt was used alongside model and token/cost details. ([#2001](https://github.com/confident-ai/deepeval/pull/2001)) <!-- pr:2001 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `ConversationalDAGMetric` and conversational DAG node types to evaluate multi-turn conversations using a DAG workflow. Supports async and sync execution with threshold/strict modes, cycle detection, and optional verbose logs and reasons. ([#2002](https://github.com/confident-ai/deepeval/pull/2002)) <!-- pr:2002 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add component-level evaluation support for PydanticAI tools by allowing `metric_collection` or `metrics` on the `@agent.tool` decorator and recording tool outputs as tracing span attributes. ([#2003](https://github.com/confident-ai/deepeval/pull/2003)) <!-- pr:2003 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add an OpenAI Agents `Runner` wrapper that collects metrics during `run`/`run_sync` and attaches inputs/results to traces. Export `Runner` from the openai_agents package for easier use in agent eval workflows. ([#2005](https://github.com/confident-ai/deepeval/pull/2005)) <!-- pr:2005 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a `function_tool` wrapper for OpenAI Agents that automatically traces tool calls with `observe` and supports passing metrics or a metric collection. Tool spans are skipped in the tracing processor to reduce noise during component evaluation. ([#2010](https://github.com/confident-ai/deepeval/pull/2010)) <!-- pr:2010 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add Markdown document support (`.md`, `.markdown`, `.mdx`) in the synthesizer loaders. Improve lazy imports and type hints so heavy optional deps like LangChain and Chroma are only required when used, with clearer errors and updated docs on required packages. ([#2018](https://github.com/confident-ai/deepeval/pull/2018)) <!-- pr:2018 --> ([Trevor Wilson](https://github.com/BloggerBust))

### Improvement

#### v3.6.0

- Add a documented, explicit way to access the active dataset golden and pass its `expected_output` during component-level evaluation. The executor now sets and resets the current golden around user code, and tests ensure `expected_output` is preserved across spans and traces with sensible override and `None` handling. ([#2096](https://github.com/confident-ai/deepeval/pull/2096)) <!-- pr:2096 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add a new CLI guide covering install, secrets, provider switching, debug flags, retries, examples, and troubleshooting. Improve Multimodal G-Eval docs by documenting `evaluation_template` behavior, expected JSON return shapes, and a minimal customization example. Fix multiple broken links across metrics, guides, integrations, and tutorials. ([#2109](https://github.com/confident-ai/deepeval/pull/2109)) <!-- pr:2109 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve the OpenAI Agents integration by simplifying agent/model processing and exposing only the supported public API (`DeepEvalTracingProcessor`, `Agent`, and `function_tool`). This reduces unused imports and avoids exporting `Runner` from the package namespace. ([#2110](https://github.com/confident-ai/deepeval/pull/2110)) <!-- pr:2110 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.9

- Add support for `name` and `comments` fields when loading goldens from CSV/JSON and when exporting datasets via `save_as`, preserving this metadata across round-trips. ([#2066](https://github.com/confident-ai/deepeval/pull/2066)) <!-- pr:2066 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix a typo in the agents getting-started guide so the end-to-end evaluation instructions read correctly. ([#2095](https://github.com/confident-ai/deepeval/pull/2095)) <!-- pr:2095 --> ([Raj Ravi](https://github.com/RajRavi05))
- Improve PydanticAI OpenTelemetry instrumentation by reviving and consolidating it under `ConfidentInstrumentationSettings`. Agent-level tracing and metric wiring is now configured via the `instrument` setting, and the old `instrument_pydantic_ai` path is deprecated. ([#2098](https://github.com/confident-ai/deepeval/pull/2098)) <!-- pr:2098 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.5

- Improve OpenAI Agents tracing and metrics by using typed `BaseMetric` lists and recording a `Prompt` on LLM spans. Also serialize streamed and non-streamed outputs for more reliable observability and downstream processing. ([#2084](https://github.com/confident-ai/deepeval/pull/2084)) <!-- pr:2084 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.3

- Improve prompt tests by asserting the pulled prompt version starts at `0`, ensuring versioning behavior is validated alongside template and message content. ([#2064](https://github.com/confident-ai/deepeval/pull/2064)) <!-- pr:2064 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix a typo in the metrics introduction docs by changing “read-to-use” to “ready-to-use” for clearer wording. ([#2065](https://github.com/confident-ai/deepeval/pull/2065)) <!-- pr:2065 --> ([Jason Smith](https://github.com/jhs))
- Add a maintainer-only GitHub Actions workflow to manually run the full test suite against a PR’s head or merge ref, with concurrency control and optional secret-based tests. ([#2069](https://github.com/confident-ai/deepeval/pull/2069)) <!-- pr:2069 --> ([trevor-cai](https://github.com/trevor-cai))

#### v3.5.2

- Improve LangChain/LangGraph tracing by using context variables to keep the active trace consistent across tool calls and nested runs. Also expose the `tool` decorator from the integration so you can attach `metric_collection` metadata and keep span attributes in the correct trace. ([#2052](https://github.com/confident-ai/deepeval/pull/2052)) <!-- pr:2052 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve the PydanticAI integration by adding safer one-time instrumentation, tracing for `run_sync`, and consistent trace argument names (e.g., `name`, `tags`, `metadata`). This also sanitizes run context data to avoid noisy or circular payloads in captured traces. ([#2060](https://github.com/confident-ai/deepeval/pull/2060)) <!-- pr:2060 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.0

- Add a provider-agnostic retry policy with env-tunable defaults and clearer transient vs non-retryable classification. OpenAI requests now use the shared policy, disable SDK internal retries to avoid double backoff, and log retries more consistently. Quota-exhausted 429s are treated as non-retryable while timeouts and 5xx errors still retry. ([#1941](https://github.com/confident-ai/deepeval/pull/1941)) <!-- pr:1941 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add a trace JSON validation flow for integration tests. Provide commands to generate trace test data and then validate the generated JSON to catch regressions earlier. ([#2019](https://github.com/confident-ai/deepeval/pull/2019)) <!-- pr:2019 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a centralized, validated Settings system and refactor CLI config commands to use it for consistent env and persistence behavior. Prevent secrets from being written to the legacy JSON store, and allow safe persistence to dotenv files when `--save` (or the default save setting) is enabled. ([#2026](https://github.com/confident-ai/deepeval/pull/2026)) <!-- pr:2026 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve example notebook formatting to satisfy `black` and fix lint errors, making the Conversational DAG example easier to run and review. ([#2028](https://github.com/confident-ai/deepeval/pull/2028)) <!-- pr:2028 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Improve OpenTelemetry handling by importing the OTLP exporter lazily and raising a clear error when the dependency is missing. This prevents import-time failures and guides you to install `opentelemetry-exporter-otlp-proto-http` when tracing is enabled. ([#2032](https://github.com/confident-ai/deepeval/pull/2032)) <!-- pr:2032 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve test setup reliability by reusing shared helpers to reset settings environment and tear down the settings singleton. Ensure the hidden store directory is created consistently and make config tests importable via a package `__init__.py`. ([#2033](https://github.com/confident-ai/deepeval/pull/2033)) <!-- pr:2033 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add `__init__.py` files to nested test directories to prevent Python import/module name collisions during test runs. ([#2037](https://github.com/confident-ai/deepeval/pull/2037)) <!-- pr:2037 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add pre-commit hooks and Ruff to provide consistent linting and formatting on changed files. Update the lockfile to include the new development dependencies. ([#2038](https://github.com/confident-ai/deepeval/pull/2038)) <!-- pr:2038 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Temporarily skip CLI and config tests that rely on environment/settings persistence while the persistence layer is being refactored. ([#2041](https://github.com/confident-ai/deepeval/pull/2041)) <!-- pr:2041 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add a simplified PydanticAI integration API by exposing `instrument_pydantic_ai` and removing the custom Agent wrapper, with updated CLI trace flag names and tests to ensure trace output is generated as expected. ([#2042](https://github.com/confident-ai/deepeval/pull/2042)) <!-- pr:2042 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v2.4.8

- Add new documentation quickstarts for AI agent evaluation, including setup for LLM tracing and both end-to-end and component-level evals across popular frameworks. Improve clarity in existing evaluation docs with updated titles and expanded dataset terminology. ([#1818](https://github.com/confident-ai/deepeval/pull/1818)) <!-- pr:1818 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve documentation site styling for collapsible sections, sidebar menu, and code blocks for a more consistent reading experience. ([#1879](https://github.com/confident-ai/deepeval/pull/1879)) <!-- pr:1879 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve tutorials by reorganizing evaluation sections, renaming pages to simpler routes, and adding a dedicated RAG QA evaluation guide with setup and synthetic data generation examples. ([#1885](https://github.com/confident-ai/deepeval/pull/1885)) <!-- pr:1885 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add support for exporting trace-level input and output fields from span attributes, so traces capture the overall request and response alongside existing trace attributes. ([#1887](https://github.com/confident-ai/deepeval/pull/1887)) <!-- pr:1887 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve telemetry tracing integration event names by standardizing them under a `deepeval.integrations.*` namespace for more consistent reporting across supported frameworks. ([#1888](https://github.com/confident-ai/deepeval/pull/1888)) <!-- pr:1888 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add support for setting a span’s `input` and `output` via `update_current_span`, so custom values are preserved and masked correctly during trace updates. ([#1893](https://github.com/confident-ai/deepeval/pull/1893)) <!-- pr:1893 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve the LLM Arena quickstart with a full walkthrough for creating `ArenaTestCase`s, defining an arena metric, and running `compare()` to pick a winner. Also fix a typo in the arena criteria example and add the page back to the docs sidebar for easier discovery. ([#1896](https://github.com/confident-ai/deepeval/pull/1896)) <!-- pr:1896 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add LangChain integration docs with end-to-end and production evaluation examples using a `CallbackHandler`, including synchronous and asynchronous workflows and guidance on supported metrics. ([#1900](https://github.com/confident-ai/deepeval/pull/1900)) <!-- pr:1900 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve CrewAI tracing by capturing agent roles, available tools, tool inputs/outputs, and completed LLM call details, and by tracing contextual memory retrieval. This makes traces more informative across agent, tool, LLM, and retriever spans. ([#1902](https://github.com/confident-ai/deepeval/pull/1902)) <!-- pr:1902 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve DeepSeek integration docs by updating the initialization example to use `model` instead of `model_name`, matching the current constructor and reducing setup confusion. ([#1906](https://github.com/confident-ai/deepeval/pull/1906)) <!-- pr:1906 --> ([Lukman Arif Sanjani](https://github.com/lukmanarifs))
- Improve tracing for CrewAI, LangChain, LlamaIndex, and PydanticAI integrations by scoping instrumentation with a context manager. This makes span capture more reliable during initialization and setup. ([#1911](https://github.com/confident-ai/deepeval/pull/1911)) <!-- pr:1911 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve G-Eval prompting to generate reasoning before the final score. This encourages more complete evaluations and can lead to more accurate, consistent scoring across judge use cases. ([#1912](https://github.com/confident-ai/deepeval/pull/1912)) <!-- pr:1912 --> ([Bofeng Huang](https://github.com/bofenghuang))
- Add `generation_kwargs` to supported LLM model wrappers so you can pass provider-specific generation options like `top_p` and `max_tokens`, with updated docs and a new MCP quickstart page in the sidebar. ([#1921](https://github.com/confident-ai/deepeval/pull/1921)) <!-- pr:1921 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve the OpenAI integration docs by adding `gpt-5`, `gpt-5-mini`, and `gpt-5-nano` to the list of commonly used models. ([#1924](https://github.com/confident-ai/deepeval/pull/1924)) <!-- pr:1924 --> ([fangshengren](https://github.com/fangshengren))
- Add and refresh end-to-end evaluation documentation for multiple frameworks, including new guides for CrewAI and Pydantic AI plus updated LangChain examples. Include clearer setup, dataset iteration, and optional trace viewing steps to help you run evals quickly. ([#1926](https://github.com/confident-ai/deepeval/pull/1926)) <!-- pr:1926 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve documentation examples for LLM tracing and agent evaluation by fixing imports, metric names, and tracing helpers. Update the walkthrough to use `EvaluationDataset.evals_iterator()` and `update_current_span` so the sample code matches current APIs. ([#1927](https://github.com/confident-ai/deepeval/pull/1927)) <!-- pr:1927 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for newer GPT-5 and o4-mini model variants, including updated pricing metadata. Automatically set `temperature=1` for models that require it to prevent invalid configuration errors. ([#1930](https://github.com/confident-ai/deepeval/pull/1930)) <!-- pr:1930 --> ([John Lemmon](https://github.com/john-lemmon-lime))
- Improve `modes` imports by defining `__all__`, making `ARCMode` and `TruthfulQAMode` the explicitly exported public API for star-imports and tooling. ([#1932](https://github.com/confident-ai/deepeval/pull/1932)) <!-- pr:1932 --> ([trevor-inflection](https://github.com/trevor-inflection))
- Improve the Confident API client by standardizing responses and surfacing clearer errors and deprecation warnings. Update endpoints and return `(data, link)` so CLI, prompts, datasets, and tracing can consume links consistently. ([#1933](https://github.com/confident-ai/deepeval/pull/1933)) <!-- pr:1933 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Upgrade the PostHog client dependency to a newer version to avoid telemetry conflicts with projects that also use PostHog. This improves compatibility when both tools are installed in the same environment. ([#1935](https://github.com/confident-ai/deepeval/pull/1935)) <!-- pr:1935 --> ([Lucas Castelo](https://github.com/castelo-software))
- Improve PydanticAI tracing by exporting spans via an OTLP HTTP endpoint and requiring a configured API key. This makes instrumentation fail fast when credentials are missing and aligns traces with standard OpenTelemetry exporters. ([#1940](https://github.com/confident-ai/deepeval/pull/1940)) <!-- pr:1940 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve benchmark `evaluate` polymorphism by standardizing interfaces and accepting extra `**kwargs`. This lets you call different benchmarks with shared arguments like `batch_size` without crashing when a benchmark does not use them. ([#1955](https://github.com/confident-ai/deepeval/pull/1955)) <!-- pr:1955 --> ([trevor-inflection](https://github.com/trevor-inflection))
- Improve trace API payloads by populating input/output, expected output, context, retrieval context, tool calls, and metadata. This makes exported traces and generated test cases more complete and easier to debug. ([#1961](https://github.com/confident-ai/deepeval/pull/1961)) <!-- pr:1961 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve the PydanticAI integration with a new `Agent` interface that supports passing `metric_collection`, `metrics`, and trace fields directly to `run`/`run_sync`. Add validation for trace and metric inputs and require OpenTelemetry to enable tracing. ([#1978](https://github.com/confident-ai/deepeval/pull/1978)) <!-- pr:1978 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add an `overwrite_metrics` option to thread offline evaluations so you can replace existing metric results when re-running evaluations. ([#1980](https://github.com/confident-ai/deepeval/pull/1980)) <!-- pr:1980 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add new LangGraph, Pydantic AI, and CrewAI cookbooks with “Open in Colab” buttons in the docs, making it easier to run the example notebooks from the integration pages. ([#1987](https://github.com/confident-ai/deepeval/pull/1987)) <!-- pr:1987 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve OpenTelemetry export by capturing span error status and description from the official status fields instead of custom attributes. Also handle trace metadata as a dict to avoid unnecessary JSON parsing and make metadata export more reliable. ([#1990](https://github.com/confident-ai/deepeval/pull/1990)) <!-- pr:1990 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve example notebooks by adding `black[jupyter]` to dev dependencies and reformatting notebook code for more consistent, readable cells. ([#2011](https://github.com/confident-ai/deepeval/pull/2011)) <!-- pr:2011 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Add an `Agent` wrapper for openai-agents that automatically traces model calls with metrics and an optional `Prompt`. Improve tracing so span and trace inputs/outputs are captured correctly, and LLM spans record the prompt when provided. ([#2012](https://github.com/confident-ai/deepeval/pull/2012)) <!-- pr:2012 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix async execution in Conversational DAG nodes by awaiting model generation and metric evaluation calls, preventing missed results during traversal. Add detailed Conversational-DAG documentation with end-to-end examples for building and running multi-turn decision-tree evaluations. ([#2014](https://github.com/confident-ai/deepeval/pull/2014)) <!-- pr:2014 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve code formatting to satisfy linting and keep tests and DAG modules consistent with Black style. ([#2016](https://github.com/confident-ai/deepeval/pull/2016)) <!-- pr:2016 --> ([Trevor Wilson](https://github.com/BloggerBust))

### Bug Fix

#### v3.6.0

- Fix Info and Caution callouts not rendering correctly in the documentation when using dark mode, improving readability and visual consistency. ([#2111](https://github.com/confident-ai/deepeval/pull/2111)) <!-- pr:2111 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))

#### v3.5.9

- Fix streaming completion handling so the final result is captured reliably and the streamed LLM output is JSON-serializable, preventing errors when consuming streamed responses. ([#2097](https://github.com/confident-ai/deepeval/pull/2097)) <!-- pr:2097 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.5.5

- Fix async evaluations by tracking and gathering only tasks created on the active event loop, preventing coroutine re-await and cross-loop errors. Normalize awaitables via `coerce_to_task()`, cancel pending tasks when clearing, and properly shut down async generators. Replace blocking sleeps in async tests and stabilize CI workflows. ([#2068](https://github.com/confident-ai/deepeval/pull/2068)) <!-- pr:2068 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix `NonAdvice` metric scoring in `strict_mode`: enforce a threshold of 1 and return 0 when the computed score falls below that threshold. ([#2070](https://github.com/confident-ai/deepeval/pull/2070)) <!-- pr:2070 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))
- Fix `mcp_use_metric` when multiple MCP servers are configured by correctly including primitives from all servers in the interaction text. ([#2076](https://github.com/confident-ai/deepeval/pull/2076)) <!-- pr:2076 --> ([Diego Rani Mazine](https://github.com/dmazine))
- Fix sidebar heading contrast in dark mode so section titles are clearly visible and easier to scan. ([#2077](https://github.com/confident-ai/deepeval/pull/2077)) <!-- pr:2077 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))
- Fix `deepeval login` failing on Python 3.9 by avoiding the unsupported `str | ProviderSlug` type union syntax, restoring compatibility for supported Python versions. ([#2079](https://github.com/confident-ai/deepeval/pull/2079)) <!-- pr:2079 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))
- Fix incorrect argument name when configuring local models by passing `model_format` to `set_local_model_env`, preventing misconfiguration in LM Studio and vLLM setup. ([#2083](https://github.com/confident-ai/deepeval/pull/2083)) <!-- pr:2083 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))

#### v3.5.6

- Fix async eval execution to use the current trace when building `LLMTestCase`, so outputs, expected output, context, and tool expectations are recorded correctly. ([#2088](https://github.com/confident-ai/deepeval/pull/2088)) <!-- pr:2088 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix incorrect model imports so faithfulness and answer relevancy scoring load `SummaCModels` and answer relevancy models from the correct modules instead of failing at runtime. ([#2089](https://github.com/confident-ai/deepeval/pull/2089)) <!-- pr:2089 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))

#### v3.5.3

- Fix `pii_leakage` metric scoring in `strict_mode` by enforcing a threshold of 1 and returning 0 when the computed score falls below that threshold. ([#2067](https://github.com/confident-ai/deepeval/pull/2067)) <!-- pr:2067 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))
- Fix the getting-started example to use `strict_mode` instead of `strict` when creating metrics, preventing confusion and failures with the current API. ([#2073](https://github.com/confident-ai/deepeval/pull/2073)) <!-- pr:2073 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))

#### v3.5.2

- Fix a typo in the getting-started chatbots guide so the “metrics” link text is spelled correctly. ([#2058](https://github.com/confident-ai/deepeval/pull/2058)) <!-- pr:2058 --> ([grant-sobkowski](https://github.com/grant-sobkowski))
- Fix passing `test_case_content` when generating conversational evaluation prompts so evaluations run correctly instead of failing due to a missing argument. ([#2059](https://github.com/confident-ai/deepeval/pull/2059)) <!-- pr:2059 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))
- Fix LocalEmbeddingModel async embedding methods to properly await embedding requests, preventing missed awaits and ensuring async calls return embeddings reliably. ([#2061](https://github.com/confident-ai/deepeval/pull/2061)) <!-- pr:2061 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix async prompt polling to work reliably with already-running event loops by reusing a general event loop and scheduling tasks instead of always blocking on `run_until_complete`. This prevents errors in async environments and keeps polling running in the background. ([#2062](https://github.com/confident-ai/deepeval/pull/2062)) <!-- pr:2062 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix duplicate arguments being passed to `update_current_trace`, preventing conflicting trace updates in online metrics tests. ([#2063](https://github.com/confident-ai/deepeval/pull/2063)) <!-- pr:2063 --> ([Sai-Suraj-27](https://github.com/Sai-Suraj-27))

#### v3.5.0

- Fix AWS Bedrock Converse requests by translating `generation_kwargs` from snake_case to the required camelCase. Prevents `ParamValidationError` when using parameters like `max_tokens`, `top_p`, `top_k`, and `stop_sequences`. ([#2017](https://github.com/confident-ai/deepeval/pull/2017)) <!-- pr:2017 --> ([Active FigureX](https://github.com/karankulshrestha))
- Fix tool correctness scoring when no tools are expected. If both expected and called tools lists are empty, the score is now 1.0 instead of 0.0, avoiding false failures in tool-free runs. ([#2027](https://github.com/confident-ai/deepeval/pull/2027)) <!-- pr:2027 --> ([Kema Uday Kiran](https://github.com/udaykiran2427))
- Fix a documentation import typo for `DeepAcyclicGraph` so the Conversational DAG example uses the correct module path. ([#2029](https://github.com/confident-ai/deepeval/pull/2029)) <!-- pr:2029 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix telemetry tests to reliably start from a clean state by removing any existing `.deepeval` directory in the temp workspace before assertions, preventing flaky failures when the hidden store already exists. ([#2035](https://github.com/confident-ai/deepeval/pull/2035)) <!-- pr:2035 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix tracing JSON serialization by stripping embedded NUL bytes from strings before writing to Postgres. This prevents `22P05` errors when storing text/jsonb payloads that contain `\x00`. ([#2036](https://github.com/confident-ai/deepeval/pull/2036)) <!-- pr:2036 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix Grok-3 Fast output token pricing by using the correct per-1e6 divisor, preventing inflated cost calculations for responses. ([#2046](https://github.com/confident-ai/deepeval/pull/2046)) <!-- pr:2046 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix Kimi `kimi-k2-0711-preview` output cost divisor so output usage is calculated with the correct scale. ([#2054](https://github.com/confident-ai/deepeval/pull/2054)) <!-- pr:2054 --> ([Trevor Wilson](https://github.com/BloggerBust))

#### v3.5.1

- Fix `generate_goldens_from_contexts` when using `source_files` so generated goldens map to the correct source file. This prevents a possible `IndexError` when `max_goldens_per_context` exceeds the number of source files. ([#2053](https://github.com/confident-ai/deepeval/pull/2053)) <!-- pr:2053 --> ([Evan Livelo](https://github.com/vandenn))

#### v2.4.8

- Fix trace posting to allow a dynamic API key set on each trace, instead of always relying on a global configured key. This prevents traces from being skipped when the per-trace key is provided at runtime. ([#1889](https://github.com/confident-ai/deepeval/pull/1889)) <!-- pr:1889 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix Conversation Simulator generating the first user turn twice, which could duplicate user messages. First-turn prompts are now only created when starting a new conversation or after an opening message. ([#1891](https://github.com/confident-ai/deepeval/pull/1891)) <!-- pr:1891 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix Ollama integration docs to use the correct `model` parameter when initializing `OllamaModel`, avoiding confusion and incorrect example code. ([#1892](https://github.com/confident-ai/deepeval/pull/1892)) <!-- pr:1892 --> ([Phil Nash](https://github.com/philnash))
- Fix CLI `identifier` handling so runs correctly propagate the identifier into evaluation and assertion flows. ([#1903](https://github.com/confident-ai/deepeval/pull/1903)) <!-- pr:1903 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix pydantic-ai agent tracing to avoid warnings and span attribute errors by safely handling missing names and non-string inputs/outputs when recording LLM test case data. ([#1904](https://github.com/confident-ai/deepeval/pull/1904)) <!-- pr:1904 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix OpenTelemetry span metadata handling by reading `confident.span.metadata` and attaching it to exported spans, instead of dumping the full span JSON. Also reduce noisy console output by swallowing conversion/validation errors during export. ([#1910](https://github.com/confident-ai/deepeval/pull/1910)) <!-- pr:1910 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix G-Eval score normalization in non-strict mode by scaling to the rubric’s actual score range instead of always dividing by 10. This also aligns normalization behavior between `measure` and `a_measure` for consistent results across different rubrics. ([#1915](https://github.com/confident-ai/deepeval/pull/1915)) <!-- pr:1915 --> ([Bofeng Huang](https://github.com/bofenghuang))
- Fix dataset iterator integration tests to use `EvaluationDataset.evals_iterator()` and load API keys from environment variables, improving reliability and avoiding hardcoded credentials. ([#1920](https://github.com/confident-ai/deepeval/pull/1920)) <!-- pr:1920 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix OpenTelemetry and PydanticAI instrumentation by setting standard trace attributes (`name`, `tags`, `thread_id`, `user_id`, `metadata`, `environment`) and ensuring tool/expected tool attributes are parsed reliably. This improves span export compatibility and corrects retriever attribute keys. ([#1934](https://github.com/confident-ai/deepeval/pull/1934)) <!-- pr:1934 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix type checker errors when overriding methods on base model classes by adding the missing return type annotations. This prevents methods from being inferred as returning `None` and incorrectly triggering type errors in subclasses. ([#1936](https://github.com/confident-ai/deepeval/pull/1936)) <!-- pr:1936 --> ([trevor-inflection](https://github.com/trevor-inflection))
- Fix model list definitions to prevent accidental string concatenation that merged entries and broke capability checks for certain model names. This corrects which models are treated as supporting structured outputs or requiring `temperature=1`. ([#1939](https://github.com/confident-ai/deepeval/pull/1939)) <!-- pr:1939 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix conversation simulation to respect `max_user_simulations` and stop generating extra user turns. Preserve any pre-seeded `turns` without inserting the opening message, and validate invalid limits with a clear error. ([#1943](https://github.com/confident-ai/deepeval/pull/1943)) <!-- pr:1943 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix trace export to handle `trace_metadata` provided as a dict or JSON string, ensuring metadata is captured correctly. Also update async trace posting to use the API’s returned link field when reporting success. ([#1944](https://github.com/confident-ai/deepeval/pull/1944)) <!-- pr:1944 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix task completion evaluation for LangChain and LangGraph traces by correctly preparing the metric test case from the root span. This prevents missing or incorrect task extraction and avoids unexpected evaluation cost being recorded. ([#1946](https://github.com/confident-ai/deepeval/pull/1946)) <!-- pr:1946 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix ToolCorrectnessMetric to avoid division-by-zero when no expected tools are provided. Return 1.0 when both expected and called tools are empty, and 0.0 when only expected tools are empty. ([#1947](https://github.com/confident-ai/deepeval/pull/1947)) <!-- pr:1947 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix duplicate items when generating synthetic datasets with `synthesizer.generate_goldens_from_docs()`. Goldens are now added only once in the generation call chain, so each generated item appears exactly once. ([#1951](https://github.com/confident-ai/deepeval/pull/1951)) <!-- pr:1951 --> ([Jaya](https://github.com/real-jiakai))
- Fix `set-openai` CLI writing `cost_per_input_token` and `cost_per_output_token` to the wrong environment keys. This prevents inverted token cost accounting and keeps any downstream cost calculations accurate. ([#1952](https://github.com/confident-ai/deepeval/pull/1952)) <!-- pr:1952 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix `set-openai` so `--cost_per_input_token` and `--cost_per_output_token` are optional for known OpenAI models, matching runtime behavior. Improve help text to clarify that costs are only required for custom or unsupported models, reducing redundant flags and misleading errors. ([#1953](https://github.com/confident-ai/deepeval/pull/1953)) <!-- pr:1953 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix the Multi-Turn Getting Started code example by importing `ConversationalGEval` instead of an unused `GEval`, so the snippet runs correctly as written. ([#1954](https://github.com/confident-ai/deepeval/pull/1954)) <!-- pr:1954 --> ([Connor Brinton](https://github.com/connorbrinton))
- Fix Arena docs example to print results from the correct variable (`arena_geval`), preventing a NameError and making the snippet runnable as written. ([#1960](https://github.com/confident-ai/deepeval/pull/1960)) <!-- pr:1960 --> ([Julius Berger](https://github.com/knulpi))
- Fix duplicated aggregate metric results by computing pass-rate summaries once per evaluation run, and handle empty result sets safely. ([#1962](https://github.com/confident-ai/deepeval/pull/1962)) <!-- pr:1962 --> ([John Lemmon](https://github.com/john-lemmon-lime))
- Fix LangChain callback `on_llm_end` handling to avoid missing-span and bad metadata issues. Model names and token usage are now extracted safely, and token counts are left unset when unavailable. ([#1963](https://github.com/confident-ai/deepeval/pull/1963)) <!-- pr:1963 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix Azure OpenAI model calls to forward constructor kwargs (like `max_tokens`) in both sync and async generation. This ensures the API receives the expected parameters and prevents `LengthFinishReasonError`. ([#1969](https://github.com/confident-ai/deepeval/pull/1969)) <!-- pr:1969 --> ([Active FigureX](https://github.com/karankulshrestha))
- Prevent endless retries in LiteLLMModel by adding a maximum retry limit (default 6) so failures stop instead of looping indefinitely. Add support for LiteLLM proxy environment variables. Move retry settings to class-level variables to simplify future configuration changes. ([#1972](https://github.com/confident-ai/deepeval/pull/1972)) <!-- pr:1972 --> ([Radosław Hęś](https://github.com/hannex))
- Fix ContextualRelevancy evaluation when a `retrieval_context` item contains no meaningful statements. The metric now handles empty or non-informative context so LLM output can be parsed reliably instead of failing when no JSON is returned. ([#1973](https://github.com/confident-ai/deepeval/pull/1973)) <!-- pr:1973 --> ([Radosław Hęś](https://github.com/hannex))
- Fix progress bar updates during conversation simulator runs, ensuring tasks advance correctly and are removed when finished. Also ensure evaluation state is always cleaned up in a `finally` block even if an error occurs. ([#1974](https://github.com/confident-ai/deepeval/pull/1974)) <!-- pr:1974 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix telemetry to fully respect opt-out by skipping all writes when `DEEPEVAL_TELEMETRY_OPT_OUT=YES` and returning a `telemetry-opted-out` sentinel ID. Also ensure the `.deepeval` directory exists before writing telemetry data, with tests covering directory creation and file writes. ([#1976](https://github.com/confident-ai/deepeval/pull/1976)) <!-- pr:1976 --> ([Trevor Wilson](https://github.com/BloggerBust))
- Fix benchmarks to work with `datasets` 4.0.0 by removing unsupported `trust_remote_code` from `load_dataset` calls. Update MMLU and MathQA to use current Parquet datasets with the required logic adjustments. ([#1977](https://github.com/confident-ai/deepeval/pull/1977)) <!-- pr:1977 --> ([Vincent Lannurien](https://github.com/khannurien))
- Fix incorrect imports in the getting-started LLM arena docs example so the sample code runs without import errors. ([#1981](https://github.com/confident-ai/deepeval/pull/1981)) <!-- pr:1981 --> ([raphaeluzan](https://github.com/raphaeluzan))
- Fix Synthesizer state tracking by clearing `synthetic_goldens` on reset and appending newly generated goldens during doc and scratch generation, so results reflect the latest run. Update the introduction docs with required dependencies and a working end-to-end example. ([#1984](https://github.com/confident-ai/deepeval/pull/1984)) <!-- pr:1984 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix notebook evaluation runs by clearing `trace_manager.integration_traces_to_evaluate` at the start of each dataset evaluation. This prevents traces from a previous run from leaking into a new run and affecting results. ([#1985](https://github.com/confident-ai/deepeval/pull/1985)) <!-- pr:1985 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix OpenTelemetry trace status so the overall trace is marked as errored when the root span fails, improving error visibility in exported traces. ([#1993](https://github.com/confident-ai/deepeval/pull/1993)) <!-- pr:1993 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix trace status reporting so traces are marked as errored when any span fails, and include a `status` field in the trace API payload for more accurate error visibility. ([#1999](https://github.com/confident-ai/deepeval/pull/1999)) <!-- pr:1999 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix `--confident-api-key` so it works again, and make login save the key to `.env.local` by default unless `--save` is set. Logout now also removes the saved key from both the JSON keystore and dotenv, and commands no longer write "None" values for optional model settings. ([#2015](https://github.com/confident-ai/deepeval/pull/2015)) <!-- pr:2015 --> ([Trevor Wilson](https://github.com/BloggerBust))

## August

August made evaluation and tracing more production-ready with refreshed docs covering component-level evaluation, tracing, and deployment patterns. Tracing gained richer LLM outputs, a v1 OpenTelemetry exporter, better span ordering, and deeper LangChain/LlamaIndex/CrewAI integrations with `metrics` and `metric_collection` support. New tutorials included the Medical Chatbot series and improved RAG guides.

### New Feature

#### v3.3.5

- Add a new Medical Chatbot tutorial series to the docs, covering development, evaluation, improvement, and deployment of a multi-turn chatbot. Improve and correct several evaluation docs examples and parameter descriptions for multi-turn test cases and datasets. ([#1802](https://github.com/confident-ai/deepeval/pull/1802)) <!-- pr:1802 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add CLI support to configure Grok, Moonshot, and DeepSeek as the LLM provider for evaluations, including setting the model name, API key, and temperature. You can switch back to the default OpenAI setup with corresponding `unset-*` commands. ([#1807](https://github.com/confident-ai/deepeval/pull/1807)) <!-- pr:1807 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add a Medical Chatbot tutorial to the docs and navigation, with updated walkthrough content and links for building, configuring, and evaluating the example app. ([#1814](https://github.com/confident-ai/deepeval/pull/1814)) <!-- pr:1814 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add support for evaluating LangGraph/LangChain traces with metrics via the callback handler. Root spans can now carry `metrics` and an optional `metric_collection`, and captured traces can be queued for evaluation instead of being posted immediately. ([#1829](https://github.com/confident-ai/deepeval/pull/1829)) <!-- pr:1829 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a CrewAI `Agent` wrapper that registers agents with an optional `metric_collection` and per-agent metrics, enabling easier evaluation and online tracing during crew runs. ([#1833](https://github.com/confident-ai/deepeval/pull/1833)) <!-- pr:1833 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a v1 OpenTelemetry span exporter that supports API key setup and trace configuration via env vars or OTel resource attributes. Improve trace handling by preserving provided trace IDs, applying trace metadata, and safely ending and clearing active traces after export. ([#1838](https://github.com/confident-ai/deepeval/pull/1838)) <!-- pr:1838 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add MCP support to conversational test cases by allowing turns to record MCP tool/prompt/resource calls and optional server metadata, with validation of MCP types to catch invalid inputs early. ([#1839](https://github.com/confident-ai/deepeval/pull/1839)) <!-- pr:1839 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add support for setting trace attributes in the LangChain callback handler. You can now pass `name`, `tags`, `metadata`, `thread_id`, and `user_id` when creating the callback to populate these fields on the completed trace. ([#1862](https://github.com/confident-ai/deepeval/pull/1862)) <!-- pr:1862 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add an `ArgumentCorrectnessMetric` to score whether tool call arguments match the user input, with optional reasons and async support. Returns a perfect score when no tool calls are provided. ([#1866](https://github.com/confident-ai/deepeval/pull/1866)) <!-- pr:1866 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add a revamped conversation simulator that generates conversational test cases from `ConversationalGolden` inputs using a provided model callback, with configurable opening message, concurrency, and async or sync execution. ([#1876](https://github.com/confident-ai/deepeval/pull/1876)) <!-- pr:1876 --> ([Kritin Vongthongsri](https://github.com/kritinv))

### Improvement

#### v3.3.5

- Improve component-level evaluation docs with clearer guidance on when to use it, what tracing means, and how to log in to view traces. Reorganize sections and examples for easier navigation and fewer confusing callouts. ([#1782](https://github.com/confident-ai/deepeval/pull/1782)) <!-- pr:1782 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve the Meeting Summarizer tutorial with a new Deployment section covering CI/CD-style continuous evaluation, dataset reuse, and optional tracing setup. Also update tutorial navigation and fix a broken docs anchor link. ([#1783](https://github.com/confident-ai/deepeval/pull/1783)) <!-- pr:1783 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Bump the package release metadata and version number for a new release. ([#1784](https://github.com/confident-ai/deepeval/pull/1784)) <!-- pr:1784 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve LLM trace output to match the updated UI by capturing structured AI responses, including role, content, and tool call details instead of only a concatenated string. ([#1786](https://github.com/confident-ai/deepeval/pull/1786)) <!-- pr:1786 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve the meeting summarizer tutorial with updated walkthrough content, refreshed screenshots, and clearer examples for generating summaries and action items using different models. ([#1788](https://github.com/confident-ai/deepeval/pull/1788)) <!-- pr:1788 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix typos and formatting across tracing integrations, tests, and documentation for clearer examples and cleaner files. ([#1789](https://github.com/confident-ai/deepeval/pull/1789)) <!-- pr:1789 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve the RAG QA Agent tutorial and navigation by adding a new tutorial section, updating sidebar links and icons, and refreshing examples to use `deepeval test run` instead of running `pytest` directly. ([#1793](https://github.com/confident-ai/deepeval/pull/1793)) <!-- pr:1793 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve docs and tutorials by switching embedded images to hosted URLs and removing bundled image assets, keeping guides lighter and images consistently available. ([#1794](https://github.com/confident-ai/deepeval/pull/1794)) <!-- pr:1794 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve SummarizationMetric schema naming and usage to reduce ambiguity and make results clearer. This refactor replaces a generic `Verdicts` schema with more descriptive Pydantic schemas, improving readability and maintainability. ([#1804](https://github.com/confident-ai/deepeval/pull/1804)) <!-- pr:1804 --> ([Shabareesh Shetty](https://github.com/ShabiShett07))
- Improve tutorial introductions by adding Tech Stack cards that show the key tools used in each guide, making it easier to understand the setup at a glance. ([#1808](https://github.com/confident-ai/deepeval/pull/1808)) <!-- pr:1808 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve tutorials and docs with updated examples and configuration names, plus refreshed navigation and UI tweaks for easier browsing. ([#1825](https://github.com/confident-ai/deepeval/pull/1825)) <!-- pr:1825 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Support passing extra `**kwargs` to underlying LLM clients across providers. This lets you customize client setup (for example timeouts, proxies, or transport settings) without modifying the model wrappers. ([#1827](https://github.com/confident-ai/deepeval/pull/1827)) <!-- pr:1827 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve contributor setup instructions by updating the dependency installation command from `make install` to `poetry install`. ([#1828](https://github.com/confident-ai/deepeval/pull/1828)) <!-- pr:1828 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add patched LlamaIndex agents that accept `metrics` and `metric_collection`, and rework LlamaIndex tracing to start and link traces correctly for workflow/agent runs. ([#1836](https://github.com/confident-ai/deepeval/pull/1836)) <!-- pr:1836 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix docs metadata and improve tutorial link cards by adding `singleTurn` tags to several metric pages and updating card layout with icons and objectives for clearer navigation. ([#1837](https://github.com/confident-ai/deepeval/pull/1837)) <!-- pr:1837 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve model CLI config handling by separating stored keys for evaluation LLMs vs embeddings, reducing key collisions when switching providers or running `unset-*` commands. ([#1855](https://github.com/confident-ai/deepeval/pull/1855)) <!-- pr:1855 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve tutorials with clearer section titles, updated wording, and expanded guidance for building and evaluating RAG QA and summarization agents, including a better focus on production eval setup. ([#1860](https://github.com/confident-ai/deepeval/pull/1860)) <!-- pr:1860 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

### Bug Fix

#### v3.3.5

- Fix LLM span cost calculation by honoring `cost_per_input_token` and `cost_per_output_token` passed to `observe`, ensuring traced runs report the correct token costs. ([#1787](https://github.com/confident-ai/deepeval/pull/1787)) <!-- pr:1787 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix async OpenAI integration by restoring `asyncio.create_task` safely after evaluation, preventing leaked monkeypatching across runs and improving stability when running concurrent test cases. ([#1790](https://github.com/confident-ai/deepeval/pull/1790)) <!-- pr:1790 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix `g_eval` to prevent a crash when accumulating evaluation cost if the initial cost is `None`. This avoids a `TypeError` during async evaluation and allows scoring to complete normally. ([#1796](https://github.com/confident-ai/deepeval/pull/1796)) <!-- pr:1796 --> ([高汝貞](https://github.com/TheNeuAra))
- Fix the docs snippet for `ConversationalGEval` by renaming the example variable to `metric`, making it consistent and easier to copy and run. ([#1799](https://github.com/confident-ai/deepeval/pull/1799)) <!-- pr:1799 --> ([Nimish Bongale](https://github.com/nimishbongale))
- Fix the few-shot example used in the Synthesizer constrained evolution template so the sample rewritten input correctly matches the solar power prompt and produces more consistent guidance. ([#1800](https://github.com/confident-ai/deepeval/pull/1800)) <!-- pr:1800 --> ([Simon M.](https://github.com/simon376))
- Prevent mixing single-turn and multi-turn goldens in a dataset by enforcing the dataset mode and raising clear `TypeError`s for invalid items. Add `add_golden` to append goldens after initialization. ([#1810](https://github.com/confident-ai/deepeval/pull/1810)) <!-- pr:1810 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix conversation eval serialization by using the correct API field aliases for `retrievalContext`, `toolsCalled`, and `additionalMetadata`, and by typing tool calls as `ToolCall` objects. ([#1811](https://github.com/confident-ai/deepeval/pull/1811)) <!-- pr:1811 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix tutorial command examples to run evaluation tests with `deepeval test run` instead of `pytest`, and improve YAML snippet formatting for the deployment guide. ([#1830](https://github.com/confident-ai/deepeval/pull/1830)) <!-- pr:1830 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix AzureOpenAIModel initialization to use the correct `model_name` argument instead of `model`, restoring compatibility with Azure OpenAI deployments. This prevents setup failures that made Azure-backed usage unusable in recent releases. ([#1832](https://github.com/confident-ai/deepeval/pull/1832)) <!-- pr:1832 --> ([StefanMojsilovic](https://github.com/StefanMojsilovic))
- Fix `LiteLLMModel` `generate`/`a_generate` to always return `(result, cost)` when a schema is provided. Prevents unpacking errors in schema-based metrics and restores consistent cost reporting. ([#1841](https://github.com/confident-ai/deepeval/pull/1841)) <!-- pr:1841 --> ([Dylan Li](https://github.com/DylanLi-Hang))
- Fix a type hint in `login_with_confident_api_key` by using `str` for the API key parameter, improving type checking and editor autocomplete. ([#1847](https://github.com/confident-ai/deepeval/pull/1847)) <!-- pr:1847 --> ([John Lemmon](https://github.com/john-lemmon-lime))
- Fix LangChain/LangGraph prompt parsing so multi-line messages and recognized roles are grouped correctly, instead of being split line-by-line or misclassified as Human messages. ([#1848](https://github.com/confident-ai/deepeval/pull/1848)) <!-- pr:1848 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix LLM tracing to accept and safely serialize non-standard output objects so responses aren’t dropped when capturing spans. ([#1849](https://github.com/confident-ai/deepeval/pull/1849)) <!-- pr:1849 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix CLI model configuration to clear previously saved evaluation or embedding settings when switching providers, preventing stale keys from overriding the newly selected model. ([#1852](https://github.com/confident-ai/deepeval/pull/1852)) <!-- pr:1852 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix code execution in the HumanEval benchmark by calling `exec` on compiled code instead of recursively invoking the secure executor, preventing infinite recursion and allowing snippets to run correctly. ([#1856](https://github.com/confident-ai/deepeval/pull/1856)) <!-- pr:1856 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix missing temperature handling in `GptModel` `generate`/`a_generate` when no schema is provided, so output randomness is consistently user-controlled instead of falling back to the provider default (often 1). ([#1857](https://github.com/confident-ai/deepeval/pull/1857)) <!-- pr:1857 --> ([Daniel Yakubov](https://github.com/DanielYakubov))
- Fix crashes in synthesizer workflows by guarding progress updates and handling fewer than 10 goldens when sampling examples. Improve test reliability by adding a `pytest.ini` config and expanding the test suite so CI runs `pytest` directly. ([#1858](https://github.com/confident-ai/deepeval/pull/1858)) <!-- pr:1858 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix OpenTelemetry trace exporting by ordering spans into parent-child trees and treating missing parents as root spans, preventing failures on incomplete span batches. Update LLM span attribute keys to the `confident.llm.*` namespace so model, token, and prompt fields are captured correctly. ([#1859](https://github.com/confident-ai/deepeval/pull/1859)) <!-- pr:1859 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix misuse metric failures by passing the correct `misuse_violations` parameter to `generate_reason` in `MisuseTemplate`. This prevents errors when running `measure`. ([#1863](https://github.com/confident-ai/deepeval/pull/1863)) <!-- pr:1863 --> ([Rohit ojha](https://github.com/siesto1elemento))
- Prevent generating more synthetic inputs than requested by enforcing `max_goldens_per_context` and truncating any extra results. This keeps dataset sizes predictable and avoids overshooting configured limits. ([#1867](https://github.com/confident-ai/deepeval/pull/1867)) <!-- pr:1867 --> ([Noah Gil](https://github.com/noah-gil))
- Fix structured output requests in the LiteLLM model by passing the Pydantic schema directly via `response_format` instead of an unsupported `json_schema` argument. Prevents `TypeError` failures when requesting JSON-formatted responses. ([#1871](https://github.com/confident-ai/deepeval/pull/1871)) <!-- pr:1871 --> ([Rohit ojha](https://github.com/siesto1elemento))
- Fix conversation relevancy windowing by grouping turns into valid user→assistant interactions and flattening them before verdict generation, preventing invalid or partial turns from skewing results. ([#1873](https://github.com/confident-ai/deepeval/pull/1873)) <!-- pr:1873 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix an ImportError caused by a circular import between the scorer module and the IFEval benchmark. The `Scorer` import is now deferred to IFEval initialization so modules load cleanly and IFEval can be imported reliably. ([#1875](https://github.com/confident-ai/deepeval/pull/1875)) <!-- pr:1875 --> ([Rohit ojha](https://github.com/siesto1elemento))
- Fix Conversation Simulator turn generation and progress tracking: `max_turns` is now validated, opening messages count toward the limit, and async vs sync callbacks are handled automatically without raising type errors. Simulated test cases now carry over scenario and metadata fields from the golden inputs. ([#1878](https://github.com/confident-ai/deepeval/pull/1878)) <!-- pr:1878 --> ([Kritin Vongthongsri](https://github.com/kritinv))

## July

July improved tracing and evaluation across agent frameworks with major upgrades to LangChain/LangGraph, CrewAI, LlamaIndex, and OpenTelemetry span handling. Safety coverage expanded with new metrics for PII leakage, role violations, non-advice, and misuse, plus IFEval benchmark support and better task-completion evaluation. The default model moved from `gpt-4o` to `gpt-4.1` with updated costs and docs.

### New Feature

#### v3.2.6

- Add a LangChain/LangGraph callback handler that captures chain, tool, LLM, and retriever events into tracing spans, and automatically starts and ends a trace for top-level runs. ([#1722](https://github.com/confident-ai/deepeval/pull/1722)) <!-- pr:1722 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a CrewAI integration to instrument `crewai.LLM.call` and capture LLM input/output in traces. Raises a clear error if CrewAI is not installed and supports optional API key login before patching. ([#1723](https://github.com/confident-ai/deepeval/pull/1723)) <!-- pr:1723 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add a revised CrewAI tracing integration with an `instrumentator()` helper that listens to CrewAI events and captures agent and LLM calls as trace spans. Also emit integration telemetry to New Relic in addition to existing PostHog tracking. ([#1724](https://github.com/confident-ai/deepeval/pull/1724)) <!-- pr:1724 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add support for the IFEval benchmark to evaluate instruction-following and format compliance. Includes rule-based verification and more detailed per-instruction reporting in verbose mode. ([#1729](https://github.com/confident-ai/deepeval/pull/1729)) <!-- pr:1729 --> ([Abhishek Ranjan](https://github.com/AbhishekRP2002))
- Add a new `dataset()` test-run interface that lets you iterate over goldens from a local list or a pulled dataset alias and track the run via `test_run` tasks, with async execution support. ([#1737](https://github.com/confident-ai/deepeval/pull/1737)) <!-- pr:1737 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add 10 new safety metrics to detect PII leakage, harmful or illegal instructions, misinformation, graphic content, prompt extraction, role boundary violations, IP issues, manipulation, and risky command execution. Improve template consistency, align parameter names, and add full test coverage for these checks. ([#1747](https://github.com/confident-ai/deepeval/pull/1747)) <!-- pr:1747 --> ([sid-murali](https://github.com/sid-murali))
- Add new safety metrics: `PIILeakageMetric` to detect SSNs/emails/addresses, `RoleViolationMetric` to flag role-breaking output, and `NonAdviceMetric` to catch financial or medical advice. Require explicit parameters like `role` and advice `types`, and switch role violations to a clear yes/no result. ([#1749](https://github.com/confident-ai/deepeval/pull/1749)) <!-- pr:1749 --> ([sid-murali](https://github.com/sid-murali))
- Add CLI support to set/unset the default OpenAI model and per-token pricing used by metrics. `GPTModel` can now read model name and pricing from saved settings, and will prompt for pricing when using an unknown model. ([#1766](https://github.com/confident-ai/deepeval/pull/1766)) <!-- pr:1766 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add the Misuse metric to detect when an LLM uses a specialized domain chatbot inappropriately (for example, asking a finance bot to write poetry). This helps keep outputs aligned with domain expertise and prevents scope creep in specialized AI use cases. ([#1773](https://github.com/confident-ai/deepeval/pull/1773)) <!-- pr:1773 --> ([sid-murali](https://github.com/sid-murali))

### Improvement

#### v3.2.6

- Prepare a new release by updating package metadata and internal version information. ([#1721](https://github.com/confident-ai/deepeval/pull/1721)) <!-- pr:1721 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Add telemetry events that record when tracing integrations are initialized (LangChain, LlamaIndex, and OpenTelemetry exporter), respecting telemetry opt-out settings. ([#1725](https://github.com/confident-ai/deepeval/pull/1725)) <!-- pr:1725 --> ([Mayank](https://github.com/spike-spiegel-21))
- Update the default OpenAI and multimodal GPT model from `gpt-4o` to `gpt-4.1`. Cost calculations and documentation examples now also default to `gpt-4.1` when a model name is not specified. ([#1727](https://github.com/confident-ai/deepeval/pull/1727)) <!-- pr:1727 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add an X (Twitter) follow icon to the README and documentation site header for quicker access to the project’s social profile. ([#1731](https://github.com/confident-ai/deepeval/pull/1731)) <!-- pr:1731 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve documentation and examples for multi-turn chatbot evaluation, clarifying conversation simulation, CI setup, and metric usage. Fix small wording issues in docs and ensure files end with a trailing newline. ([#1732](https://github.com/confident-ai/deepeval/pull/1732)) <!-- pr:1732 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve task completion evaluations by supporting span-based tracing. TaskCompletionMetric can now run without an `LLMTestCase` when it’s the only metric, and it attaches the trace to produce suggested fixes while giving a clearer error for other metrics missing `update_current_span()`. ([#1734](https://github.com/confident-ai/deepeval/pull/1734)) <!-- pr:1734 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve CrewAI tracing by capturing tool usage and memory search as dedicated spans, with inputs/outputs recorded for easier debugging. LLM spans no longer fail when a parent span can’t be found. ([#1740](https://github.com/confident-ai/deepeval/pull/1740)) <!-- pr:1740 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve LlamaIndex instrumentation by unifying event and span handling, generating stable span UUIDs, and properly starting/ending traces when spans are dropped or completed. This makes LLM and tool spans more consistent and avoids lingering spans in trace output. ([#1745](https://github.com/confident-ai/deepeval/pull/1745)) <!-- pr:1745 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve OpenAI integration by evaluating captured OpenAI test case/metric pairs when no traces are available, and by recording the latest OpenAI hyperparameters in the test run. Also clear stored OpenAI pairs after a run to avoid leaking state between evaluations. ([#1746](https://github.com/confident-ai/deepeval/pull/1746)) <!-- pr:1746 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve LangChain and LangGraph integration with clearer message roles, better tool call/result handling, and cleaner inputs. Fix span naming plus fallback/metadata behavior and make outputs visible in LangChain. Update docs with function descriptions; token usage and cost reporting is still pending. ([#1752](https://github.com/confident-ai/deepeval/pull/1752)) <!-- pr:1752 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix a typo in the README explanation of `expected_output` and `GEval` to make the quickstart guidance clearer. ([#1754](https://github.com/confident-ai/deepeval/pull/1754)) <!-- pr:1754 --> ([Chetan Shinde](https://github.com/css911))
- Add comprehensive docs for `NonAdviceMetric`, `PIILeakageMetric`, and `RoleViolationMetric`, including usage examples, parameters, and scoring rubrics. Improve consistency by standardizing metric names, schema fields, and clarifying parameter naming for these metrics. ([#1755](https://github.com/confident-ai/deepeval/pull/1755)) <!-- pr:1755 --> ([sid-murali](https://github.com/sid-murali))
- Improve the tutorials onboarding experience by grouping Getting Started pages in the sidebar and refreshing the Introduction with clearer guidance and a first evaluation walkthrough. ([#1759](https://github.com/confident-ai/deepeval/pull/1759)) <!-- pr:1759 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve compatibility by loosening the `click` version restriction so newer `click` releases can be used, reducing dependency conflicts and avoiding the need to pin an outdated version. ([#1760](https://github.com/confident-ai/deepeval/pull/1760)) <!-- pr:1760 --> ([lwarsaame](https://github.com/lwarsaame))
- Improve the tutorial introduction and setup docs with a clearer getting-started flow, curated tutorial cards, and tightened wording. Add a concrete `OPENAI_API_KEY` export example and clarify the required `test_` filename prefix. ([#1761](https://github.com/confident-ai/deepeval/pull/1761)) <!-- pr:1761 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add a blog sidebar that lists all posts and expand the tutorials sidebar with a new Meeting Summarizer section. Improve tutorials navigation by renaming the tutorial card component to `LinkCards` and enabling sidebar icons on tutorial routes. ([#1767](https://github.com/confident-ai/deepeval/pull/1767)) <!-- pr:1767 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Support passing extra client options to Azure OpenAI model initialization via `kwargs`. This lets you customize the underlying Azure OpenAI client without modifying the tool’s source code. ([#1772](https://github.com/confident-ai/deepeval/pull/1772)) <!-- pr:1772 --> ([Aaryan Verma](https://github.com/Aaryanverma))
- Improve tutorials and docs navigation with refreshed summarization content, clearer headings, and new example visuals. Add optional numbered tutorial link cards and temporarily hide the Meeting Summarizer section from the sidebar. ([#1775](https://github.com/confident-ai/deepeval/pull/1775)) <!-- pr:1775 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve dependency compatibility by loosening the `tenacity` version constraint to allow newer releases while keeping a safe supported range. ([#1776](https://github.com/confident-ai/deepeval/pull/1776)) <!-- pr:1776 --> ([Andy Freeland](https://github.com/rouge8))
- Improve dataset handling by aligning dataset endpoints, making golden lists optional, and supporting extra conversational metadata like `scenario`, `userDescription`, and `comments` when sending test runs. ([#1777](https://github.com/confident-ai/deepeval/pull/1777)) <!-- pr:1777 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve the TaskCompletionMetric docs with a clearer tracing example, including the correct `Golden` input format and updated imports for `evaluate` and `ToolCall`. This makes it easier to run the sample code without adjustments. ([#1779](https://github.com/confident-ai/deepeval/pull/1779)) <!-- pr:1779 --> ([Mayank](https://github.com/spike-spiegel-21))

### Bug Fix

#### v3.2.6

- Fix the quickstart link shown after CLI login so it points to the correct setup page. ([#1726](https://github.com/confident-ai/deepeval/pull/1726)) <!-- pr:1726 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix OpenAI Completions examples in the docs to use the current `OpenAI()` client and `chat.completions.create`, preventing runtime errors and incorrect response parsing in sample code. ([#1728](https://github.com/confident-ai/deepeval/pull/1728)) <!-- pr:1728 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix `AnthropicModel.calculate_cost` indentation so cost calculation and fallback pricing warning run correctly when pricing is missing. ([#1739](https://github.com/confident-ai/deepeval/pull/1739)) <!-- pr:1739 --> ([nsking02](https://github.com/nsking02))
- Fix component-level evaluation serialization by converting test run payloads into JSON-safe data before sending them, preventing failures when metrics or complex objects are included. ([#1744](https://github.com/confident-ai/deepeval/pull/1744)) <!-- pr:1744 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix synthetic golden sample generation when `context_size` is 1 by making the context generator always return a consistent list-of-lists shape. This prevents type mismatches in `Golden` creation when a document has only one chunk. ([#1748](https://github.com/confident-ai/deepeval/pull/1748)) <!-- pr:1748 --> ([Nicolas Torres](https://github.com/ntgussoni))
- Improve JSON tool-call reliability when using `instructor` TOOLS mode with custom LLMs by renaming internal `Reason` schemas so models don’t skip tool calls and return plain content. This prevents exceptions and keeps structured outputs coming from `tool_calls` as expected. ([#1753](https://github.com/confident-ai/deepeval/pull/1753)) <!-- pr:1753 --> ([Radosław Hęś](https://github.com/hannex))
- Fix `EvaluationDataset.evaluate` type hints to accept all supported metric base types and explicitly annotate the `EvaluationResult` return type, avoiding circular import issues. ([#1756](https://github.com/confident-ai/deepeval/pull/1756)) <!-- pr:1756 --> ([AI](https://github.com/yalishanda42))
- Fix an error when calculating OpenAI costs by handling a missing model value and falling back to the default model when none is provided. ([#1768](https://github.com/confident-ai/deepeval/pull/1768)) <!-- pr:1768 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix component-level metric data not showing up in test results by extracting and appending trace and span-level metric outputs to the reported results. ([#1769](https://github.com/confident-ai/deepeval/pull/1769)) <!-- pr:1769 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix syntax errors in the evaluation test case documentation examples so `ToolCall` snippets parse correctly and can be copied into Python without edits. ([#1770](https://github.com/confident-ai/deepeval/pull/1770)) <!-- pr:1770 --> ([Dhanesh Gujrathi](https://github.com/dhanesh24g))
- Fix the Task Completion metric documentation example by using valid sample inputs for `destination` and `days`, preventing the snippet from failing when copied and run. ([#1778](https://github.com/confident-ai/deepeval/pull/1778)) <!-- pr:1778 --> ([Kritin Vongthongsri](https://github.com/kritinv))

## June

June made evaluations and tracing more robust across providers and async workloads with fixes to prevent crashes and broken serialization. Tracing matured with improved OpenAI/OTEL integrations and new hooks for OpenAI Agents and LlamaIndex via `trace_manager.configure`. Evaluation added native LiteLLM support, `MultimodalGEval`, arena-style `GEval`, and `jsonl` dataset saving.

### Backward Incompatible Change

#### v3.1.5

- Remove the `client` parameter from `observe()` and rely on `trace_manager.configure(openai_client=...)` for LLM spans. LLM tracing now requires either a `model` in `observe` or a configured `openai_client`, otherwise a clear error is raised. ([#1667](https://github.com/confident-ai/deepeval/pull/1667)) <!-- pr:1667 --> ([Mayank](https://github.com/spike-spiegel-21))

#### v3.0.8

- Improve the packaged API by removing the `monitor` helpers from top-level imports, leaving only `send_feedback` and `a_send_feedback` available via `deepeval`. ([#1673](https://github.com/confident-ai/deepeval/pull/1673)) <!-- pr:1673 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### New Feature

#### v3.1.9

- Add a LlamaIndex integration entry point via `instrument_llama_index` to hook into LlamaIndex instrumentation and capture agent runs for monitoring. ([#1714](https://github.com/confident-ai/deepeval/pull/1714)) <!-- pr:1714 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add expanded OpenAI multimodal model support, including newer GPT-4.1 and o-series options. Improve structured output handling by using native parsing when available and falling back to JSON parsing when needed, while tracking log-prob limitations for unsupported models. ([#1716](https://github.com/confident-ai/deepeval/pull/1716)) <!-- pr:1716 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add arena-style evaluation to `GEval` by allowing a list of test cases and selecting the best output. Validate that all candidates share the same input and expose `best_test_case` and `best_test_case_index` for easier comparisons. ([#1717](https://github.com/confident-ai/deepeval/pull/1717)) <!-- pr:1717 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v3.1.5

- Add `MultimodalGEval`, a GEval-based metric to score multimodal test cases using configurable criteria, rubrics, and evaluation steps. Supports async evaluation and can incorporate inputs like context, retrieval context, and tool calls. Also improve image encoding by converting non-RGB images before JPEG serialization. ([#1684](https://github.com/confident-ai/deepeval/pull/1684)) <!-- pr:1684 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add OpenAI Agents tracing integration via `DeepEvalTracingProcessor`, capturing agent, tool, and LLM spans and mapping key metadata like prompts, responses, and token usage into the tracing system. ([#1699](https://github.com/confident-ai/deepeval/pull/1699)) <!-- pr:1699 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add broader multimodal test case support in the platform API by sending expected output, context, and retrieval context fields. Improve handling of local image inputs by detecting `file://` paths, capturing filenames and MIME types, and embedding file data as Base64. ([#1704](https://github.com/confident-ai/deepeval/pull/1704)) <!-- pr:1704 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v3.0.8

- Add native LiteLLM model support so you can run evaluations with any LiteLLM-supported provider. Includes sync/async text generation, schema validation, cost tracking, and improved error handling, plus tests and updated docs. ([#1670](https://github.com/confident-ai/deepeval/pull/1670)) <!-- pr:1670 --> ([Prahlad Sahu](https://github.com/ps2program))

#### v3.0.6

- Add support for saving datasets in `jsonl` format, making it easier to write large datasets without loading everything into memory. This is especially useful for generating and exporting datasets with more than 10k rows. ([#1652](https://github.com/confident-ai/deepeval/pull/1652)) <!-- pr:1652 --> ([Yudhiesh Ravindranath](https://github.com/yudhiesh))

### Improvement

#### v3.1.9

- Bump package version metadata for a new release, updating the published version string and release date. ([#1710](https://github.com/confident-ai/deepeval/pull/1710)) <!-- pr:1710 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve the RoleAdherenceMetric documentation by fixing wording, removing a duplicate argument entry, and clarifying how assistant turns are evaluated against `chatbot_role` using prior context. ([#1711](https://github.com/confident-ai/deepeval/pull/1711)) <!-- pr:1711 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add pricing support for `claude-opus-4` and `claude-sonnet-4`. Raise a clear `ValueError` when cost pricing is missing for an unknown Anthropic model, preventing silent fallbacks and `TypeError` crashes. ([#1715](https://github.com/confident-ai/deepeval/pull/1715)) <!-- pr:1715 --> ([Abhishek Ranjan](https://github.com/AbhishekRP2002))
- Add a new blog guide on building and evaluating multi-turn chatbots, covering conversation simulation, metrics for memory and tone, and CI-friendly regression testing. ([#1718](https://github.com/confident-ai/deepeval/pull/1718)) <!-- pr:1718 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.1.5

- Bump the package version metadata for a new release. ([#1676](https://github.com/confident-ai/deepeval/pull/1676)) <!-- pr:1676 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve telemetry for `traceable evaluate()` runs by tracking them as a separate component evaluation feature. This records the correct feature status and updates the last-used feature accordingly. ([#1678](https://github.com/confident-ai/deepeval/pull/1678)) <!-- pr:1678 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add a new blog post covering an evaluation-first approach to building and testing RAG apps, including automated test data generation, retriever/generator metrics, and CI test integration. Add a new blog author profile and related images. ([#1686](https://github.com/confident-ai/deepeval/pull/1686)) <!-- pr:1686 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Add links in the README to translated versions in multiple languages, making it easier for non-English readers to find localized documentation. ([#1687](https://github.com/confident-ai/deepeval/pull/1687)) <!-- pr:1687 --> ([neo](https://github.com/dowithless))
- Improve the RAG evaluation blog guide with updated wording, clearer code examples, and revised diagrams. Rename the article file and slug to better reflect its focus, and simplify CI/CD integration examples for easier copy-paste. ([#1694](https://github.com/confident-ai/deepeval/pull/1694)) <!-- pr:1694 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))

#### v3.0.8

- Prepare a new release by updating the package version metadata and reported `__version__`. ([#1668](https://github.com/confident-ai/deepeval/pull/1668)) <!-- pr:1668 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v3.0.6

- Prepare the 3.0.0 release by updating package version metadata and release date. ([#1631](https://github.com/confident-ai/deepeval/pull/1631)) <!-- pr:1631 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve multimodal metrics docs by fixing the Answer Relevancy example to use `MultimodalAnswerRelevancyMetric`, and by aligning output and bulk-evaluation snippets to print score and reason consistently. ([#1635](https://github.com/confident-ai/deepeval/pull/1635)) <!-- pr:1635 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve the faithfulness verdict prompt wording by fixing grammar and removing threatening language, making instructions clearer and more professional for LLM evaluations. ([#1636](https://github.com/confident-ai/deepeval/pull/1636)) <!-- pr:1636 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve AnswerRelevancy prompt templates to produce valid, parseable JSON more reliably. Clarify when ambiguous fragments count as statements and add clearer examples and end markers to reduce malformed outputs. ([#1642](https://github.com/confident-ai/deepeval/pull/1642)) <!-- pr:1642 --> ([Aaron McClintock](https://github.com/Spectavi))
- Improve conversation simulation progress output by switching to Rich traceable progress bars and showing per-conversation and per-step progress during scenario setup and turn simulation, in both sync and async modes. ([#1649](https://github.com/confident-ai/deepeval/pull/1649)) <!-- pr:1649 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve tracing internals by moving current span/trace state to context variables and reorganizing attribute and type definitions. This makes trace updates more consistent across sync and async execution, and enables centralized OpenAI client patching via the trace manager. ([#1651](https://github.com/confident-ai/deepeval/pull/1651)) <!-- pr:1651 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Bug Fix

#### v3.1.9

- Fix JSON serialization failures when a dictionary contains non-string keys by converting keys to strings during tracing serialization. ([#1712](https://github.com/confident-ai/deepeval/pull/1712)) <!-- pr:1712 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v3.1.5

- Fix import failures on read-only file systems by skipping telemetry-related filesystem setup when `DEEPEVAL_TELEMETRY_OPT_OUT` is set. This prevents evaluations from failing in restricted environments like serverless runtimes. ([#1654](https://github.com/confident-ai/deepeval/pull/1654)) <!-- pr:1654 --> ([Leo Kacenjar](https://github.com/lkacenja))
- Fix OpenAI model initialization to pass `base_url`, enabling proxy or custom endpoint configurations in both sync and async clients. ([#1703](https://github.com/confident-ai/deepeval/pull/1703)) <!-- pr:1703 --> ([jnchen](https://github.com/jnchen))
- Fix `evaluate` so it no longer raises TypeError when a single `TestResult` is passed. The metric pass rate aggregation now wraps non-list results into a list before processing. ([#1705](https://github.com/confident-ai/deepeval/pull/1705)) <!-- pr:1705 --> ([Aditya Bharadwaj](https://github.com/adityabharadwaj198))
- Fix an `IndexError` in `Synthesizer.generate_goldens_from_docs()` by safely handling missing or shorter `source_files`, preventing crashes when generating goldens from documentation inputs. ([#1706](https://github.com/confident-ai/deepeval/pull/1706)) <!-- pr:1706 --> ([Aditya Bharadwaj](https://github.com/adityabharadwaj198))

#### v3.0.6

- Fix GSM8K benchmark crashes when a model returns a tuple or other non-standard response. Prediction extraction now handles `NumberSchema`, tuples, strings, dicts, and `.text`/`.content` objects, and avoids unsafe `.values()` unpacking to prevent `AttributeError`/`TypeError`. ([#1628](https://github.com/confident-ai/deepeval/pull/1628)) <!-- pr:1628 --> ([Muhammad Hussain](https://github.com/SYED-M-HUSSAIN))
- Fix traceable span evaluation traversal so child spans are always processed and recorded, even when a parent span has no metrics or test case. This prevents missing spans in trace output and avoids incomplete evaluations. ([#1632](https://github.com/confident-ai/deepeval/pull/1632)) <!-- pr:1632 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix TruthfulQA evaluation with AnthropicModel by handling JSON parsing failures and falling back to text-based prompting when structured output isn’t supported. This prevents crashes from uncaught errors and improves robustness across models. ([#1638](https://github.com/confident-ai/deepeval/pull/1638)) <!-- pr:1638 --> ([Pradyun Magal](https://github.com/PradyMagal))
- Fix the OpenAI tracing integration so LLM span attributes are applied correctly and tracing data is recorded as expected. ([#1639](https://github.com/confident-ai/deepeval/pull/1639)) <!-- pr:1639 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix async golden generation to call `a_embed_text` instead of the blocking `embed_text` when building contexts. This prevents event-loop blocking, improves parallel performance, and avoids runtime errors like `asyncio.run()` being called from a running loop. ([#1641](https://github.com/confident-ai/deepeval/pull/1641)) <!-- pr:1641 --> ([Andreas Gabrielsson](https://github.com/andreasgabrielsson))
- Fix OTEL exporter crashes when span or event attributes are missing by handling `None` values and returning empty objects or `None` instead of raising type conversion errors. ([#1646](https://github.com/confident-ai/deepeval/pull/1646)) <!-- pr:1646 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix `expected_output` serialization for span test cases by correcting the `expectedOutput` field alias so optional expected outputs are sent and parsed correctly. ([#1650](https://github.com/confident-ai/deepeval/pull/1650)) <!-- pr:1650 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix the traceable evaluation progress bar so it updates correctly during runs, including async execution, by using the proper progress bar ID. ([#1655](https://github.com/confident-ai/deepeval/pull/1655)) <!-- pr:1655 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix trace posting when a Confident AI API key is provided directly, so traces are no longer skipped due to the environment not being detected as Confident. ([#1656](https://github.com/confident-ai/deepeval/pull/1656)) <!-- pr:1656 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix a typo in the conversation simulator docs so the `user_intentions` example is valid Python and can be copied and run without errors. ([#1664](https://github.com/confident-ai/deepeval/pull/1664)) <!-- pr:1664 --> ([Eduardo Arndt](https://github.com/eduardoarndt))
- Fix a circular import in the tracing API by importing `current_trace_context` from the context module, preventing import-time errors when using tracing. ([#1665](https://github.com/confident-ai/deepeval/pull/1665)) <!-- pr:1665 --> ([Mayank](https://github.com/spike-spiegel-21))

## May

May made evaluations and tracing more robust and configurable. LLM wrappers gained configurable `temperature`, new providers including Amazon Bedrock, and PEP 561 support for static analysis. Tracing improved with cleaner defaults, richer metadata, optional sampling/masking, and better OpenTelemetry interoperability while respecting opt-out more consistently.

### Backward Incompatible Change

#### v2.8.5

- Rename the tracing callback parameter from `traceable_callback` to `observed_callback` in `evaluate()` and `assert_test()` when running agentic golden tests, improving naming consistency for traced runs. ([#1561](https://github.com/confident-ai/deepeval/pull/1561)) <!-- pr:1561 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.8.4

- Remove the LangChain dependency so installs are lighter and avoid importing LangChain modules. Update conversational GEval to use OpenAI `ChatCompletion` responses directly when parsing content and logprobs. ([#1544](https://github.com/confident-ai/deepeval/pull/1544)) <!-- pr:1544 --> ([Kritin Vongthongsri](https://github.com/kritinv))

### New Feature

#### v3.0

- Add utility functions to write evaluation logs to a file, making it easier to track results when running large batches without a web app. This also helps spot missing results caused by connection errors. ([#1601](https://github.com/confident-ai/deepeval/pull/1601)) <!-- pr:1601 --> ([Daehui Kim](https://github.com/daehuikim))
- Add an OpenTelemetry span exporter that detects `gen_ai` operations and converts spans into LLM, tool, agent, and retriever traces with inputs, outputs, token usage, and cost metadata for export. ([#1603](https://github.com/confident-ai/deepeval/pull/1603)) <!-- pr:1603 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add optional `thread_id` to traces and support sending it as `threadId` in the tracing API. This lets you associate a trace with a specific conversation thread when updating the current trace. ([#1604](https://github.com/confident-ai/deepeval/pull/1604)) <!-- pr:1604 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for setting a trace `userId` so you can associate traces with a specific end user when updating and exporting trace data. ([#1605](https://github.com/confident-ai/deepeval/pull/1605)) <!-- pr:1605 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `input` and `output` fields to trace data so you can record request payloads and final results at the trace level, including via `update_current_trace`. ([#1606](https://github.com/confident-ai/deepeval/pull/1606)) <!-- pr:1606 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.9.0

- Add support for tracing `LlmAttributes` on the OpenAI client by patching it into `Observer`, so `@observe(type="llm", client=...)` captures LLM call attributes automatically. ([#1560](https://github.com/confident-ai/deepeval/pull/1560)) <!-- pr:1560 --> ([Mayank](https://github.com/spike-spiegel-21))
- Add `AmazonBedrockModel` to run LLM-based evaluations using Amazon Bedrock, with async and sync generation plus optional Pydantic schema parsing. Includes usage docs and recognizes Bedrock models as native for metric execution. ([#1570](https://github.com/confident-ai/deepeval/pull/1570)) <!-- pr:1570 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for setting per-span `metadata` via `update_current_span`, and include it when exporting spans to the tracing API. ([#1575](https://github.com/confident-ai/deepeval/pull/1575)) <!-- pr:1575 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add trace-level tags and metadata, plus an optional environment label for better trace filtering and context. Support masking trace inputs/outputs via a configurable mask function. Allow sampling with `CONFIDENT_SAMPLE_RATE` to skip posting a portion of traces. ([#1578](https://github.com/confident-ai/deepeval/pull/1578)) <!-- pr:1578 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.9.1

- Add a more flexible conversation simulator: generate a configurable number of conversations per intent, accept either `user_profile_items` or predefined `user_profiles`, and optionally stop early using a `stopping_criteria`. Progress tracking now reflects the total conversations generated across intents. ([#1584](https://github.com/confident-ai/deepeval/pull/1584)) <!-- pr:1584 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.8.5

- Add `get_actual_model_name()` helper to extract the underlying model ID from provider-prefixed strings like `openai/gpt-4.1-mini`, as used by proxies such as LiteLLM. This makes it easier to work with provider/model formats consistently. ([#1555](https://github.com/confident-ai/deepeval/pull/1555)) <!-- pr:1555 --> ([Serghei Iakovlev](https://github.com/sergeyklay))

#### v2.8.4

- Add support for `gpt-4.1` in structured output mode by including it in the list of supported models. This lets you use `gpt-4.1` where structured outputs are required without extra configuration. ([#1547](https://github.com/confident-ai/deepeval/pull/1547)) <!-- pr:1547 --> ([Serghei Iakovlev](https://github.com/sergeyklay))

### Improvement

#### v3.0

- Support passing through unknown command-line options from `deepeval test run` to pytest, so third-party and custom pytest plugins can receive their flags without the CLI rejecting them. ([#1589](https://github.com/confident-ai/deepeval/pull/1589)) <!-- pr:1589 --> ([Matt Barr](https://github.com/marr75))
- Improve telemetry and tracing reliability by propagating an internal `_in_component` flag through metric evaluation and wrapping trace flush sends with capture logic, reducing noisy progress output and ensuring in-flight tasks are cleaned up more safely. ([#1596](https://github.com/confident-ai/deepeval/pull/1596)) <!-- pr:1596 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Bump package version to 2.9.1 for the latest release. ([#1600](https://github.com/confident-ai/deepeval/pull/1600)) <!-- pr:1600 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Add support for saving `expected_output` when exporting datasets, so expected results are preserved alongside inputs and other golden fields. ([#1602](https://github.com/confident-ai/deepeval/pull/1602)) <!-- pr:1602 --> ([Nail Khusainov](https://github.com/nkhus))
- Add default trace input/output capture when they are not explicitly set, using the observed function’s kwargs and result. This ensures traces include basic I/O data without requiring manual `update_current_trace` calls. ([#1620](https://github.com/confident-ai/deepeval/pull/1620)) <!-- pr:1620 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Remove the SIGINT/SIGTERM signal handler from tracing so the tool no longer overrides your process signal handling during shutdown. ([#1621](https://github.com/confident-ai/deepeval/pull/1621)) <!-- pr:1621 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve `assert_test` AssertionError messages by including the failure reason in the thrown metrics string. This makes it easier to understand failures when logging exceptions, abstracting tests, or running under pytest. ([#1623](https://github.com/confident-ai/deepeval/pull/1623)) <!-- pr:1623 --> ([Orel Lazri](https://github.com/orellazri))

#### v2.9.0

- Update package metadata and internal version to 2.8.5 for the new release. ([#1567](https://github.com/confident-ai/deepeval/pull/1567)) <!-- pr:1567 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve tracing span updates by consolidating `update_current_span_test_case` and `update_current_span_attributes` into a single `update_current_span` API. This makes it easier to attach both span attributes and an `LLMTestCase`, and updates docs and error messages to match the new call pattern. ([#1574](https://github.com/confident-ai/deepeval/pull/1574)) <!-- pr:1574 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add the PEP 561 `py.typed` marker so type checkers like mypy can analyze installed package imports without reporting missing stubs or `import-untyped` errors. ([#1592](https://github.com/confident-ai/deepeval/pull/1592)) <!-- pr:1592 --> ([Sigurd Spieckermann](https://github.com/sisp))

#### v2.9.1

- Bump the package release to 2.9.0 and update version metadata across the project. ([#1597](https://github.com/confident-ai/deepeval/pull/1597)) <!-- pr:1597 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.8.5

- Update package metadata and internal `__version__` to reflect the latest release. ([#1558](https://github.com/confident-ai/deepeval/pull/1558)) <!-- pr:1558 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Prevent trace status logs from printing during evaluations unless `CONFIDENT_TRACE_VERBOSE` explicitly enables them, reducing noisy console output while running eval traces. ([#1565](https://github.com/confident-ai/deepeval/pull/1565)) <!-- pr:1565 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.8.4

- Improve type safety and simplify golden/context generation by removing legacy `_nodes` paths. Add a ChromaDB availability check and clearer error messages to fail fast when optional dependencies are missing. ([#1534](https://github.com/confident-ai/deepeval/pull/1534)) <!-- pr:1534 --> ([Rami Pellumbi](https://github.com/ramipellumbi))
- Add configurable `temperature` to supported LLM model wrappers (including Anthropic, Azure OpenAI, and Gemini) and pass it through on generation calls. Prevent invalid settings by rejecting negative temperatures with a clear error. ([#1541](https://github.com/confident-ai/deepeval/pull/1541)) <!-- pr:1541 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve type hints in the MMLU benchmark by making `tasks` optional and simplifying prompt variable typing for better static analysis and editor support. ([#1550](https://github.com/confident-ai/deepeval/pull/1550)) <!-- pr:1550 --> ([Serghei Iakovlev](https://github.com/sergeyklay))
- Fix typos across benchmark prompts, comments, and tests to improve wording clarity and reduce confusion when reading task names and evaluation steps. ([#1552](https://github.com/confident-ai/deepeval/pull/1552)) <!-- pr:1552 --> ([João Matias](https://github.com/joaopmatias))
- Move telemetry, cache, temp test-run data, and key storage into a `.deepeval/` folder to reduce clutter in the project root. Automatically migrates legacy files to the new location when found. ([#1556](https://github.com/confident-ai/deepeval/pull/1556)) <!-- pr:1556 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve tracing logs with clearer success/failure messages, a queue-size status, and an exit warning when traces are still pending. Add optional flushing on shutdown via `CONFIDENT_TRACE_FLUSH`, and control log verbosity with `CONFIDENT_TRACE_VERBOSE`. ([#1557](https://github.com/confident-ai/deepeval/pull/1557)) <!-- pr:1557 --> ([Kritin Vongthongsri](https://github.com/kritinv))

### Bug Fix

#### v3.0

- Fix `TaskNodeOutput` response format types so list and dict outputs are fully specified and accepted by OpenAI. This prevents confusing bad request errors that only appeared when the model tried to emit those previously invalid shapes. ([#1599](https://github.com/confident-ai/deepeval/pull/1599)) <!-- pr:1599 --> ([Matt Barr](https://github.com/marr75))
- Restrict `typer` and `click` dependency versions to improve compatibility and prevent install issues with newer releases. ([#1607](https://github.com/confident-ai/deepeval/pull/1607)) <!-- pr:1607 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix `ToolCorrectnessMetric` input parameter comparison so identical dictionaries are treated as a full match, improving scoring consistency when tool inputs are the same. ([#1608](https://github.com/confident-ai/deepeval/pull/1608)) <!-- pr:1608 --> ([Nathan-Kr](https://github.com/Nathan-Kr))
- Fix temp directory cleanup on Windows by adding a safer `rmtree` with retries and forced garbage collection to reduce failures from locked files. Also register an exit cleanup hook to help release resources before deletion. ([#1609](https://github.com/confident-ai/deepeval/pull/1609)) <!-- pr:1609 --> ([Propet40](https://github.com/Propet40))
- Fix telemetry opt-out so no analytics events or traces are captured when opt-out is enabled across evaluation, metrics, dataset pulls, and trace sending. ([#1614](https://github.com/confident-ai/deepeval/pull/1614)) <!-- pr:1614 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix a `ZeroDivisionError` when running the HellaSwag benchmark with no predictions for a task by returning an accuracy of 0 instead of dividing by zero. ([#1616](https://github.com/confident-ai/deepeval/pull/1616)) <!-- pr:1616 --> ([Mikhail Salnikov](https://github.com/mdsalnikov))
- Fix a ValueError when running the TruthfulQA benchmark by including the expected output in each recorded prediction row, keeping result data aligned during evaluation. ([#1619](https://github.com/confident-ai/deepeval/pull/1619)) <!-- pr:1619 --> ([Mikhail Salnikov](https://github.com/mdsalnikov))
- Fix `ToolCall.__hash__` to support unhashable input/output values like lists and nested dicts. Hashing now converts complex nested structures into stable hashable forms, preventing `TypeError` during comparisons and test runs. ([#1625](https://github.com/confident-ai/deepeval/pull/1625)) <!-- pr:1625 --> ([Muhammad Hussain](https://github.com/SYED-M-HUSSAIN))
- Fix a `FileNotFoundError` in telemetry by using a consistent temp run data filename when moving it into the `.deepeval` directory. This prevents failures caused by a mismatch between dotted and non-dotted filenames. ([#1630](https://github.com/confident-ai/deepeval/pull/1630)) <!-- pr:1630 --> ([Jakub Koněrza](https://github.com/konerzajakub))

#### v2.9.0

- Fix Azure OpenAI initialization to use the correct `deployment_name` when setting `azure_deployment`, preventing misconfigured clients and failed requests. ([#1571](https://github.com/confident-ai/deepeval/pull/1571)) <!-- pr:1571 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix Amazon Bedrock model imports to avoid unnecessary dependencies being loaded when using the Bedrock LLM integration. ([#1573](https://github.com/confident-ai/deepeval/pull/1573)) <!-- pr:1573 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix a typo in the MMLU benchmark that could cause an assertion failure when validating the example dataset, so `load_benchmark` and prediction run as expected. ([#1580](https://github.com/confident-ai/deepeval/pull/1580)) <!-- pr:1580 --> ([Tri Dao](https://github.com/dmtri35))
- Fix broken integration documentation links for LlamaIndex and Hugging Face so the README points to the correct pages. ([#1582](https://github.com/confident-ai/deepeval/pull/1582)) <!-- pr:1582 --> ([Wey Gu](https://github.com/wey-gu))
- Fix client patching during tracing context setup by skipping type checks when the client is `None`, preventing errors when no client is configured. ([#1585](https://github.com/confident-ai/deepeval/pull/1585)) <!-- pr:1585 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix a syntax error in the synthesizer generate-from-scratch documentation example by adding a missing trailing comma in `StylingConfig`, making the snippet copy-pasteable. ([#1587](https://github.com/confident-ai/deepeval/pull/1587)) <!-- pr:1587 --> ([Shun Liang](https://github.com/shun-liang))
- Fix `OllamaModel.a_generate()` to use the model name set in the constructor. This keeps async generation consistent with `OllamaModel.generate()` and prevents using the wrong Ollama model. ([#1594](https://github.com/confident-ai/deepeval/pull/1594)) <!-- pr:1594 --> ([Sigurd Spieckermann](https://github.com/sisp))

#### v2.8.5

- Fix trace queue handling so queued and in-flight traces are posted more reliably on exit or interruption. Add SIGINT/SIGTERM handling and improve warnings to report remaining traces and support optional flushing via `CONFIDENT_TRACE_FLUSH`. ([#1559](https://github.com/confident-ai/deepeval/pull/1559)) <!-- pr:1559 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix the exit warning to only appear when there are pending traces to post. This prevents misleading warnings when the trace queue and in-flight tasks are empty. ([#1566](https://github.com/confident-ai/deepeval/pull/1566)) <!-- pr:1566 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.8.4

- Fix MMLU evaluation when `model.generate()` returns a tuple or list by extracting the first result before reading `.answer`. This prevents `AttributeError`/`TypeError` and improves compatibility across different model implementations. ([#1546](https://github.com/confident-ai/deepeval/pull/1546)) <!-- pr:1546 --> ([krishna0125](https://github.com/krishna0125))

## April

April made evaluations more traceable and easier to configure. Native model support expanded with Gemini and Anthropic, plus improved Azure OpenAI and Ollama setup. New metadata fields (`token_cost`, `completion_time`, `additional_metadata`) and tracing upgrades made multi-turn test generation and debugging smoother, while robustness fixes reduced import failures and crashes.

### Backward Incompatible Change

#### v2.7.6

- Remove async from `get_model_name` on the base embedding model interface, making model name retrieval a synchronous call for simpler implementations and call sites. ([#1516](https://github.com/confident-ai/deepeval/pull/1516)) <!-- pr:1516 --> ([Rami Pellumbi](https://github.com/ramipellumbi))

#### v2.7.3

- Remove the `auto_evaluate` helper from the public API to streamline the tracing-focused surface area and reduce unused functionality. ([#1513](https://github.com/confident-ai/deepeval/pull/1513)) <!-- pr:1513 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### New Feature

#### v2.7.7

- Add traceable eval runs so agent/tool/LLM steps can be captured and attached to each test case during evaluation. This improves debugging and makes it easier to understand how outputs were produced, including when running evals over pulled datasets. ([#1523](https://github.com/confident-ai/deepeval/pull/1523)) <!-- pr:1523 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for named goldens and allow `assert_test` to run traceable evals using a Golden plus callback, in both sync and async modes. Improve input validation for `assert_test` to prevent invalid argument combinations. ([#1532](https://github.com/confident-ai/deepeval/pull/1532)) <!-- pr:1532 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.7.6

- Add `min_context_length` and `min_contexts_per_document` to Synthesizer document context generation, so you can enforce a minimum context size and minimum number of contexts per document while still capping with the existing max settings. ([#1508](https://github.com/confident-ai/deepeval/pull/1508)) <!-- pr:1508 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.7.3

- Add `generate_goldens_from_goldens` to expand an existing set of Goldens into new ones, reusing available contexts for grounded generation or falling back to scratch generation when context is missing. Optionally generates expected outputs and can infer prompt styling from the provided examples. ([#1506](https://github.com/confident-ai/deepeval/pull/1506)) <!-- pr:1506 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.6.8

- Add native Gemini model support, including multimodal judging and structured outputs. Configure it via `set-gemini` using either a Google API key or Vertex AI project/location, and disable it with `unset-gemini` to revert to the default provider. ([#1493](https://github.com/confident-ai/deepeval/pull/1493)) <!-- pr:1493 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for running evaluations with Anthropic Claude models via a new `AnthropicModel`, including sync/async generation and token cost tracking. ([#1495](https://github.com/confident-ai/deepeval/pull/1495)) <!-- pr:1495 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.6.6

- Add a conversation simulator that generates multi-turn conversational test cases from user profile items and intentions, with optional opening messages. Supports async concurrency and tracks simulation cost when using native models. ([#1481](https://github.com/confident-ai/deepeval/pull/1481)) <!-- pr:1481 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Improvement

#### v2.7.7

- Prepare a new release by updating the package version metadata. ([#1525](https://github.com/confident-ai/deepeval/pull/1525)) <!-- pr:1525 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Allow LLM and retriever spans to be recorded without calling `update_current_span_attributes`. Missing attributes no longer raise errors, and span conversion skips optional fields when they aren’t provided. Improve error handling for non-JSON API responses. ([#1530](https://github.com/confident-ai/deepeval/pull/1530)) <!-- pr:1530 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve how `LLMTestCase` is converted to a string for g-eval prompts by centralizing the formatting and ensuring tool-call values are rendered consistently via `repr()`. ([#1531](https://github.com/confident-ai/deepeval/pull/1531)) <!-- pr:1531 --> ([João Matias](https://github.com/joaopmatias))

#### v2.7.9

- Improve documentation by clarifying CLI usage (`deepeval test run`), updating command examples to `bash`, and fixing links to the correct evaluation guide sections. ([#1537](https://github.com/confident-ai/deepeval/pull/1537)) <!-- pr:1537 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Prepare a new package release by bumping the project version metadata. ([#1539](https://github.com/confident-ai/deepeval/pull/1539)) <!-- pr:1539 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Bump the package version to 2.7.8 for the latest release metadata. ([#1540](https://github.com/confident-ai/deepeval/pull/1540)) <!-- pr:1540 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.7.6

- Add a new documentation article showcasing popular G-Eval metric examples, with sample code and guidance for defining custom LLM-judge criteria and RAG-focused evaluations. ([#1517](https://github.com/confident-ai/deepeval/pull/1517)) <!-- pr:1517 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve the G-Eval documentation with research context, clearer RAG evaluation criteria, and a new advanced section explaining limitations and when to use DAG-based metrics, including an end-to-end example. ([#1519](https://github.com/confident-ai/deepeval/pull/1519)) <!-- pr:1519 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix typos and improve wording in synthesizer prompt templates to make instructions clearer and reduce confusion in generated outputs. ([#1521](https://github.com/confident-ai/deepeval/pull/1521)) <!-- pr:1521 --> ([Song Luar](https://github.com/luarss))
- Improve import-time dependency resolution by deferring optional integration imports, reducing startup failures when LangChain or LlamaIndex aren’t installed. Change update checks to be opt-in via `DEEPEVAL_UPDATE_WARNING_OPT_IN`. ([#1524](https://github.com/confident-ai/deepeval/pull/1524)) <!-- pr:1524 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.7.3

- Fix a typo in the QA agent metrics tutorial by correcting “weather” to “whether” in the `Faithfulness` description, improving documentation clarity. ([#1505](https://github.com/confident-ai/deepeval/pull/1505)) <!-- pr:1505 --> ([Justin Nauman](https://github.com/jrnt30))
- Fix typos in the benchmarks introduction docs to use the correct `prompts` variable name and improve wording for clarity. ([#1511](https://github.com/confident-ai/deepeval/pull/1511)) <!-- pr:1511 --> ([Russell-Day](https://github.com/Russell-Day))

#### v2.6.8

- Add retention analytics by sending PostHog events for evaluation runs and synthesizer invocations when telemetry is enabled, improving visibility into feature usage over time. ([#1486](https://github.com/confident-ai/deepeval/pull/1486)) <!-- pr:1486 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add log-probability support for Azure OpenAI in `GEval`, including Azure models in log-probability compatibility checks and enabling raw response generation with cost tracking via the LangChain client. ([#1492](https://github.com/confident-ai/deepeval/pull/1492)) <!-- pr:1492 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `google-genai` and `posthog` as dependencies and refresh the lockfile to pull in required transitive packages. ([#1499](https://github.com/confident-ai/deepeval/pull/1499)) <!-- pr:1499 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.6.6

- Add a new comparison blog post and author profile to the documentation, expanding the site’s blog content and attribution. ([#1471](https://github.com/confident-ai/deepeval/pull/1471)) <!-- pr:1471 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve Ollama embedding configuration by using the same underlying `ollama` module as the chat model. This aligns `base_url` handling so embeddings and chat can share the same Ollama host without requiring different `/v1` URL variants, reducing setup confusion. ([#1474](https://github.com/confident-ai/deepeval/pull/1474)) <!-- pr:1474 --> ([Paul Lewis](https://github.com/paul91))
- Add a new documentation blog post comparing the tool with Langfuse, and update existing comparison content for clearer messaging about provider integration and metric support. ([#1475](https://github.com/confident-ai/deepeval/pull/1475)) <!-- pr:1475 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `token_cost` and `completion_time` fields to LLM and multimodal test cases, and include them in the API test case payload as `tokenCost` and `completionTime`. ([#1476](https://github.com/confident-ai/deepeval/pull/1476)) <!-- pr:1476 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `additional_metadata` to test results so extra per-test details are preserved and returned for conversational, multimodal, and standard evaluations. ([#1477](https://github.com/confident-ai/deepeval/pull/1477)) <!-- pr:1477 --> ([Mayank](https://github.com/spike-spiegel-21))
- Improve the conversation simulator API by moving `model_callback`, turn limits, and conversation count into `simulate()` and adding clearer progress reporting during generation for both sync and async runs. ([#1491](https://github.com/confident-ai/deepeval/pull/1491)) <!-- pr:1491 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Bug Fix

#### v2.7.7

- Fix invalid enum errors in tracing by aligning span status values to use `ERRORED` instead of `ERROR`, so failed spans serialize and report correctly. ([#1536](https://github.com/confident-ai/deepeval/pull/1536)) <!-- pr:1536 --> ([Mayank](https://github.com/spike-spiegel-21))
- Fix agentic `assert_test` runs so they no longer always disable saving results. Test runs now respect the `save_to_disk` setting and correctly reuse or create the current test run by identifier. ([#1538](https://github.com/confident-ai/deepeval/pull/1538)) <!-- pr:1538 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.7.6

- Fix `FiltrationConfig.synthetic_input_quality_threshold` to use a `float` instead of an `int`, matching its default value and preventing type-related configuration errors. ([#1515](https://github.com/confident-ai/deepeval/pull/1515)) <!-- pr:1515 --> ([Rami Pellumbi](https://github.com/ramipellumbi))
- Fix the Bias metric docs example to import `evaluate` from `deepeval`, so the sample code runs as written. ([#1520](https://github.com/confident-ai/deepeval/pull/1520)) <!-- pr:1520 --> ([snsk](https://github.com/snsk))

#### v2.7.3

- Fix Gemini model wrappers to stop hardcoding an allowlist of model names. You can now pass newer or custom Gemini model IDs without getting an unnecessary "Invalid model" error. ([#1503](https://github.com/confident-ai/deepeval/pull/1503)) <!-- pr:1503 --> ([Mete Atamel](https://github.com/meteatamel))
- Fix Anthropic model initialization and async generation by treating `AnthropicModel` as a native provider and loading the client in async mode, preventing failures when calling `a_generate`. ([#1504](https://github.com/confident-ai/deepeval/pull/1504)) <!-- pr:1504 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.6.8

- Fix synthetic dataset generation from documents failing with `UnicodeDecodeError` on non-UTF-8 text. Default to auto-detecting file encoding instead of Windows defaults, and allow manually setting an encoding for edge cases. ([#1485](https://github.com/confident-ai/deepeval/pull/1485)) <!-- pr:1485 --> ([Aahil Shaikh](https://github.com/AahilShaikh))
- Fix type hints for `context_quality_threshold` and `context_similarity_threshold` to use `float`, matching their default values and preventing misleading type checking. ([#1490](https://github.com/confident-ai/deepeval/pull/1490)) <!-- pr:1490 --> ([Jakub Koněrza](https://github.com/konerzajakub))

#### v2.6.6

- Fix Azure OpenAI setup by separating `openai_model_name` from the deployment name and using the deployment name when creating the client. The CLI now prompts for `--openai-model-name` and stores/clears it alongside other Azure settings. ([#1480](https://github.com/confident-ai/deepeval/pull/1480)) <!-- pr:1480 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix the QA agent evaluation tutorial to import `EvaluationDataset` from `deepeval.dataset`, matching the current package structure and preventing import errors when following the docs. ([#1483](https://github.com/confident-ai/deepeval/pull/1483)) <!-- pr:1483 --> ([Anton](https://github.com/tonton-golio))
- Fix ToolCorrectness metric crashing with an unhashable type error when a tool call output is a list and expected tools are provided without a guaranteed order. This lets tool-correctness evaluation run reliably for list outputs. ([#1487](https://github.com/confident-ai/deepeval/pull/1487)) <!-- pr:1487 --> ([Sai Pavan Kumar](https://github.com/pavan555))

## March

March made evaluations and synthesis more reliable. Defaults improved for Ollama and Azure OpenAI, broader model support landed (including `gpt-4.5-preview`), and structured outputs became more consistent. Large runs gained resilience with expanded retry handling for transient failures, plus fixes for async scoring, G-Eval strict mode, and benchmark parsing.

### New Feature

#### v2.6.5

- Add support for the `gpt-4.5-preview-2025-02-27` model, including pricing metadata and compatibility flags for features like structured outputs and JSON mode. ([#1453](https://github.com/confident-ai/deepeval/pull/1453)) <!-- pr:1453 --> ([John Lemmon](https://github.com/john-lemmon-lime))
- Add `file_name` and `quiet` options to `Synthesizer.save_as()` so you can control the output filename and suppress console output. Improve validation for file types and synthetic goldens, with updated docs and tests. ([#1455](https://github.com/confident-ai/deepeval/pull/1455)) <!-- pr:1455 --> ([Serghei Iakovlev](https://github.com/sergeyklay))

#### v2.5.9

- Support additional native model providers when initializing metrics and evaluators, including Azure OpenAI, Ollama, and local models. Model selection can now be driven by configuration without changing code. ([#1441](https://github.com/confident-ai/deepeval/pull/1441)) <!-- pr:1441 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.5.8

- Add optional `cost_tracking` to Synthesizer to enable full API cost tracking, disabled by default. When enabled, generation runs report detailed cost information alongside the output. ([#1406](https://github.com/confident-ai/deepeval/pull/1406)) <!-- pr:1406 --> ([Chuqing Gao](https://github.com/chuqingG))

### Improvement

#### v2.6.5

- Update package metadata for a new release, including the published version and release date. ([#1446](https://github.com/confident-ai/deepeval/pull/1446)) <!-- pr:1446 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve resilience of large runs by retrying on additional OpenAI connection-related exceptions, not just rate limits. This reduces failures from transient network issues during long parallel evaluations. ([#1450](https://github.com/confident-ai/deepeval/pull/1450)) <!-- pr:1450 --> ([John Lemmon](https://github.com/john-lemmon-lime))
- Improve reliability of uploads to Confident AI by adding retries on transient HTTPS/SSL failures, especially for large batch test runs, so evaluations are more likely to complete successfully. ([#1452](https://github.com/confident-ai/deepeval/pull/1452)) <!-- pr:1452 --> ([John Lemmon](https://github.com/john-lemmon-lime))

#### v2.5.9

- Update package metadata to the latest release version for more accurate reporting in builds and tooling. ([#1445](https://github.com/confident-ai/deepeval/pull/1445)) <!-- pr:1445 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.5.8

- Bump package metadata to the latest release version. ([#1399](https://github.com/confident-ai/deepeval/pull/1399)) <!-- pr:1399 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve Ollama model configuration by defaulting the base URL to `http://localhost:11434` and removing the response format option from `set-ollama`. This reduces mismatches with Ollama endpoints and keeps CLI setup focused on LLM configuration. ([#1401](https://github.com/confident-ai/deepeval/pull/1401)) <!-- pr:1401 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve documentation for JSON correctness metrics by showing how to validate `actual_output` that is a list of JSON objects using a Pydantic `RootModel` list schema. ([#1403](https://github.com/confident-ai/deepeval/pull/1403)) <!-- pr:1403 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Update the Task Completion metric docs to use `gpt-4o` instead of `gpt-4` in the example configuration. ([#1415](https://github.com/confident-ai/deepeval/pull/1415)) <!-- pr:1415 --> ([Obada Khalili](https://github.com/obadakhalili))
- Fix a typo in the RAG evaluation guide example input, changing “gow” to “how” for clearer documentation. ([#1431](https://github.com/confident-ai/deepeval/pull/1431)) <!-- pr:1431 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve `prettify_list()` JSON formatting by enabling `ensure_ascii`, making output consistently ASCII-escaped for non-ASCII characters and easier to paste into logs and terminals. ([#1437](https://github.com/confident-ai/deepeval/pull/1437)) <!-- pr:1437 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Improve benchmark imports by loading `datasets` only when needed, reducing import-time failures for users who don’t use those benchmarks. Update packaging metadata to broaden the supported Python range and remove the legacy `setup.py`. ([#1440](https://github.com/confident-ai/deepeval/pull/1440)) <!-- pr:1440 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Bug Fix

#### v2.6.5

- Fix infinite verbose output in notebooks by only constructing verbose logs when verbose mode is enabled, and by writing logs via `sys.stdout` with an explicit flush. ([#1444](https://github.com/confident-ai/deepeval/pull/1444)) <!-- pr:1444 --> ([fetz236](https://github.com/fetz236))
- Fix a typo in the tracing example prompt so the sample question reads correctly when you run the demo. ([#1448](https://github.com/confident-ai/deepeval/pull/1448)) <!-- pr:1448 --> ([Mert Doğruca](https://github.com/meroo36))
- Fix Azure OpenAI initialization to always use the configured deployment name from settings, ensuring the correct `azure_deployment` is passed to sync and async clients. Improve the docs for `set-azure-openai` with clearer endpoint examples and a minimum required API version note. ([#1451](https://github.com/confident-ai/deepeval/pull/1451)) <!-- pr:1451 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix incorrect metadata propagation in conversational test cases so each turn keeps its own `additional_metadata` and `comments` instead of inheriting the parent test case values. ([#1456](https://github.com/confident-ai/deepeval/pull/1456)) <!-- pr:1456 --> ([Xiaopei](https://github.com/xiaopeiwu))
- Fix synthesizer compatibility with Azure OpenAI by handling `generate()` responses that return plain strings or `(result, cost)` tuples, preventing tuple attribute errors when extracting synthetic data. ([#1459](https://github.com/confident-ai/deepeval/pull/1459)) <!-- pr:1459 --> ([Nicolas Torres](https://github.com/ntgussoni))
- Fix `set-ollama --base-url` so Ollama requests use the configured base URL from `.deepeval` instead of falling back to the default localhost setting. ([#1460](https://github.com/confident-ai/deepeval/pull/1460)) <!-- pr:1460 --> ([Paul Lewis](https://github.com/paul91))
- Fix native model handling in the synthesizer and multimodal metrics by using structured outputs when a schema is provided, returning typed results instead of parsing JSON strings. Add CLI commands to set and unset Ollama embeddings, and use the configured embedding initializer instead of a hardcoded OpenAI embedder. ([#1461](https://github.com/confident-ai/deepeval/pull/1461)) <!-- pr:1461 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix the red-teaming guide example so the `chat.completions.create` call uses the correct `messages` argument and returns the message content, making the snippet runnable as written. ([#1463](https://github.com/confident-ai/deepeval/pull/1463)) <!-- pr:1463 --> ([Karthick Nagarajan](https://github.com/karthick965938))
- Fix async `measure` to return `self.score` when `async_mode=True`, instead of returning `None`. Async and sync metric execution now produce a consistent, non-empty score value. ([#1464](https://github.com/confident-ai/deepeval/pull/1464)) <!-- pr:1464 --> ([Roman Makeev](https://github.com/RomaanMkv))

#### v2.5.8

- Fix Ragas metrics failing with an “async_mode is missing” error by explicitly running metric tracking in non-async mode during evaluation. ([#1402](https://github.com/confident-ai/deepeval/pull/1402)) <!-- pr:1402 --> ([Tanay Agrawal](https://github.com/tanayag))
- Fix the import path for `LLMTestCaseParams` in the metrics selection tutorial so the example code runs without import errors. ([#1407](https://github.com/confident-ai/deepeval/pull/1407)) <!-- pr:1407 --> ([Obada Khalili](https://github.com/obadakhalili))
- Fix a typo in the synthetic input generation template to clarify instructions about avoiding repetitive `input`. ([#1408](https://github.com/confident-ai/deepeval/pull/1408)) <!-- pr:1408 --> ([John D. McDonald](https://github.com/Rasputin2))
- Fix tool correctness reason messages so the `expected` and `called` tool names are reported in the right order when using exact match checks. ([#1409](https://github.com/confident-ai/deepeval/pull/1409)) <!-- pr:1409 --> ([Casey Lewiston](https://github.com/shredinger137))
- Fix the dataset synthesis tutorial to use the correct `StylingConfig` keyword argument, replacing `expected_output` with `expected_output_format` so the example code runs as intended. ([#1411](https://github.com/confident-ai/deepeval/pull/1411)) <!-- pr:1411 --> ([Obada Khalili](https://github.com/obadakhalili))
- Fix a typo in `__all__` by restoring a missing comma so `auto_evaluate` and `assert_test` are exported correctly from the package. ([#1412](https://github.com/confident-ai/deepeval/pull/1412)) <!-- pr:1412 --> ([88roy88](https://github.com/88roy88))
- Fix benchmark prediction generation to fall back more reliably by also handling `AttributeError` when extracting the model answer. ([#1414](https://github.com/confident-ai/deepeval/pull/1414)) <!-- pr:1414 --> ([Stan Kirdey](https://github.com/skirdey-inflection))
- Fix G-Eval strict mode to use a dedicated prompt and return a binary score (0/1) with an explicit reason, instead of scaling scores and post-adjusting them against the threshold. ([#1416](https://github.com/confident-ai/deepeval/pull/1416)) <!-- pr:1416 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix SQuAD benchmark answer parsing by using `StringSchema` for enforced model generation instead of a multiple-choice schema, improving compatibility with model outputs. ([#1423](https://github.com/confident-ai/deepeval/pull/1423)) <!-- pr:1423 --> ([Diogo Carvalho](https://github.com/carvalho28))
- Fix the documented Azure OpenAI embedding setup command by correcting the flag name to `--embedding-deployment-name`, so the example works as shown. ([#1424](https://github.com/confident-ai/deepeval/pull/1424)) <!-- pr:1424 --> ([Amali Matharaarachchi](https://github.com/AmaliMatharaarachchi))
- Prevent G-Eval from requesting log probabilities on unsupported GPT models (such as `o1` and `o3-mini`). This avoids errors when generating raw responses and lets evaluations run normally by falling back when logprobs aren’t available. ([#1425](https://github.com/confident-ai/deepeval/pull/1425)) <!-- pr:1425 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix `login_with_confident_api_key()` to reject missing API keys by raising a clear `ValueError`, preventing confusing behavior when the key is empty or not provided. ([#1427](https://github.com/confident-ai/deepeval/pull/1427)) <!-- pr:1427 --> ([Vamshi Adimalla](https://github.com/A-Vamshi))
- Fix the LLM monitoring docs example to use the correct variable name for the monitored response, so the async `a_monitor` call matches the returned output. ([#1432](https://github.com/confident-ai/deepeval/pull/1432)) <!-- pr:1432 --> ([Lucas Le Ray](https://github.com/LucasLeRay))
- Fix document-based golden generation to rebuild the vector index each run instead of reusing cached state, avoiding stale chunks in repeated notebook executions. Add validation to prevent `chunk_overlap` from exceeding `chunk_size - 1`, and relax the `chromadb` install requirement to any compatible version. ([#1433](https://github.com/confident-ai/deepeval/pull/1433)) <!-- pr:1433 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix the DAG non-binary verdict prompt to require a consistent JSON response with `verdict` and `reason`, including an example format. This reduces malformed outputs and makes results easier to parse reliably. ([#1434](https://github.com/confident-ai/deepeval/pull/1434)) <!-- pr:1434 --> ([Hani Cierlak](https://github.com/shrimpnoodles))
- Fix synthesizer chunking with ChromaDB by handling missing collections more robustly, avoiding failures when the collection error type differs across versions. ([#1442](https://github.com/confident-ai/deepeval/pull/1442)) <!-- pr:1442 --> ([Kritin Vongthongsri](https://github.com/kritinv))

## February

February improved evaluation reliability and expanded customization. Fixes landed for batching detection, async `auto_evaluate`, custom LLM validation, and concurrent evaluation stability. Metrics gained injectable templates including `FaithfulnessTemplate`, improved DAG reasoning with `include_reason`, and `MultimodalToolCorrectnessMetric`, plus conversational metadata and `Prompt` hyperparameters.

### New Feature

#### v2.4.6

- Add `MultimodalToolCorrectnessMetric` to score whether an MLLM called the expected tools correctly. Evaluation can check tool name, input parameters, and outputs, with optional exact-match and ordering rules. Results now include expected and called tool data in API test cases. ([#1386](https://github.com/confident-ai/deepeval/pull/1386)) <!-- pr:1386 --> ([Umut Hope YILDIRIM](https://github.com/umuthopeyildirim))
- Support passing `Prompt` objects as hyperparameters in test runs and monitoring, preserving prompt version metadata when available. Improve prompt pulling and validation so prompts can be created from an alias or a manually provided template. ([#1387](https://github.com/confident-ai/deepeval/pull/1387)) <!-- pr:1387 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.3.9

- Add `deepeval recommend metrics`, an interactive CLI flow that asks a few yes/no questions and returns recommended evaluation metrics for your use case. ([#1342](https://github.com/confident-ai/deepeval/pull/1342)) <!-- pr:1342 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for passing `additional_metadata` on conversational test cases, and include it in the generated API payload as `additionalMetadata`. This preserves extra context when creating and evaluating test runs. ([#1352](https://github.com/confident-ai/deepeval/pull/1352)) <!-- pr:1352 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add CLI support for running LLM-based evaluations with local Ollama models via `set-ollama` and `unset-ollama`, including configurable base URL and response format. Documentation was updated with setup and usage guidance. ([#1360](https://github.com/confident-ai/deepeval/pull/1360)) <!-- pr:1360 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for injecting a custom `FaithfulnessTemplate` into `FaithfulnessMetric` for dynamic prompt generation. This lets you plug in domain-specific or few-shot templates without overriding claim generation methods. ([#1367](https://github.com/confident-ai/deepeval/pull/1367)) <!-- pr:1367 --> ([Lei WANG](https://github.com/realei))

#### v2.3.1

- Add support for the `o3-mini` and `o3-mini-2025-01-31` models, including pricing metadata and enabling use in structured outputs and JSON mode where supported. ([#1331](https://github.com/confident-ai/deepeval/pull/1331)) <!-- pr:1331 --> ([Song Luar](https://github.com/luarss))

### Improvement

#### v2.4.7

- Update package metadata and internal `__version__` to match the latest release. ([#1392](https://github.com/confident-ai/deepeval/pull/1392)) <!-- pr:1392 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Add support for injecting custom evaluation templates into metrics, making it easier to customize the prompts used to generate statements, verdicts, and reasons. ([#1393](https://github.com/confident-ai/deepeval/pull/1393)) <!-- pr:1393 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix a typo in the getting started guide so the `GEval` description correctly refers to evaluating outputs on any custom metric. ([#1394](https://github.com/confident-ai/deepeval/pull/1394)) <!-- pr:1394 --> ([Christian Bernhard](https://github.com/ChristianBernhard))
- Fix a typo in the getting started guide to improve clarity when describing `GEval` and recommending `DAGMetric` for deterministic scoring. ([#1395](https://github.com/confident-ai/deepeval/pull/1395)) <!-- pr:1395 --> ([Christian Bernhard](https://github.com/ChristianBernhard))
- Fix a typo in the getting-started guide by correcting “somewhre” to “somewhere” for clearer documentation. ([#1396](https://github.com/confident-ai/deepeval/pull/1396)) <!-- pr:1396 --> ([Christian Bernhard](https://github.com/ChristianBernhard))

#### v2.4.6

- Improve dependency compatibility by relaxing the `grpcio` pin to allow newer 1.x releases while staying below 2.0. This reduces install and resolver conflicts across environments. ([#1383](https://github.com/confident-ai/deepeval/pull/1383)) <!-- pr:1383 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Bump the package release metadata to 2.4.3 so the published version and citation information reflect the latest release. ([#1385](https://github.com/confident-ai/deepeval/pull/1385)) <!-- pr:1385 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Update package metadata and internal version to 2.4.4 for the new release. ([#1388](https://github.com/confident-ai/deepeval/pull/1388)) <!-- pr:1388 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve metric parameter validation by moving each metric’s required test-case fields into the metric class, ensuring consistent checks in both sync and async evaluation. ([#1389](https://github.com/confident-ai/deepeval/pull/1389)) <!-- pr:1389 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.4.3

- Add telemetry for dataset pulls, capturing login method, environment, and basic user identifiers to help monitor usage and diagnose issues. ([#1377](https://github.com/confident-ai/deepeval/pull/1377)) <!-- pr:1377 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.3.9

- Update package metadata for a new release, including the version and release date. ([#1334](https://github.com/confident-ai/deepeval/pull/1334)) <!-- pr:1334 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve CLI login by opening a paired browser flow and recording the login provider for telemetry. Evaluation and run events now include a `logged_in_with` attribute to help diagnose usage patterns. ([#1341](https://github.com/confident-ai/deepeval/pull/1341)) <!-- pr:1341 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix typos and small wording issues in the contextual precision and contextual recall metric templates to make the generated prompts clearer and more consistent. ([#1344](https://github.com/confident-ai/deepeval/pull/1344)) <!-- pr:1344 --> ([Filippo Paganelli](https://github.com/FilippoPaganelli))
- Add telemetry for the `recommend metrics` CLI flow to capture usage context when telemetry is enabled. Mark runs as incomplete when the command errors out. ([#1346](https://github.com/confident-ai/deepeval/pull/1346)) <!-- pr:1346 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `include_reason` support to DAG-based metrics and generate clearer, path-based reasons from the DAG traversal. Improve verbose output by recording per-node execution steps, and normalize static node scores to a 0–1 range. ([#1348](https://github.com/confident-ai/deepeval/pull/1348)) <!-- pr:1348 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve documentation navigation and onboarding by reorganizing the Guides sidebar and adding an early `deepeval login` step in the tutorial introduction to help users set up their API key before starting. ([#1353](https://github.com/confident-ai/deepeval/pull/1353)) <!-- pr:1353 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add documentation for integrating Elasticsearch as a vector database, including setup steps and examples for evaluating and tuning retrieval with contextual metrics. ([#1354](https://github.com/confident-ai/deepeval/pull/1354)) <!-- pr:1354 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve Elasticsearch integration documentation with clearer setup steps and an expanded walkthrough for preparing `LLMTestCase`s and running contextual retrieval metrics to evaluate and tune retriever performance. ([#1355](https://github.com/confident-ai/deepeval/pull/1355)) <!-- pr:1355 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add integration docs for Chroma, including setup and examples for evaluating retrieval quality with contextual metrics and tuning retriever hyperparameters. ([#1357](https://github.com/confident-ai/deepeval/pull/1357)) <!-- pr:1357 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve the Chroma integration docs with clearer setup and retrieval evaluation examples, including persistent client usage and `n_results` (top-K) tuning guidance. ([#1361](https://github.com/confident-ai/deepeval/pull/1361)) <!-- pr:1361 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve metric docs with a clearer example of using `evaluate()` to generate reports or run multiple metrics on a test case, plus an explicit alternative showing how to call `metric.measure()` directly. ([#1364](https://github.com/confident-ai/deepeval/pull/1364)) <!-- pr:1364 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add telemetry for metrics run mode by recording whether a metric is executed in async mode. This improves observability when diagnosing performance and runtime behavior across different execution paths. ([#1365](https://github.com/confident-ai/deepeval/pull/1365)) <!-- pr:1365 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve the PGVector integration guide with clearer setup and retrieval steps, expanded evaluation guidance, and updated examples for embedding models and tuning `LIMIT`/top-k. Reorganize content to better explain how PGVector fits into a RAG pipeline. ([#1366](https://github.com/confident-ai/deepeval/pull/1366)) <!-- pr:1366 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix a typo in the tutorial introduction so the guidance on choosing evaluation criteria reads correctly. ([#1370](https://github.com/confident-ai/deepeval/pull/1370)) <!-- pr:1370 --> ([JonasHildershavnUke](https://github.com/JonasHildershavnUke))

#### v2.3.1

- Prepare a new release by updating package metadata and reported version. ([#1328](https://github.com/confident-ai/deepeval/pull/1328)) <!-- pr:1328 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Bug Fix

#### v2.4.7

- Fix a typo in the Faithfulness metric docs by correcting a sentence in the `truths_extraction_limit` parameter description. ([#1391](https://github.com/confident-ai/deepeval/pull/1391)) <!-- pr:1391 --> ([Christian Bernhard](https://github.com/ChristianBernhard))

#### v2.4.6

- Fix cleanup of test case instance IDs so concurrent `evaluate` calls with multiple non-conversational metrics no longer crash in the same process. ([#1384](https://github.com/confident-ai/deepeval/pull/1384)) <!-- pr:1384 --> ([cancelself](https://github.com/cancelself))

#### v2.4.3

- Fix the faithfulness prompt example to use the correct `truths` JSON key instead of `claims`. ([#1373](https://github.com/confident-ai/deepeval/pull/1373)) <!-- pr:1373 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix initialization of the faithfulness metric by ensuring the prompt template is created during construction. This prevents missing template errors and makes metric setup more reliable. ([#1374](https://github.com/confident-ai/deepeval/pull/1374)) <!-- pr:1374 --> ([Jaime Enríquez](https://github.com/ebjaime))
- Fix ValidationErrors when evaluating with a custom LLM after the verdict-based schema change, ensuring custom models validate correctly and evaluation runs without failing. ([#1375](https://github.com/confident-ai/deepeval/pull/1375)) <!-- pr:1375 --> ([Tyler Ball](https://github.com/tyler-ball))
- Relax the `grpcio` dependency to `^1.67.1` instead of pinning `1.67.1`. This reduces pip upgrade conflicts in projects that already require a newer `grpcio` (for example via `grpcio-status`). ([#1379](https://github.com/confident-ai/deepeval/pull/1379)) <!-- pr:1379 --> ([Dmitriy Vasilyuk](https://github.com/reasonmethis))
- Fix the first README example by adding missing imports and providing `expected_output` in `LLMTestCase`, so the snippet runs without NameError and matches the documented setup. ([#1382](https://github.com/confident-ai/deepeval/pull/1382)) <!-- pr:1382 --> ([dokato](https://github.com/dokato))

#### v2.3.9

- Fix the broken link to the G-Eval paper in the `ConversationalGEval` documentation so readers can access the referenced source directly. ([#1336](https://github.com/confident-ai/deepeval/pull/1336)) <!-- pr:1336 --> ([Jonathan du Mesnil](https://github.com/j-mesnil))
- Fix `auto_evaluate` async execution by passing the correct `async_mode` flag, and export `auto_evaluate` at the package top level so it can be imported directly from the main module. ([#1338](https://github.com/confident-ai/deepeval/pull/1338)) <!-- pr:1338 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix CLI login pairing flow by starting the local server on an available port and opening a direct pairing URL. Show which provider you logged in with after login (and on failure) to make troubleshooting easier. ([#1345](https://github.com/confident-ai/deepeval/pull/1345)) <!-- pr:1345 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix DAG template examples to use valid JSON booleans (`true`/`false`) so generated verdict outputs are JSON-compliant and easier to parse. ([#1349](https://github.com/confident-ai/deepeval/pull/1349)) <!-- pr:1349 --> ([Aaron McClintock](https://github.com/Spectavi))
- Fix `red_teamer.scan` documentation by adding the missing comma in the example call, so the code block parses correctly and can be copied without syntax errors. ([#1351](https://github.com/confident-ai/deepeval/pull/1351)) <!-- pr:1351 --> ([Akshay Rahatwal](https://github.com/amrakshay))
- Fix prompt wording so `verdict` is only set to 'yes' when the instruction is completely followed, reducing ambiguous interpretations in generated results. ([#1369](https://github.com/confident-ai/deepeval/pull/1369)) <!-- pr:1369 --> ([Daniel Abraján](https://github.com/seorc))
- Fix the CybersecurityGuard API by renaming `CyberattackType` to `CyberattackCategory` and switching configuration from `vulnerabilities` to `categories`. Remove stray debug prints and make input/output guard type selection consistent. ([#1372](https://github.com/confident-ai/deepeval/pull/1372)) <!-- pr:1372 --> ([Jeffrey Ip](https://github.com/penguine-ip))

#### v2.3.1

- Fix `should_use_batch` detection by checking for a `batch_generate` method instead of calling it and swallowing errors. This prevents false negatives when `batch_generate` requires extra arguments (for example `schemas`) and ensures batching is enabled when supported. ([#1327](https://github.com/confident-ai/deepeval/pull/1327)) <!-- pr:1327 --> ([Ruiqi(Ricky) Zhu](https://github.com/ruiqizhu-ricky))
- Fix typos in generated telemetry output to improve accuracy and readability of telemetry files. ([#1329](https://github.com/confident-ai/deepeval/pull/1329)) <!-- pr:1329 --> ([Paul-Louis NECH](https://github.com/PLNech))
- Fix passing document paths to the context generator when building embeddings, preventing incorrect argument mapping during golden generation from docs. ([#1330](https://github.com/confident-ai/deepeval/pull/1330)) <!-- pr:1330 --> ([Kritin Vongthongsri](https://github.com/kritinv))

## January

January made evaluations and red-teaming easier to adopt with documentation cleanups, new tutorials, and clearer configuration patterns like `target_model_callback` and `ignore_errors`. Observability improved with expanded telemetry, run identifiers, and `synthesis_cost` tracking. Features advanced with new ARC benchmark runners, structured ToolCall support, an upgraded `TaskCompletionMetric`, and a revamped Guardrails API.

### New Feature

#### v2.2.7

- Add `auto_evaluate` to automatically generate evaluation datasets from captured LangChain or LlamaIndex context, run a target model, and score results with selected metrics. Supports async execution and optional dataset/result caching. ([#1283](https://github.com/confident-ai/deepeval/pull/1283)) <!-- pr:1283 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `TaskCompletionMetric` to score whether an agent completed the user’s goal based on the actual outcome and tools called, with optional reasons and async support. ([#1295](https://github.com/confident-ai/deepeval/pull/1295)) <!-- pr:1295 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add a new Legal Document Summarizer tutorial series, covering how to define summarization criteria, pick metrics, run evaluations, iterate on hyperparameters, and catch regressions by comparing test runs. ([#1323](https://github.com/confident-ai/deepeval/pull/1323)) <!-- pr:1323 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add a new RAG QA Agent tutorial in the docs, including guidance on choosing metrics, running evaluations, and improving hyperparameters. The tutorials sidebar now includes this section and surfaces it by default. ([#1326](https://github.com/confident-ai/deepeval/pull/1326)) <!-- pr:1326 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.2.2

- Add three new multimodal evaluation metrics: `ImageCoherenceMetric`, `ImageHelpfulnessMetric`, and `ImageReferenceMetric` for scoring how well images align with surrounding context, user intent, and provided references. ([#1230](https://github.com/confident-ai/deepeval/pull/1230)) <!-- pr:1230 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add an optional `identifier` to tag and persist test runs, available via the CLI flag `--identifier` and the pytest plugin option. This helps you distinguish and group results across multiple runs more easily. ([#1237](https://github.com/confident-ai/deepeval/pull/1237)) <!-- pr:1237 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Add an ARC benchmark runner with ARC-Easy and ARC-Challenge modes, configurable `n_shots` and problem count, and built-in accuracy reporting with per-example predictions. Expand the docs to include new benchmark pages and navigation entries for additional benchmark suites. ([#1239](https://github.com/confident-ai/deepeval/pull/1239)) <!-- pr:1239 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add multimodal RAG evaluation support, including test cases with image inputs and retrieval context plus new multimodal metrics for recall, relevancy, precision, answer relevancy, and faithfulness. ([#1241](https://github.com/confident-ai/deepeval/pull/1241)) <!-- pr:1241 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add a revamped guardrails API with built-in guard classes (e.g., privacy, prompt-injection, jailbreaking, topical, cybersecurity) and support for running multiple guards in one call, returning per-guard scores and breakdowns. ([#1247](https://github.com/confident-ai/deepeval/pull/1247)) <!-- pr:1247 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add `max_context_length` to control how many chunks are grouped into each generated context during document-based synthesis, letting you tune context size for generation. Also adjust context grouping defaults and de-duplication to produce more consistent context groups. ([#1289](https://github.com/confident-ai/deepeval/pull/1289)) <!-- pr:1289 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add ToolCall support for tool evaluation data. Datasets can now load `tools_called` and `expected_tools` from JSON/CSV into structured ToolCall objects, with more robust JSON parsing. Metrics like ToolCorrectness and GEval now handle ToolCall values when evaluating and formatting outputs. ([#1290](https://github.com/confident-ai/deepeval/pull/1290)) <!-- pr:1290 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add configurable tool correctness scoring to validate tool names, input parameters, or outputs. Improve verbose logs by showing expected vs called values and the final score and reason, making tool-call mismatches easier to diagnose. ([#1293](https://github.com/confident-ai/deepeval/pull/1293)) <!-- pr:1293 --> ([Kritin Vongthongsri](https://github.com/kritinv))

### Improvement

#### v2.2.7

- Bump package version metadata to 2.2.2 for the latest release. ([#1302](https://github.com/confident-ai/deepeval/pull/1302)) <!-- pr:1302 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve the G-Eval documentation by adding guidance for running evaluations on Confident AI, including the `deepeval login` step to get started. ([#1303](https://github.com/confident-ai/deepeval/pull/1303)) <!-- pr:1303 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix a typo in the dataset push success message and docs, correcting “Confidnet” to “Confident” for clearer branding and guidance. ([#1307](https://github.com/confident-ai/deepeval/pull/1307)) <!-- pr:1307 --> ([Rahul Shah](https://github.com/r-sniper))
- Add an `ignore_errors` option to red teaming scans so attack generation and evaluation can surface failures without aborting the run. Also rename the async concurrency setting to `max_concurrent` for clearer configuration. ([#1309](https://github.com/confident-ai/deepeval/pull/1309)) <!-- pr:1309 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve the Task Completion metric documentation by clarifying that it evaluates tool-calling agents using `input`, `tools_called`, and `actual_output`. Expand the calculation section to explain task/outcome extraction and alignment scoring, with additional examples for context. ([#1310](https://github.com/confident-ai/deepeval/pull/1310)) <!-- pr:1310 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve Jailbreaking Crescendo JSON schema generation by adding stricter system prompts to confine outputs to the expected keys and moving the `description` field to the eval schema. Also ensure remote attack generation initializes the API client with an explicit API key value. ([#1311](https://github.com/confident-ai/deepeval/pull/1311)) <!-- pr:1311 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix the MMLU benchmark docs by updating the example to use `MMLUTask`, helping users get started with the correct setup. This addresses an issue in the MMLU introduction, though some guidance gaps remain around long outputs and batching with varying prompt lengths. ([#1313](https://github.com/confident-ai/deepeval/pull/1313)) <!-- pr:1313 --> ([Matthew Khoriaty](https://github.com/AMindToThink))
- Improve tool correctness evaluation by supporting multiple `ToolCallParams` at once and generating clearer scoring and verbose logs for exact-match and ordering checks. ([#1317](https://github.com/confident-ai/deepeval/pull/1317)) <!-- pr:1317 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve synthesizer docs by clarifying that for RAG evaluation only certain evolution types reliably stick to the provided context, and annotate the examples accordingly. ([#1319](https://github.com/confident-ai/deepeval/pull/1319)) <!-- pr:1319 --> ([Sebastian](https://github.com/sobs0))
- Add a new RAG QA Agent tutorial series covering synthetic dataset generation, evaluation criteria, and metric selection, and reorganize the tutorials sidebar to keep other sections collapsed by default. ([#1325](https://github.com/confident-ai/deepeval/pull/1325)) <!-- pr:1325 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.2.2

- Improve red-teaming 2.0 documentation with clearer setup and scan examples, including how to define vulnerabilities and a target model callback. Reorganize the docs sidebar to add OWASP guidance and a dedicated vulnerabilities section for easier navigation. ([#1209](https://github.com/confident-ai/deepeval/pull/1209)) <!-- pr:1209 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Bump package version to 2.0.5. ([#1217](https://github.com/confident-ai/deepeval/pull/1217)) <!-- pr:1217 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Add tracking of `synthesis_cost` when synthesizing goldens by accumulating model call costs, so you can see the estimated spend for synthesis runs. ([#1218](https://github.com/confident-ai/deepeval/pull/1218)) <!-- pr:1218 --> ([Vytenis Šliogeris](https://github.com/vjsliogeris))
- Improve dependency compatibility by updating the `tenacity` requirement to allow up to version 9.0.0, reducing install conflicts with newer environments. ([#1226](https://github.com/confident-ai/deepeval/pull/1226)) <!-- pr:1226 --> ([Anindyadeep](https://github.com/Anindyadeep))
- Fix a grammar issue in the RAG evaluation guide to clarify that prompts are constructed from both the initial input and the retrieved context. ([#1233](https://github.com/confident-ai/deepeval/pull/1233)) <!-- pr:1233 --> ([Nishant Mahesh](https://github.com/nishant-mahesh))
- Improve benchmark docs with clearer descriptions, supported modes/tasks, and copy-paste examples for ARC, BBQ, and Winogrande. Also tidy benchmark exports and naming to make imports and evaluation parameters more consistent. ([#1240](https://github.com/confident-ai/deepeval/pull/1240)) <!-- pr:1240 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Prepare a new release by bumping the package version to 2.1.0. ([#1245](https://github.com/confident-ai/deepeval/pull/1245)) <!-- pr:1245 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve benchmark runs by adding more built-in benchmark imports, optional verbose per-problem logging, and configurable answer-format confinement instructions to reduce parsing errors and make results easier to inspect. ([#1246](https://github.com/confident-ai/deepeval/pull/1246)) <!-- pr:1246 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve red-teaming documentation by renaming the target model function parameter to `target_model_callback` and updating sync/async examples to match, reducing confusion when wiring up scans. ([#1250](https://github.com/confident-ai/deepeval/pull/1250)) <!-- pr:1250 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Change the default Guardrails API base URL to `https://deepeval.confident-ai.com/` instead of `http://localhost:8000`, so it connects to the hosted service by default. ([#1252](https://github.com/confident-ai/deepeval/pull/1252)) <!-- pr:1252 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Update package metadata by bumping the release version and refreshing the project description. ([#1254](https://github.com/confident-ai/deepeval/pull/1254)) <!-- pr:1254 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve Guardrails API configuration by using the shared `BASE_URL` from the guardrails API module instead of a hardcoded localhost URL. ([#1255](https://github.com/confident-ai/deepeval/pull/1255)) <!-- pr:1255 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add an `IS_CONFIDENT` environment toggle to switch the API base URL to a local server (using `PORT`) instead of the default hosted endpoint. ([#1258](https://github.com/confident-ai/deepeval/pull/1258)) <!-- pr:1258 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve guardrails base classes and typing by introducing `BaseGuard`/`BaseDecorativeGuard` and a shared `GuardType` enum. This makes guard metadata and guardrail configuration more consistent across built-in guards. ([#1259](https://github.com/confident-ai/deepeval/pull/1259)) <!-- pr:1259 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Add a configurable `top_logprobs` setting to better support OpenAI and Azure OpenAI deployments where logprobs limits vary by model/version. This helps avoid failures or unexpected clamping when a service only supports smaller values (for example, 5 instead of 20). ([#1261](https://github.com/confident-ai/deepeval/pull/1261)) <!-- pr:1261 --> ([Dave Erickson](https://github.com/derickson))
- Add PostHog analytics tracking to the documentation site, with tracking disabled in development to avoid collecting local activity. ([#1268](https://github.com/confident-ai/deepeval/pull/1268)) <!-- pr:1268 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Update package metadata for a new release. ([#1270](https://github.com/confident-ai/deepeval/pull/1270)) <!-- pr:1270 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix typos in the README by correcting “continous” to “continuous” in multiple places. ([#1273](https://github.com/confident-ai/deepeval/pull/1273)) <!-- pr:1273 --> ([Ikko Eltociear Ashimine](https://github.com/eltociear))
- Improve telemetry spans for evaluations, synthesizer, red teaming, guardrails, and benchmarks by capturing more run details and consistently tagging an anonymous `unique_id` (and public IP when available). This makes usage and performance monitoring more consistent across features. ([#1276](https://github.com/confident-ai/deepeval/pull/1276)) <!-- pr:1276 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add support for additional OpenAI GPT model IDs, including versioned `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, and `gpt-3.5-turbo-instruct` variants, so model validation accepts more current options out of the box. ([#1277](https://github.com/confident-ai/deepeval/pull/1277)) <!-- pr:1277 --> ([Song Luar](https://github.com/luarss))
- Add an opt-out for automatic update warnings via the `DEEPEVAL_UPDATE_WARNING_OPT_OUT=YES` environment variable, so you can suppress update checks in non-interactive or CI environments. Documentation was added for this setting. ([#1278](https://github.com/confident-ai/deepeval/pull/1278)) <!-- pr:1278 --> ([Song Luar](https://github.com/luarss))
- Bump the package version for a new release. ([#1279](https://github.com/confident-ai/deepeval/pull/1279)) <!-- pr:1279 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Improve telemetry by tagging spans with the runtime environment (Jupyter notebook vs other) to better understand where evaluations and tools are run. ([#1280](https://github.com/confident-ai/deepeval/pull/1280)) <!-- pr:1280 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve OpenAI-native model calls by using structured outputs with explicit schemas, returning typed fields directly instead of parsing JSON strings. This makes metric verdicts/reasons/statements more reliable and reduces parsing failures. ([#1285](https://github.com/confident-ai/deepeval/pull/1285)) <!-- pr:1285 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Update OpenAI model lists so `gpt_model` and `gpt_model_schematic` stay in sync, including refreshed multimodal model support. Adjust validation and pricing data to match the latest available models and costs. ([#1287](https://github.com/confident-ai/deepeval/pull/1287)) <!-- pr:1287 --> ([Song Luar](https://github.com/luarss))
- Update the default API base URL used by the red teaming attack synthesizer to point to the hosted service instead of localhost. ([#1288](https://github.com/confident-ai/deepeval/pull/1288)) <!-- pr:1288 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Improve documentation with a new Cognee integration guide and corrected guardrails example usage, plus small styling and copy updates across the site. ([#1291](https://github.com/confident-ai/deepeval/pull/1291)) <!-- pr:1291 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix typos in the custom LLMs guide to clarify the exception note and correct the `instantiate` instruction. ([#1294](https://github.com/confident-ai/deepeval/pull/1294)) <!-- pr:1294 --> ([Christian Bernhard](https://github.com/ChristianBernhard))
- Add telemetry attributes to record whether each feature run is considered `new` or `old`, and persist that status after a feature is used. This improves feature-usage reporting across evaluation, synthesizer, red teaming, guardrails, and benchmarks. ([#1296](https://github.com/confident-ai/deepeval/pull/1296)) <!-- pr:1296 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Add validation and pricing metadata for OpenAI `o1` models (`o1`, `o1-preview`, `o1-2024-12-17`) so they can be used with JSON mode and structured outputs where supported. ([#1299](https://github.com/confident-ai/deepeval/pull/1299)) <!-- pr:1299 --> ([Song Luar](https://github.com/luarss))
- Add a `--display` option to control which test cases are shown in the final results output, so you can view all, only failing, or only passing cases in CLI runs and `evaluate()` printing. ([#1301](https://github.com/confident-ai/deepeval/pull/1301)) <!-- pr:1301 --> ([Jeffrey Ip](https://github.com/penguine-ip))

### Bug Fix

#### v2.2.7

- Fix structured (`schema`) responses when using non-OpenAI models (including Azure/local) by correctly invoking the loaded model and returning the parsed JSON along with the tracked cost. ([#1304](https://github.com/confident-ai/deepeval/pull/1304)) <!-- pr:1304 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix circular imports involving `Scorer` by deferring its import in benchmark modules, preventing import-time crashes when loading benchmarks. ([#1315](https://github.com/confident-ai/deepeval/pull/1315)) <!-- pr:1315 --> ([Song Luar](https://github.com/luarss))
- Fix async tracing in the LangChain callback by making trace state thread-safe and correctly linking parent/child spans. This prevents missing or mis-associated traces when runs execute concurrently. ([#1318](https://github.com/confident-ai/deepeval/pull/1318)) <!-- pr:1318 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix leftover `.vector_db` collections when chunking fails by cleaning up the generated collection folders before raising an error. Also handle invalid Chroma collections explicitly so document loading can recover more reliably. ([#1320](https://github.com/confident-ai/deepeval/pull/1320)) <!-- pr:1320 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix context generation from docs by passing `document_paths` explicitly, preventing incorrect argument binding. Also skip the MULTICONTEXT evolution when transforming evolution distributions to avoid generating unsupported prompt evolutions. ([#1321](https://github.com/confident-ai/deepeval/pull/1321)) <!-- pr:1321 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix local Ollama embedding requests by routing through the OpenAI client when the base URL points to localhost. This restores embedding support for both single text and batch inputs without changing cloud OpenAI behavior. ([#1322](https://github.com/confident-ai/deepeval/pull/1322)) <!-- pr:1322 --> ([Kritin Vongthongsri](https://github.com/kritinv))

#### v2.2.2

- Prevent endless verdict generation in ContextualPrecision by including the explicit document count in the prompt, helping LLMs stay aligned on long or complex context lists. ([#1222](https://github.com/confident-ai/deepeval/pull/1222)) <!-- pr:1222 --> ([enrico-stauss](https://github.com/enrico-stauss))
- Fix `MMLUTemplate.format_subject` to be a static method, allowing it to be called without an instance and preventing incorrect usage in MMLU prompt formatting. ([#1229](https://github.com/confident-ai/deepeval/pull/1229)) <!-- pr:1229 --> ([Terrasse](https://github.com/Jerry-Terrasse))
- Prevent OpenTelemetry from loading on import when telemetry is opted out. This avoids importing protobuf dependencies unnecessarily and reduces conflicts with other libraries. ([#1231](https://github.com/confident-ai/deepeval/pull/1231)) <!-- pr:1231 --> ([Mykhailo Chalyi (Mike Chaliy)](https://github.com/chaliy))
- Fix red teaming risk-category mapping to use the updated `*Type` vulnerability enums, keeping vulnerability classification consistent after recent naming changes. ([#1236](https://github.com/confident-ai/deepeval/pull/1236)) <!-- pr:1236 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix synthetic data generation when ChromaDB raises `InvalidCollectionException` by catching the correct exception type in `a_chunk_doc`, ensuring fallback handling runs instead of stopping early. ([#1242](https://github.com/confident-ai/deepeval/pull/1242)) <!-- pr:1242 --> ([Mizuki Nakano](https://github.com/Mizuki8783))
- Fix text-to-image metric semantic consistency evaluation to use the generated output image instead of an input image, improving scoring accuracy for text-only prompts. ([#1253](https://github.com/confident-ai/deepeval/pull/1253)) <!-- pr:1253 --> ([Kritin Vongthongsri](https://github.com/kritinv))
- Fix docs to use the correct import paths for sensitive information disclosure attack types (`PIILeakageType`, `PromptLeakageType`, `IntellectualPropertyType`), preventing import errors when following the example code. ([#1256](https://github.com/confident-ai/deepeval/pull/1256)) <!-- pr:1256 --> ([Mohammad-Reza Azizi](https://github.com/mrazizi))
- Fix guardrails API calls to use the updated `/guardrails` endpoint instead of the old multiple-guard path. ([#1257](https://github.com/confident-ai/deepeval/pull/1257)) <!-- pr:1257 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix guardrails API schema so `input` and `response` are defined at the request level instead of per-guard, preventing invalid payloads when multiple guards are used. ([#1260](https://github.com/confident-ai/deepeval/pull/1260)) <!-- pr:1260 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix MMLU task reloading so the benchmark dataset is fetched fresh for the selected task instead of reusing a previously cached dataset. This prevents running evaluations against the wrong task data when switching tasks. ([#1267](https://github.com/confident-ai/deepeval/pull/1267)) <!-- pr:1267 --> ([Yuyao Huang](https://github.com/exhyy))
- Fix synthesizer cost tracking to handle unset `synthesis_cost`. This prevents errors when generating data if cost accounting is disabled or not initialized. ([#1271](https://github.com/confident-ai/deepeval/pull/1271)) <!-- pr:1271 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Fix batched `evaluate()` results so prediction rows include the expected output alongside the input, prediction, and score, keeping benchmark output consistent and easier to inspect. ([#1274](https://github.com/confident-ai/deepeval/pull/1274)) <!-- pr:1274 --> ([BjarniH](https://github.com/BjarniHaukur))
- Fix documentation “Edit this page” links to point to the correct `docs/` directory so edits open in the right place on GitHub. ([#1292](https://github.com/confident-ai/deepeval/pull/1292)) <!-- pr:1292 --> ([Jeffrey Ip](https://github.com/penguine-ip))
- Prevent installing the `tests` folder into site-packages by excluding it from the package install. This avoids name conflicts when your project also includes a `tests` directory. ([#1300](https://github.com/confident-ai/deepeval/pull/1300)) <!-- pr:1300 --> ([冯键](https://github.com/fj11))
