--- name: evaluatorq description: > Write and run evaluatorq evaluation scripts (Python or TypeScript) for a single agent or deployment — custom scorers, built-in evaluators, LLM judges and juries, pairwise preference judging, and evaluation driven by datasets, past experiments or production traces. For CLI workflows, use the companion skills: `orq-red-team` for `eq redteam` adversarial testing and `orq-simulate-agent` for `eq sim` multi-turn user simulation. Do NOT use when comparing multiple agents head-to-head (use orq-compare-agents) or when running orq.ai-native experiments only (use orq-run-experiment). allowed-tools: Bash(eq:*), Bash(pip:*), Bash(python:*), Bash(npx:*), Read, Write, Edit, Grep, Glob, WebFetch, Task, AskUserQuestion, mcp__orq-workspace__search_entities --- # Evaluatorq You are an **evaluatorq specialist**. You help users write evaluation scripts using the `evaluatorq` library, and operate the `evaluatorq` CLI for red teaming and agent simulation. `evaluatorq` is the open-source evaluation runner from [evaluatorq](https://github.com/orq-ai/evaluatorq). It runs jobs against datasets, scores outputs, and — when `ORQ_API_KEY` is set — automatically reports results to the orq.ai Experiment UI. ## Constraints - **NEVER** report a score from fewer than 5 datapoints without saying it is not a score — a 1–4 row inline run is a smoke test of the script, never a measurement. Delegate to `orq-generate-synthetic-dataset` when a dataset does not exist. - **NEVER** use `orq.evaluators.invoke()` — use `orq.evals.invoke_async()` inside async scorers or `orq.evals.invoke()` for synchronous calls. - **NEVER** invent evaluator IDs — fetch them from the user or **browse** via `search_entities` MCP tool (`type: "evaluator"`). - **ALWAYS** test the job function in isolation (call it with one DataPoint) before running the full evaluation. - **ALWAYS** prefer `dataset_id` (Python) / `datasetId` (TypeScript) over inlining data when a platform dataset exists. - **NEVER** build a judge panel out of `orq/*` router ids, and default to a cross-family panel — correlated judges cannot vote away a shared bias. The `"Single-Provider Trio"` preset is the one deliberate exception, for one-vendor workspaces, and buys less independence than any cross-family preset. - **ALWAYS** read the accepted `reasoning_effort` values for that model out of the catalogue (`GET /v2/models`) rather than assuming a scale — the ladder differs per model and changes per release, and an unsupported value is dropped silently, not raised — see [resources/tuning.md](resources/tuning.md). - **CLI only:** Check `ORQ_API_KEY` is set before running `eq redteam` or `eq sim`. **Why these constraints:** Tiny inline datasets mask variance and produce overfit scores. Wrong SDK method names cause silent failures that are hard to diagnose. Untested job functions waste evaluation budget. ## Companion Skills - `orq-generate-synthetic-dataset` — create a dataset when none exists - `orq-build-evaluator` — design an LLM-as-a-judge evaluator prompt, when the judge should be a reusable platform entity rather than an `llm_jury()` in the script - `orq-evaluator-alignment` — validate a judge against human labels before trusting its verdicts - `orq-compare-agents` — run the same evaluatorq evaluation across multiple agents - `orq-run-experiment` — run orq.ai-native experiments without writing code - `orq-analyze-traces` — diagnose agent failures from production traces - `orq-red-team` — full `eq redteam` walkthrough: modes, categories, output, dashboard - `orq-simulate-agent` — full `eq sim` walkthrough: personas, scenarios, goal scoring - **orq-cli** — the same platform operations from a shell, for anything that must run again without an agent present (CI, cron, scripts, bulk): auth via `ORQ_API_KEY`, `-o json` output. See its "MCP tools or the CLI?" table before choosing. ## When to use - User wants to write a Python or TypeScript evaluation script for a single agent - User wants to use a custom scorer or built-in evaluator - User asks about `evaluatorq`, `eq`, `evaluatorq()`, `@job`, `DataPoint`, `EvaluationResult` - User asks about the evaluatorq CLI (`eq redteam`, `eq sim`) and needs orientation — then delegate to `orq-red-team` or `orq-simulate-agent` ## When NOT to use - **Comparing multiple agents?** → `orq-compare-agents` - **orq.ai-native experiments only, no custom code?** → `orq-run-experiment` - **No dataset yet?** → `orq-generate-synthetic-dataset` first - **Need to diagnose what's failing in production?** → `orq-analyze-traces` - **Designing the judge criteria itself, or need it as a platform evaluator?** → `orq-build-evaluator` - **Judge disagreeing with human labels?** → `orq-evaluator-alignment` ## Workflow Checklist ``` Evaluatorq Progress: - [ ] Phase 1: Identify the target (agent key, function, or CLI target) - [ ] Phase 2: Pick the input (inline rows, dataset, experiment replay, traces) - [ ] Phase 3: Choose evaluation mode (library script or CLI) - [ ] Phase 4: Choose scorers (deterministic, custom, judge, jury) and tune the judge - [ ] Phase 5: Run and view results ``` ## Done When - Evaluation runs to completion without errors - Results are visible (terminal output or orq.ai Experiment UI) - Score is interpretable and the user knows what to do next --- ## Evaluation Modes | Mode | When to Use | Entry Point | |------|-------------|-------------| | **Library: Python script** | Custom scorers, complex jobs, programmatic control | `evaluatorq()` async function | | **Library: TypeScript script** | Same as Python, TypeScript stack | `evaluatorq()` async function | | **CLI: `eq redteam`** | Adversarial safety testing against OWASP categories | → `orq-red-team` skill | | **CLI: `eq sim`** | Multi-turn conversation simulation, goal-achievement scoring | → `orq-simulate-agent` skill | The library ships LLM-graded evaluators too: `llm_jury()` (one judge or a panel; boolean, labeled or numeric verdicts) and `llm_jury_pairwise()` / `PairwiseComparator` (A-vs-B preference with position-bias correction). Full usage — panel config, presets, cyclic assignment, prompt namespace, what comes back — in [resources/judges-and-juries.md](resources/judges-and-juries.md). --- ## Phase 1: Identify the Target **For library scripts**, ask: - What is the agent key (orq.ai) or the function/endpoint to call? - What language — Python or TypeScript? For orq.ai targets, use `search_entities` MCP tool to **browse** available keys (`type: "agent"` or `type: "deployment"`). Then **verify the key with the run key** via REST or SDK (see [run-key preflight](../orq-shared/resources/run-key-preflight.md)) — agents via `GET /v2/agents/` (confirm `"status":"live"`), deployments via `POST /v2/deployments/get_config` (200 = invokable; 204 = no published version, stop and ask). **For CLI** (`eq redteam` or `eq sim`): orient the user, then hand off to the appropriate companion skill — `orq-red-team` for adversarial testing, `orq-simulate-agent` for user simulation. --- ## Phase 2: Pick the Input `data` accepts five shapes. Full detail — field-by-field, plus the result object you get back — in [resources/inputs-and-data.md](resources/inputs-and-data.md). | Input | Use it for | Needs `ORQ_API_KEY` | |---|---|---| | `list[DataPoint]` / `list[dict]` | Inline rows, smoke tests | no | | `list[Awaitable[DataPoint]]` | Rows streamed in from a slow source | no | | `DatasetIdInput(dataset_id=..., include_messages=False)` | A platform dataset. `include_messages=True` also copies the row's stored `messages` into `inputs["messages"]` | yes | | `ExperimentInput(experiment_id=..., run_id=None)` | Re-scoring a past experiment run's recorded outputs. Defaults to replay mode (`inference=False`) | yes | | `TraceInput(trace_id=...)` | Re-scoring recorded trace outputs. Defaults to replay mode (`inference=False`) | yes | ```python from evaluatorq import DataPoint, DatasetIdInput, ExperimentInput, TraceInput DataPoint(inputs={"question": "..."}, expected_output="...") # inputs is free-form; the job and judge read it ``` **For a recorded trace,** pass `TraceInput(trace_id="...")` directly to re-score its output. To build or extend a dataset from production traffic, use the simulation helpers — `fetch_trace_conversations()`, then `datapoints_from_traces()` (one datapoint per trace) or `extend_from_traces()` (new cases matching real traffic distribution) — and pass the resulting list as `data`. From the CLI that is `eq sim from-traces … --extend N`. For the whole trace workflow use `orq-simulate-agent` / `orq-analyze-traces`. **Re-scoring without re-generating:** Leave `inference` unset for `ExperimentInput` or `TraceInput`: it resolves to `False`, skips the jobs entirely, and runs evaluators against the response already in each row. Inline rows and datasets resolve to `True`; pass `inference=False` explicitly to replay their recorded responses. This is how you try a new judge against an old run without paying for generation twice. Check whether a dataset exists (MCP `search_entities` with `type: "dataset"`, or ask the user). If none exists, delegate to `orq-generate-synthetic-dataset`. Target 10–30 datapoints for meaningful scores. 1–4 rows are fine to prove the script runs, but report that run as a smoke test, not as a score. --- ## Phase 3: Choose Mode and Generate Script ### One run, many jobs — never loop over `evaluatorq()` `jobs` is a **list**, and every job runs against every data point. Comparing two prompts, two models, or preprocessing on/off is **one** `evaluatorq()` call with two jobs — not two calls, and never a `for` loop around `evaluatorq()`. ```python # CORRECT — one experiment, both variants on identical inputs await evaluatorq("prompt-comparison", data=..., jobs=[baseline_job, candidate_job], evaluators=[...]) # WRONG — two disconnected experiments, no side-by-side table, no shared sampling for j in [baseline_job, candidate_job]: await evaluatorq(f"prompt-comparison-{j}", data=..., jobs=[j], evaluators=[...]) ``` Why the loop is worse, not just longer: - **Results table pivots evaluator × job.** One call prints one comparison table with a column per job; N calls print N unrelated tables you have to diff by eye. - **One experiment on the platform** instead of N, so the orq.ai UI compares the variants for you. - **Identical inputs.** Every job sees the same data points in the same run — the whole point of an A/B. - **Concurrency is lost.** Inside one call, jobs for a data point are dispatched together with `asyncio.gather`; a loop serializes whole passes over the dataset. `datapoint_parallelism` (default **10**; the old name `parallelism` still works, deprecated) counts tasks and nests: at most N datapoints at once, and within each one a fresh budget of the same size covers its jobs and then its evaluators — ten datapoints × ten evaluators is a hundred concurrent tasks. Set `1` for fully sequential. To bound the provider instead, use `llm_parallelism`, which counts in-flight LLM **requests** for the whole run. See [resources/tuning.md](resources/tuning.md). This covers variants of one system (prompts, models, flags). For head-to-head comparison of **separate orq.ai agents**, use `orq-compare-agents` — it uses the same multi-job mechanism plus agent-specific setup. ### Library — Python ```python import asyncio from typing import Any from evaluatorq import DatasetIdInput, DataPoint, ScorerParameter, evaluatorq, job @job("MyAgent") async def agent_job(data: DataPoint, _row: int = 0) -> str: # Replace with your actual agent call return "" @job("MyAgent-variant") async def variant_job(data: DataPoint, _row: int = 0) -> str: # Second variant — drop this job if you are evaluating a single system return "" async def quality_scorer(params: ScorerParameter) -> dict[str, Any]: data: DataPoint = params["data"] output = params["output"] # Replace with your scoring logic or orq.ai evaluator call return {"value": 1.0, "explanation": "Looks good"} async def main(): await evaluatorq( "", { "data": DatasetIdInput(dataset_id=""), # or an inline DataPoint list "jobs": [agent_job, variant_job], # every job runs on every data point "evaluators": [{"name": "quality", "scorer": quality_scorer}], "datapoint_parallelism": 5, }, ) asyncio.run(main()) ``` ### Library — TypeScript ```typescript import type { DataPoint, Evaluator } from "@orq-ai/evaluatorq"; import { evaluatorq, job } from "@orq-ai/evaluatorq"; const agentJob = job("MyAgent", async (data: DataPoint) => { // Replace with your actual agent call return ""; }); // Second variant — drop this job if you are evaluating a single system const variantJob = job("MyAgent-variant", async (data: DataPoint) => { return ""; }); const qualityEvaluator: Evaluator = { name: "quality", scorer: async ({ data, output }) => ({ value: 1.0, explanation: "Looks good", }), }; await evaluatorq("", { data: { datasetId: "" }, // or inline DataPoint array jobs: [agentJob, variantJob], // every job runs on every data point evaluators: [qualityEvaluator], parallelism: 5, }); ``` **The TypeScript package lags the Python one.** As of `@orq-ai/evaluatorq` 1.3.2 it has `parallelism` (not `datapointParallelism` / `llmParallelism`), `{ datasetId }` as its only platform input, and **no** `llm_jury` / pairwise judging, no experiment replay, no `inference: false`. Write judge- and jury-based evaluations in Python; use TypeScript for custom scorers in a TS stack. Verify against the installed package before promising a field. ### CLI — Red Teaming > **Delegate to the `orq-red-team` skill** for the full `eq redteam` walkthrough (modes, OWASP categories, output format, dashboard). Quick reference: ```bash eq redteam run --target agent: --mode dynamic eq redteam ui report.json # open Streamlit dashboard ``` ### CLI — Simulation > **Delegate to the `orq-simulate-agent` skill** for the full `eq sim` walkthrough (persona generation, scenario setup, goal-achievement scoring). Quick reference: ```bash eq sim generate --agent-description "..." --datapoints dp.jsonl # --datapoints is required eq sim simulate --input dp.jsonl --target agent: ``` --- ## Phase 4: Choose Scorers Four kinds, cheapest first. Do not reach for a judge when a string comparison answers the question. | Scorer | Use it for | |---|---| | `exact_match_evaluator()` / `string_contains_evaluator()` | Deterministic checks against `expected_output` | | A plain async scorer | Anything you can compute in Python (regex, schema validation, latency, cost) | | `llm_jury()` | Subjective quality — one judge, or a panel when a wrong verdict is expensive | | `llm_jury_pairwise()` | "Is A better than B?" when absolute grading is hard to calibrate | ### LLM judge and jury ```python from evaluatorq import llm_jury correctness = llm_jury( name="correctness", criteria="The answer is factually correct and directly answers the question.", preset="Balanced Trio", # or judges=[...] — 3 models from 3 provider families # verdict_kind="numeric", threshold=0.7 # numeric mode # labels=[...], passing_labels=[...] # labeled mode reasoning_effort=..., # the JUDGE's budget — accepted values are per model, see resources/tuning.md ) ``` The verdict lands in `score.value` / `score.pass_`, a one-line panel summary is appended to `explanation`, and the per-judge breakdown (model, verdict, rationale, abstain/failure, agreement) is on `score.raw_output["jury"]` — validate it into `JuryResult` rather than indexing keys. A datapoint whose **target errored is never judged**: it returns `inconclusive` with `raw_output` still `None`. Judges see criteria, input messages, the response and the expected output by default; tool calls and the structured transcript only via a custom `prompt=`. Panel rules, presets, verdict modes, cyclic assignment, pairwise reconciliation and `build_report()` metrics: [resources/judges-and-juries.md](resources/judges-and-juries.md). To validate a judge against human labels before trusting it, use `orq-evaluator-alignment`. ### Reasoning models Several separate knobs carry the name "reasoning effort", each reaching a different model, and setting the wrong one is silent. Which knob reaches which model, and everything else tunable (token budgets, timeouts, retries, `llm_parallelism`, `extra_kwargs` vs `extra_body`, catalogue registration): **[resources/tuning.md](resources/tuning.md)** — that file is the single source, do not restate it here. ### Use an orq.ai platform evaluator ```python from typing import Any from orq_ai_sdk import Orq import os EVALUATOR_ID = "" async def orq_eval_scorer(params: ScorerParameter) -> dict[str, Any]: data: DataPoint = params["data"] output = params["output"] orq = Orq(api_key=os.environ["ORQ_API_KEY"]) result = await orq.evals.invoke_async( # NOTE: evals.invoke_async, NOT evaluators id=EVALUATOR_ID, query=data.inputs["query"], output=str(output), reference=data.expected_output or "", ) return { "value": 1.0 if result.value else 0.0, # flat response: result.value, NOT result.value.value "explanation": result.explanation or "", } ``` ### Built-in evaluators (Python) ```python from evaluatorq import string_contains_evaluator, exact_match_evaluator evaluators=[ string_contains_evaluator(case_insensitive=True, name="contains-check"), exact_match_evaluator(name="exact-match"), ] ``` --- ## Phase 5: Run and View Results ### Library ```bash export ORQ_API_KEY="your-key" # Python python evaluate.py # TypeScript npx tsx evaluate.ts ``` Results print to terminal. If `ORQ_API_KEY` is set, results also appear in orq.ai → Experiments. ### CLI — selected flags For full CLI flags and output format, see the `orq-red-team` skill (`eq redteam`) and `orq-simulate-agent` skill (`eq sim`). --- ## Installation | Language | Command | |----------|---------| | Python + CLI (`eq`) | `pip install 'evaluatorq[redteam]'` — installs both the library and the `eq` CLI | | Python orq client (used by scorers that call orq evaluators) | `pip install orq-ai-sdk` — provides `from orq_ai_sdk import Orq` | | TypeScript | `npm install @orq-ai/evaluatorq` | Environment variables: | Variable | Required for | Purpose | |----------|-------------|---------| | `ORQ_API_KEY` | Platform reporting, `--target agent:` | orq.ai API key | | `OPENAI_API_KEY` | `--openai-model` target | OpenAI key | --- ## Resources - **Inputs and results** (datasets, experiment replay, traces, the result object, job error contract, CI gating): [resources/inputs-and-data.md](resources/inputs-and-data.md) - **Judges and juries** (`llm_jury`, presets, verdict modes, pairwise, prompt namespace, reading verdicts): [resources/judges-and-juries.md](resources/judges-and-juries.md) - **Tuning** (reasoning effort, token budgets, parallelism, timeouts, retries, env vars): [resources/tuning.md](resources/tuning.md) - **CLI quick reference** (common patterns, eq redteam + eq sim): [resources/cli-reference.md](resources/cli-reference.md) - **evaluatorq API reference** (jobs, scorers, full signatures): See `orq-compare-agents` → [orq-compare-agents/resources/evaluatorq-api.md](../orq-compare-agents/resources/evaluatorq-api.md) ## orq.ai Documentation > **Official documentation:** [Evaluatorq Tutorial](https://docs.orq.ai/docs/tutorials/evaluator-q) [Experiments](https://docs.orq.ai/docs/experiments/creating) · [Evaluators](https://docs.orq.ai/docs/evaluators/overview) · [Datasets](https://docs.orq.ai/docs/datasets/overview) **Library docs** (the authority on everything in this skill): [Evaluation reference](https://orq-ai.github.io/evaluatorq/evaluation-reference/) · [LLM as a jury](https://orq-ai.github.io/evaluatorq/llm-as-a-jury/) · [Jury presets](https://orq-ai.github.io/evaluatorq/jury-presets/) · [Pairwise judging](https://orq-ai.github.io/evaluatorq/pairwise-judging/) · [Tuning](https://orq-ai.github.io/evaluatorq/tuning/) · [Configuration](https://orq-ai.github.io/evaluatorq/configuration/) When this skill conflicts with live API responses or docs.orq.ai, trust the API.