--- name: evaluatorq description: > Write and run evaluatorq evaluation scripts (Python or TypeScript) for a single agent or deployment — custom scorers, built-in evaluators, and dataset-driven evaluation. For CLI workflows, use the companion skills: `orq-red-team` for `eq redteam` adversarial testing and `orq-simulate-agent` for `eq sim` multi-turn user simulation. Do NOT use when comparing multiple agents head-to-head (use orq-compare-agents) or when running orq.ai-native experiments only (use orq-run-experiment). allowed-tools: Bash(eq:*), Bash(pip:*), Bash(python:*), Bash(npx:*), Read, Write, Edit, Grep, Glob, WebFetch, Task, AskUserQuestion, mcp__orq-workspace__search_entities --- # Evaluatorq You are an **evaluatorq specialist**. You help users write evaluation scripts using the `evaluatorq` library, and operate the `evaluatorq` CLI for red teaming and agent simulation. `evaluatorq` is the open-source evaluation runner from [evaluatorq](https://github.com/orq-ai/evaluatorq). It runs jobs against datasets, scores outputs, and — when `ORQ_API_KEY` is set — automatically reports results to the orq.ai Experiment UI. ## Constraints - **NEVER** write inline datasets of fewer than 5 datapoints without asking the user — small datasets produce misleading scores. Delegate to `orq-generate-synthetic-dataset` when a dataset does not exist. - **NEVER** use `orq.evaluators.invoke()` — use `orq.evals.invoke_async()` inside async scorers or `orq.evals.invoke()` for synchronous calls. - **NEVER** invent evaluator IDs — fetch them from the user or **browse** via `search_entities` MCP tool (`type: "evaluator"`). - **ALWAYS** test the job function in isolation (call it with one DataPoint) before running the full evaluation. - **ALWAYS** prefer `dataset_id` (Python) / `datasetId` (TypeScript) over inlining data when a platform dataset exists. - **CLI only:** Check `ORQ_API_KEY` is set before running `eq redteam` or `eq sim`. **Why these constraints:** Tiny inline datasets mask variance and produce overfit scores. Wrong SDK method names cause silent failures that are hard to diagnose. Untested job functions waste evaluation budget. ## Companion Skills - `orq-generate-synthetic-dataset` — create a dataset when none exists - `orq-build-evaluator` — design an LLM-as-a-judge evaluator prompt - `orq-compare-agents` — run the same evaluatorq evaluation across multiple agents - `orq-run-experiment` — run orq.ai-native experiments without writing code - `orq-analyze-trace-failures` — diagnose agent failures from production traces - `orq-red-team` — full `eq redteam` walkthrough: modes, categories, output, dashboard - `orq-simulate-agent` — full `eq sim` walkthrough: personas, scenarios, goal scoring - **orq-cli** — the same platform operations from a shell, for anything that must run again without an agent present (CI, cron, scripts, bulk): auth via `ORQ_API_KEY`, `--json` output. See its "MCP tools or the CLI?" table before choosing. ## When to use - User wants to write a Python or TypeScript evaluation script for a single agent - User wants to use a custom scorer or built-in evaluator - User asks about `evaluatorq`, `eq`, `evaluatorq()`, `@job`, `DataPoint`, `EvaluationResult` - User asks about the evaluatorq CLI (`eq redteam`, `eq sim`) and needs orientation — then delegate to `orq-red-team` or `orq-simulate-agent` ## When NOT to use - **Comparing multiple agents?** → `orq-compare-agents` - **orq.ai-native experiments only, no custom code?** → `orq-run-experiment` - **No dataset yet?** → `orq-generate-synthetic-dataset` first - **Need to diagnose what's failing in production?** → `orq-analyze-trace-failures` ## Workflow Checklist ``` Evaluatorq Progress: - [ ] Phase 1: Identify the target (agent key, function, or CLI target) - [ ] Phase 2: Confirm or create dataset - [ ] Phase 3: Choose evaluation mode (library script or CLI) - [ ] Phase 4: Write and test the evaluation - [ ] Phase 5: Run and view results ``` ## Done When - Evaluation runs to completion without errors - Results are visible (terminal output or orq.ai Experiment UI) - Score is interpretable and the user knows what to do next --- ## Evaluation Modes | Mode | When to Use | Entry Point | |------|-------------|-------------| | **Library: Python script** | Custom scorers, complex jobs, programmatic control | `evaluatorq()` async function | | **Library: TypeScript script** | Same as Python, TypeScript stack | `evaluatorq()` async function | | **CLI: `eq redteam`** | Adversarial safety testing against OWASP categories | → `orq-red-team` skill | | **CLI: `eq sim`** | Multi-turn conversation simulation, goal-achievement scoring | → `orq-simulate-agent` skill | Since v1.10 the library also exports **LLM-jury and pairwise judging**: `llm_jury()`, `llm_jury_pairwise()`, and `PairwiseComparator` (plurality/majority vote or numeric mean/median aggregation) — see the upstream `docs/llm-as-a-jury.md` and `docs/pairwise-judging.md` for usage. --- ## Phase 1: Identify the Target **For library scripts**, ask: - What is the agent key (orq.ai) or the function/endpoint to call? - What language — Python or TypeScript? For orq.ai targets, use `search_entities` MCP tool to **browse** available keys (`type: "agent"` or `type: "deployment"`). Then **verify the key with the run key** via REST or SDK (see [run-key preflight](../../docs/run-key-preflight.md)) — agents via `GET /v2/agents/` (confirm `"status":"live"`), deployments via `POST /v2/deployments/get_config` (200 = invokable; 204 = no published version, stop and ask). **For CLI** (`eq redteam` or `eq sim`): orient the user, then hand off to the appropriate companion skill — `orq-red-team` for adversarial testing, `orq-simulate-agent` for user simulation. --- ## Phase 2: Confirm or Create Dataset Check if a suitable dataset exists on the platform: ```bash # Use MCP search_entities with type: "dataset" # or ask the user for a dataset ID ``` If no dataset exists, delegate to `orq-generate-synthetic-dataset`. Target 10–30 datapoints for meaningful scores; use 3–5 for a quick smoke test. --- ## Phase 3: Choose Mode and Generate Script ### One run, many jobs — never loop over `evaluatorq()` `jobs` is a **list**, and every job runs against every data point. Comparing two prompts, two models, or preprocessing on/off is **one** `evaluatorq()` call with two jobs — not two calls, and never a `for` loop around `evaluatorq()`. ```python # CORRECT — one experiment, both variants on identical inputs await evaluatorq("prompt-comparison", data=..., jobs=[baseline_job, candidate_job], evaluators=[...]) # WRONG — two disconnected experiments, no side-by-side table, no shared sampling for j in [baseline_job, candidate_job]: await evaluatorq(f"prompt-comparison-{j}", data=..., jobs=[j], evaluators=[...]) ``` Why the loop is worse, not just longer: - **Results table pivots evaluator × job.** One call prints one comparison table with a column per job; N calls print N unrelated tables you have to diff by eye. - **One experiment on the platform** instead of N, so the orq.ai UI compares the variants for you. - **Identical inputs.** Every job sees the same data points in the same run — the whole point of an A/B. - **Concurrency is lost.** Inside one call, jobs for a data point are dispatched together with `asyncio.gather`; a loop serializes whole passes over the dataset. `parallelism` (default **10**) gates two semaphores: concurrent data points, and concurrent jobs within a data point. Lower it when your provider rate-limits; set `1` for fully sequential execution. This covers variants of one system (prompts, models, flags). For head-to-head comparison of **separate orq.ai agents**, use `orq-compare-agents` — it uses the same multi-job mechanism plus agent-specific setup. ### Library — Python ```python import asyncio from typing import Any from evaluatorq import evaluatorq, job, DataPoint, ScorerParameter @job("MyAgent") async def agent_job(data: DataPoint, _row: int = 0) -> str: # Replace with your actual agent call return "" @job("MyAgent-variant") async def variant_job(data: DataPoint, _row: int = 0) -> str: # Second variant — drop this job if you are evaluating a single system return "" async def quality_scorer(params: ScorerParameter) -> dict[str, Any]: data: DataPoint = params["data"] output = params["output"] # Replace with your scoring logic or orq.ai evaluator call return {"value": 1.0, "explanation": "Looks good"} async def main(): await evaluatorq( "", { "data": {"dataset_id": ""}, # or inline DataPoint list "jobs": [agent_job, variant_job], # every job runs on every data point "evaluators": [{"name": "quality", "scorer": quality_scorer}], "parallelism": 5, }, ) asyncio.run(main()) ``` ### Library — TypeScript ```typescript import type { DataPoint, Evaluator } from "@orq-ai/evaluatorq"; import { evaluatorq, job } from "@orq-ai/evaluatorq"; const agentJob = job("MyAgent", async (data: DataPoint) => { // Replace with your actual agent call return ""; }); // Second variant — drop this job if you are evaluating a single system const variantJob = job("MyAgent-variant", async (data: DataPoint) => { return ""; }); const qualityEvaluator: Evaluator = { name: "quality", scorer: async ({ data, output }) => ({ value: 1.0, explanation: "Looks good", }), }; await evaluatorq("", { data: { datasetId: "" }, // or inline DataPoint array jobs: [agentJob, variantJob], // every job runs on every data point evaluators: [qualityEvaluator], parallelism: 5, }); ``` ### CLI — Red Teaming > **Delegate to the `orq-red-team` skill** for the full `eq redteam` walkthrough (modes, OWASP categories, output format, dashboard). Quick reference: ```bash eq redteam run --target agent: --mode dynamic eq redteam ui report.json # open Streamlit dashboard ``` ### CLI — Simulation > **Delegate to the `orq-simulate-agent` skill** for the full `eq sim` walkthrough (persona generation, scenario setup, goal-achievement scoring). Quick reference: ```bash eq sim generate --agent-description "..." --datapoints dp.jsonl # --datapoints is required eq sim simulate --input dp.jsonl --target agent: ``` --- ## Phase 4: Customize Scorers ### Use an orq.ai LLM-as-a-Judge evaluator ```python from typing import Any from orq_ai_sdk import Orq import os EVALUATOR_ID = "" async def orq_eval_scorer(params: ScorerParameter) -> dict[str, Any]: data: DataPoint = params["data"] output = params["output"] orq = Orq(api_key=os.environ["ORQ_API_KEY"]) result = await orq.evals.invoke_async( # NOTE: evals.invoke_async, NOT evaluators id=EVALUATOR_ID, query=data.inputs["query"], output=str(output), reference=data.expected_output or "", ) return { "value": 1.0 if result.value.value else 0.0, "explanation": result.value.explanation or "", } ``` ### Built-in evaluators (Python) ```python from evaluatorq import string_contains_evaluator, exact_match_evaluator evaluators=[ string_contains_evaluator(case_insensitive=True, name="contains-check"), exact_match_evaluator(name="exact-match"), ] ``` --- ## Phase 5: Run and View Results ### Library ```bash export ORQ_API_KEY="your-key" # Python python evaluate.py # TypeScript npx tsx evaluate.ts ``` Results print to terminal. If `ORQ_API_KEY` is set, results also appear in orq.ai → Experiments. ### CLI — selected flags For full CLI flags and output format, see the `orq-red-team` skill (`eq redteam`) and `orq-simulate-agent` skill (`eq sim`). --- ## Installation | Language | Command | |----------|---------| | Python + CLI (`eq`) | `pip install 'evaluatorq[redteam]'` — installs both the library and the `eq` CLI | | Python orq client (used by scorers that call orq evaluators) | `pip install orq-ai-sdk` — provides `from orq_ai_sdk import Orq` | | TypeScript | `npm install @orq-ai/evaluatorq` | Environment variables: | Variable | Required for | Purpose | |----------|-------------|---------| | `ORQ_API_KEY` | Platform reporting, `--target agent:` | orq.ai API key | | `OPENAI_API_KEY` | `--openai-model` target | OpenAI key | --- ## Resources - **CLI quick reference** (common patterns, eq redteam + eq sim): [resources/cli-reference.md](resources/cli-reference.md) - **evaluatorq API reference** (jobs, scorers, full signatures): See `orq-compare-agents` → [orq-compare-agents/resources/evaluatorq-api.md](../orq-compare-agents/resources/evaluatorq-api.md) ## orq.ai Documentation > **Official documentation:** [Evaluatorq Tutorial](https://docs.orq.ai/docs/tutorials/evaluator-q) [Experiments](https://docs.orq.ai/docs/experiments/creating) · [Evaluators](https://docs.orq.ai/docs/evaluators/overview) · [Datasets](https://docs.orq.ai/docs/datasets/overview) When this skill conflicts with live API responses or docs.orq.ai, trust the API.