# Generated by API Evangelist (build-phrasing.py). Our phrasing, not observed demand. overlay: 1.0.0 info: title: API Evangelist conversational phrasing for OpenAI Evals API version: 1.0.0 extends: openapi/openai-evals-api-openapi.yml actions: - target: $.info update: x-apievangelist-phrasing: method: generated generated: '2026-09-26' generator: build-phrasing.py label: Generated by API Evangelist operations: 12 - target: $.paths['/evals'].get update: x-apievangelist-phrasing: intent: List evaluations in a project effect: read questions: - Which evals have I set up in this project? - Can I sort my evaluations by when they were last updated? instructions: - text: List the evaluations in my project. - text: Show my evals ordered by {order_by}. slots: order_by: query.order_by method: generated generated: '2026-09-26' - target: $.paths['/evals'].post update: x-apievangelist-phrasing: intent: Create an evaluation effect: write questions: - How do I define an eval to test how well a model performs on my data? - What do I need to specify when setting up a new evaluation's grading criteria? instructions: - text: Create an eval named {name} with data source config {data_source_config} and testing criteria {testing_criteria}. slots: name: requestBody.name data_source_config: requestBody.data_source_config testing_criteria: requestBody.testing_criteria - text: Set up a new evaluation using graders {testing_criteria} over data shaped like {data_source_config}. slots: testing_criteria: requestBody.testing_criteria data_source_config: requestBody.data_source_config method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}'].get update: x-apievangelist-phrasing: intent: Retrieve an evaluation effect: read questions: - What testing criteria does one of my evals use? - Can I look up the configuration of an eval by its ID? instructions: - text: Get eval {eval_id}. slots: eval_id: path.eval_id - text: Show the testing criteria and data source config of eval {eval_id}. slots: eval_id: path.eval_id method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}'].post update: x-apievangelist-phrasing: intent: Rename or retag an evaluation effect: write questions: - Can I rename an evaluation after I've created it? - Which properties of an existing eval am I allowed to change? instructions: - text: Rename eval {eval_id} to {name}. slots: eval_id: path.eval_id name: requestBody.name - text: Update the metadata on eval {eval_id} to {metadata}. slots: eval_id: path.eval_id metadata: requestBody.metadata method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}'].delete update: x-apievangelist-phrasing: intent: Delete an evaluation effect: destructive questions: - How do I remove an evaluation I no longer need? - Can I delete an entire eval definition by ID? instructions: - text: Delete eval {eval_id}. slots: eval_id: path.eval_id - text: Permanently remove the evaluation {eval_id} and its definition. slots: eval_id: path.eval_id method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs'].get update: x-apievangelist-phrasing: intent: List the runs of an evaluation effect: read questions: - What runs have been executed against one of my evals? - Can I see only the eval runs that failed or were canceled? instructions: - text: List the runs for eval {eval_id}. slots: eval_id: path.eval_id - text: Show runs of eval {eval_id} with status {status}. slots: eval_id: path.eval_id status: query.status method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs'].post update: x-apievangelist-phrasing: intent: Start an evaluation run effect: write questions: - How do I kick off a run to test a model against an existing eval? - Can I choose which data source and model config an eval run uses? instructions: - text: Start a run of eval {eval_id} using data source {data_source}. slots: eval_id: path.eval_id data_source: requestBody.data_source - text: Kick off an eval run named {name} on eval {eval_id} with data source {data_source}. slots: name: requestBody.name eval_id: path.eval_id data_source: requestBody.data_source method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs/{run_id}'].get update: x-apievangelist-phrasing: intent: Check an evaluation run effect: read questions: - Has my eval run finished, and what were the results? - Can I fetch one evaluation run to see its status and scores? instructions: - text: Get run {run_id} of eval {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id - text: Show the status and results of eval run {run_id} on {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs/{run_id}'].post update: x-apievangelist-phrasing: intent: Cancel an ongoing evaluation run effect: destructive questions: - Can I stop an eval run that's still going? - How do I cancel an evaluation run I started with the wrong config? instructions: - text: Cancel eval run {run_id} on eval {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id - text: Stop the ongoing run {run_id} of evaluation {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs/{run_id}'].delete update: x-apievangelist-phrasing: intent: Delete an evaluation run effect: destructive questions: - Can I delete a single run from an eval while keeping the eval itself? - What's the way to clean up old eval runs I don't need? instructions: - text: Delete eval run {run_id} from eval {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id - text: Remove the old run {run_id} but keep evaluation {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs/{run_id}/output_items'].get update: x-apievangelist-phrasing: intent: List output items from an evaluation run effect: read questions: - Which individual test cases passed or failed in an eval run? - Can I list just the failed output items from a run? instructions: - text: List the output items of run {run_id} on eval {eval_id}. slots: run_id: path.run_id eval_id: path.eval_id - text: Show the {status} output items from eval run {run_id} of {eval_id}. slots: status: query.status run_id: path.run_id eval_id: path.eval_id method: generated generated: '2026-09-26' - target: $.paths['/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}'].get update: x-apievangelist-phrasing: intent: Retrieve one output item from an evaluation run effect: read questions: - What did the model output and how was it graded for one specific test case? - Can I inspect a single eval output item by its ID? instructions: - text: Get output item {output_item_id} from run {run_id} of eval {eval_id}. slots: output_item_id: path.output_item_id run_id: path.run_id eval_id: path.eval_id - text: Show the grading details for output item {output_item_id} in eval run {run_id} on {eval_id}. slots: output_item_id: path.output_item_id run_id: path.run_id eval_id: path.eval_id method: generated generated: '2026-09-26'