openapi: 3.1.0 info: title: Confident AI Metrics API version: 1.0.0 servers: - x-fern-server-name: Confident AI url: https://api.confident-ai.com/ tags: - name: Dashboards description: Dashboards API - name: Metric Collections description: Metric Collection API externalDocs: description: Learn about metric collection - name: Metrics description: Metrics API externalDocs: description: Learn about metrics - name: Datasets description: Confident AI Datasets API externalDocs: description: Learn about datasets - name: Evaluation description: Run your evals using API externalDocs: description: Learn about evaluations - name: Tracing description: Run your evals using API externalDocs: description: Learn about tracing - name: Threads description: Threads API externalDocs: description: Learn about threads - name: Prompt description: Run your evals using API externalDocs: description: Learn about prompts - name: Annotations description: Annotations API externalDocs: description: Learn about annotations - name: Annotation Queues description: Annotation Queues API externalDocs: description: Learn about annotation queues - name: Projects description: Projects API externalDocs: description: Learn about projects - name: Metrics Data description: Metrics Data API externalDocs: description: Learn about metric data - name: Project API Keys x-displayName: API Keys description: Create and manage project-scoped API keys externalDocs: description: Learn about API keys - name: Project Members x-displayName: Members description: Manage the members of a project externalDocs: description: Learn about members - name: Project Invitations x-displayName: Invitations description: Invite users to a project externalDocs: description: Learn about invitations - name: Project Roles x-displayName: Roles description: Manage project roles externalDocs: description: Learn about roles - name: Project Policies x-displayName: Policies description: Manage project policies externalDocs: description: Learn about policies - name: Project Permissions x-displayName: Permissions description: List assignable project permissions externalDocs: description: Learn about permissions - name: Organization API Keys x-displayName: API Keys description: Create and manage organization-scoped API keys externalDocs: description: Learn about API keys - name: Organization description: Manage your organization's settings externalDocs: description: Learn about organizations - name: Organization Members x-displayName: Members description: Manage the members of your organization externalDocs: description: Learn about members - name: Organization Invitations x-displayName: Invitations description: Invite users to your organization externalDocs: description: Learn about invitations - name: Organization Roles x-displayName: Roles description: Manage organization roles externalDocs: description: Learn about roles - name: Organization Policies x-displayName: Policies description: Manage organization policies externalDocs: description: Learn about policies - name: Organization Permissions x-displayName: Permissions description: List assignable organization permissions externalDocs: description: Learn about permissions - name: Organization Governance Policies x-displayName: Governance Policies description: List governance policies and assign projects to them externalDocs: description: Learn about AI governance - name: AI Connections description: List the AI connections in your project externalDocs: description: Learn about AI connections - name: Risk Assessments description: Run red-team risk assessments and list available frameworks externalDocs: description: Learn about risk assessments - name: Legacy description: Deprecated endpoints kept for backward compatibility. Avoid using these in new integrations. externalDocs: description: Learn about legacy endpoints paths: /v1/metrics: get: x-fern-server-name: Confident AI tags: - Metrics summary: List Metrics operationId: getCustomMetric description: Retrieves all the metrics from your Confident AI project. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetMetricResponse" example: success: true data: metrics: - id: "METRIC-ID-1" name: Correctness criteria: Determine if the `actual output` is correct based on the `input`. evaluationSteps: null rubric: null multiTurn: false requiredParameters: - input - actualOutput - id: "METRIC-ID-2" name: Relevancy criteria: Determine if the assistant answers are relevant to what the user is asking. evaluationSteps: null rubric: null multiTurn: true requiredParameters: - role - content deprecated: false post: x-fern-server-name: Confident AI tags: - Metrics summary: Create Metrics operationId: createCustomMetric description: Creates a new metric on your Confident AI project. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreateMetric" examples: Single-Turn: name: Correctness criteria: Determine if the `actual output` is correct based on the `expected output`. evaluationParams: - actualOutput - expectedOutput multiTurn: false Multi-Turn: name: Relevancy criteria: Determine if the assistant answers are relevant to what the user is asking. multiTurn: true With-Rubric: name: Accuracy criteria: Determine if the `actual output` is correct based on the `expected output`. evaluationParams: - actualOutput - expectedOutput multiTurn: false rubric: - scoreRange: - 0 - 2 expectedOutcome: "Factually incorrect." - scoreRange: - 10 - 10 expectedOutcome: "100% correct." responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateMetricResponse" examples: Single-Turn: success: true data: id: METRIC-ID deprecated: false Multi-Turn: success: true data: id: METRIC-ID deprecated: false With-Rubric: success: true data: id: METRIC-ID deprecated: false /v1/metrics/{id}: put: x-fern-server-name: Confident AI tags: - Metrics summary: Update Metrics operationId: updateCustomMetric description: Updates a custom metric on your Confident AI project. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/UpdateMetric" example: criteria: Determine if the `actual output` is correct based on the `input`. evaluationParams: - actualOutput - expectedOutput responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/UpdateMetricResponse" example: success: true deprecated: false /v1/metric/{name}: get: x-fern-server-name: Confident AI tags: - Metrics summary: Pull Metric operationId: pullCustomMetric description: Retrieves a single metric by `name` from your Confident AI project. parameters: - in: path name: name required: true schema: type: string description: The name of the metric. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PullMetricResponse" examples: Single-Turn: value: success: true data: id: "METRIC-ID-1" name: Correctness criteria: Determine if the `actual output` is correct based on the `expected output`. evaluationSteps: null rubric: null multiTurn: false requiredParameters: - actualOutput - expectedOutput deprecated: false Multi-Turn: value: success: true data: id: "METRIC-ID-2" name: Professionalism criteria: Determine whether the assistant has acted professionally throughout the conversation. evaluationSteps: null rubric: null multiTurn: true requiredParameters: - role - content deprecated: false /v1/metrics/batch: post: x-fern-server-name: Confident AI tags: - Metrics summary: Batch Create operationId: createMetricsBatch description: Creates a batch of new metrics on your Confident AI project. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreateBatchMetrics" example: metrics: - name: Correctness criteria: Determine if the `actual output` is correct based on the `expected output`. evaluationParams: - actualOutput - expectedOutput multiTurn: false responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateBatchMetricsResponse" example: success: true data: ids: - METRIC-ID deprecated: false /v1/batch-metrics: post: x-fern-server-name: Confident AI tags: - Legacy summary: Batch Create operationId: createBatchMetrics description: >- Deprecated. Use `POST /v1/metrics/batch` instead. Creates a batch of new metrics on your Confident AI project. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreateBatchMetrics" example: metrics: - name: Correctness criteria: Determine if the `actual output` is correct based on the `expected output`. evaluationParams: - actualOutput - expectedOutput multiTurn: false responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateBatchMetricsResponse" example: success: true data: ids: - METRIC-ID /v1/metric-collections: post: x-fern-server-name: Confident AI tags: - Metric Collections summary: Create Collection operationId: createMetricCollection description: Creates a metric collection with the `name` and the `metricSettings` you specified. requestBody: description: "" required: true content: application/json: schema: $ref: "#/components/schemas/CreateMetricCollection" examples: Single-Turn: name: "Collection Name" multiTurn: false metricSettings: - metric: name: Answer Relevancy threshold: 0.8 Multi-Turn: name: "Collection Name" multiTurn: true metricSettings: - metric: name: Turn Relevancy threshold: 0.8 responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateMetricCollectionResponse" examples: Single-Turn: success: true data: id: "COLLECTION-ID" deprecated: false Multi-Turn: success: true data: id: "COLLECTION-ID" deprecated: false get: x-fern-server-name: Confident AI tags: - Metric Collections summary: List Metric Collections operationId: listMetricCollections description: Lists all the available metric collections in your Confident AI project. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetMetricCollectionResponse" examples: GetMetricCollections: success: true data: metricCollections: - id: "COLLECTION-ID" name: "Collection Name" multiTurn: false metricSettings: - metric: name: "Faithfulness" activated: true threshold: 0.5 includeReason: true strictMode: false /v1/metric-collections/{id}: put: x-fern-server-name: Confident AI tags: - Metric Collections summary: Update Collection operationId: updateMetricCollection description: Updates a metric collection's `name` and `metricSettings`. requestBody: description: "" required: true content: application/json: schema: $ref: "#/components/schemas/UpdateMetricCollection" examples: Single-Turn: name: "New Collection Name" multiTurn: false metricSettings: - metric: name: "Faithfulness" threshold: 0.5 Multi-Turn: name: "New Collection Name" multiTurn: true metricSettings: - metric: name: "Role Adherence" threshold: 0.5 responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/UpdateMetricCollectionResponse" examples: Single-Turn: success: true data: id: "COLLECTION-ID" deprecated: false Multi-Turn: success: true data: id: "COLLECTION-ID" deprecated: false /v1/datasets: get: x-fern-server-name: Confident AI tags: - Datasets summary: List Datasets operationId: listDatasets description: Lists all the available datasets in your Confident AI project. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetDatasetsResponse" examples: GetDatasets: success: true data: datasets: - id: "DATASET-ID" alias: "DATASET-ALIAS" multiTurn: false /v1/datasets/{alias}: get: x-fern-server-name: Confident AI tags: - Datasets summary: Pull Dataset operationId: pullDataset description: Retrieves a list of `Golden`s or `ConversationalGolden`s from your dataset. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetDatasetResponse" examples: Single-Turn: success: true data: id: "DATASET-ID" version: "00.00.01" goldens: - input: "How's the weather like in NYC?" expectedOutput: "No idea" Multi-Turn: success: true data: id: "DATASET-ID" version: "00.00.01" conversationalGoldens: - scenario: "Booking a hotel" expectedOutcome: "Successfully booked" userDescription: "Finds hotels in NYC" post: x-fern-server-name: Confident AI tags: - Datasets summary: Push Dataset operationId: pushDataset description: Pushes a list of `Golden`s or `ConversationalGolden`s to your dataset. requestBody: description: "" required: true content: application/json: schema: $ref: "#/components/schemas/PushDatasetRequestData" examples: Single-Turn: finalized: true goldens: - input: "How is the weather like in NYC?" expectedOutput: "No idea" Multi-Turn: finalized: true conversationalGoldens: - scenario: "Booking a hotel" expectedOutcome: "Successfully booked" userDescription: "Finds hotels in NYC" Custom-Columns: finalized: true goldens: - input: "What is 2 + 2?" customColumnKeyValues: key: "value" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdateGoldenRequestData" examples: Single-Turn: link: "https://app.confident-ai.com/project//datasets/" Multi-Turn: link: "https://app.confident-ai.com/project//datasets/" Custom-Columns: link: "https://app.confident-ai.com/project//datasets/" delete: x-fern-server-name: Confident AI tags: - Datasets summary: Delete Dataset operationId: deleteDataset description: | Permanently deletes a dataset and all its associated data. **Warning:** This action cannot be undone. All goldens or conversational goldens in the dataset will be deleted. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteDatasetResponse" example: success: true data: id: "dataset-id" /v1/datasets/{alias}/versions: get: x-fern-server-name: Confident AI tags: - Datasets summary: List Dataset Versions operationId: getDatasetVersions description: Returns all versions of the dataset, newest first. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetDatasetVersionsResponse" example: success: true data: versions: - id: "abc-123" version: "00.00.02" createdAt: "2026-05-28T13:35:16.268Z" - id: "def-456" version: "00.00.01" createdAt: "2026-05-28T13:05:24.777Z" deprecated: false post: x-fern-server-name: Confident AI tags: - Datasets summary: Create Dataset Version operationId: createDatasetVersion description: | Snapshots the current state of the dataset as a new immutable version. If this is the first version, all existing unversioned goldens are backfilled onto it. Subsequent versions snapshot all goldens from the previous version (with new IDs). responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateDatasetVersionResponse" example: success: true data: id: "abc-123" version: "00.00.02" deprecated: false /v1/datasets/{alias}/queue: post: x-fern-server-name: Confident AI tags: - Datasets summary: Queue Goldens operationId: queueDatasetGoldens description: | Queues `Golden`s or `ConversationalGolden`s to a dataset as unfinalized goldens for later review. If the dataset alias does not exist yet, a new dataset is created automatically. Provide either `goldens` (single-turn) or `conversationalGoldens` (multi-turn), but not both. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/QueueDatasetGoldensRequestData" examples: Single-Turn: goldens: - input: "How is the weather like in NYC?" expectedOutput: "No idea" Multi-Turn: conversationalGoldens: - scenario: "Booking a hotel" expectedOutcome: "Successfully booked" userDescription: "Finds hotels in NYC" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/QueueDatasetGoldensResponse" example: link: "https://app.confident-ai.com/project//datasets/" /v1/evaluate: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Run LLM Evals operationId: evaluateLLM description: Run online evals for your test cases using the metrics in `metricCollection`. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/EvaluateRequest" examples: Single-Turn: metricCollection: "Collection Name" llmTestCases: - input: "How tall is mount everest?" actualOutput: "No clue, pretty tall I guess?" Multi-Turn: metricCollection: "Multi-Turn Collection Name" conversationalTestCases: - turns: - role: user content: "How tall is Mount Everest?" - role: assistant content: "Mount Everest is approximately 8,848 meters tall." - role: user content: "Wow, that is really high! Has that changed recently?" - role: assistant content: "Yes, a 2020 survey by China and Nepal revised the height to 8,848.86 meters." Logging Parameters: metricCollection: "Collection Name" llmTestCases: input: "How tall is mount everest?" actualOutput: "No clue, pretty tall I guess?" hyperparameters: model: "gpt-4o-mini" prompt-version: "ai_generation_v2" Logging Identifier: metricCollection: "Collection Name" llmTestCases: input: "How tall is mount everest?" actualOutput: "No clue, pretty tall I guess?" name: "Your Test Case Name" identifier: "run-399-102" responses: "200": content: application/json: schema: $ref: "#/components/schemas/EvaluateResponse" examples: Single-Turn: success: true data: id: "TEST-RUN-ID" deprecated: false Multi-Turn: success: true data: id: "TEST-RUN-ID" deprecated: false Logging Parameters: success: true data: id: "TEST-RUN-ID" deprecated: false Logging Identifier: success: true data: id: "TEST-RUN-ID" deprecated: false /v1/simulate: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Simulate Conversation operationId: simulateConversation description: Simulate the next conversation turn from a conversational golden. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/SimulateRequest" examples: SimulationExample: conversationalGolden: scenario: "A frustrated user asking for a refund." userDescription: "A white male who is a customer for over 2 years." responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/SimulateResponse" examples: SimulationExample: success: true data: simulationId: "SIMULATION-ID" completed: false userResponse: "I'd like my refund please." turns: - role: "assistant" content: "Hey, how can I help you today?" - role: "user" content: "I'd like my refund please." deprecated: false /v1/evaluate/spans/{spanUuid}: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Evaluate Span operationId: evaluateSpan requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/EvaluateSpanRequest" examples: EvaluateSpan: metricCollection: "Collection Name" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/EvaluateSpanResponse" examples: EvaluateSpan: success: true data: id: "SPAN-ID" deprecated: false /v1/evaluate/traces/{traceUuid}: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Evaluate Trace operationId: evaluateTrace requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/EvaluateTraceRequest" examples: EvaluateTrace: metricCollection: "Collection Name" responses: "200": description: Successful response from evaluate trace API content: application/json: schema: $ref: "#/components/schemas/EvaluateTraceResponse" examples: EvaluateTrace: success: true data: id: "TRACE-ID" deprecated: false /v1/evaluate/threads/{threadId}: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Evaluate Thread description: Triggers evaluation of a thread using a specified metric collection and optional chatbot role instructions. operationId: evaluateThread requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/EvaluateThreadRequest" examples: EvaluateThread: metricCollection: "Collection Name" chatbotRole: "You are a rich, powerful..." responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/EvaluateThreadResponse" examples: EvaluateThread: success: true data: id: "THREAD-ID" deprecated: false /v1/governance/assess: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Assess Governance operationId: assessGovernance description: Assesses all governance controls for the project against the governance policy it belongs to, and returns whether every control passed. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/AssessGovernanceResponse" examples: AssessGovernance: success: true data: passed: false governancePolicy: id: "GOVERNANCE-POLICY-ID" name: "EU AI Act" deprecated: false /v1/test-runs/evaluate/{testCaseId}: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Submit Test Case Result operationId: submitTestCaseResult description: >- Submit the result for a single test case in a long-running agent evaluation. Confident AI automatically evaluates the test case and finalizes the test run once every result has been received. Long-running mode is available for single-turn AI connection evaluations only. requestBody: description: "" required: true content: application/json: schema: $ref: "#/components/schemas/SubmitTestCaseResultRequest" example: actualOutput: "The capital of France is Paris." responses: "200": description: The result was accepted for evaluation. content: application/json: schema: $ref: "#/components/schemas/SubmitTestCaseResultResponse" example: success: true data: testCaseId: "" status: "accepted" "400": description: >- Invalid request body, the test run has no metric collection, or the test run is conversational (long-running mode is single-turn only). "404": description: No test run was found for the given testCaseId in this project. "409": description: The test run is no longer accepting results (already finalized). "410": description: >- Unknown or expired testCaseId — the result window for this test case has closed. /v1/traces: post: x-fern-server-name: Confident AI tags: - Tracing summary: Trace Ingestion operationId: createTrace description: Creates a new trace on Confident AI. requestBody: description: "" required: true content: application/json: schema: $ref: "#/components/schemas/Trace" examples: Base-Span: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" baseSpans: - uuid: "" name: "Agent" input: "What is the capital of France?" output: "Let me look that up for you." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:02Z" LLM-Span: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" llmSpans: - uuid: "" name: "OpenAI Call" provider: "OpenAI" integration: "LangChain" model: "gpt-4o" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:02Z" Test-Run-Id: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" testRunId: "" metricCollection: "Collection Name" baseSpans: - uuid: "" name: "Agent" input: "What is the capital of France?" output: "Let me look that up for you." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:02Z" Test-Run-Component-Level: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" testRunId: "" metricCollection: "Collection Name" retrieverSpans: - uuid: "" name: "retrieve_context" embedder: "text-embedding-3-small" input: "capital of France" retrievalContext: - "Paris is the capital and most populous city of France." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:01Z" metricCollection: "Retriever Collection Name" llmSpans: - uuid: "" parentUuid: "" name: "generate_answer" model: "gpt-4o" input: "Answer using the retrieved context." output: "The capital of France is Paris." startTime: "2025-01-15T10:30:01Z" endTime: "2025-01-15T10:30:05Z" metricCollection: "LLM Collection Name" Thread-Id: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" threadId: "" baseSpans: - uuid: "" name: "Agent" input: "What is the capital of France?" output: "Let me look that up for you." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:02Z" Thread-Metadata: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" threadId: "" thread: id: "" metadata: dvaVersion: "1.4.2" client: "acme-corp" agentId: "support-agent" tags: - "vip" - "billing" baseSpans: - uuid: "" name: "Agent" input: "What is the capital of France?" output: "Let me look that up for you." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:02Z" User-Id: uuid: "" input: "What is the capital of France?" output: "The capital of France is Paris." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" userId: "" baseSpans: - uuid: "" name: "Agent" input: "What is the capital of France?" output: "Let me look that up for you." startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:02Z" Multimodal-Attachments: uuid: "" name: "PDF summarization" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" status: "SUCCESS" input: messages: - "Summarize this document: [DEEPEVAL:PDF:doc-id-1]" attachments: doc-id-1: mimeType: "application/pdf" dataBase64: "" llmSpans: - uuid: "" name: "LLM" model: "gpt-4o" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" status: "SUCCESS" input: text: "User attached [DEEPEVAL:PDF:doc-1]" output: "Summary of the document..." responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateTraceResponse" examples: Base-Span: success: true data: id: "TRACE-ID" link: "https://app.confident-ai.com/project//observatory/traces/TRACE-ID" deprecated: false LLM-Span: success: true data: id: "TRACE-ID" link: "https://app.confident-ai.com/project//observatory/traces/TRACE-ID" deprecated: false Thread-Id: success: true data: id: "TRACE-ID" link: "https://app.confident-ai.com/project//observatory/traces/TRACE-ID" deprecated: false Thread-Metadata: success: true data: id: "TRACE-ID" link: "https://app.confident-ai.com/project//observatory/traces/TRACE-ID" deprecated: false User-Id: success: true data: id: "TRACE-ID" link: "https://app.confident-ai.com/project//observatory/traces/TRACE-ID" deprecated: false get: x-fern-server-name: Confident AI tags: - Tracing summary: List Traces operationId: getTrace description: Retrieves a list of traces from your Confident AI project. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetTraceResponse" example: success: true data: traces: name: "Trace Name" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" uuid: "TRACE-UUID" projectId: "PROJECT-ID" environment: "production" threadId: "THREAD-ID" userId: "USER-ID" tags: - "General QA" totalTraces: 1 deprecated: false /v1/traces/{traceUuid}: get: x-fern-server-name: Confident AI tags: - Tracing summary: Retrieve Trace operationId: fetchTrace description: Retrieves an existing trace on your Confident AI project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: traceUuid required: true schema: type: string description: This is the trace UUID you wish to retrieve. responses: responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/RetrieveTraceResponse" example: success: true data: uuid: "TRACE-ID" name: "Trace Name" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" projectId: "PROJECT-ID" environment: "production" spans: - id: "SPAN-ID" uuid: "SPAN-UUID" name: "Span Name" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" traceUuid: "TRACE-UUID" status: "SUCCESS" type: "LLM" provider: "OpenAI" integration: "LangChain" model: "gpt-4o" deprecated: false /v1/spans: get: x-fern-server-name: Confident AI tags: - Tracing summary: List Spans operationId: getSpan description: Retrieves a list of spans from your Confident AI project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the threads to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 description: This specifies the maximum number of threads per page. Defaulted to 25. - in: query name: type description: Filter by the specific type of span. schema: type: string enum: [SPAN, AGENT, TOOL, RETRIEVER, LLM] - in: query name: traceUuid description: Filter spans that belong to a specific trace UUID. schema: type: string format: uuid - in: query name: name description: Filter spans by their exact name. schema: type: string - in: query name: hasError description: Filter for spans that either failed (true) or succeeded (false). schema: type: boolean - in: query name: model description: Filter LLM spans by the model used (e.g., 'gpt-4'). schema: type: string - in: query name: promptAlias required: false schema: type: string description: This filters the spans by the prompt alias used. - in: query name: promptCommitHash required: false schema: type: string description: This filters the spans by the exact prompt commit hash used. - in: query name: promptVersion required: false schema: type: string description: This filters the spans by the prompt version used. - in: query name: promptLabel required: false schema: type: string description: This filters the spans by the prompt label used. - in: query name: embedder description: Filter retriever spans by the embedder model used. schema: type: string - in: query name: topK description: Filter retriever spans by the topK value. schema: type: integer minimum: 0 - in: query name: chunkSize description: Filter retriever spans by the chunk size. schema: type: integer minimum: 0 - in: query name: environment required: false schema: type: string description: This filters the threads by the environment where the thread was created, and returns threads from all environments if not specified. - in: query name: start required: false schema: type: string format: date-time description: This filters for threads created after the specified start datetime. Defaulted to 30 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for threads created before the specified end datetime. Defaulted to the current time. - in: query name: sortBy required: false schema: type: string enum: [createdAt, lastActivity] description: This determines the field to sort by. Defaulted to `lastActivity`. - in: query name: ascending required: false schema: type: string enum: ["true", "false"] description: This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListSpansResponse" example: success: true data: spans: - id: "SPAN-ID" uuid: "SPAN-UUID" name: "Span Name" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" traceUuid: "TRACE-UUID" status: "SUCCESS" type: "LLM" provider: "OpenAI" integration: "LangChain" model: "gpt-4o" totalSpans: 1 deprecated: false /v1/spans/{spanUuid}: get: x-fern-server-name: Confident AI tags: - Tracing summary: Retrieve Span operationId: retrieveSpan description: Retrieves a single, detailed span by its unique identifier. parameters: - name: spanUuid in: path required: true description: The unique UUID of the span you want to retrieve. schema: type: string format: uuid responses: "200": description: Successfully retrieved the detailed span. content: application/json: schema: $ref: "#/components/schemas/RetrieveSpanResponse" examples: LLMSpan: success: true data: id: "SPAN-ID" uuid: "SPAN-UUID" name: "Span Name" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" traceUuid: "TRACE-UUID" status: "SUCCESS" type: "LLM" provider: "OpenAI" integration: "LangChain" model: "gpt-4o" promptAlias: "Prompt Alias" deprecated: false /v1/threads: get: x-fern-server-name: Confident AI tags: - Threads summary: List Threads operationId: listThreads description: Retrieves a list of threads from your Confident AI project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the threads to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 description: This specifies the maximum number of threads per page. Defaulted to 25. - in: query name: environment required: false schema: type: string description: This filters the threads by the environment where the thread was created, and returns threads from all environments if not specified. - in: query name: start required: false schema: type: string format: date-time description: This filters for threads created after the specified start datetime. Defaulted to 30 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for threads created before the specified end datetime. Defaulted to the current time. - in: query name: sortBy required: false schema: type: string enum: [createdAt, lastActivity] description: This determines the field to sort by. Defaulted to `lastActivity`. - in: query name: ascending required: false schema: type: string enum: ["true", "false"] description: This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListThreadsResponse" example: success: true data: threads: - threadId: "thread-123" createdAt: "2025-01-15T10:30:00Z" lastActivity: "2025-01-15T11:45:00Z" metadata: userId: "user-456" tags: - "support" metricCollectionName: "conversation-metrics" totalTraces: 5 totalThreads: 100 deprecated: false /v1/threads/{threadId}: get: x-fern-server-name: Confident AI tags: - Threads summary: Retrieve Thread operationId: retrieveThread description: Retrieves a thread by ID from your Confident AI project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: threadId required: true schema: type: string description: This is the thread ID you wish to retrieve. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetThreadResponse" example: success: true data: threadId: "thread-123" createdAt: "2025-01-15T10:30:00Z" lastActivity: "2025-01-15T11:45:00Z" metadata: userId: "user-456" tags: - "support" metricCollectionName: "conversation-metrics" totalTraces: 5 metricsData: - metricName: "Answer Relevancy" score: 0.95 annotations: - id: "annotation-1" rating: 5 traces: - id: "TRACE-ID" uuid: "trace-uuid-1" name: "User Message" startTime: "2025-01-15T10:30:00Z" endTime: "2025-01-15T10:30:05Z" input: "Hello, I need help" output: "Hi! How can I assist you today?" environment: "production" deprecated: false /v1/prompts: get: x-fern-server-name: Confident AI tags: - Prompt summary: List Prompts operationId: listPrompts description: Lists all the available prompts in your Confident AI project. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetPromptsResponse" examples: GetPrompts: success: true data: prompts: - id: "PROMPT-ID" alias: "PROMPT-ALIAS" type: "TEXT" post: x-fern-server-name: Confident AI tags: - Prompt summary: Push Prompts operationId: pushPrompt description: Creates a new commit for an existing prompt, or creates a new prompt with the given `alias` otherwise. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreatePromptRequest" examples: Text-Prompt: alias: "Prompt Name" text: "Hello, {{name}}!" interpolationType: "FSTRING" outputType: "TEXT" List-Prompt: alias: "Prompt Name" messages: - role: user content: "What is the weather like in {{city}}?" interpolationType: "FSTRING" outputType: "TEXT" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreatePromptResponse" examples: Text-Prompt: success: true data: promptId: "prm_xyz456" hash: "bab04ce" link: "https://app.confident-ai.com/project//prompt-studio/editor/prm_xyz456" deprecated: false List-Prompt: success: true data: promptId: "prm_xyz456" hash: "bab04ce" link: "https://app.confident-ai.com/project//prompt-studio/editor/prm_xyz456" deprecated: false /v1/prompts/{alias}/labels/{label}: get: x-fern-server-name: Confident AI tags: - Prompt summary: Pull Prompts By Label operationId: getPromptByLabel description: Retrieves a prompt with `alias` and `label` from your Confident AI account. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetPromptResponse" examples: Text-Example: value: success: true data: id: "prv_abc123" version: "00.00.02" label: "my-label" hash: "bab04ce" text: "Hello, {{name}}!" type: "TEXT" interpolationType: "FSTRING" outputType: "TEXT" modelSettings: provider: "OPEN_AI" name: "gpt-4o" temperature: 0.7 deprecated: false Message-Example: value: success: true data: id: "prv_abc123" version: "00.00.02" label: "my-label" hash: "bab04ce" messages: - role: system content: "Hello, {{name}}" type: "LIST" interpolationType: "FSTRING" outputType: "TEXT" modelSettings: provider: "OPEN_AI" name: "gpt-4o" temperature: 0.7 deprecated: false /v1/prompts/{alias}/versions/{version}: get: x-fern-server-name: Confident AI tags: - Prompt summary: Pull Prompts By Version operationId: getPromptByVersion description: Retrieves a prompt with `alias` and `version` from your Confident AI account. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetPromptResponse" examples: Text-Example: value: success: true data: id: "prv_abc123" version: "00.00.01" label: "my-label" hash: "bab04ce" text: "Hello, {{name}}!" type: "TEXT" interpolationType: "FSTRING" outputType: "TEXT" modelSettings: provider: "OPEN_AI" name: "gpt-4o" temperature: 0.7 deprecated: false Message-Example: value: success: false data: id: "prv_abc123" version: "00.00.02" label: "my-label" hash: "bab04ce" messages: - role: system content: "Hello, {{name}}" type: "LIST" interpolationType: "FSTRING" outputType: "TEXT" modelSettings: provider: "OPEN_AI" name: "gpt-4o" temperature: 0.7 deprecated: false /v1/prompts/{alias}/commits/{hash}: get: x-fern-server-name: Confident AI tags: - Prompt summary: Pull Prompts By Commit operationId: getPromptByCommit description: Retrieves a prompt with `alias` and the specific `hash` from your Confident AI account. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetPromptResponse" examples: Text-Example: value: success: true data: id: "prv_abc123" version: "00.00.01" label: "my-label" hash: "bab04ce" text: "Hello, {{name}}!" type: "TEXT" interpolationType: "FSTRING" outputType: "TEXT" modelSettings: provider: "OPEN_AI" name: "gpt-4o" temperature: 0.7 deprecated: false Message-Example: value: success: false data: id: "prv_abc123" version: "00.00.02" label: "my-label" hash: "bab04ce" messages: - role: system content: "Hello, {{name}}" type: "LIST" interpolationType: "FSTRING" outputType: "TEXT" modelSettings: provider: "OPEN_AI" name: "gpt-4o" temperature: 0.7 deprecated: false /v1/prompts/{alias}/versions: get: x-fern-server-name: Confident AI tags: - Prompt summary: List Versions operationId: getPromptVersions description: Retrieves a list of all versions associated with a given prompt `alias`. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PromptVersionsResponse" example: success: true data: textVersions: - id: "prm_xyz456" version: "00.00.01" messagesVersions: null deprecated: false post: x-fern-server-name: Confident AI tags: - Prompt summary: Create Version operationId: createVersion description: Creates a new version of the specified or latest commit on your Confident AI project. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreatePromptVersionRequest" examples: With-Hash: hash: "bab04ce" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreatePromptVersionResponse" examples: With-Hash: success: true data: version: "00.00.01" hash: "bab04ce" deprecated: false Without-Hash: success: true data: version: "00.00.01" hash: "9ace048" deprecated: false /v1/prompts/{alias}/commits: get: x-fern-server-name: Confident AI tags: - Prompt summary: List Commits operationId: getPromptCommits description: Retrieves a list of all the commits associated with a prompt responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PromptCommitsResponse" example: success: true data: commits: - id: "144aa01d-af4d-4054-83fa-78e2301d94fb" hash: "bab04ce" message: "Committed from API" deprecated: false /v1/prompts/{alias}/branches: get: x-fern-server-name: Confident AI tags: - Prompt summary: List Branches operationId: getPromptBranches description: Retrieves a list of all the branches associated with a prompt responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListPromptBranchesResponse" example: success: true data: branches: - id: "br_123456" name: "main" - id: "br_123789" name: "NewBranch" deprecated: false post: x-fern-server-name: Confident AI tags: - Prompt summary: Create Branch operationId: createPromptBranch description: Creates a new branch with the specified name diverging from the main branch's head commit of your prompt. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreatePromptBranchRequest" example: branch: "NewBranch" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreatePromptBranchResponse" examples: With-Hash: success: true data: name: "NewBranch" id: "br_123456" link: "https://app.confident-ai.com/project//prompt-studio/?branch=NewBranch" deprecated: false /v1/prompts/{alias}/branches/{name}: put: x-fern-server-name: Confident AI tags: - Prompt summary: Update Branch operationId: updatePromptBranch description: Updates the name of an existing branch in your prompt on Confident AI. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/UpdatePromptBranchRequest" example: name: "RenamedBranch" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/UpdatePromptBranchResponse" example: success: true data: id: "br_123456" deprecated: false delete: x-fern-server-name: Confident AI tags: - Prompt summary: Delete Branch operationId: deletePromptBranch description: Deletes an existing branch from your prompt on Confident AI. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeletePromptBranchResponse" example: success: true data: id: "br_123456" deprecated: false /v1/ai-connections: get: x-fern-server-name: Confident AI tags: - AI Connections summary: List AI Connections operationId: listAIConnections description: Lists the AI connections in your project. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListAIConnectionsResponse" example: success: true data: aiConnections: - id: "AI-CONNECTION-ID" name: "Production Chatbot" endpoint: "https://api.example.com/chat" active: true /v1/risk-assessments/frameworks: get: x-fern-server-name: Confident AI tags: - Risk Assessments summary: List Frameworks operationId: listRiskAssessmentFrameworks description: | Lists the red-team frameworks in your project along with their risk categories. Use the returned risk category names when [running a risk assessment](https://www.confident-ai.com/docs/api-reference/run-assessments/run-risk-assessment). Requires an Enterprise plan. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListRiskAssessmentFrameworksResponse" example: success: true data: frameworks: - id: "FRAMEWORK-ID" name: "OWASP Top 10 for LLMs" description: "Standard LLM risk framework" riskCategories: - name: "Prompt Injection" numVulnerabilityTypes: 4 numAttackMethods: 6 /v1/risk-assessments/frameworks/{frameworkId}/run: post: x-fern-server-name: Confident AI tags: - Risk Assessments summary: Run Risk Assessment operationId: runRiskAssessment description: | Starts a risk assessment run asynchronously against a framework's risk categories, then returns a link to the assessment. Target your application with either an AI connection (by name) or a prompt (by alias and commit). Provide exactly one, or set `generationMode` to disambiguate. Requires an Enterprise plan. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/RunRiskAssessmentRequest" examples: AI Connection: value: identifier: "Weekly production scan" riskCategories: ["Prompt Injection", "PII Leakage"] exposure: "MEDIUM" aiConnectionId: "AI-CONNECTION-ID" Prompt: value: riskCategories: ["Prompt Injection"] exposure: "HIGH" promptAlias: "support-agent" promptCommit: "bab04ce" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/RunRiskAssessmentResponse" example: success: true data: id: "RISK-ASSESSMENT-ID" link: "https://app.confident-ai.com/project//risk-profile/assessments/" /v1/test-runs/{testRunId}: get: x-fern-server-name: Confident AI tags: - Evaluation summary: Retrieve Test Run operationId: getTestRun description: Retrieves a list of test cases and their respective metrics from a test run. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/TestRunResponse" examples: Single-Turn: value: success: true data: metricsScores: - metric: "Answer Correctness" scores: [0] passes: 0 fails: 1 errors: 0 traceMetricsScores: [] conversational: false identifier: "test-run-001" status: "COMPLETED" testsPassed: 0 testsFailed: 1 totalTests: 1 runDuration: 1.5 testCases: - id: "TEST-CASE-ID" name: "Test Case 1" input: "What's the capital of France?" expectedOutput: "Paris" actualOutput: "San Francisco" success: false context: ["The capital of France is Paris."] retrievalContext: ["The capital of France is Paris."] runDuration: 1.2 evaluationCost: 0.001 metricsData: - id: "METRIC-ID" score: 0 reason: "The capital of France is Paris." success: false threshold: 0.5 evaluationModel: "gpt-4.1" strictMode: false evaluationCost: 0.001 name: "Answer Correctness" verboseLogs: "..." deprecated: false Multi-Turn: value: success: true data: metricsScores: - metric: "Conversation Completeness" scores: [0.85] passes: 1 fails: 0 errors: 0 traceMetricsScores: [] conversational: true identifier: "test-run-002" status: "COMPLETED" testsPassed: 1 testsFailed: 0 totalTests: 1 runDuration: 2.3 testCases: - id: "CONV-TEST-CASE-ID" name: "Mount Everest Conversation" scenario: "User asking about Mount Everest" expectedOutcome: "Assistant provides accurate height information" userDescription: "A curious user asking geography questions" context: ["Mount Everest is 8,848 meters tall."] success: true runDuration: 2.1 evaluationCost: 0.002 turns: - role: "user" content: "How tall is Mount Everest?" - role: "assistant" content: "Mount Everest is approximately 8,848 meters (29,029 feet) tall." metricsData: - id: "METRIC-ID" score: 0.85 reason: "The assistant provided accurate information about Mount Everest's height." success: true threshold: 0.7 evaluationModel: "gpt-4.1" strictMode: false evaluationCost: 0.002 name: "Conversation Completeness" verboseLogs: "..." deprecated: false /v1/metrics-data: get: x-fern-server-name: Confident AI tags: - Metrics Data summary: List Metrics Data operationId: listMetricsData description: | Retrieves a paginated list of metric data (evaluation results) from your Confident AI project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the metrics data to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 maximum: 100 description: This specifies the maximum number of metrics data per page. Defaulted to 25, capped at 100. - in: query name: start required: false schema: type: string format: date-time description: This filters for metrics data created after the specified start datetime. Defaulted to 60 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for metrics data created before the specified end datetime. Defaulted to the current time. - in: query name: sortBy required: false schema: type: string enum: [createdAt] description: This determines the field to sort by. Defaulted to `createdAt`. - in: query name: ascending required: false schema: type: string enum: ["true", "false"] description: This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`. responses: "200": description: Successfully retrieved list of metrics data content: application/json: schema: $ref: "#/components/schemas/ListMetricsDataResponse" example: success: true data: metricsData: - id: "metric-data-uuid-1" projectId: "PROJECT-ID" traceUuid: null spanUuid: null testCaseId: "test-case-uuid-1" testRunId: null threadId: null name: "answer_relevancy" multiTurn: false score: 0.85 reason: "The answer is relevant to the question" success: true createdAt: "2025-11-12T10:30:00.000Z" evaluatedAt: "2025-11-12T10:30:02.000Z" threshold: 0.7 strictMode: false skipped: false evaluationModel: "gpt-4o" error: null evaluationCost: 0.002 verboseLogs: null updatedAt: "2025-11-12T10:30:02.000Z" - id: "metric-data-uuid-2" projectId: "PROJECT-ID" traceUuid: null spanUuid: null testCaseId: "test-case-uuid-2" testRunId: null threadId: null name: "faithfulness" multiTurn: false score: 0.92 reason: "The output is faithful to the context" success: true createdAt: "2025-11-12T09:15:00.000Z" evaluatedAt: "2025-11-12T09:15:03.000Z" threshold: 0.8 strictMode: false skipped: false evaluationModel: "gpt-4o" error: null evaluationCost: 0.003 verboseLogs: null updatedAt: "2025-11-12T09:15:03.000Z" totalMetricsData: 150 page: 1 pageSize: 25 /v1/metric-data: get: x-fern-server-name: Confident AI tags: - Legacy summary: List Metric Data operationId: listMetricData description: >- Deprecated. Use `GET /v1/metrics-data` instead. Retrieves a paginated list of metric data (evaluation results) from your Confident AI project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the metrics data to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 maximum: 100 description: This specifies the maximum number of metrics data per page. Defaulted to 25, capped at 100. - in: query name: start required: false schema: type: string format: date-time description: This filters for metrics data created after the specified start datetime. Defaulted to 60 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for metrics data created before the specified end datetime. Defaulted to the current time. responses: "200": description: Successfully retrieved list of metrics data content: application/json: schema: $ref: "#/components/schemas/ListMetricsDataResponse" example: success: true data: metricsData: - id: "metric-data-uuid-1" projectId: "PROJECT-ID" traceUuid: null spanUuid: null testCaseId: "test-case-uuid-1" testRunId: null threadId: null name: "answer_relevancy" multiTurn: false score: 0.85 reason: "The answer is relevant to the question" success: true createdAt: "2025-11-12T10:30:00.000Z" evaluatedAt: "2025-11-12T10:30:02.000Z" threshold: 0.7 strictMode: false skipped: false evaluationModel: "gpt-4o" error: null evaluationCost: 0.002 verboseLogs: null updatedAt: "2025-11-12T10:30:02.000Z" totalMetricsData: 150 page: 1 pageSize: 25 /v1/test-runs: post: x-fern-server-name: Confident AI tags: - Evaluation summary: Create Test Run operationId: createTestRun description: >- Creates a new in-progress test run and returns its id. Use this id as the `testRunId` when ingesting traces (`POST /v1/traces`) so that each trace becomes one test case in this run. requestBody: description: "" required: false content: application/json: schema: $ref: "#/components/schemas/CreateTestRun" example: identifier: "my-test-run" metricCollection: "Agent Quality" responses: "200": description: The created test run id. content: application/json: schema: $ref: "#/components/schemas/CreateTestRunResponse" example: success: true data: id: "" link: "https://app.confident-ai.com/project//test-runs/" deprecated: false get: x-fern-server-name: Confident AI tags: - Evaluation summary: List Test Runs operationId: listTestRuns description: >- Retrieves a paginated list of test runs for the authorized project. Results can be filtered and sorted by status, dates, conversational (multi-turn) or metrics. Pagination is controlled with `page` and `pageSize`. Filtering and sorting options include `status`, `multiTurn`, `sortField`, and `ascending`. responses: "200": description: A paginated list of test runs and total count. content: application/json: schema: type: object properties: success: type: boolean example: true data: type: object properties: testRuns: type: array items: type: object properties: id: type: string description: Test run unique identifier. createdAt: type: string format: datetime identifier: type: string nullable: true description: Optional test run identifier. status: type: string enum: [COMPLETED, ERRORED, IN_PROGRESS, CANCELLED] multiTurn: type: boolean description: Whether the test run is conversational. testsPassed: type: integer nullable: true testsFailed: type: integer nullable: true totalTests: type: integer nullable: true metricsScores: type: array items: type: object properties: metric: type: string scores: type: array items: type: number nullable: true passes: type: integer nullable: true fails: type: integer nullable: true errors: type: integer nullable: true runDuration: type: number nullable: true evaluationCost: type: number nullable: true datasetAlias: type: string nullable: true testFile: type: string nullable: true summary: type: object nullable: true properties: topicSummaries: type: array items: type: object properties: topic: type: string testCaseIds: type: array items: oneOf: - type: integer - type: string summaryPoints: type: array items: type: object properties: grade: type: number content: type: string testCaseIds: type: array items: oneOf: - type: integer - type: string summaryOverview: type: object properties: summary: type: array items: type: string actionItems: type: array items: type: string totalTestRuns: type: integer description: Total number of test runs matching filters. page: type: integer example: 1 description: Current page number. pageSize: type: integer example: 25 description: Page size. required: - success - data example: success: true data: testRuns: - id: "342bd7a6" createdAt: "2024-06-01T12:34:56Z" identifier: "run042" status: "COMPLETED" multiTurn: false testsPassed: 8 testsFailed: 2 totalTests: 10 metricsScores: - metric: "Answer Correctness" scores: [0.9, 1.0] passes: 8 fails: 2 errors: 0 runDuration: 15.2 evaluationCost: 0.254 datasetAlias: "agent" testFile: "test-file-1.jsonl" summary: "8/10 passed" totalTestRuns: 113 page: 1 pageSize: 25 /v1/annotations: get: x-fern-server-name: Confident AI tags: - Annotations summary: List Annotations operationId: listAnnotations description: | Retrieves a paginated list of annotations from your Confident AI project. Annotations are user feedback on traces, spans, or threads that include ratings, expected outputs/outcomes, and explanations. Use query parameters to filter annotations by trace UUID, span UUID, thread ID, type, or rating range. Results are returned in descending order by creation date with pagination support. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: query name: traceUuid schema: type: string description: Filter annotations by trace UUID. - in: query name: spanUuid schema: type: string description: Filter annotations by span UUID. - in: query name: threadId schema: type: string description: Filter annotations by thread ID. - in: query name: type schema: type: string enum: [THUMBS_RATING, FIVE_STAR_RATING] description: Filter annotations by type. - in: query name: minRating schema: type: string description: Filter annotations with minimum rating (inclusive). - in: query name: maxRating schema: type: string description: Filter annotations with maximum rating (inclusive). - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the annotations to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 description: This specifies the maximum number of annotations per page. Defaulted to 25. - in: query name: start required: false schema: type: string format: date-time description: This filters for annotations created after the specified start datetime. Defaulted to 30 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for annotations created before the specified end datetime. Defaulted to the current time. - in: query name: sortBy required: false schema: type: string enum: [createdAt] description: This determines the field to sort by. Defaulted to `createdAt`. - in: query name: ascending required: false schema: type: string enum: ["true", "false"] description: This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`. responses: "200": description: Successfully retrieved list of annotations content: application/json: schema: $ref: "#/components/schemas/ListAnnotationsResponse" example: success: true data: annotations: - id: "annotation-uuid-1" rating: 1 type: "THUMBS_RATING" name: "Quality Check" expectedOutput: "Provide clear password reset instructions" explanation: "Response correctly guides user through password reset" createdAt: "2025-11-12T10:30:00Z" traceUuid: "trace-uuid-1" user: id: "user-uuid-1" email: "user1@example.com" name: "User One" image: "https://example.com/user1.png" - id: "annotation-uuid-2" rating: 5 type: "FIVE_STAR_RATING" name: "Customer Service" expectedOutcome: "Agent successfully resolves user issue" explanation: "Excellent resolution with follow-up" createdAt: "2025-11-12T09:15:00Z" threadId: "thread-id-1" user: id: "user-uuid-2" email: "user2@example.com" name: "User Two" image: "https://example.com/user2.png" total: 150 page: 1 pageSize: 25 post: x-fern-server-name: Confident AI tags: - Annotations summary: Create Annotation operationId: createAnnotation x-fern-examples: - name: Thumbs Rating headers: CONFIDENT_API_KEY: "" request: traceUuid: "" rating: 1 type: "THUMBS_RATING" expectedOutput: "Provide a clear, step-by-step password reset flow." explanation: "Response acknowledges issue and guides the user through reset steps." response: body: success: true data: id: "ANNOTATION-ID" deprecated: false - name: Five Star Rating headers: CONFIDENT_API_KEY: "" request: spanUuid: "" rating: 5 type: "FIVE_STAR_RATING" expectedOutput: "Include verification, link to reset page, and confirmation messaging." explanation: "High-quality answer with actionable steps and safety considerations." response: body: success: true data: id: "ANNOTATION-ID" deprecated: false description: | Creates a new annotation for a trace, span, or thread in your Confident AI project. Annotations capture human feedback including ratings, expected outputs/outcomes, and explanations. **Important validation rules:** - For traces and spans: Use `expectedOutput` (not `expectedOutcome`) - For threads: Use `expectedOutcome` (not `expectedOutput`) - Rating must be 0 or 1 for THUMBS_RATING, or 1-5 for FIVE_STAR_RATING - You must provide either `traceUuid`, `spanUuid`, or `threadId` parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateAnnotation" responses: "200": description: Successfully created annotation content: application/json: schema: $ref: "#/components/schemas/CreateAnnotationResponse" /v1/annotations/{annotationId}: get: x-fern-server-name: Confident AI tags: - Annotations summary: Get Annotation operationId: getAnnotation description: | Retrieves a specific annotation by its unique ID. Returns the complete annotation details including rating, type, expected output/outcome, explanation, and associated trace/span/thread identifiers. This endpoint verifies that the annotation belongs to your project before returning it. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: annotationId required: true schema: type: string description: The ID of the annotation to retrieve. responses: "200": description: Successfully retrieved annotation content: application/json: schema: $ref: "#/components/schemas/GetAnnotationResponse" example: success: true data: annotation: id: "annotation-uuid" rating: 1 type: "THUMBS_RATING" name: "Quality Check" expectedOutput: "Provide clear password reset instructions" explanation: "Response correctly guides user through password reset" createdAt: "2025-11-12T10:30:00Z" traceUuid: "trace-uuid-1" user: id: "user-uuid-1" email: "user1@example.com" name: "User One" image: "https://example.com/user1.png" "404": description: Annotation not found content: application/json: schema: $ref: "#/components/schemas/ErrorResponse" put: x-fern-server-name: Confident AI tags: - Annotations summary: Update Annotation operationId: updateAnnotation description: | Updates an existing annotation's properties such as rating, type, expected output/outcome, or explanation. **Validation rules:** - Thread annotations cannot have `expectedOutput` (use `expectedOutcome` instead) - Trace/span annotations cannot have `expectedOutcome` (use `expectedOutput` instead) - The annotation must belong to your project parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: annotationId required: true schema: type: string description: The ID of the annotation to update. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateAnnotation" example: rating: 5 type: "FIVE_STAR_RATING" expectedOutput: "Provide detailed password reset steps with security considerations" explanation: "Updated: Response now includes security best practices" responses: "200": description: Successfully updated annotation content: application/json: schema: $ref: "#/components/schemas/UpdateAnnotationResponse" example: success: true data: id: "ANNOTATION-ID" deprecated: false "404": description: Annotation not found content: application/json: schema: $ref: "#/components/schemas/ErrorResponse" /v1/annotation-queues: get: x-fern-server-name: Confident AI tags: - Annotation Queues summary: List Queues operationId: listAnnotationQueues description: | Retrieves all annotation queues in your project with statistics and pagination support. Each queue includes completion metrics, item counts, and assignment breakdowns. Filter results by queue type (TRACE, SPAN, THREAD, GOLDEN, TEST_RUN) or search by name. Queues are returned in descending order by creation date. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the annotation queues to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 description: This specifies the maximum number of annotation queues per page. Defaulted to 25. - in: query name: start required: false schema: type: string format: date-time description: This filters for annotation queues created after the specified start datetime. Defaulted to 30 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for annotation queues created before the specified end datetime. Defaulted to the current time. - in: query name: sortBy required: false schema: type: string enum: [createdAt] description: This determines the field to sort by. Defaulted to `createdAt`. - in: query name: ascending required: false schema: type: string enum: ["true", "false"] description: This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`. - in: query name: type schema: type: string enum: [TRACE, SPAN, THREAD, GOLDEN, TEST_RUN] description: Filter by queue type - in: query name: searchTerm schema: type: string description: Search queues by name responses: "200": description: Successfully retrieved list of annotation queues content: application/json: schema: $ref: "#/components/schemas/ListAnnotationQueuesResponse" example: success: true data: annotationQueues: - id: "queue-uuid-1" name: "Trace Review Queue" type: "TRACE" createdAt: "2025-11-12T10:00:00Z" updatedAt: "2025-11-12T11:00:00Z" completedItems: 45 totalItems: 100 completionPercentage: 45 - id: "queue-uuid-2" name: "Span Quality Check" type: "SPAN" createdAt: "2025-11-11T09:00:00Z" updatedAt: "2025-11-12T10:00:00Z" completedItems: 20 totalItems: 50 completionPercentage: 40 total: 2 post: x-fern-server-name: Confident AI tags: - Annotation Queues summary: Create Queue operationId: createAnnotationQueue description: | Creates a new annotation queue for organizing traces, spans, threads, golden datasets, or test runs for systematic review and annotation. **Queue types:** - `TRACE`: Queue for reviewing complete trace executions - `SPAN`: Queue for reviewing individual spans within traces - `THREAD`: Queue for reviewing conversation threads - `GOLDEN`: Queue for creating golden datasets - `TEST_RUN`: Queue for reviewing test run results Queue names must be unique within your project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateAnnotationQueue" responses: "200": description: Successfully created annotation queue content: application/json: schema: $ref: "#/components/schemas/CreateAnnotationQueueResponse" example: success: true data: id: "queue-uuid" /v1/annotation-queues/{queueId}: get: x-fern-server-name: Confident AI tags: - Annotation Queues summary: Get Queue operationId: getAnnotationQueue description: | Retrieves detailed information about a specific annotation queue including comprehensive statistics. Returns queue metadata, completion metrics (total items, completed items, pending items, completion percentage), and assignment breakdowns showing how many items are assigned to each user and their completion status. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: queueId required: true schema: type: string description: The ID of the queue to retrieve. responses: "200": description: Successfully retrieved annotation queue content: application/json: schema: $ref: "#/components/schemas/GetAnnotationQueueResponse" example: success: true data: annotationQueue: id: "queue-uuid" name: "Trace Review Queue" type: "TRACE" createdAt: "2025-11-12T10:00:00Z" updatedAt: "2025-11-12T11:00:00Z" completedItems: 45 totalItems: 100 pendingItems: 55 completionPercentage: 45 assignedItems: 60 assignmentBreakdown: "user1@example.com": assigned: 30 completed: 25 "user2@example.com": assigned: 30 completed: 20 "404": description: Queue not found delete: x-fern-server-name: Confident AI tags: - Annotation Queues summary: Delete Queue operationId: deleteAnnotationQueue description: | Permanently deletes an annotation queue and all its associated queue items. **Warning:** This action cannot be undone. All queue items (assignments and completion status) will be deleted. However, any annotations created from this queue will remain in your project. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: queueId required: true schema: type: string description: The ID of the queue to delete. responses: "200": description: Successfully deleted annotation queue content: application/json: schema: $ref: "#/components/schemas/DeleteAnnotationQueueResponse" example: success: true data: id: "queue-uuid" /v1/annotation-queues/{queueId}/items: get: x-fern-server-name: Confident AI tags: - Annotation Queues summary: List Queue Items operationId: listQueueItems description: | Retrieves all items in an annotation queue with pagination and optional filtering by completion status. Use the `completed` query parameter to filter for pending items (`false`) or completed items (`true`). Omit this parameter to retrieve all items regardless of completion status. Items are returned in ascending order by the time they were added to the queue. parameters: - in: header name: CONFIDENT_API_KEY required: true description: The API key of your Confident AI project. schema: type: string example: "" - in: path name: queueId required: true schema: type: string description: The ID of the queue. - in: query name: page required: false schema: type: integer minimum: 1 description: This specifies the page number of the queue items to return. Defaulted to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 description: This specifies the maximum number of queue items per page. Defaulted to 25. - in: query name: start required: false schema: type: string format: date-time description: This filters for queue items added after the specified start datetime. Defaulted to 30 days ago. - in: query name: end required: false schema: type: string format: date-time description: This filters for queue items added before the specified end datetime. Defaulted to the current time. - in: query name: sortBy required: false schema: type: string enum: [addedAt] description: This determines the field to sort by. Defaulted to `addedAt`. - in: query name: ascending required: false schema: type: string enum: ["true", "false"] description: This determines if the field specified in `sortBy` should be in ascending order. Defaults to `false`. - in: query name: completed schema: type: string enum: [true, false] description: Filter by completion status (true for completed, false for pending). Omit to return all items. responses: "200": description: Successfully retrieved list of queue items content: application/json: schema: $ref: "#/components/schemas/ListQueueItemsResponse" example: success: true data: items: - id: "item-uuid-1" traceUuid: "trace-uuid-1" addedAt: "2025-11-12T10:00:00Z" completed: false assignedToEmail: "user@example.com" - id: "item-uuid-2" spanUuid: "span-uuid-1" addedAt: "2025-11-12T10:01:00Z" completed: true assignedToEmail: null /v1/annotation-queues/{itemId}/annotate: post: x-fern-server-name: Confident AI tags: - Annotation Queues summary: Annotate Queue Item operationId: annotateItem description: | Annotates a single queue item, associating the work with the item's trace, span, or thread automatically. Supports both plain queues and queues with an attached form. Submit criteria ratings (the flat `rating` fields or the `annotations` array) and, for form queues, custom-field answers (`formResponses`). At least one must be present. **Validation rules:** - For thread items: Use `expectedOutcome` (not `expectedOutput`). - For trace/span items: Use `expectedOutput` (not `expectedOutcome`). - A criterion's `name` must match an available criterion — omit it for the default criterion, otherwise it must match a configured custom criterion or a form criteria field. A mismatched name is rejected. - Form `formResponses` are addressed by each field's visible label, accepted only for queues with an attached form. - Set `markAsCompleted: false` to leave the item open (default: true). For form queues, completing enforces the form's required fields. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/AnnotateItem" examples: simple: summary: Single default rating (plain queue) value: rating: 1 type: "THUMBS_RATING" explanation: "Response is accurate and helpful" markAsCompleted: true responses: "200": description: Successfully annotated the queue item content: application/json: schema: $ref: "#/components/schemas/AnnotateItemResponse" example: success: true data: id: "annotation-uuid" annotationIds: - "annotation-uuid" formResponseIds: - "form-response-uuid-1" - "form-response-uuid-2" /v1/annotation-queues/{name}/batch-annotate: post: x-fern-server-name: Confident AI tags: - Annotation Queues summary: Batch Annotate Queue Items operationId: batchAnnotateItems description: | Annotates many items in a queue in one request. Each entry in `items` carries a `queueItemId` plus the same body as the single-item annotate endpoint. `annotatorEmail` and `markAsCompleted` set at the top level act as defaults applied to any item that omits them; a per-item value takes precedence. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/BatchAnnotate" example: annotatorEmail: "annotator@yourcompany.com" markAsCompleted: true items: - queueItemId: "queue-item-uuid-1" rating: 1 type: "THUMBS_RATING" name: "Correctness" explanation: "Matches the expected answer" formResponses: - label: "Clarity" value: "Very clear" - label: "Overall" value: 10 - queueItemId: "queue-item-uuid-2" rating: 0 type: "THUMBS_RATING" name: "Correctness" responses: "200": description: Batch processed; inspect each result's `success` flag content: application/json: schema: $ref: "#/components/schemas/BatchAnnotateResponse" example: success: true data: results: - queueItemId: "queue-item-uuid-1" success: true id: "annotation-uuid-1" annotationIds: - "annotation-uuid-1" formResponseIds: - "form-response-uuid-1" - "form-response-uuid-2" - queueItemId: "queue-item-uuid-2" success: false error: "No matching criterion for annotation 'Correctness' (THUMBS_RATING). Valid criteria: ..." /v1/projects: get: x-fern-server-name: Confident AI tags: - Projects summary: List Projects operationId: listProjects description: Retrieves all projects within your organization. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetProjectsResponse" example: success: true data: projects: - id: "PROJECT-ID-1" name: "Production App" description: "Main production application" organizationId: "ORGANIZATION-ID" created_at: "2024-12-04T23:00:00.000Z" - id: "PROJECT-ID-2" name: "Staging Environment" description: null organizationId: "ORGANIZATION-ID" created_at: "2024-12-03T15:30:00.000Z" post: x-fern-server-name: Confident AI tags: - Projects summary: Create Project operationId: createProject description: | Creates a new project within your organization. A default project-scoped API key is provisioned with the project — its full `value` is returned **once** in this response. Optionally pass `email` to assign an existing organization member (by email) as the project's Owner. requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/CreateProject" example: name: "New Project" description: "This is a new project for testing" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/CreateProjectResponse" example: success: true data: project: id: "PROJECT-ID" name: "New Project" description: "This is a new project for testing" organizationId: "ORGANIZATION-ID" created_at: "2024-12-04T23:00:00.000Z" apiKey: id: 12 name: "Default Key" valid: true value: "confident_proj_9b8a7c6d5e4f3a2b1c0d9e8f7a6b5c4d" created_at: "2024-12-04T23:00:00.000Z" lastUsed: null /v1/projects/{projectId}: get: x-fern-server-name: Confident AI tags: - Projects summary: Retrieve Project operationId: getProject description: Retrieves a single project by id. The project must belong to the organization the API key is scoped to. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project to retrieve responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GetProjectResponse" example: success: true data: project: id: "PROJECT-ID" name: "Production App" description: "Main production application" organizationId: "ORGANIZATION-ID" created_at: "2024-12-04T23:00:00.000Z" put: x-fern-server-name: Confident AI tags: - Projects summary: Update Project operationId: updateProject description: Updates an existing project's name or description. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project to update requestBody: required: true description: "" content: application/json: schema: $ref: "#/components/schemas/UpdateProject" example: name: "Updated Project Name" description: "Updated description" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/UpdateProjectResponse" example: success: true data: project: id: "PROJECT-ID" name: "Updated Project Name" description: "Updated description" organizationId: "ORGANIZATION-ID" created_at: "2024-12-04T23:00:00.000Z" delete: x-fern-server-name: Confident AI tags: - Projects summary: Delete Project operationId: deleteProject description: | Permanently deletes a project and all of its associated data (traces, datasets, metrics, API keys, etc.). This action cannot be undone. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project to delete responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteProjectResponse" example: success: true data: id: "PROJECT-ID" deleted: true /v1/organization/api-keys: get: x-fern-server-name: Confident AI tags: - Organization API Keys summary: List Organization API Keys operationId: listOrganizationApiKeys description: | Lists all organization-scoped API keys. Each key's `value` is redacted (only the last 6 characters are shown) — the full value is only ever returned once, at creation. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyListResponse" example: success: true data: apiKeys: - id: 12 name: "CI/CD key" valid: true value: "***************a1b2c3" created_at: "2024-12-04T23:00:00.000Z" lastUsed: "2024-12-05T10:15:00.000Z" post: x-fern-server-name: Confident AI tags: - Organization API Keys summary: Create Organization API Key operationId: createOrganizationApiKey description: | Mints a new organization-scoped API key. The raw `value` is returned **exactly once** in this response and can never be retrieved again — store it securely. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateApiKey" example: name: "CI/CD key" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyResponse" example: success: true data: apiKey: id: 12 name: "CI/CD key" valid: true value: "confident_org_a3f1c9e2b7d84f06a1c2e3d4f5a6b7c8" created_at: "2024-12-04T23:00:00.000Z" lastUsed: null /v1/organization/api-keys/{apiKeyId}: get: x-fern-server-name: Confident AI tags: - Organization API Keys summary: Get Organization API Key operationId: getOrganizationApiKey description: Retrieves a single organization-scoped API key by id. The `value` is redacted — the full value is only ever returned once, at creation. parameters: - name: apiKeyId in: path required: true schema: type: integer description: The unique identifier of the API key responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyResponse" example: success: true data: apiKey: id: 12 name: "CI/CD key" valid: true value: "***************a1b2c3" created_at: "2024-12-04T23:00:00.000Z" lastUsed: "2024-12-05T10:15:00.000Z" put: x-fern-server-name: Confident AI tags: - Organization API Keys summary: Update Organization API Key operationId: updateOrganizationApiKey description: Activates or deactivates an organization-scoped API key. A deactivated key is rejected on authentication. parameters: - name: apiKeyId in: path required: true schema: type: integer description: The unique identifier of the API key requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateApiKey" example: valid: false responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyResponse" example: success: true data: apiKey: id: 12 name: "CI/CD key" valid: false value: "***************a1b2c3" created_at: "2024-12-04T23:00:00.000Z" lastUsed: "2024-12-05T10:15:00.000Z" delete: x-fern-server-name: Confident AI tags: - Organization API Keys summary: Revoke Organization API Key operationId: deleteOrganizationApiKey description: Permanently revokes an organization-scoped API key. This action cannot be undone. parameters: - name: apiKeyId in: path required: true schema: type: integer description: The unique identifier of the API key responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteApiKeyResponse" example: success: true data: id: 12 deleted: true /v1/projects/{projectId}/api-keys: get: x-fern-server-name: Confident AI tags: - Project API Keys summary: List Project API Keys operationId: listProjectApiKeys description: | Lists all API keys scoped to the project. Each key's `value` is redacted (only the last 6 characters are shown) — the full value is only ever returned once, at creation. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyListResponse" example: success: true data: apiKeys: - id: 34 name: "Production agent key" valid: true value: "***************d4e5f6" created_at: "2024-12-04T23:00:00.000Z" lastUsed: "2024-12-05T10:15:00.000Z" post: x-fern-server-name: Confident AI tags: - Project API Keys summary: Create Project API Key operationId: createProjectApiKey description: | Mints a new project-scoped API key. The raw `value` is returned **exactly once** in this response and can never be retrieved again — store it securely. This is the key your application uses to send traces and run evaluations against the project. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateApiKey" example: name: "Production agent key" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyResponse" example: success: true data: apiKey: id: 34 name: "Production agent key" valid: true value: "confident_proj_9b8a7c6d5e4f3a2b1c0d9e8f7a6b5c4d" created_at: "2024-12-04T23:00:00.000Z" lastUsed: null /v1/projects/{projectId}/api-keys/{apiKeyId}: get: x-fern-server-name: Confident AI tags: - Project API Keys summary: Get Project API Key operationId: getProjectApiKey description: Retrieves a single project-scoped API key by id. The `value` is redacted — the full value is only ever returned once, at creation. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: apiKeyId in: path required: true schema: type: integer description: The unique identifier of the API key responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyResponse" example: success: true data: apiKey: id: 34 name: "Production agent key" valid: true value: "***************d4e5f6" created_at: "2024-12-04T23:00:00.000Z" lastUsed: "2024-12-05T10:15:00.000Z" put: x-fern-server-name: Confident AI tags: - Project API Keys summary: Update Project API Key operationId: updateProjectApiKey description: Activates or deactivates a project-scoped API key. A deactivated key is rejected on authentication. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: apiKeyId in: path required: true schema: type: integer description: The unique identifier of the API key requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateApiKey" example: valid: false responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ApiKeyResponse" example: success: true data: apiKey: id: 34 name: "Production agent key" valid: false value: "***************d4e5f6" created_at: "2024-12-04T23:00:00.000Z" lastUsed: "2024-12-05T10:15:00.000Z" delete: x-fern-server-name: Confident AI tags: - Project API Keys summary: Revoke Project API Key operationId: deleteProjectApiKey description: Permanently revokes a project-scoped API key. This action cannot be undone. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: apiKeyId in: path required: true schema: type: integer description: The unique identifier of the API key responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteApiKeyResponse" example: success: true data: id: 34 deleted: true /v1/projects/{projectId}/members: get: x-fern-server-name: Confident AI tags: - Project Members summary: List Project Members operationId: listProjectMembers description: Lists the members of a project, along with each member's project role. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - in: query name: page required: false schema: type: integer minimum: 1 default: 1 description: The page number to return. Defaults to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 maximum: 100 default: 25 description: The maximum number of members per page (max 100). Defaults to 25. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListProjectMembersResponse" example: success: true data: members: - id: "user-uuid-1" email: "alice@example.com" name: "Alice" image: null projectRole: id: "project-role-uuid-1" name: "Owner" - id: "user-uuid-2" email: "bob@example.com" name: "Bob" image: null projectRole: id: "project-role-uuid-2" name: "Member" total: 2 /v1/projects/{projectId}/members/{userId}: put: x-fern-server-name: Confident AI tags: - Project Members summary: Update Project Member Role operationId: updateProjectMemberRole description: | Changes a member's project role. Assigning the `Owner` role to another member transfers ownership, demoting the current Owner to `Manager`. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: userId in: path required: true schema: type: string description: The unique identifier of the member requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateMemberRole" example: roleId: "project-role-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ProjectMemberResponse" example: success: true data: member: id: "user-uuid-2" email: "bob@example.com" name: "Bob" image: null projectRole: id: "project-role-uuid-2" name: "Member" delete: x-fern-server-name: Confident AI tags: - Project Members summary: Remove Project Member operationId: removeProjectMember description: | Removes a member from the project. The member is detached from the project and their pending project invitations are cleared. The project Owner cannot be removed. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: userId in: path required: true schema: type: string description: The unique identifier of the member responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/RemoveMemberResponse" example: success: true data: id: "user-uuid-2" removed: true /v1/projects/{projectId}/invitations: get: x-fern-server-name: Confident AI tags: - Project Invitations summary: List Project Invitations operationId: listProjectInvitations description: Lists the project's pending and declined invitations. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListProjectInvitationsResponse" example: success: true data: invitations: - id: 201 email: "carol@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" projectRoleId: "project-role-uuid-2" post: x-fern-server-name: Confident AI tags: - Project Invitations summary: Create Project Invitations operationId: createProjectInvitations description: | Invites one or more users to a project by email. Emails that are already members or already invited are skipped. Optionally assign a project role to the invitees — the `Owner` role cannot be assigned. Inviting members is not available on the Free plan outside of the trial period. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateProjectInvitations" example: emails: - "carol@example.com" - "dave@example.com" projectRoleId: "project-role-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListProjectInvitationsResponse" example: success: true data: invitations: - id: 201 email: "carol@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" projectRoleId: "project-role-uuid-2" - id: 202 email: "dave@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" projectRoleId: "project-role-uuid-2" /v1/projects/{projectId}/invitations/{invitationId}: delete: x-fern-server-name: Confident AI tags: - Project Invitations summary: Revoke Project Invitation operationId: deleteProjectInvitation description: Revokes a pending project invitation. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: invitationId in: path required: true schema: type: integer description: The unique identifier of the invitation responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteInvitationResponse" example: success: true data: id: 201 deleted: true put: x-fern-server-name: Confident AI tags: - Project Invitations summary: Resend Project Invitation operationId: resendProjectInvitation description: Resends a pending project invitation email to the invitee. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ProjectInvitationResponse" example: success: true data: invitation: id: 201 email: "carol@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" projectRoleId: "project-role-uuid-2" /v1/projects/{projectId}/roles: get: x-fern-server-name: Confident AI tags: - Project Roles summary: List Project Roles operationId: listProjectRoles description: | Lists the roles available to a project. This includes both global, system-defined roles (where `projectId` is `null`) and custom roles defined for the project. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListProjectRolesResponse" example: success: true data: roles: - id: "project-role-uuid-1" name: "Owner" description: "Owner of the project with full access to all resources." projectId: null policies: [] - id: "project-role-uuid-3" name: "Analyst" description: "Read-only project access" projectId: "PROJECT-ID" policies: - id: "policy-uuid-1" name: "View traces" post: x-fern-server-name: Confident AI tags: - Project Roles summary: Create Project Role operationId: createProjectRole description: | Creates a custom project role from a set of policies. The role name must be unique within the project and cannot collide with a system-defined role name. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdateRole" example: name: "Analyst" description: "Read-only project access" policyIds: - "policy-uuid-1" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ProjectRoleResponse" example: success: true data: role: id: "project-role-uuid-3" name: "Analyst" description: "Read-only project access" projectId: "PROJECT-ID" policies: - id: "policy-uuid-1" name: "View traces" /v1/projects/{projectId}/roles/{roleId}: put: x-fern-server-name: Confident AI tags: - Project Roles summary: Update Project Role operationId: updateProjectRole description: Updates a custom project role's name, description, or policies. Global roles cannot be modified. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: roleId in: path required: true schema: type: string description: The unique identifier of the role requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdateRole" example: name: "Analyst" description: "Read and comment on project traces" policyIds: - "policy-uuid-1" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ProjectRoleResponse" example: success: true data: role: id: "project-role-uuid-3" name: "Analyst" description: "Read and comment on project traces" projectId: "PROJECT-ID" policies: - id: "policy-uuid-1" name: "View traces" delete: x-fern-server-name: Confident AI tags: - Project Roles summary: Delete Project Role operationId: deleteProjectRole description: | Deletes a custom project role. A role that is still assigned to one or more members cannot be deleted. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: roleId in: path required: true schema: type: string description: The unique identifier of the role responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteRoleResponse" example: success: true data: id: "project-role-uuid-3" deleted: true /v1/projects/{projectId}/policies: get: x-fern-server-name: Confident AI tags: - Project Policies summary: List Project Policies operationId: listProjectPolicies description: Lists the custom policies defined for a project. Each policy is a named collection of permissions. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListProjectPoliciesResponse" example: success: true data: policies: - id: "policy-uuid-1" name: "View traces" description: "Grants read access to traces" permissions: - id: "perm-uuid-1" name: "trace:read" post: x-fern-server-name: Confident AI tags: - Project Policies summary: Create Project Policy operationId: createProjectPolicy description: Creates a custom project policy from a set of permissions. Use `GET /v1/organization/permissions` to discover assignable permission ids. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdatePolicy" example: name: "View traces" description: "Grants read access to traces" permissionIds: - "perm-uuid-1" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PolicyResponse" example: success: true data: policy: id: "policy-uuid-1" name: "View traces" description: "Grants read access to traces" permissions: - id: "perm-uuid-1" name: "trace:read" /v1/projects/{projectId}/policies/{policyId}: put: x-fern-server-name: Confident AI tags: - Project Policies summary: Update Project Policy operationId: updateProjectPolicy description: Updates a custom project policy's name, description, or permissions. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: policyId in: path required: true schema: type: string description: The unique identifier of the policy requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdatePolicy" example: name: "View traces" description: "Grants read and export access to traces" permissionIds: - "perm-uuid-1" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PolicyResponse" example: success: true data: policy: id: "policy-uuid-1" name: "View traces" description: "Grants read and export access to traces" permissions: - id: "perm-uuid-1" name: "trace:read" delete: x-fern-server-name: Confident AI tags: - Project Policies summary: Delete Project Policy operationId: deleteProjectPolicy description: Deletes a custom project policy. parameters: - name: projectId in: path required: true schema: type: string description: The unique identifier of the project - name: policyId in: path required: true schema: type: string description: The unique identifier of the policy responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeletePolicyResponse" example: success: true data: id: "policy-uuid-1" deleted: true /v1/organization: get: x-fern-server-name: Confident AI tags: - Organization summary: Retrieve Organization operationId: getOrganization description: Retrieves the organization that the API key is scoped to. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/OrganizationResponse" example: success: true data: organization: id: "ORGANIZATION-ID" name: "Acme Inc." plan: "PREMIUM" created_at: "2024-12-04T23:00:00.000Z" put: x-fern-server-name: Confident AI tags: - Organization summary: Update Organization operationId: updateOrganization description: Updates the organization's name. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateOrganization" example: name: "Acme Inc." responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/OrganizationResponse" example: success: true data: organization: id: "ORGANIZATION-ID" name: "Acme Inc." plan: "PREMIUM" created_at: "2024-12-04T23:00:00.000Z" /v1/organization/members: get: x-fern-server-name: Confident AI tags: - Organization Members summary: List Organization Members operationId: listMembers description: Lists the members of your organization, along with each member's organization role. parameters: - in: query name: page required: false schema: type: integer minimum: 1 default: 1 description: The page number to return. Defaults to 1. - in: query name: pageSize required: false schema: type: integer minimum: 1 maximum: 100 default: 25 description: The maximum number of members per page (max 100). Defaults to 25. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListMembersResponse" example: success: true data: members: - id: "user-uuid-1" email: "alice@example.com" name: "Alice" image: null organizationRole: id: "role-uuid-1" name: "Admin" - id: "user-uuid-2" email: "bob@example.com" name: "Bob" image: null organizationRole: id: "role-uuid-2" name: "Member" total: 2 /v1/organization/members/{userId}: put: x-fern-server-name: Confident AI tags: - Organization Members summary: Update Organization Member Role operationId: updateMemberRole description: | Changes a member's organization role. Assigning the `Owner` role to another member transfers ownership, demoting the current Owner to `Admin`. parameters: - name: userId in: path required: true schema: type: string description: The unique identifier of the member requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateMemberRole" example: roleId: "role-uuid-1" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/MemberResponse" example: success: true data: member: id: "user-uuid-2" email: "bob@example.com" name: "Bob" image: null organizationRole: id: "role-uuid-1" name: "Admin" delete: x-fern-server-name: Confident AI tags: - Organization Members summary: Remove Organization Member operationId: removeMember description: | Removes a member from the organization. The member is detached from all projects and their pending invitations are cleared. The organization Owner cannot be removed. parameters: - name: userId in: path required: true schema: type: string description: The unique identifier of the member responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/RemoveMemberResponse" example: success: true data: id: "user-uuid-2" removed: true /v1/organization/invitations: get: x-fern-server-name: Confident AI tags: - Organization Invitations summary: List Organization Invitations operationId: listInvitations description: Lists the organization's pending and declined invitations. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListInvitationsResponse" example: success: true data: invitations: - id: 101 email: "carol@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" organizationRoleId: "role-uuid-2" post: x-fern-server-name: Confident AI tags: - Organization Invitations summary: Create Organization Invitations operationId: createInvitations description: | Invites one or more users to your organization by email. Emails that are already members or already invited are skipped. Optionally assign an organization role to the invitees — the `Owner` role cannot be assigned. Inviting members is not available on the Free plan outside of the trial period. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateInvitations" example: emails: - "carol@example.com" - "dave@example.com" organizationRoleId: "role-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListInvitationsResponse" example: success: true data: invitations: - id: 101 email: "carol@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" organizationRoleId: "role-uuid-2" - id: 102 email: "dave@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" organizationRoleId: "role-uuid-2" /v1/organization/invitations/{invitationId}: delete: x-fern-server-name: Confident AI tags: - Organization Invitations summary: Revoke Organization Invitation operationId: deleteInvitation description: Revokes a pending organization invitation. parameters: - name: invitationId in: path required: true schema: type: integer description: The unique identifier of the invitation responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteInvitationResponse" example: success: true data: id: 101 deleted: true put: x-fern-server-name: Confident AI tags: - Organization Invitations summary: Resend Organization Invitation operationId: resendInvitation description: Resends a pending organization invitation email to the invitee. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/InvitationResponse" example: success: true data: invitation: id: 101 email: "carol@example.com" status: "PENDING" created_at: "2024-12-04T23:00:00.000Z" organizationRoleId: "role-uuid-2" /v1/organization/roles: get: x-fern-server-name: Confident AI tags: - Organization Roles summary: List Organization Roles operationId: listRoles description: | Lists the roles available to your organization. This includes both global, system-defined roles (where `organizationId` is `null`) and custom roles defined by your organization. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListRolesResponse" example: success: true data: roles: - id: "role-uuid-1" name: "Admin" description: "Full administrative access" organizationId: null policies: - id: "policy-uuid-1" name: "Manage projects" - id: "role-uuid-3" name: "Billing Manager" description: "Can manage billing only" organizationId: "ORGANIZATION-ID" policies: - id: "policy-uuid-2" name: "Manage billing" post: x-fern-server-name: Confident AI tags: - Organization Roles summary: Create Organization Role operationId: createRole description: | Creates a custom organization role from a set of policies. The role name must be unique within your organization and cannot collide with a system-defined role name. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdateRole" example: name: "Billing Manager" description: "Can manage billing only" policyIds: - "policy-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/RoleResponse" example: success: true data: role: id: "role-uuid-3" name: "Billing Manager" description: "Can manage billing only" organizationId: "ORGANIZATION-ID" policies: - id: "policy-uuid-2" name: "Manage billing" /v1/organization/roles/{roleId}: put: x-fern-server-name: Confident AI tags: - Organization Roles summary: Update Organization Role operationId: updateRole description: Updates a custom organization role's name, description, or policies. Global roles cannot be modified. parameters: - name: roleId in: path required: true schema: type: string description: The unique identifier of the role requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdateRole" example: name: "Billing Manager" description: "Can manage billing and view usage" policyIds: - "policy-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/RoleResponse" example: success: true data: role: id: "role-uuid-3" name: "Billing Manager" description: "Can manage billing and view usage" organizationId: "ORGANIZATION-ID" policies: - id: "policy-uuid-2" name: "Manage billing" delete: x-fern-server-name: Confident AI tags: - Organization Roles summary: Delete Organization Role operationId: deleteRole description: | Deletes a custom organization role. A role that is still assigned to one or more members cannot be deleted. parameters: - name: roleId in: path required: true schema: type: string description: The unique identifier of the role responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeleteRoleResponse" example: success: true data: id: "role-uuid-3" deleted: true /v1/organization/policies: get: x-fern-server-name: Confident AI tags: - Organization Policies summary: List Organization Policies operationId: listPolicies description: Lists the custom policies defined by your organization. Each policy is a named collection of permissions. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListPoliciesResponse" example: success: true data: policies: - id: "policy-uuid-2" name: "Manage billing" description: "Grants access to billing settings" permissions: - id: "perm-uuid-1" name: "billing:read" - id: "perm-uuid-2" name: "billing:write" post: x-fern-server-name: Confident AI tags: - Organization Policies summary: Create Organization Policy operationId: createPolicy description: Creates a custom organization policy from a set of permissions. Use `GET /v1/organization/permissions` to discover assignable permission ids. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdatePolicy" example: name: "Manage billing" description: "Grants access to billing settings" permissionIds: - "perm-uuid-1" - "perm-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PolicyResponse" example: success: true data: policy: id: "policy-uuid-2" name: "Manage billing" description: "Grants access to billing settings" permissions: - id: "perm-uuid-1" name: "billing:read" - id: "perm-uuid-2" name: "billing:write" /v1/organization/policies/{policyId}: put: x-fern-server-name: Confident AI tags: - Organization Policies summary: Update Organization Policy operationId: updatePolicy description: Updates a custom organization policy's name, description, or permissions. parameters: - name: policyId in: path required: true schema: type: string description: The unique identifier of the policy requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateOrUpdatePolicy" example: name: "Manage billing" description: "Grants full access to billing settings" permissionIds: - "perm-uuid-1" - "perm-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/PolicyResponse" example: success: true data: policy: id: "policy-uuid-2" name: "Manage billing" description: "Grants full access to billing settings" permissions: - id: "perm-uuid-1" name: "billing:read" - id: "perm-uuid-2" name: "billing:write" delete: x-fern-server-name: Confident AI tags: - Organization Policies summary: Delete Organization Policy operationId: deletePolicy description: Deletes a custom organization policy. parameters: - name: policyId in: path required: true schema: type: string description: The unique identifier of the policy responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/DeletePolicyResponse" example: success: true data: id: "policy-uuid-2" deleted: true /v1/organization/governance-policies: get: x-fern-server-name: Confident AI tags: - Organization Governance Policies summary: List Governance Policies operationId: listGovernancePolicies description: Lists your organization's governance policies. Each policy includes the projects assigned to it and the controls it enforces. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListGovernancePoliciesResponse" example: success: true data: governancePolicies: - id: "gov-policy-uuid-1" name: "Production Gate" description: "Pre-deployment gate for production agents" projectsCount: 1 controls: - id: "control-uuid-1" name: "Logs traces before production" type: "PRE_DEPLOYMENT_EVALS" /v1/organization/governance-policies/{policyId}/assign: post: x-fern-server-name: Confident AI tags: - Organization Governance Policies summary: Assign Projects to Governance Policy operationId: assignProjectsToGovernancePolicy description: Assigns one or more projects to a governance policy. A project already on a different policy is moved to this one. This is a partial-success operation. Projects that exist are assigned and returned in `assignedProjectIds`, while ids that do not exist in the organization are reported in `notFoundProjectIds` instead of failing the request. `count` is the number now assigned (re-assigning an already-enrolled project still counts it). parameters: - name: policyId in: path required: true schema: type: string description: The unique identifier of the governance policy requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/AssignGovernancePolicyProjects" example: projectIds: - "project-uuid-1" - "project-uuid-2" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GovernancePolicyAssignmentResponse" example: success: true data: governancePolicy: id: "gov-policy-uuid-1" name: "Production Gate" assignedProjectIds: - "project-uuid-1" - "project-uuid-2" notFoundProjectIds: [] count: 2 /v1/organization/governance-policies/{policyId}/unassign: post: x-fern-server-name: Confident AI tags: - Organization Governance Policies summary: Unassign Projects from Governance Policy operationId: unassignProjectsFromGovernancePolicy description: Removes one or more projects from a governance policy. This is a partial-success operation. Projects currently on this policy are removed and returned in `unassignedProjectIds`, while ids that are not on this policy (unknown, foreign, or on another policy) are reported in `skippedProjectIds` instead of failing the request. `count` is the number removed. parameters: - name: policyId in: path required: true schema: type: string description: The unique identifier of the governance policy requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/AssignGovernancePolicyProjects" example: projectIds: - "project-uuid-1" responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GovernancePolicyUnassignmentResponse" example: success: true data: governancePolicy: id: "gov-policy-uuid-1" name: "Production Gate" unassignedProjectIds: - "project-uuid-1" skippedProjectIds: [] count: 1 /v1/organization/governance-policies/{policyId}/projects: get: x-fern-server-name: Confident AI tags: - Organization Governance Policies summary: List Governance Policy Projects operationId: listGovernancePolicyProjects description: Lists the projects assigned to a governance policy, paginated. parameters: - name: policyId in: path required: true schema: type: string description: The unique identifier of the governance policy responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/GovernancePolicyProjectsResponse" example: success: true data: projects: - id: "project-uuid-1" name: "Acme Support Agent" total: 1 /v1/organization/permissions: get: x-fern-server-name: Confident AI tags: - Organization Permissions summary: List Organization Permissions operationId: listPermissions description: | Lists every assignable organization permission. Permissions are named `resource:action` (e.g. `billing:read`) and are the building blocks of policies. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListPermissionsResponse" example: success: true data: permissions: - id: "perm-uuid-1" name: "billing:read" description: null - id: "perm-uuid-2" name: "billing:write" description: null /v1/projects/{projectId}/permissions: get: x-fern-server-name: Confident AI tags: - Project Permissions summary: List Project Permissions operationId: listProjectPermissions description: | Lists every assignable project permission. Permissions are named `resource:action` (e.g. `traces:read`) and are the building blocks of project policies. responses: "200": description: "" content: application/json: schema: $ref: "#/components/schemas/ListPermissionsResponse" example: success: true data: permissions: - id: "perm-uuid-1" name: "traces:read" description: null - id: "perm-uuid-2" name: "traces:write" description: null /v1/dashboards: get: x-fern-server-name: Confident AI tags: - Dashboards summary: List Dashboards operationId: listDashboards description: Lists dashboard overviews in your Confident AI project — metadata and widget counts only. Widget configurations are returned by the dashboard detail endpoint, and computed widget data by the query endpoints. responses: "200": description: The dashboard overviews in your project. content: application/json: schema: $ref: "#/components/schemas/DashboardListResponse" examples: List Dashboards: success: true data: dashboards: - id: "DASHBOARD-ID" name: "Production Overview" description: null private: false projectId: "PROJECT-ID" userId: null widgetCount: 2 createdAt: "2024-01-01T00:00:00.000Z" updatedAt: "2024-01-01T00:00:00.000Z" deprecated: false post: x-fern-server-name: Confident AI tags: - Dashboards summary: Create Dashboard operationId: createDashboard description: Creates a dashboard, optionally with starter widget configurations, and returns the id of the created dashboard. Use the dashboard detail endpoint for its full configuration, or a query endpoint for computed widget data. requestBody: description: The dashboard configuration to create. required: true content: application/json: schema: $ref: "#/components/schemas/CreateDashboardRequest" examples: Empty: name: "Production Overview" WithWidget: name: "Production Overview" private: false widgets: - name: "Trace Count" type: LINE unit: COUNT mode: TIME_SERIES lines: - name: "Count" dataModel: TRACE aggregation: COUNT responses: "200": description: The id of the created dashboard. content: application/json: schema: $ref: "#/components/schemas/DashboardIdResponse" example: success: true data: id: "DASHBOARD-ID" deprecated: false /v1/dashboards/{dashboardId}: get: x-fern-server-name: Confident AI tags: - Dashboards summary: Get Dashboard operationId: getDashboard description: Retrieves a single dashboard with embedded widget configurations and lines. This endpoint does not compute widget data; use a query endpoint for rendered data. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. responses: "200": description: The requested dashboard with embedded widget configurations. content: application/json: schema: $ref: "#/components/schemas/DashboardObjectResponse" example: success: true data: dashboard: id: "DASHBOARD-ID" name: "Production Overview" description: null private: false projectId: "PROJECT-ID" userId: null createdAt: "2024-01-01T00:00:00.000Z" updatedAt: "2024-01-01T00:00:00.000Z" widgets: - id: "WIDGET-ID" name: "Trace Count" description: null type: LINE unit: COUNT mode: TIME_SERIES bucketMode: null dimension: null topK: null startTime: null endTime: null layout: x: 0 y: 0 w: 6 h: 2 lines: - id: "LINE-ID" name: "Count" color: BLUE dataModel: TRACE aggregation: COUNT filters: null extraQueryParams: null deprecated: false put: x-fern-server-name: Confident AI tags: - Dashboards summary: Update Dashboard operationId: updateDashboard description: Updates a dashboard's metadata and returns the id of the updated dashboard. Only the provided fields are changed. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. requestBody: description: The dashboard metadata fields to update. required: true content: application/json: schema: $ref: "#/components/schemas/UpdateDashboardRequest" example: name: "Renamed Dashboard" responses: "200": description: The id of the updated dashboard. content: application/json: schema: $ref: "#/components/schemas/DashboardIdResponse" example: success: true data: id: "DASHBOARD-ID" deprecated: false delete: x-fern-server-name: Confident AI tags: - Dashboards summary: Delete Dashboard operationId: deleteDashboard description: Permanently deletes a dashboard. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. responses: "200": description: The id of the deleted dashboard. content: application/json: schema: $ref: "#/components/schemas/DashboardIdResponse" example: success: true data: id: "DASHBOARD-ID" deprecated: false /v1/dashboards/{dashboardId}/query: post: x-fern-server-name: Confident AI tags: - Dashboards summary: Query Dashboard Data operationId: queryDashboardData description: Fetches computed data for all widgets on a dashboard, or for a provided subset of widget IDs. Request time range fields override widget defaults for this query only. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. requestBody: description: Optional time range and widget filter for the dashboard data query. Send `{}` to use each widget's saved/default time range. required: true content: application/json: schema: $ref: "#/components/schemas/DashboardQueryRequest" example: startTime: "2024-01-01T00:00:00.000Z" endTime: "2024-01-31T23:59:59.999Z" granularity: day widgetIds: - "WIDGET-ID" responses: "200": description: The computed data for the requested dashboard widgets. content: application/json: schema: $ref: "#/components/schemas/DashboardQueryResponse" example: success: true data: results: - widgetId: "WIDGET-ID" status: ok type: LINE mode: TIME_SERIES kind: TIME_SERIES unit: COUNT xAxis: type: time series: - key: "Count" name: "Count" color: BLUE lineId: "LINE-ID" points: - x: "2024-01-01T00:00:00.000Z" y: 42 deprecated: false /v1/dashboards/{dashboardId}/widgets: post: x-fern-server-name: Confident AI tags: - Dashboards summary: Create Widget operationId: createDashboardWidget description: Adds a widget configuration to a dashboard and returns the id of the created widget. Use the dashboard detail endpoint for its full configuration, or a query endpoint for computed widget data. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. requestBody: description: The widget configuration to add to the dashboard. required: true content: application/json: schema: $ref: "#/components/schemas/DashboardWidgetInput" examples: Time Series: name: "Trace Count" type: LINE unit: COUNT mode: TIME_SERIES lines: - name: "Count" dataModel: TRACE aggregation: COUNT Dimension Breakdown: name: "p90 Latency by Model" type: BAR unit: MILLISECONDS mode: DIMENSION_SERIES dimension: model topK: limit: 10 orderBy: p90_latency direction: desc lines: - name: "p90" dataModel: LLM_SPAN aggregation: P90_LATENCY responses: "200": description: The id of the created widget. content: application/json: schema: $ref: "#/components/schemas/DashboardIdResponse" example: success: true data: id: "WIDGET-ID" deprecated: false /v1/dashboards/{dashboardId}/widgets/{widgetId}: put: x-fern-server-name: Confident AI tags: - Dashboards summary: Update Widget operationId: updateDashboardWidget description: Updates a widget configuration and returns the id of the updated widget. Provided fields replace its configuration, omitted scalar fields are cleared, and lines replace the existing lines when provided. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. - in: path name: widgetId required: true schema: type: string description: The id of the widget. requestBody: description: The replacement widget configuration. If `lines` is provided, it replaces the widget's existing lines. required: true content: application/json: schema: $ref: "#/components/schemas/DashboardWidgetInput" example: name: "Trace Count" type: BAR unit: COUNT mode: TIME_SERIES lines: - name: "Count" dataModel: TRACE aggregation: COUNT responses: "200": description: The id of the updated widget. content: application/json: schema: $ref: "#/components/schemas/DashboardIdResponse" example: success: true data: id: "WIDGET-ID" deprecated: false delete: x-fern-server-name: Confident AI tags: - Dashboards summary: Delete Widget operationId: deleteDashboardWidget description: Detaches a widget from a dashboard. The widget is hard-deleted only when it is no longer attached to any other dashboard. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. - in: path name: widgetId required: true schema: type: string description: The id of the widget. responses: "200": description: The id of the affected widget. content: application/json: schema: $ref: "#/components/schemas/DashboardIdResponse" example: success: true data: id: "WIDGET-ID" deprecated: false /v1/dashboards/{dashboardId}/widgets/{widgetId}/query: post: x-fern-server-name: Confident AI tags: - Dashboards summary: Query Widget Data operationId: queryDashboardWidgetData description: Fetches computed data for one widget. Request time range fields override widget defaults for this query only. parameters: - in: path name: dashboardId required: true schema: type: string description: The id of the dashboard. - in: path name: widgetId required: true schema: type: string description: The id of the widget. requestBody: description: Optional time range override for the widget data query. Send `{}` to use the widget's saved/default time range. required: true content: application/json: schema: $ref: "#/components/schemas/DashboardWidgetQueryRequest" example: startTime: "2024-01-01T00:00:00.000Z" endTime: "2024-01-31T23:59:59.999Z" granularity: day responses: "200": description: The computed data for the requested widget. content: application/json: schema: $ref: "#/components/schemas/DashboardWidgetQueryResponse" examples: TimeSeries: success: true data: widgetId: "WIDGET-ID" type: LINE mode: TIME_SERIES kind: TIME_SERIES unit: COUNT xAxis: type: time series: - key: "Count" name: "Count" color: BLUE lineId: "LINE-ID" points: - x: "2024-01-01T00:00:00.000Z" y: 42 deprecated: false BigNumber: success: true data: widgetId: "WIDGET-ID" type: BIG_NUMBER mode: DIMENSION_SERIES kind: BIG_NUMBER unit: COUNT values: - key: "Count" name: "Count" color: BLUE lineId: "LINE-ID" value: 42 deprecated: false /v1/widgets/query: post: x-fern-server-name: Confident AI tags: - Dashboards summary: Query Ad-hoc Widget Data operationId: queryWidgetData description: Computes widget data from a widget definition supplied inline, without creating a dashboard or saving a widget. Returns the same data shape as the dashboard widget query. Scoped to the project of the API key. At most 20 lines and a `topK.limit` of 100 are allowed, and an explicit query range may not exceed 366 days. requestBody: description: The widget definition to compute, plus an optional query time range that overrides the widget's own time fields for this query. required: true content: application/json: schema: $ref: "#/components/schemas/WidgetQueryRequest" examples: Time Series: widget: name: "Trace Count" type: LINE unit: COUNT mode: TIME_SERIES lines: - name: "Count" dataModel: TRACE aggregation: COUNT startTime: "2024-01-01T00:00:00.000Z" endTime: "2024-01-31T23:59:59.999Z" granularity: day Dimension Breakdown: widget: name: "p90 Latency by Model" type: BAR unit: MILLISECONDS mode: DIMENSION_SERIES dimension: model topK: limit: 10 orderBy: p90_latency direction: desc lines: - name: "p90" dataModel: LLM_SPAN aggregation: P90_LATENCY startTime: "2024-01-01T00:00:00.000Z" endTime: "2024-01-31T23:59:59.999Z" responses: "200": description: The computed data for the supplied widget. content: application/json: schema: $ref: "#/components/schemas/WidgetQueryResponse" examples: Time Series: success: true data: type: LINE mode: TIME_SERIES kind: TIME_SERIES unit: COUNT xAxis: type: time series: - key: "Count" name: "Count" color: BLUE lineId: "line-0" points: - x: "2024-01-01T00:00:00.000Z" y: 42 deprecated: false Dimension Breakdown: success: true data: type: BAR mode: DIMENSION_SERIES kind: DIMENSION unit: MILLISECONDS xAxis: type: category series: - key: "p90" name: "p90" color: BLUE lineId: "line-0" points: - x: "gpt-4o" y: 812 - x: "gpt-4o-mini" y: 342 deprecated: false components: schemas: Metric: type: object properties: id: type: string description: This is the unique id of the metric. name: type: string description: This is the name of the metric, it's unique to all metrics. criteria: type: string nullable: true description: This is the criteria this metric uses to evaluate test cases. evaluationSteps: type: array nullable: true description: An alternative to criteria — a list of steps used to evaluate test cases. items: type: string requiredParameters: description: The parameters required by this metric for evaluation. oneOf: - type: array title: singleTurnRequiredParameters items: type: string enum: [ input, actualOutput, expectedOutput, context, expectedTools, toolsCalled, retrievalContext, metadata, tags, ] - type: array title: multiTurnRequiredParameters items: type: string enum: [ content, role, scenario, expectedOutcome, toolsCalled, retrievalContext, metadata, tags, ] multiTurn: type: boolean description: This is true if the metric is used to evaluate multi-turn test cases. rubric: type: array nullable: true description: A list of score ranges (0–10 inclusive). Must be in order and non-overlapping. Click here to [learn more](https://deepeval.com/docs/metrics-llm-evals#rubric) items: $ref: "#/components/schemas/Rubric" required: - id - name - multiTurn - requiredParameters CreateBatchMetrics: type: object properties: metrics: type: array description: This is the list of metrics you want to add to your project. items: $ref: "#/components/schemas/Metric" required: - metrics CreateBatchMetricsResponse: type: object properties: success: type: boolean description: "This is true if the metrics were created successfully." data: type: object description: "This maps to the ids of the metrics created." properties: ids: type: array description: The ids of the batch metrics created. items: type: string required: - ids deprecated: type: boolean description: "This is true if this batch metrics endpoint is deprecated." required: - success - data - deprecated GetMetricResponse: type: object properties: success: type: boolean description: "This is true if the metrics were successfully retrieved." data: type: object description: "This maps to all the metrics retrieved." properties: metrics: type: array items: $ref: "#/components/schemas/Metric" required: - metrics deprecated: type: boolean description: "This is true if this metrics endpoint is deprecated." required: - success - data - deprecated PullMetricResponse: type: object properties: success: type: boolean description: "This is true if the metric was successfully retrieved." data: $ref: "#/components/schemas/Metric" description: "The metric retrieved." deprecated: type: boolean description: "This is true if this metric endpoint is deprecated." required: - success - data - deprecated CreateMetric: type: object properties: name: type: string description: The name of the metric you're creating in your project. criteria: type: string description: The criteria for this particular metric, that will be used to evaluate test cases later. evaluationSteps: type: string description: An alternative to criteria, it is a list of steps to take to evaluate your test cases. evaluationParams: description: It is an array of the params that will be used to evaluate your test cases. oneOf: - type: array title: llmTestCaseParams items: type: string enum: [ input, actualOutput, expectedOutput, context, toolsCalled, expectedTools, retrievalContext, ] - type: array title: conversationalTestCaseParams items: type: string enum: [ role, content, scenario, toolsCalled, expectedOutcome, retrievalContext, ] multiTurn: type: boolean description: This is true if your metric is used for evaluating multi-turn test cases. rubric: type: array description: A list of score ranges (0–10 inclusive). Must be in order and non-overlapping. Click here to [learn more](https://deepeval.com/docs/metrics-llm-evals#rubric) items: $ref: "#/components/schemas/Rubric" required: - name - multiTurn Rubric: type: object description: A list of score ranges (0–10 inclusive). Must be in order and non-overlapping. Click here to [learn more](https://deepeval.com/docs/metrics-llm-evals#rubric) properties: scoreRange: type: array description: An array consisting of the ranges of scores to generate. items: type: number expectedOutcome: type: string description: The expected outcome for your evaluation. required: - scoreRange - expectedOutcome CreateMetricResponse: type: object properties: success: type: boolean description: "This is true if the metric was created successfully." data: type: object description: "This maps to the id of the metric created." properties: id: type: string description: The id of the metric created required: - id deprecated: type: boolean description: "This is true if this metrics endpoint is deprecated." required: - success - data - deprecated UpdateMetric: type: object properties: name: type: string description: The name of the metric you're updating. criteria: type: string description: The new criteria for this particular metric, that you want to update to. evaluationSteps: type: string description: The new evaluation steps that you want to update for this metric. evaluationParams: description: It is an array of the params that will be used to evaluate your test cases. oneOf: - type: array title: llmTestCaseParams items: type: string enum: [ input, actualOutput, expectedOutput, context, toolsCalled, expectedTools, retrievalContext, ] - type: array title: conversationalTestCaseParams items: type: string enum: [ turns, scenario, expectedOutcome, userDescription, context, chatbotRole, ] rubric: type: array description: A list of score ranges (0–10 inclusive). Must be in order and non-overlapping. Click here to [learn more](https://deepeval.com/docs/metrics-llm-evals#rubric) items: $ref: "#/components/schemas/Rubric" UpdateMetricResponse: type: object properties: success: type: boolean description: "This is true if the metric was updated successfully." deprecated: type: boolean description: "This is true if this metrics endpoint is deprecated." required: - success - deprecated CreateMetricCollection: type: object description: Request schema to create or update a metric collection properties: name: type: string description: This is the unique name of the metric collection you wish to create. multiTurn: type: boolean description: This is true if your metric collection is multi-turn, which contains only multi-turn metrics. Defaulted to `false`. metricSettings: type: array description: This is the list of metric settings for the collection. items: $ref: "#/components/schemas/MetricSetting" required: - name UpdateMetricCollection: type: object description: Request schema to create or update a metric collection properties: name: type: string description: The name of the metric collection, which must be unique within your project. multiTurn: type: boolean description: This is true if your metric collection is multi-turn, which contains only multi-turn metrics. Defaulted to `false`. metricSettings: type: array description: This is the list of metric settings for the collection. items: $ref: "#/components/schemas/MetricSetting" required: MetricSetting: type: object properties: metric: type: object description: This is a metric object, which contains the metric name. properties: name: type: string description: This is the name of the metric. required: - name activated: type: boolean default: true description: "This determines if the metric is activated. Only activated metrics are used for evaluations. Non-activated metrics are skipped." threshold: type: number minimum: 0 maximum: 1 default: 0.5 description: "This determines the threshold for the metric which determines if the metric passes or fails depending on if the metric score is equal or greater than the threshold." includeReason: type: boolean default: true description: "This determines if the reason for the metric score should be generated during evaluations." strictMode: type: boolean default: false description: "This determines if the metric is in strict mode. Metrics in strict mode output a binary score of 0 or 1, indicating pass or fail, as opposed to a continuous score from 0 to 1." sampleRate: type: number minimum: 0 maximum: 1 default: 1 description: "This determines the probability of the metric being ran for evaluation." required: - metric CreateMetricCollectionResponse: type: object properties: success: type: boolean description: "This is true if the metric collection was created successfully." data: type: object description: "This maps to the id of the metric collection." properties: id: description: "This is the id of the metric collection." type: string required: - id deprecated: type: boolean description: "This is true if this metric collection endpoint is deprecated." required: - success - data - deprecated UpdateMetricCollectionResponse: type: object properties: success: type: boolean description: "This is true if the metric collection was updated successfully." data: type: object description: "This maps to the id of the metric collection." properties: id: description: "This is the id of the metric collection." type: string required: - id deprecated: type: boolean description: "This is true if this metric collection endpoint is deprecated." required: - success - data - deprecated GetMetricCollectionResponse: type: object properties: success: type: boolean description: "This is true if the list of metric collections was retrieved successfully." data: type: object description: "This maps to a list of metric collections, which can be used to [run evals](/docs/api-reference/evaluate/evaluate-llm) remotely." properties: metricCollections: type: array description: "This is a list of metric collection objects." items: type: object properties: id: type: string description: "This is the id of the metric collection." name: type: string description: "This is the name of the metric collection, which should be supplied to the [evals API](/docs/api-reference/evaluate/evaluate-llm) to run evaluations remotely." multiTurn: type: boolean description: "This is true if the metric collection is a multi-turn collection, which only contains multi-turn metrics for multi-turn evaluations." metricSettings: type: array description: "This is a list of settings for each metric in the collection, which can be configured on [Confident AI's](https://confidentai.com) metric collection page." items: type: object properties: metric: type: object description: "This is a metric object, which contains the metric name." properties: name: type: string description: "This is the name of the metric." activated: type: boolean description: "This is true if the metric is activated. Only activated metrics are used for evaluations. Non-activated metrics are skipped." threshold: type: number minimum: 0 maximum: 1 description: "This is the threshold for the metric, which determines if the metric passes or fails depending on if the metric score is above or below the threshold." includeReason: type: boolean description: "This is true if a metric's `include_reason` setting is toggled on the Confident platform. If true, the reason for the metric score will be generated during evaluations." strictMode: type: boolean description: "This is true if the metric is in strict mode. Metrics in strict mode output a binary score of 0 or 1, indicating pass or fail, as opposed to a continuous score from 0 to 1." sampleRate: type: number minimum: 0 maximum: 1 description: "This is the probability of the metric being ran for evaluation." required: - metric - activated - threshold - includeReason - strictMode - sampleRate required: - id - name - multiTurn - metricSettings required: - metricCollections required: - success - data GetDatasetsResponse: type: object properties: success: type: boolean description: "This is true if the datasets were successfully retrieved." data: type: object description: "This maps to all the datasets retrieved." properties: datasets: type: array items: type: object properties: id: type: string description: This is the unique id of the dataset. alias: type: string description: This is the alias of the dataset, which is unique within your project. multiTurn: type: boolean description: This is true if the dataset is multi-turn, which contains multi-turn test cases. Single-turn datasets have `multi_turn` set to false and contain single-turn test cases. required: - id - alias - multiTurn required: - datasets deprecated: type: boolean description: "This is true if this datasets endpoint is deprecated." required: - success - data - deprecated RetrievedGolden: type: object properties: input: type: string description: "This is the input to your LLM application." actualOutput: type: string description: "This is the actual output of your LLM application." expectedOutput: type: string description: "This is the expected output of your LLM application, which is the ideal actual output." retrievalContext: type: array items: type: string description: "This is the retrieval context of your LLM application." context: type: array items: type: string description: "This is the ideal retrieval context of your LLM application." toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: "This is the tools called by your LLM application." expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: "This is the expected tools to be called by the LLM application." additionalMetadata: type: object additionalProperties: true description: "This is any additional metadata you wish to associate with the golden." comments: type: string description: "This is any comments you wish to associate with the golden." sourceFile: type: string description: "This is the source file from which the golden was retrieved." customColumnKeyValues: type: object additionalProperties: true description: "This is the custom column key values of the LLM application." required: - input RetrievedConversationalGolden: type: object properties: scenario: type: string description: "This is a description of the conversation context." userDescription: type: string description: "This is the description of the user in the conversation." expectedOutcome: type: string description: "This describes the expected outcome, or ideal conversation flow, of the conversation." turns: type: array items: $ref: "#/components/schemas/Turn" description: "This is the list of turns in the conversation." context: type: array items: type: string description: "This is the context of the conversation." additionalMetadata: type: object additionalProperties: true description: "This is any additional metadata you wish to associate with the golden." comments: type: string description: "This is any comments you wish to associate with the golden." sourceFile: type: string description: "This is the source file from which the golden was retrieved." customColumnKeyValues: type: object additionalProperties: true description: "This is the custom column key values of the LLM application." required: - scenario Golden: type: object properties: input: type: string description: "This is the input to your LLM application." actualOutput: type: string description: "This is the actual output of your LLM application." expectedOutput: type: string description: "This is the expected output of your LLM application, which is the ideal actual output." retrievalContext: type: array items: type: string description: "This is the retrieval context of your LLM application." context: type: array items: type: string description: "This is the ideal retrieval context of your LLM application." toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: "This is the tools called by your LLM application." expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: "This is the expected tools to be called by the LLM application." additionalMetadata: type: object additionalProperties: true description: "This is any additional metadata you wish to associate with the golden." comments: type: string description: "This is any comments you wish to associate with the golden." sourceFile: type: string description: "This is the source file from which the golden was retrieved." finalized: type: boolean description: "This determines whether the golden has been finalized." customColumnKeyValues: type: object additionalProperties: true description: "Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset." required: - input ConversationalGolden: type: object properties: scenario: type: string description: "This is a description of the conversation context." userDescription: type: string description: "This is the description of the user in the conversation." expectedOutcome: type: string description: "This describes the expected outcome, or ideal conversation flow, of the conversation." turns: type: array items: $ref: "#/components/schemas/Turn" description: "This is the list of turns in the conversation." context: type: array items: type: string description: "This is the context of the conversation." additionalMetadata: type: object additionalProperties: true description: "This is any additional metadata you wish to associate with the golden." comments: type: string description: "This is any comments you wish to associate with the golden." sourceFile: type: string description: "This is the source file from which the golden was retrieved." finalized: type: boolean description: "This determines whether the golden has been finalized." customColumnKeyValues: type: object additionalProperties: true description: "Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset." required: - scenario SimulationConversationalGolden: type: object properties: scenario: type: string description: "This is a description of the conversation context." userDescription: type: string description: "This is the description of the user in the conversation." expectedOutcome: type: string description: "This describes the expected outcome, or ideal conversation flow, of the conversation." turns: type: array items: $ref: "#/components/schemas/Turn" description: "This is the list of turns in the conversation." context: type: array items: type: string description: "This is the context of the conversation." additionalMetadata: type: object additionalProperties: true description: "This is any additional metadata you wish to associate with the golden." comments: type: string description: "This is any comments you wish to associate with the golden." name: type: string description: "This is the name of the golden." customColumnKeyValues: type: object additionalProperties: true description: "Key-value pairs representing custom table column data for this golden. Keys correspond to the custom column keys defined in the dataset." required: - scenario Turn: type: object properties: role: type: string enum: ["user", "assistant"] description: "The role of the turn, either user or assistant." content: type: string description: "The message content of the turn." userId: type: string description: "The user ID associated with the turn." retrievalContext: type: array items: type: string description: "The contexts retrieved to generate the LLM response for this turn." toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: "The tools called to generate the LLM response for this turn." required: - role - content ToolCallData: type: object properties: name: type: string description: "This is the name of the tool." description: type: string description: "This is the description of the tool." inputParameters: type: object additionalProperties: true description: "This is the input parameters that are passed to the tool." output: type: string description: "This is the output of the tool." reasoning: type: string description: "This is the reasoning your LLM provided for the tool call." required: - name - description GetDatasetResponse: type: object properties: success: type: boolean description: "This is true if the dataset was successfully pulled." data: oneOf: - type: object title: Single-Turn description: Dataset with single-turn goldens properties: id: type: string description: A unique identifier for a dataset version: type: string nullable: true description: | The resolved version of the dataset returned. `null` if the dataset has no versions yet. goldens: type: array items: $ref: "#/components/schemas/Golden" required: - id - goldens - type: object title: Multi-Turn description: Dataset with multi-turn goldens properties: id: type: string description: A unique identifier for a dataset version: type: string nullable: true description: | The resolved version of the dataset returned. `null` if the dataset has no versions yet. conversationalGoldens: type: array items: $ref: "#/components/schemas/ConversationalGolden" required: - id - conversationalGoldens required: - success - data PushDatasetRequestData: type: object properties: finalized: type: boolean description: "Determines whether goldens are finalized when pushed to the dataset." version: type: string description: | Optional dataset version (e.g. `"00.00.01"`) to push goldens onto. When the dataset already has versions, omitting this pushes to the latest version. When the dataset has no versions yet, omitting this leaves the goldens unversioned. Specifying a `version` on a brand-new dataset alias returns 400. goldens: type: array items: $ref: "#/components/schemas/Golden" description: "This is a list of single-turn goldens to push. If you are pushing a multi-turn dataset, this should be `null`." conversationalGoldens: type: array items: $ref: "#/components/schemas/ConversationalGolden" description: "This is a list of conversational goldens to push. If you are pushing a single-turn dataset, this should be `null`." required: - finalized QueueDatasetGoldensRequestData: type: object properties: goldens: type: array items: $ref: "#/components/schemas/Golden" description: "This is a list of single-turn goldens to queue. If you are queueing conversational goldens, this should be `null`." conversationalGoldens: type: array items: $ref: "#/components/schemas/ConversationalGolden" description: "This is a list of conversational goldens to queue. If you are queueing single-turn goldens, this should be `null`." QueueDatasetGoldensResponse: type: object properties: link: type: string description: "A link to the dataset the goldens were queued to." required: - link DeleteDatasetResponse: type: object properties: success: type: boolean description: "This is true if the dataset was successfully deleted." data: type: object properties: id: type: string description: "This is the ID of the deleted dataset." required: - success - data DatasetVersion: type: object properties: id: type: string description: A unique identifier for the dataset version. version: type: string description: 'The version label (e.g. `"00.00.01"`).' createdAt: type: string format: date-time description: ISO timestamp of when the version was created. required: - id - version GetDatasetVersionsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: versions: type: array items: $ref: "#/components/schemas/DatasetVersion" required: - versions required: - success - data CreateDatasetVersionResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: string version: type: string required: - id - version required: - success - data CreateOrUpdateGoldenRequestData: type: object properties: link: type: string description: "This is the URL to the dataset you updated." required: - link LLMTestCase: type: object properties: input: type: string description: "This is the input to your LLM application." actualOutput: type: string description: "This is the actual output of your LLM application." name: type: string description: "This is the name of your test case, it allows you to search and match test cases across different test runs." expectedOutput: type: string description: "This is the expected output of your LLM application, which is the ideal actual output." retrievalContext: type: array items: type: string description: "This is the retrieval context of your LLM application." context: type: array items: type: string description: "This is the ideal retrieval context of your LLM application." toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: "This is the tools called by your LLM application." expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: "This is the expected tools to be called by the LLM application." required: - input - actualOutput ConversationalTestCase: type: object properties: turns: type: array items: $ref: "#/components/schemas/Turn" description: "This is the list of turns in the conversation." scenario: type: string description: "This is a description of the conversation context." name: type: string description: "This is the name of your test case, it allows you to search and match test cases across different test runs." expectedOutcome: type: string description: "This describes the expected outcome, or ideal conversation flow, of the conversation." userDescription: type: string description: "This is the description of the user in the conversation." chatbotRole: type: string description: "This is the role of the chatbot in the conversation." required: - turns EvaluateRequest: type: object properties: metricCollection: type: string description: "The name of the metric collection you wish to use for evaluation." llmTestCases: type: array description: "This is a list of single-turn test cases to evaluate. If you are evaluating multi-turn test cases, this should be `null`." items: $ref: "#/components/schemas/LLMTestCase" conversationalTestCases: type: array description: "This is a list of multi-turn test cases to evaluate. If you are evaluating single-turn test cases, this should be `null`." items: $ref: "#/components/schemas/ConversationalTestCase" hyperparameters: type: object additionalProperties: true description: "This is any hyperparameters like model or prompt you wish to associate with the test run." identifier: type: string description: "A unique identifier for the test run." required: - metricCollection - testCases EvaluateResponse: type: object properties: success: type: boolean description: "This is true if the test cases were successfully evaluated." data: type: object properties: id: type: string description: "This is the unique ID for the test run. This ID is generated by Confident AI and is not to be confused with the identifier provided by the user." deprecated: type: boolean description: "This is true if this endpoint is deprecated." required: - success - data - deprecated SimulateRequest: type: object properties: conversationalGolden: $ref: "#/components/schemas/ConversationalGolden" description: A Conversational Golden that is used to simulate your conversations required: - conversationalGolden SimulateResponse: type: object properties: success: type: boolean description: "This is true if the next turn in the conversation was successfully simulated." data: type: object properties: simulationId: type: string description: "This is the unique ID for the simulation." completed: type: boolean description: "This is true if the conversation is complete, which means the expected outcome has been reached." userResponse: type: string description: "This is the simulated user response of the last turn in the conversation." turns: type: array items: $ref: "#/components/schemas/Turn" description: "This is the list of all the turns in the conversation." deprecated: type: boolean description: "This is true if this endpoint is deprecated." required: - success - data - deprecated BaseSpan: type: object properties: uuid: type: string description: This is the unique identifier of the span. name: type: string description: This is the name of the span. input: type: string description: This is the input to the span. output: type: string description: This is the output of the span. error: type: string description: This is the error message, if an error occurred inside the span. status: type: string enum: [SUCCESS, ERRORED] description: This represents the error status of the span. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. parentUuid: type: string format: uuid description: This is the unique identifier of the span's parent span. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. metricCollection: type: string description: This is the metric collection to be used for evaluating the span. type: type: string description: This is a string that represents the type of span. retrievalContext: type: array items: type: string description: This is the retrieval context of your span, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your span, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your span, which is the ideal actual output and to be used for evaluation. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called by your span, which is to be used for evaluation. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools to be called by the span, which is to be used for evaluation. integration: type: string description: This is the integration associated with the span. required: - uuid - name - startTime - endTime LlmSpan: type: object properties: uuid: type: string description: This is the unique identifier of the span. name: type: string description: This is the name of the span. input: type: string description: This is the input to the span. output: type: string description: This is the output of the span. error: type: string description: This is the error message, if an error occurred inside the span. status: type: string enum: [SUCCESS, ERRORED] description: This represents the error status of the span. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. parentUuid: type: string format: uuid description: This is the unique identifier of the span's parent span. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. metricCollection: type: string description: This is the metric collection to be used for evaluating the span. type: type: string description: This is a string that represents the type of span. retrievalContext: type: array items: type: string description: This is the retrieval context of your span, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your span, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your span, which is the ideal actual output and to be used for evaluation. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called by your span, which is to be used for evaluation. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools to be called by the span, which is to be used for evaluation. model: type: string description: This is the LLM model used in the span. provider: type: string description: This is the provider of the generation model used in the span. integration: type: string description: This is the integration associated with the span. promptAlias: type: string description: This is the alias of your prompt which is stored on Confident AI. promptCommitHash: type: string description: This is the hash of the current prompt being logged in the llm span. promptLabel: type: string description: This is the label assigned to a specific version of prompt on the Confident AI platform. promptVersion: type: string description: This is the version assigned to your prompt on Confident AI. costPerInputToken: type: number format: float description: This is the cost per input token of the LLM model. costPerOutputToken: type: number format: float description: This is the cost per output token of the LLM model. inputTokenCount: type: integer description: This is the number of input tokens passed to the LLM model. outputTokenCount: type: integer description: This is the number of output tokens generated by the LLM model. required: - model RetrieverSpan: type: object properties: uuid: type: string description: This is the unique identifier of the span. name: type: string description: This is the name of the span. input: type: string description: This is the input to the span. output: type: string description: This is the output of the span. error: type: string description: This is the error message, if an error occurred inside the span. status: type: string enum: [SUCCESS, ERRORED] description: This represents the error status of the span. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. parentUuid: type: string format: uuid description: This is the unique identifier of the span's parent span. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. metricCollection: type: string description: This is the metric collection to be used for evaluating the span. type: type: string description: This is a string that represents the type of span. retrievalContext: type: array items: type: string description: This is the retrieval context of your span, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your span, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your span, which is the ideal actual output and to be used for evaluation. integration: type: string description: This is the integration associated with the span. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called by your span, which is to be used for evaluation. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools to be called by the span, which is to be used for evaluation. embedder: type: string description: This is the embedder model used in the span. topK: type: integer description: This is the top K chunks retrieved from your knowledge base. chunkSize: type: integer description: This is the chunk size of each retrieved context. required: - embedder ToolSpan: type: object properties: uuid: type: string description: This is the unique identifier of the span. name: type: string description: This is the name of the span. input: type: string description: This is the input to the span. output: type: string description: This is the output of the span. error: type: string description: This is the error message, if an error occurred inside the span. integration: type: string description: This is the integration associated with the span. status: type: string enum: [SUCCESS, ERRORED] description: This represents the error status of the span. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. parentUuid: type: string format: uuid description: This is the unique identifier of the span's parent span. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. metricCollection: type: string description: This is the metric collection to be used for evaluating the span. type: type: string description: This is a string that represents the type of span. retrievalContext: type: array items: type: string description: This is the retrieval context of your span, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your span, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your span, which is the ideal actual output and to be used for evaluation. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called by your span, which is to be used for evaluation. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools to be called by the span, which is to be used for evaluation. description: type: string description: This is the description of the tool used in the span. AgentSpan: type: object properties: uuid: type: string description: This is the unique identifier of the span. name: type: string description: This is the name of the span. input: type: string description: This is the input to the span. output: type: string description: This is the output of the span. integration: type: string description: This is the integration associated with the span. error: type: string description: This is the error message, if an error occurred inside the span. status: type: string enum: [SUCCESS, ERRORED] description: This represents the error status of the span. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. parentUuid: type: string format: uuid description: This is the unique identifier of the span's parent span. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. metricCollection: type: string description: This is the metric collection to be used for evaluating the span. type: type: string description: This is a string that represents the type of span. retrievalContext: type: array items: type: string description: This is the retrieval context of your span, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your span, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your span, which is the ideal actual output and to be used for evaluation. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called by your span, which is to be used for evaluation. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools to be called by the span, which is to be used for evaluation. availableTools: type: array description: This is the list of names of available tools to be used in the span. items: type: string agentHandoffs: type: array description: This is the list of potential agent handoffs in the span. items: type: string TraceAttachment: type: object description: Payload for a multimodal attachment referenced by id in `[DEEPEVAL:IMAGE:…]` or `[DEEPEVAL:PDF:…]` markers. Provide either a `url`, or `dataBase64` with `mimeType`. properties: url: type: string description: Public URL of the attachment. dataBase64: type: string description: Base64-encoded file bytes. mimeType: type: string description: MIME type of the attachment (required when using `dataBase64`). Trace: type: object required: - uuid - startTime - endTime properties: uuid: type: string description: This is the unique identifier of the trace. name: type: string description: This is the name of the trace. input: type: string description: This is the input to the trace. output: type: string description: This is the output of the trace. startTime: type: string format: date-time description: This is the time the trace started. endTime: type: string format: date-time description: This is the time the trace ended. environment: type: string enum: [production, development, staging, testing] description: This is the environment where your trace was posted, which helps with separating and debugging traces from different environments on the Confident AI platform. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the trace. tags: type: array items: type: string description: This is any tags associated with the trace, which helps with grouping traces and filtering them on the Confident AI platform. spans: type: array description: This is the list of base spans associated with the trace. items: oneOf: - $ref: "#/components/schemas/BaseSpan" title: "BaseSpan" - $ref: "#/components/schemas/LlmSpan" title: "LlmSpan" - $ref: "#/components/schemas/RetrieverSpan" title: "RetrieverSpan" - $ref: "#/components/schemas/ToolSpan" title: "ToolSpan" - $ref: "#/components/schemas/AgentSpan" title: "AgentSpan" threadId: type: string description: This is the unique identifier of the thread associated with the trace. thread: type: object description: Thread-level fields applied to the thread record. `thread.id` is an alternate way to specify the thread (must match top-level `threadId` if both are provided). `metadata` and `tags` only take effect when a thread id is resolvable; successive ingestions merge metadata keys, while tags replace any prior value. properties: id: type: string description: The thread id. Equivalent to top-level `threadId`; if both are set they must match. metadata: type: object additionalProperties: true nullable: true description: Custom key/value metadata to attach to the thread. Values can be any JSON-serializable type and are stringified server-side. Successive ingestions for the same thread merge metadata keys. tags: type: array items: type: string nullable: true description: Tags to set on the thread. Replaces any previously stored tags. userId: type: string description: This is the unique identifier for your end user for the trace. metricCollection: type: string description: This is the metric collection you wish to use to evaluate the trace. testRunId: type: string description: This is the unique identifier of the test run to associate the trace with. When set, the trace becomes one test case in that test run, and `metricCollection` is required. Create a test run with the `POST /v1/test-runs` endpoint to get this id. retrievalContext: type: array items: type: string description: This is the retrieval context of your trace, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your trace, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your trace, which is the ideal actual output and to be used for evaluation. toolsCalled: type: array description: This is the tools called by your trace, which is to be used for evaluation. items: $ref: "#/components/schemas/ToolCallData" expectedTools: type: array description: This is the expected tools to be called by the trace, which is to be used for evaluation. items: $ref: "#/components/schemas/ToolCallData" attachments: type: object additionalProperties: $ref: "#/components/schemas/TraceAttachment" description: Map of attachment ids to payloads for all `[DEEPEVAL:IMAGE:…]` and `[DEEPEVAL:PDF:…]` markers in this trace. Define attachments at the trace level with same ids for same instances. SubmitTestCaseResultRequest: type: object properties: actualOutput: type: string description: The actual output produced by your agent. retrievalContext: type: array items: type: string description: The retrieval context your agent used, if any. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: The tools your agent called while producing the output. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: The tools you expected to be called for this test case. metadata: type: object additionalProperties: true description: Optional additional metadata to attach to the test case. SubmitTestCaseResultResponse: type: object properties: success: type: boolean description: A boolean indicating the success or failure of the API call. data: type: object description: The recorded test case id and its status. properties: testCaseId: type: string description: The test case id the result was recorded for. status: type: string enum: - accepted - already_received description: "`accepted` when queued for evaluation; `already_received` on an idempotent retry." required: - testCaseId - status required: - success - data CreateTraceResponse: type: object properties: success: type: boolean description: A boolean indicating the success or failure of the API call data: type: object description: This maps to the trace id. properties: id: type: string description: This is the uuid of the trace. link: type: string description: This is the URL to the trace on the Confident AI platform. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated - link CreateTestRun: type: object properties: metricCollection: type: string description: This is the metric collection used to evaluate the test cases formed from traces ingested into this test run. identifier: type: string description: This is an optional human-readable identifier for the test run, shown on the Confident AI platform. CreateTestRunResponse: type: object properties: success: type: boolean description: A boolean indicating the success or failure of the API call data: type: object description: This maps to the test run id. properties: id: type: string description: This is the unique identifier of the created test run. Pass it as `testRunId` when ingesting traces. link: type: string description: This is the URL to the test run on the Confident AI platform. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated - link example: success: true data: id: "" link: "https://app.confident-ai.com/project//test-runs//test-cases" deprecated: false UpdateTrace: type: object properties: input: type: string description: This is the input to the trace. output: type: string description: This is the output of the trace. endTime: type: string format: date-time description: This is the time the trace ended. name: type: string description: This is the name of the trace. environment: type: string enum: [production, development, staging, testing] description: This is the environment where your trace was posted, which helps with separating and debugging traces from different environments on the Confident AI platform. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the trace. tags: type: array description: This is any tags associated with the trace, which helps with grouping traces and filtering them on the Confident AI platform. items: type: string baseSpans: type: array description: This is the list of base spans associated with the trace. items: $ref: "#/components/schemas/BaseSpan" llmSpans: type: array description: This is the list of LLM spans associated with the trace. items: $ref: "#/components/schemas/LlmSpan" retrieverSpans: type: array description: This is the list of retriever spans associated with the trace. items: $ref: "#/components/schemas/RetrieverSpan" toolSpans: type: array description: This is the list of tool spans associated with the trace. items: $ref: "#/components/schemas/ToolSpan" agentSpans: type: array description: This is the list of agent spans associated with the trace. items: $ref: "#/components/schemas/AgentSpan" UpdateTraceResponse: type: object properties: success: type: boolean description: A boolean indicating the success or failure of the API call data: type: object properties: id: type: string description: This is the uuid of the trace. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated ListSpansResponse: type: object properties: success: type: boolean description: This is true if the spans were successfully retrieved. data: type: object description: The payload containing the paginated spans and total count. properties: spans: type: array description: The list of spans for the current page. items: $ref: "#/components/schemas/GetSpan" totalSpans: type: integer description: The total number of spans matching the query across all pages. required: - spans - totalSpans deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated RetrieveSpanResponse: type: object properties: success: type: boolean description: This is true if the span was successfully retrieved. data: description: The payload containing the detailed span data. oneOf: - $ref: "#/components/schemas/LlmSpan" title: "LLM Span" - $ref: "#/components/schemas/RetrieverSpan" title: "Retriever Span" - $ref: "#/components/schemas/ToolSpan" title: "Tool Span" - $ref: "#/components/schemas/AgentSpan" title: "Agent Span" - $ref: "#/components/schemas/BaseSpan" title: "Base Span" deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated EvaluateThreadRequest: type: object properties: metricCollection: type: string description: This is the name of the multi-turn metric collection to evaluate the thread. chatbotRole: type: string description: The role or purpose of the chatbot in the thread. required: - metricCollection EvaluateThreadResponse: type: object properties: success: type: boolean description: This is true if the thread was successfully evaluated. data: type: object properties: id: type: string description: This is the id of the thread generated by Confident AI, not to be confused with the thread id you supplied. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated AssessGovernanceResponse: type: object properties: success: type: boolean description: This is true if the governance controls were successfully assessed. data: type: object properties: passed: type: boolean description: This is true if every governance control passed the assessment. governancePolicy: type: object description: The governance policy the project was assessed against. properties: id: type: string description: The unique identifier of the governance policy. name: type: string description: The name of the governance policy. required: - id - name required: - passed - governancePolicy deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated GovernanceControl: type: object properties: id: type: string description: The unique identifier of the control. name: type: string description: The name of the control. type: type: string description: The control type. enum: - RUNTIME - PRE_DEPLOYMENT_EVALS - PRE_DEPLOYMENT_RED_TEAMING - OPERATIONAL required: - id - name - type GovernancePolicy: type: object properties: id: type: string description: The unique identifier of the governance policy. name: type: string description: The name of the governance policy. description: type: string nullable: true description: Optional description of the governance policy. projectsCount: type: integer description: The number of projects assigned to this policy. controls: type: array description: The controls this policy enforces. items: $ref: "#/components/schemas/GovernanceControl" required: - id - name - projectsCount - controls ListGovernancePoliciesResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: governancePolicies: type: array description: The organization's governance policies items: $ref: "#/components/schemas/GovernancePolicy" required: - governancePolicies required: - success - data AssignGovernancePolicyProjects: type: object properties: projectIds: type: array description: The ids of the projects to assign or unassign. minItems: 1 items: type: string required: - projectIds GovernancePolicyAssignmentResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: governancePolicy: type: object description: The governance policy that was modified. properties: id: type: string description: The unique identifier of the governance policy. name: type: string description: The name of the governance policy. required: - id - name assignedProjectIds: type: array description: Ids of the projects now assigned to this policy, including any that were already enrolled. items: type: string notFoundProjectIds: type: array description: Ids that do not exist in this organization. They are skipped (not assigned) and reported here rather than failing the request. items: type: string count: type: integer description: The number of projects now assigned (the length of assignedProjectIds). required: - governancePolicy - assignedProjectIds - notFoundProjectIds - count required: - success - data GovernancePolicyUnassignmentResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: governancePolicy: type: object description: The governance policy that was modified. properties: id: type: string description: The unique identifier of the governance policy. name: type: string description: The name of the governance policy. required: - id - name unassignedProjectIds: type: array description: Ids of the projects removed from this policy. items: type: string skippedProjectIds: type: array description: Ids that were not on this policy (unknown, foreign, or on another policy). They are skipped and reported here rather than failing the request. items: type: string count: type: integer description: The number of projects removed (the length of unassignedProjectIds). required: - governancePolicy - unassignedProjectIds - skippedProjectIds - count required: - success - data GovernancePolicyProjectsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: projects: type: array description: The projects assigned to this governance policy (paginated). items: type: object properties: id: type: string description: The unique identifier of the project. name: type: string description: The name of the project. required: - id - name total: type: integer description: Total number of projects assigned to this policy. required: - projects - total required: - success - data EvaluateTraceRequest: type: object properties: metricCollection: type: string description: The name of the single-turn metric collection to evaluate the trace. required: - metricCollection EvaluateTraceResponse: type: object properties: success: type: boolean description: This is true if the trace was successfully evaluated. data: type: object properties: id: type: string description: This is the id of the trace, not to be confused with the UUID of the trace. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated EvaluateSpanRequest: type: object properties: metricCollection: type: string description: The name of the metric collection to use required: - metricCollection EvaluateSpanResponse: type: object properties: success: type: boolean description: This is true if the span was successfully evaluated. data: type: object properties: id: type: string description: This is the id of the span, not to be confused with the UUID of the trace. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated CreatePromptBranchRequest: type: object properties: branch: type: string description: The unique name of the branch to create in the prompt. required: - branch CreatePromptBranchResponse: type: object properties: success: type: boolean description: "This is true if the prompt branch was successfully created." data: type: object properties: name: type: string description: "This is the name of the newly created prompt branch." id: type: string description: "This is the ID of the newly created prompt branch." required: - name - id link: type: string description: "This is the URL to the prompt branch on the Confident AI platform." deprecated: type: boolean description: "This is true if this endpoint is deprecated." required: - success - data - link - deprecated UpdatePromptBranchRequest: type: object properties: name: type: string description: The new name of the branch you're trying to update. required: - branch UpdatePromptBranchResponse: type: object properties: success: type: boolean description: "This is true if the prompt branch was successfully deleted." data: type: object properties: id: type: string description: "This is the id of the updated prompt branch." required: - id deprecated: type: boolean description: "This is true if this endpoint is deprecated." required: - success - data - deprecated ListPromptBranchesResponse: type: object properties: success: type: boolean description: "This is true if the prompt branches were successfully retrieved." data: type: object properties: branches: type: array items: type: object properties: id: type: string description: "This is the unique id of the prompt branch." name: type: string description: "This is the name of the prompt branch." required: - id - name required: - branches deprecated: type: boolean description: "This is true if this endpoint is deprecated." required: - success - data - deprecated DeletePromptBranchResponse: type: object properties: success: type: boolean description: "This is true if the prompt branch was successfully deleted." data: type: object properties: id: type: string description: "This is the ID of the deleted prompt branch." required: - id deprecated: type: boolean description: "This is true if this endpoint is deprecated." required: - success - data - deprecated GetPromptsResponse: type: object properties: success: type: boolean description: "This is true if the prompts were successfully retrieved." data: type: object description: "This maps to all the prompts retrieved." properties: prompts: type: array items: type: object properties: id: type: string description: This is the unique id of the dataset. alias: type: string description: This is the alias of the dataset, which is unique within your project. type: type: string enum: [TEXT, LIST] description: This is the type of the prompt (TEXT or LIST). required: - id - alias - type required: - prompts deprecated: type: boolean description: "This is true if this prompts endpoint is deprecated." required: - success - data - deprecated CreatePromptRequest: type: object properties: alias: type: string description: The unique alias of the prompt. text: type: string description: The text content of the prompt. Supply this only if you are creating a text-based prompt. messages: type: array items: $ref: "#/components/schemas/PromptMessage" description: The list of messages that make up the prompt. Supply this only if you are creating a list-based prompt. interpolationType: type: string enum: [MUSTACHE, MUSTACHE_WITH_SPACE, FSTRING, DOLLAR_BRACKETS] default: FSTRING description: The type of interpolation format used in the prompt to insert dynamic variables. modelSettings: $ref: "#/components/schemas/ModelSettings" description: This is the model settings for the prompt. outputType: type: string enum: [TEXT, JSON, SCHEMA] default: TEXT description: The type of output expected from the prompt. outputSchema: $ref: "#/components/schemas/OutputSchema" description: This is the output schema definition when outputType is SCHEMA. tools: type: array items: $ref: "#/components/schemas/Tool" branch: type: string description: The name of the branch you want to push the new commit to. Defaults to "main" if not specified. required: - alias CreatePromptVersionRequest: type: object properties: hash: type: string description: The hash of the commit you wanna release a new version of. Only commits above the last versioned commit can be released as new versions. If ommited, the most recent commit will be versioned. CreatePromptVersionResponse: type: object properties: success: type: boolean description: This is true prompt version was successfully created data: type: object properties: version: type: string description: The version id generated by Confident AI of the new version you just released, it is always incremental. hash: type: string description: The hash code of the commit that was just versioned. This is the same hash you pass in request or the hash of the most recent commit on the Confident AI platform. required: - version - hash deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated ModelSettings: type: object additionalProperties: false properties: provider: $ref: "#/components/schemas/EvaluationModelProvider" description: This is the model provider for evaluation. name: type: string description: This is the name of the model. temperature: type: number format: float description: This controls randomness in the model's output. Higher values make output more random. maxTokens: type: integer description: This is the maximum number of tokens to generate. topP: type: number format: float description: This controls diversity via nucleus sampling. Lower values focus on more likely tokens. frequencyPenalty: type: number format: float description: This is the penalty for tokens based on their frequency in the text so far. presencePenalty: type: number format: float description: This is the penalty for tokens based on whether they appear in the text so far. stopSequence: type: array items: type: string description: This is the sequences where the model will stop generating further tokens. reasoningEffort: type: string enum: [MINIMAL, LOW, MEDIUM, HIGH] description: This is the level of reasoning effort for the model. verbosity: type: string enum: [LOW, MEDIUM, HIGH] description: This is the verbosity level for model output. OutputSchema: type: object properties: name: type: string description: This is the name of the output schema. fields: type: array items: $ref: "#/components/schemas/OutputSchemaField" description: This is the array of fields that define the output schema structure. required: - fields - name OutputSchemaField: type: object properties: id: type: string description: This is the unique identifier for the schema field. name: type: string description: This is the name of the schema field. type: type: string enum: [OBJECT, ARRAY, STRING, FLOAT, INTEGER, BOOLEAN, NULL] description: This is the data type for schema fields. required: type: boolean default: false description: This indicates whether the field is required in the output. parentId: type: string description: This is the ID of the parent field for nested structures. required: - id - type - name Tool: type: object properties: id: type: string description: This is the unique identifier for the schema field. name: type: string description: This is the name of the tool description: type: string description: This is the description of the tool mode: type: string enum: [STRICT, ADDITIONAL, NO_ADDITIONAL] description: This is the mode for your tool input fields structuredSchema: $ref: "#/components/schemas/OutputSchema" description: This is the schema for your tool input required: - name - description - mode # UpdatePromptRequest: # type: object # properties: # text: # type: string # description: This is the text content of the prompt. Supply this only if you are updating a text-based prompt. # messages: # type: array # items: # $ref: "#/components/schemas/PromptMessage" # description: This is the list of messages that make up the prompt. Supply this only if you are updating a list-based prompt. # interpolationType: # type: string # enum: [MUSTACHE, MUSTACHE_WITH_SPACE, FSTRING, DOLLAR_BRACKETS] # description: This is the type of interpolation format used in the prompt to insert dynamic variables. # modelSettings: # $ref: "#/components/schemas/ModelSettings" # description: This is the model settings for the prompt. # outputType: # type: string # enum: [TEXT, JSON, SCHEMA] # description: This is the type of output expected from the prompt. # outputSchema: # $ref: "#/components/schemas/OutputSchema" # description: This is the output schema definition when outputType is SCHEMA. # tools: # type: array # items: # $ref: "#/components/schemas/Tool" # description: These are the list of tools your agent has access to. # UpdatePromptResponse: # type: object # properties: # success: # type: boolean # description: This is true if the prompt was successfully updated. # data: # type: object # properties: # id: # type: string # description: This is the id of the updated prompt version, not to be confused with the alias you supplied or version number. # message: # type: string # description: A message that acknowledges that the prompt update was successful # required: # - id # - message # deprecated: # type: boolean # description: This is true if this endpoint is deprecated. # required: # - success # - data # - deprecated EvaluationModelProvider: type: string enum: [OPEN_AI, ANTHROPIC] description: This is the provider of the evaluation model. PromptMessage: type: object properties: role: type: string description: This is the role of the message, which can be user, assistant, system, or developer. content: type: string description: This is the text content of the message. required: - role - content GetPromptResponse: type: object properties: success: type: boolean description: This is true if the prompt was successfully retrieved. data: type: object properties: id: description: This is the id of the prompt version generated by Confident AI, not to be confused with the alias you supplied or version number. type: string version: type: string description: The version number of the prompt. hash: type: string description: This is the commit hash of the prompt pulled label: type: string description: The user-defined label for a specific version of the prompt. text: description: This is the text content of the prompt, which is null if the prompt is a list prompt. type: string messages: description: This is the list of messages associated with the prompt, which is null if the prompt is a text prompt. type: array items: $ref: "#/components/schemas/PromptMessage" type: description: This is the type of the prompt, which can be either simple text or a list of messages. type: string enum: [TEXT, LIST] interpolationType: type: string enum: [MUSTACHE, MUSTACHE_WITH_SPACE, FSTRING, DOLLAR_BRACKETS] description: The type of interpolation format used in the prompt to insert dynamic variables. modelSettings: $ref: "#/components/schemas/ModelSettings" description: This is the model settings for the prompt. outputType: type: string enum: [TEXT, JSON, SCHEMA] description: The type of output expected from the prompt. outputSchema: $ref: "#/components/schemas/OutputSchema" description: This is the output schema definition when outputType is SCHEMA. tools: type: array items: $ref: "#/components/schemas/Tool" required: - id - type - interpolationType - hash - outputType deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated CreatePromptResponse: type: object properties: success: type: boolean description: This is true if the prompt was successfully created. data: type: object description: This maps to the prompt version id. properties: promptId: type: string description: This is the id of the prompt generated by Confident AI, not to be confused with the alias you supplied or version number. hash: type: string description: This is the hash of the commit created by Confident AI for pushing this prompt, not to be confused with version number. required: - promptId - hash link: type: string description: This is the URL to redirect to after the prompt is created. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - link - deprecated PromptVersionsResponse: type: object properties: success: type: boolean description: This is true if the prompt versions were successfully retrieved. data: type: object additionalProperties: false properties: textVersions: description: This is the list of versions associated with the text prompt, which is null if the prompt is a messages prompt. type: array items: $ref: "#/components/schemas/PromptVersion" messagesVersions: description: This is the list of versions associated with the messages prompt, which is null if the prompt is a text prompt. type: array items: $ref: "#/components/schemas/PromptVersion" deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated PromptCommit: type: object properties: id: type: string description: The id of a commit generated by Confident AI, not to be confused with prompt id or commit hash. hash: type: string description: The hash of a commit generated by Confident AI. message: type: string description: The message associated with a commit, not to be confused with a prompt's message. required: - id - hash - message PromptCommitsResponse: type: object properties: success: type: boolean description: This is true if the prompt versions were successfully retrieved. data: type: object additionalProperties: false properties: commits: type: array items: $ref: "#/components/schemas/PromptCommit" deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated PromptVersion: type: object additionalProperties: false properties: id: type: string description: This is the id of the prompt version generated by Confident AI, not to be confused with the version. version: type: string description: This is the version number of a prompt version. required: - id - version UserReference: type: object properties: id: type: string description: The id of the user. email: type: string description: The email address of the user. name: type: string nullable: true description: The name of the user. image: type: string nullable: true description: The image of the user. required: - id - email - name - image Annotation: type: object properties: id: type: string description: This is the id of the annotation generated by Confident AI, not to be confused with the alias you supplied or version number. rating: type: integer description: This is the annotated rating score. type: type: enum enum: [THUMBS_RATING, FIVE_STAR_RATING] description: This is the type of annotation, which can be either thumbs rating or five star rating. name: type: string description: The name of the annotation. expectedOutcome: type: string description: This is the annotated expected outcome, for conversation annotations. expectedOutput: type: string description: This is the annotated expected output, for span and trace annotations. explanation: type: string description: This is the explanation for the annotation. createdAt: type: string format: date-time description: The timestamp when the annotation was created. traceUuid: type: string description: The UUID of the trace associated with this annotation, if applicable. spanUuid: type: string description: The UUID of the span associated with this annotation, if applicable. threadId: type: string description: The ID of the thread associated with this annotation, if applicable. testCaseId: type: string description: The ID of the test case associated with this annotation, if applicable. user: allOf: - $ref: "#/components/schemas/UserReference" nullable: true description: The user who created this annotation. userEmail: type: string nullable: true deprecated: true description: The email address of the user created this annotation. The field is being deprecated. Please use `user.email` instead. required: - id - rating - type CreateAnnotation: type: object description: Request schema to create an annotation for a trace, span, or thread. properties: traceUuid: type: string description: This is the trace UUID to annotate. Exactly one of traceUuid, spanUuid, or threadId must be provided. spanUuid: type: string description: This is the span UUID to annotate. Exactly one of traceUuid, spanUuid, or threadId must be provided. threadId: type: string description: This is the thread ID to annotate. Exactly one of traceUuid, spanUuid, or threadId must be provided. rating: type: number description: This is the annotated rating score, which must be 0 or 1 if the annotation is a thumb rating and an integer from 1 to 5 if the annotation is a five star rating. type: type: string enum: [THUMBS_RATING, FIVE_STAR_RATING] default: THUMBS_RATING description: This is the annotation typem which defaults to THUMBS_RATING. expectedOutput: type: string description: This is the expected output for trace/span annotations, which mustn't be provided when annotating a thread. expectedOutcome: type: string description: This is the expected outcome for thread annotations, which mustn't be provided when annotating a trace or span. explanation: type: string description: This is an explanation for the annotation. userId: type: string description: This can be any user ID that you want to associate with the annotation. required: - rating CreateAnnotationResponse: type: object properties: success: type: boolean description: This is true if the annotation was successfully created. data: type: object description: This maps to the id of the created annotation. properties: id: type: string description: This is the id of the created annotation. required: - id deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated GetTraceResponse: type: object properties: success: type: boolean description: This is true if the traces were successfully retrieved. data: type: object description: This maps to the list of traces retrieved. properties: traces: type: array description: This is the list of traces retrieved. items: $ref: "#/components/schemas/Trace" totalTraces: type: integer description: This is the total number of traces retrieved. required: - traces - totalTraces deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated RetrieveTraceResponse: type: object properties: success: type: boolean description: This is true if the trace was successfully retrieved. data: $ref: "#/components/schemas/GetTrace" description: This maps to the retrieved trace data. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated ListThreadsResponse: type: object properties: success: type: boolean description: This is true if the threads were successfully retrieved. data: type: object description: This maps to the list of threads retrieved. properties: threads: type: array description: This is the list of threads retrieved. items: $ref: "#/components/schemas/ThreadSummary" totalThreads: type: integer description: This is the total number of threads retrieved. required: - threads - totalThreads deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated GetThreadResponse: type: object properties: success: type: boolean description: This is true if the thread was successfully retrieved. data: $ref: "#/components/schemas/ThreadDetail" description: This maps to the thread retrieved. deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated ThreadSummary: type: object properties: threadId: type: string description: This is the thread ID you supplied when creating the thread. createdAt: type: string format: date-time description: This is when the thread was created. lastActivity: type: string format: date-time description: This is when the thread was last active. metadata: type: object additionalProperties: true description: This is the custom metadata attached to the thread. tags: type: array items: type: string description: This is the list of tags associated with the thread. metricCollectionName: type: string description: This is the name of the metric collection assigned to evaluate the thread. totalTraces: type: integer description: This is the total number of traces in this thread. required: - threadId - createdAt - lastActivity - totalTraces ThreadDetail: type: object properties: threadId: type: string description: This is the thread ID you supplied when creating the thread. createdAt: type: string format: date-time description: This is when the thread was created. lastActivity: type: string format: date-time description: This is when the thread was last active. metadata: type: object additionalProperties: true description: This is the custom metadata attached to the thread. tags: type: array items: type: string description: This is the list of tags associated with the thread. metricCollectionName: type: string description: This is the name of the metric collection assigned to evaluate the thread. totalTraces: type: integer description: This is the total number of traces in this thread. metricsData: type: array items: $ref: "#/components/schemas/MetricsData" description: This is the evaluation metrics data for the thread. annotations: type: array items: $ref: "#/components/schemas/Annotation" description: This is the list of annotations associated with the thread. traces: type: array items: $ref: "#/components/schemas/GetTrace" description: This is the list of traces in this thread. required: - threadId - createdAt - lastActivity - totalTraces GetTrace: type: object properties: uuid: type: string format: uuid description: This is the uuid of the trace, not to be confused with the trace id. input: additionalProperties: true description: This is the input to the trace. output: additionalProperties: true description: This is the output of the trace. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. name: type: string description: This is the name of the trace. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. environment: type: string enum: [production, development, staging, testing] description: This is the environment where the trace was created. threadId: type: string description: This is the thread id of the trace, which groups traces in the same thread into a conversation. testCaseId: type: string description: This is the test case id of the trace, which is only set if the trace was created in a testing environment. userId: type: string description: This is the user id you provided for this trace. projectId: type: string description: This is the id of the project where the trace lives. metricCollectionName: type: string description: This is the name of the metric collection assigned to evaluate the trace. retrievalContext: type: array description: This is the retrieval context associated with the trace, to be used for evaluations. items: type: string context: type: array description: This is the ideal retrieval context associated with the trace, to be used for evaluations. items: type: string expectedOutput: type: array description: This is the expected output associated with the trace, to be used for evaluations. items: $ref: "#/components/schemas/ToolCallData" expectedTools: type: array description: This is the list of expected tools associated with the trace, to be used for evaluations. items: $ref: "#/components/schemas/ToolCallData" metricsData: type: array description: This is the list of metrics data associated with the trace after running evaluations. items: $ref: "#/components/schemas/MetricsData" annotation: $ref: "#/components/schemas/Annotation" description: This is the text annotation for the trace. tags: type: array items: type: string description: This is the list of tags associated with the trace, which is useful for grouping and filtering for traces. spans: type: array items: $ref: "#/components/schemas/GetSpan" description: This is the list of spans in the trace. required: - id - uuid - startTime - endTime - projectId - spans MetricsData: type: object properties: id: type: string description: The unique identifier of the metrics data entry. projectId: type: string description: The project this metric data belongs to. traceUuid: type: string nullable: true description: The UUID of the trace this metric data is associated with, if any. spanUuid: type: string nullable: true description: The UUID of the span this metric data is associated with, if any. testCaseId: type: string nullable: true description: The ID of the test case this metric data is associated with, if any. testRunId: type: string nullable: true description: The ID of the test run this metric data is associated with, if any. threadId: type: string nullable: true description: The ID of the thread this metric data is associated with, if any. name: type: string description: The name of the metric. multiTurn: type: boolean description: Whether this metric was evaluated on a multi-turn conversation. score: type: number nullable: true description: The final metric score. reason: type: string nullable: true description: The reason for the metric score, generated by the evaluation model at evaluation time. success: type: boolean nullable: true description: Whether the metric score is above the threshold. createdAt: type: string format: date-time description: The time the metric data was created. evaluatedAt: type: string format: date-time nullable: true description: The time the metric was evaluated. threshold: type: number format: float description: The threshold for the metric, which determines if the metric is passing or failing. strictMode: type: boolean description: Whether the metric was run in strict mode, which outputs a binary score of 0 or 1. skipped: type: boolean description: Whether the metric evaluation was skipped. evaluationModel: type: string nullable: true description: The evaluation model used to run the evaluation. error: type: string nullable: true description: The error message if the evaluation failed. evaluationCost: type: number format: float nullable: true description: The cost of running the evaluation. verboseLogs: type: string nullable: true description: The verbose logs of the evaluation, which breaks down the metric score calculation into individual steps. required: - id - projectId - name - threshold - strictMode - skipped - multiTurn - createdAt GetSpan: type: object properties: id: type: string description: This is the id of the span generated by Confident AI, not to be confused with the uuid of the span. uuid: type: string format: uuid description: This is the uuid of the span, not to be confused with the span id. name: type: string description: This is the name of the span. input: description: This is the input to the span. output: description: This is the output of the span. error: type: string description: This is the error string that caused the span to fail, if an error occurred. parentUuid: type: string description: This is the uuid of the parent span, if any. startTime: type: string format: date-time description: This is the time the span started. endTime: type: string format: date-time description: This is the time the span ended. traceUuid: type: string description: This is the uuid of the trace containing the span. agentHandoffs: type: array items: string description: This is the list of agent handoffs associated with an agent span. availableTools: type: array items: string description: This is the list of available tools associated with an agent span. chunkSize: type: integer description: This is the chunk size of each retrieved context for a retriever span. costPerInputToken: type: number format: float description: This is the cost per input token of the LLM model for an LLM span. costPerOutputToken: type: number format: float description: This is the cost per output token of the LLM model for an LLM span. description: type: string description: This is a description if the span is a tool span. embedder: type: string description: This is the embedder model used in a retriever span. inputTokenCost: type: number format: float description: This is the total cost of the input tokens passed to the LLM model in an LLM span. inputTokenCount: type: integer description: This is the total number of input tokens passed to the LLM model in an LLM span. model: type: string description: This is the LLM model used in an LLM span. provider: type: string description: This is the LLM provider used in an LLM span. integration: type: string description: This is the integration associated with the span. outputTokenCost: type: number format: float description: This is the total cost of the output tokens generated by the LLM model in an LLM span. outputTokenCount: type: integer description: This is the total number of output tokens generated by the LLM model in an LLM span. status: type: string enum: [SUCCESS, FAILED, PENDING] description: This is the error status of the span. topK: type: integer description: This is the top K chunks retrieved from your knowledge base. type: type: string description: This is the type of the span. metricCollectionName: type: string description: This is the name of the metric collection to evaluate the span. retrievalContext: type: array items: type: string description: This is the retrieval context of your span, which is to be used for evaluation. context: type: array items: type: string description: This is the ideal retrieval context of your span, which is to be used for evaluation. expectedOutput: type: string description: This is the expected output of your span, which is the ideal actual output and to be used for evaluation. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called by your span, which is to be used for evaluation. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools to be called by the span, which is to be used for evaluation. metricsData: type: array items: $ref: "#/components/schemas/MetricsData" description: This is the metrics data associated with the span. annotation: $ref: "#/components/schemas/Annotation" description: This is the text annotation for the span. environment: type: string enum: [production, development, staging, testing] description: This is the environment where your span was posted, which helps with separating and debugging spans from different environments on the Confident AI platform. metadata: type: object additionalProperties: true description: This is any additional metadata associated with the span. required: - id - uuid - name - startTime - endTime - traceUuid - status - type TestRunResponse: type: object properties: success: type: boolean description: This is true if the test run was successfully retrieved. data: type: object additionalProperties: false properties: metricsScores: type: array items: $ref: "#/components/schemas/MetricScores" description: The aggregated metric scores across all test cases. testCases: type: array items: oneOf: - $ref: "#/components/schemas/TestRunTestCase" title: "LLMTestCase" - $ref: "#/components/schemas/TestRunConversationalTestCase" title: "ConversationalTestCase" description: The test cases in this test run. Will contain either single-turn test cases or conversational test cases, but not both. multiTurn: type: boolean description: Whether this test run contains multi-turn test cases. identifier: type: string description: A unique identifier for the test run. status: type: string enum: ["IN_PROGRESS", "COMPLETED", "ERRORED", "CANCELLED"] description: The current status of the test run. testsPassed: type: number description: The number of test cases that passed. testsFailed: type: number description: The number of test cases that failed. totalTests: type: number description: The total number of test cases in this test run. runDuration: type: number format: float description: The total duration of the test run in seconds. required: - testCases - metricsScores deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated TestRunTestCase: type: object additionalProperties: false properties: id: type: string description: This is the id of the test case generated by Confident AI. name: type: string description: This is the name of the test case. input: type: string description: This is the input of the test case. actualOutput: type: string description: This is the actual output of the test case. context: type: array items: type: string description: This is the context of the test case. retrievalContext: type: array items: type: string description: This is the retrieval context of the test case. expectedOutput: type: string description: This is the expected output of the test case. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the tools called of the test case. expectedTools: type: array items: $ref: "#/components/schemas/ToolCallData" description: This is the expected tools of the test case. metricsData: type: array items: $ref: "#/components/schemas/MetricsData" description: The metric evaluation results for this test case. comments: type: string description: Any comments associated with this test case. additionalMetadata: type: object additionalProperties: true description: Additional metadata associated with this test case. success: type: boolean description: Whether this test case passed all metric thresholds. runDuration: type: number format: float description: The duration of the test case evaluation in seconds. evaluationCost: type: number format: float description: The cost of evaluating this test case. trace: $ref: "#/components/schemas/Trace" description: "This is the trace dictionary of a test case." required: - id - name - input TestRunConversationalTestCase: type: object additionalProperties: false properties: id: type: string description: This is the id of the conversational test case generated by Confident AI. name: type: string description: This is the name of the conversational test case. turns: type: array items: $ref: "#/components/schemas/TestRunTurn" description: The list of turns in the conversation. scenario: type: string description: A description of the conversation context. expectedOutcome: type: string description: The expected outcome or ideal conversation flow. userDescription: type: string description: A description of the user in the conversation. context: type: array items: type: string description: The context provided for the conversation. comments: type: string description: Any comments associated with this test case. additionalMetadata: type: object additionalProperties: true description: Additional metadata associated with this test case. metricsData: type: array items: $ref: "#/components/schemas/MetricsData" description: The metric evaluation results for this test case. success: type: boolean description: Whether this test case passed all metric thresholds. runDuration: type: number format: float description: The duration of the test case evaluation in seconds. evaluationCost: type: number format: float description: The cost of evaluating this test case. required: - id - name - turns TestRunTurn: type: object properties: role: type: string enum: ["user", "assistant"] description: The role of the turn, either user or assistant. content: type: string description: The message content of the turn. toolsCalled: type: array items: $ref: "#/components/schemas/ToolCallData" description: The tools called to generate the LLM response for this turn. retrievalContext: type: array items: type: string description: The contexts retrieved to generate the LLM response for this turn. required: - role - content MetricScores: type: object additionalProperties: false properties: fails: type: number description: This is the number of times this metric failed to pass the threshold. errors: type: number description: This is the number of times this metric errored during evaluation. metric: type: string description: This is the name of the metric. passes: type: number description: This is the number of times this metric passed the threshold. scores: type: array items: type: number description: This is an array of scores for the metric across test cases. required: - fails - errors - metric - passes - scores ListAnnotationsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: Contains the list of annotations and pagination information. properties: annotations: type: array items: $ref: "#/components/schemas/Annotation" description: List of annotations matching the filter criteria. total: type: integer description: Total number of annotations matching the filter criteria. page: type: integer description: Current page number of annotations returned in this response. pageSize: type: integer description: Maximum number of annotations returned in this response. required: - annotations - total - page - pageSize required: - success - data GetAnnotationResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: Contains the annotation information. properties: annotation: $ref: "#/components/schemas/Annotation" description: The annotation details. required: - annotation required: - success - data UpdateAnnotation: type: object description: Request schema to update an existing annotation. properties: rating: type: number description: Updated rating score. Must be 0 or 1 for THUMBS_RATING, 1-5 for FIVE_STAR_RATING. type: type: string enum: [THUMBS_RATING, FIVE_STAR_RATING] description: Updated annotation type. expectedOutput: type: string description: Updated expected output for trace/span annotations. expectedOutcome: type: string description: Updated expected outcome for thread annotations. explanation: type: string description: Updated explanation for the annotation. UpdateAnnotationResponse: type: object properties: success: type: boolean description: This is true if the annotation was successfully updated. data: type: object description: This maps to the id of the updated annotation. properties: id: type: string description: This is the id of the updated annotation. required: - id deprecated: type: boolean description: This is true if this endpoint is deprecated. required: - success - data - deprecated ErrorResponse: type: object properties: message: type: string description: Error message describing what went wrong. required: - message ListAIConnectionsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object properties: aiConnections: type: array items: type: object properties: id: type: string description: The unique identifier of the AI connection. name: type: string description: The name of the AI connection. endpoint: type: string nullable: true description: The endpoint of the AI connection where requests are made. active: type: boolean description: Whether the AI connection is active. required: - id - name - active required: - aiConnections required: - success - data ListRiskAssessmentFrameworksResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object properties: frameworks: description: List of all frameworks present in this project type: array items: type: object properties: id: type: string description: The unique identifier of the framework. Use this value when running an assessment. name: type: string description: The name of the framework. description: type: string nullable: true description: The description of the framework. riskCategories: type: array items: type: object properties: name: type: string description: The name of the risk category. Use this value when running an assessment. numVulnerabilityTypes: type: integer description: The number of vulnerability types in this category. numAttackMethods: type: integer description: The number of attack methods in this category. required: - name - numVulnerabilityTypes - numAttackMethods required: - id - name - riskCategories required: - frameworks required: - success - data RunRiskAssessmentRequest: type: object required: - riskCategories - exposure properties: riskCategories: type: array items: type: string minItems: 1 description: The names of the risk categories to assess, exactly as returned by the list frameworks endpoint. exposure: type: string enum: [LOW, MEDIUM, HIGH] description: The exposure level of the application under test. identifier: type: string description: An optional label for the run. aiConnectionId: type: string description: The ID of the AI connection to target, you can find this in the Project Settings → AI Connections on the platform. Required for `AI_CONNECTION` mode. promptAlias: type: string description: The alias of the prompt to target. Required for `PROMPT` mode. promptCommit: type: string description: The prompt commit hash to target. Defaults to the latest commit on the prompt's main branch. generationMode: type: string enum: [AI_CONNECTION, PROMPT] description: The target type. Optional when exactly one of `aiConnectionId` or `promptAlias` is provided. attackEngine: type: object description: Optional attack generation settings. properties: generationGuidelines: type: array items: type: string description: Freeform guidelines that are used to guide the simulator model when generating attacks, give clear instructions to generate attacks that are more customized for your use case. RunRiskAssessmentResponse: type: object properties: success: type: boolean description: Indicates if the risk assessment run was started. data: type: object properties: id: type: string description: The unique identifier of the created risk assessment. required: - id link: type: string description: A link to view the risk assessment. required: - success - data CreateAnnotationQueue: type: object required: - name - type properties: name: type: string description: The name of the annotation queue type: type: string enum: [TRACE, SPAN, THREAD, GOLDEN, TEST_RUN] description: The type of items this queue will contain ListAnnotationQueuesResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object properties: annotationQueues: type: array items: type: object properties: id: type: string description: The unique identifier of the annotation queue. name: type: string description: The name of the annotation queue. type: type: string enum: [TRACE, SPAN, THREAD, GOLDEN, TEST_RUN] description: The type of items this queue contains. createdAt: type: string format: date-time description: The timestamp when the annotation queue was created. updatedAt: type: string format: date-time description: The timestamp when the annotation queue was last updated. completedItems: type: integer description: The number of items in the queue that have been completed. totalItems: type: integer description: The total number of items in the queue. completionPercentage: type: integer description: The percentage of items completed in the queue (0-100). total: type: integer description: The total number of annotation queues matching the filter criteria. limit: type: integer description: The maximum number of annotation queues returned in this response. offset: type: integer description: The number of annotation queues skipped in this response. required: - annotationQueues - total - limit - offset required: - success - data CreateAnnotationQueueResponse: type: object properties: success: type: boolean description: Indicates if the annotation queue was successfully created. data: type: object description: Contains the created annotation queue information. properties: id: type: string description: The unique identifier of the newly created annotation queue. required: - id required: - success - data GetAnnotationQueueResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: Contains the annotation queue information. properties: annotationQueue: type: object description: The annotation queue details with comprehensive statistics. properties: id: type: string description: The unique identifier of the annotation queue. name: type: string description: The name of the annotation queue. type: type: string enum: [TRACE, SPAN, THREAD, GOLDEN, TEST_RUN] description: The type of items this queue contains. createdAt: type: string format: date-time description: The timestamp when the annotation queue was created. updatedAt: type: string format: date-time description: The timestamp when the annotation queue was last updated. testRunId: type: string description: The ID of the test run associated with this queue, if applicable. completedItems: type: integer description: The number of items in the queue that have been completed. totalItems: type: integer description: The total number of items in the queue. pendingItems: type: integer description: The number of items in the queue that are still pending completion. completionPercentage: type: integer description: The percentage of items completed in the queue (0-100). assignedItems: type: integer description: The number of items in the queue that have been assigned to users. assignmentBreakdown: type: object description: A breakdown of assignments by user email, showing how many items are assigned to each user and how many they have completed. additionalProperties: type: object properties: assigned: type: integer description: The number of items assigned to this user. completed: type: integer description: The number of items completed by this user. required: - annotationQueue required: - success - data DeleteAnnotationQueueResponse: type: object properties: success: type: boolean description: Indicates if the annotation queue was successfully deleted. data: type: object description: Contains the deleted annotation queue information. properties: id: type: string description: The unique identifier of the deleted annotation queue. required: - id required: - success - data ListQueueItemsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: Contains the list of queue items. properties: items: type: array description: List of items in the annotation queue. items: type: object description: A single item in the annotation queue. properties: id: type: string description: The unique identifier of the queue item. traceUuid: type: string description: The UUID of the trace associated with this queue item, if applicable. spanUuid: type: string description: The UUID of the span associated with this queue item, if applicable. threadId: type: string description: The ID of the thread associated with this queue item, if applicable. testCaseId: type: string description: The ID of the test case associated with this queue item, if applicable. addedAt: type: string format: date-time description: The timestamp when this item was added to the queue. completed: type: boolean description: Whether this queue item has been completed. assignedToEmail: type: string description: The email address of the user assigned to this queue item, if any. required: - items required: - success - data AnnotateItem: type: object description: | Provide criteria ratings (the flat `rating` fields below, or the `annotations` array) and/or custom form-field answers (`responses`). At least one of these must be present. properties: rating: type: integer description: Rating value for a single criterion (0-1 for THUMBS_RATING, 1-5 for FIVE_STAR_RATING). Shorthand for a one-entry `annotations` array. type: type: string enum: [THUMBS_RATING, FIVE_STAR_RATING] default: THUMBS_RATING description: Type of the rating. name: type: string description: Criterion name. Omit for the default criterion; otherwise it must match a configured custom criterion (or a form criteria field), or the annotation will not appear. expectedOutcome: type: string description: Expected outcome (for thread items only). expectedOutput: type: string description: Expected output (for trace/span items only). explanation: type: string description: Explanation for the rating. annotations: type: array description: One entry per criterion. Use instead of the flat fields to rate multiple criteria in a single request. items: $ref: "#/components/schemas/AnnotateCriteria" formResponses: type: array description: Answers to a form's custom fields, addressed by each field's visible label. Accepted only for queues with an attached form, and requires `annotatorEmail`. items: $ref: "#/components/schemas/AnnotateFormResponse" annotatorEmail: type: string format: email description: Project member to attribute the work to. Required when submitting `formResponses`, otherwise they will not appear in the platform. markAsCompleted: type: boolean default: true description: Whether to mark the queue item as completed. For form queues, completing enforces the form's required fields. AnnotateCriteria: type: object required: - rating properties: rating: type: integer description: Rating value (0-1 for THUMBS_RATING, 1-5 for FIVE_STAR_RATING). type: type: string enum: [THUMBS_RATING, FIVE_STAR_RATING] default: THUMBS_RATING description: Type of the rating. name: type: string description: Criterion name. Omit for the default criterion; otherwise it must match a configured custom criterion or a form criteria field. explanation: type: string description: Explanation for the rating. expectedOutput: type: string description: Expected output (for trace/span items only). expectedOutcome: type: string description: Expected outcome (for thread items only). AnnotateFormResponse: type: object required: - label properties: label: type: string description: The visible label of the custom field being answered. Must be unique within the form and must not be a criteria field's label. value: nullable: true description: The answer, shaped to the field type. Null clears the answer. oneOf: - type: string description: Text or single-select fields. - type: number description: Number or decimal fields. - type: boolean description: Yes/No fields (also accepts the strings "Yes"/"No"). - type: array items: type: string description: Multi-select fields. AnnotateItemResponse: type: object properties: success: type: boolean description: Indicates if the queue item was successfully annotated. data: type: object description: Contains the created annotation and form-response identifiers. properties: id: type: string nullable: true description: The first created annotation's id, kept for backward compatibility. Null when only form responses were submitted. annotationIds: type: array items: type: string description: Identifiers of the created annotations. formResponseIds: type: array items: type: string description: Identifiers of the created form responses. required: - success - data BatchAnnotateItem: allOf: - type: object required: - queueItemId properties: queueItemId: type: string description: The id of the queue item to annotate. - $ref: "#/components/schemas/AnnotateItem" BatchAnnotate: type: object required: - items properties: annotatorEmail: type: string format: email description: Default annotator for items that omit it. A per-item `annotatorEmail` takes precedence. Required (here or per item) whenever an item submits `formResponses`. markAsCompleted: type: boolean description: Default completion flag for items that omit it (effectively `true`). A per-item value takes precedence. items: type: array minItems: 1 description: The items to annotate. Processed independently (best-effort). items: $ref: "#/components/schemas/BatchAnnotateItem" BatchAnnotateResult: type: object description: The outcome for a single item. `success` distinguishes the success and failure shapes. required: - queueItemId - success properties: queueItemId: type: string success: type: boolean id: type: string nullable: true description: First created annotation's id (success only; null when only form responses were submitted). annotationIds: type: array items: type: string description: Identifiers of the created annotations (success only). formResponseIds: type: array items: type: string description: Identifiers of the created form responses (success only). error: type: string description: Why this item failed (failure only). BatchAnnotateResponse: type: object required: - success - data properties: success: type: boolean description: Indicates the batch request was well-formed and processed. Per-item outcomes are in `data.results`. data: type: object required: - results properties: results: type: array items: $ref: "#/components/schemas/BatchAnnotateResult" # Metrics Data Schemas MetricDataItem: type: object required: - name - success - threshold - strictMode properties: name: type: string description: The name of the metric (e.g., "Correctness", "Relevancy") score: type: number description: The metric score (optional, typically 0-1) success: type: boolean description: Whether the metric evaluation passed threshold: type: number description: The threshold used for evaluation strictMode: type: boolean description: Whether strict mode was enabled reason: type: string description: Explanation for the metric result evaluationModel: type: string description: The model used for evaluation (e.g., "gpt-4") evaluationCost: type: number description: Cost of the evaluation in USD verboseLogs: type: string description: Detailed logs from the evaluation error: type: string description: Error message if evaluation failed DirectMetricData: type: object required: - metricData properties: metricData: type: array items: $ref: "#/components/schemas/MetricDataItem" description: Array of metric data items to post traceUuid: type: string description: UUID of the trace to attach metrics to (mutually exclusive with other entity IDs) spanUuid: type: string description: UUID of the span to attach metrics to (mutually exclusive with other entity IDs) threadId: type: string description: ID of the thread to attach metrics to (mutually exclusive with other entity IDs) testCaseId: type: integer description: ID of the test case to attach metrics to (mutually exclusive with other entity IDs) conversationalTestCaseId: type: string description: ID of the conversational test case to attach metrics to (mutually exclusive with other entity IDs) DirectMetricDataResponse: type: object properties: success: type: boolean description: Whether the operation was successful message: type: string description: Success message data: type: object properties: metricsCreated: type: integer description: Number of metric data items created ListMetricsDataResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: metricsData: type: array description: List of metric data entries items: $ref: "#/components/schemas/MetricsData" totalMetricsData: type: integer description: Total number of metric data entries matching the query page: type: integer description: Current page number pageSize: type: integer description: Number of items per page required: - success - data # Projects Schemas Project: type: object properties: id: type: string description: Unique identifier for the project name: type: string description: Name of the project description: type: string nullable: true description: Optional description of the project organizationId: type: string description: ID of the organization this project belongs to created_at: type: string format: date-time description: ISO 8601 timestamp of when the project was created governancePolicy: type: object nullable: true description: The governance policy the project is enrolled in, or null if it is not enrolled. properties: id: type: string description: The unique identifier of the governance policy. name: type: string description: The name of the governance policy. GetProjectsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: projects: type: array description: List of projects in the organization items: $ref: "#/components/schemas/Project" required: - success - data CreateProject: type: object properties: name: type: string description: Name of the project (must be unique within the organization) description: type: string description: Optional description of the project email: type: string format: email description: Optional email of an existing organization member to assign as the project's Owner. If omitted, the project has no member and is accessible via its API key and to organization admins. required: - name CreateProjectResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: project: $ref: "#/components/schemas/Project" apiKey: allOf: - $ref: "#/components/schemas/ApiKey" description: A default project-scoped API key provisioned with the project. Its full `value` is returned only once, in this response. required: - success - data UpdateProject: type: object properties: name: type: string description: New name for the project description: type: string description: New description for the project UpdateProjectResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: project: $ref: "#/components/schemas/Project" required: - success - data GetProjectResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: project: $ref: "#/components/schemas/Project" required: - success - data DeleteProjectResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: string description: ID of the deleted project deleted: type: boolean description: Always `true` when the project was deleted required: - success - data # API Key Schemas ApiKey: type: object properties: id: type: integer description: Unique identifier for the API key name: type: string nullable: true description: Human-readable label for the API key valid: type: boolean description: Whether the key is active. A deactivated key is rejected on authentication. value: type: string description: | The API key value. This is redacted (only the last 6 characters are shown, prefixed with asterisks) on every response **except** the create response, where the full value is returned exactly once. created_at: type: string format: date-time description: ISO 8601 timestamp of when the key was created lastUsed: type: string format: date-time nullable: true description: ISO 8601 timestamp of when the key was last used to authenticate, or `null` if never used required: - id - valid - value - created_at CreateApiKey: type: object properties: name: type: string description: Human-readable label for the API key required: - name UpdateApiKey: type: object properties: valid: type: boolean description: Set to `false` to deactivate the key, or `true` to reactivate it required: - valid ApiKeyResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: apiKey: $ref: "#/components/schemas/ApiKey" required: - success - data ApiKeyListResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: apiKeys: type: array items: $ref: "#/components/schemas/ApiKey" required: - success - data DeleteApiKeyResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: integer description: ID of the revoked API key deleted: type: boolean description: Always `true` when the key was revoked required: - success - data # Organization Schemas Organization: type: object properties: id: type: string description: Unique identifier for the organization name: type: string description: Name of the organization plan: type: string enum: [TRIAL, FREE, STARTER, ENTERPRISE, TEAM, PREMIUM] description: The organization's current billing plan created_at: type: string format: date-time description: ISO 8601 timestamp of when the organization was created required: - id - name - plan - created_at UpdateOrganization: type: object properties: name: type: string description: New name for the organization required: - name OrganizationResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: organization: $ref: "#/components/schemas/Organization" required: - success - data # Member Schemas Member: type: object properties: id: type: string description: Unique identifier for the member email: type: string description: The member's email address name: type: string nullable: true description: The member's display name image: type: string nullable: true description: URL of the member's avatar image organizationRole: allOf: - $ref: "#/components/schemas/NamedRef" nullable: true description: The member's organization role, or `null` if they have no role required: - id - email UpdateMemberRole: type: object properties: roleId: type: string description: The id of the role to assign to the member. required: - roleId ListMembersResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: members: type: array description: List of members in the organization items: $ref: "#/components/schemas/Member" total: type: integer description: Total number of members in the organization required: - success - data MemberResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: member: $ref: "#/components/schemas/Member" required: - success - data RemoveMemberResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: string description: ID of the removed member removed: type: boolean description: Always `true` when the member was removed required: - success - data # Invitation Schemas Invitation: type: object properties: id: type: integer description: Unique identifier for the invitation email: type: string description: The invited email address status: type: string enum: [PENDING, ACCEPTED, DECLINED] description: The current status of the invitation created_at: type: string format: date-time description: ISO 8601 timestamp of when the invitation was created organizationRoleId: type: string nullable: true description: The id of the organization role the invitee will receive, or `null` for the default required: - id - email - status - created_at CreateInvitations: type: object properties: emails: type: array description: One or more email addresses to invite items: type: string format: email organizationRoleId: type: string format: uuid description: Optional organization role id to assign to all invitees. The `Owner` role cannot be assigned. required: - emails ListInvitationsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: invitations: type: array description: List of organization invitations items: $ref: "#/components/schemas/Invitation" required: - success - data DeleteInvitationResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: integer description: ID of the revoked invitation deleted: type: boolean description: Always `true` when the invitation was revoked required: - success - data # Role & Policy Schemas NamedRef: type: object properties: id: type: string description: Unique identifier name: type: string description: Name required: - id - name Role: type: object properties: id: type: string description: Unique identifier for the role name: type: string description: Name of the role description: type: string nullable: true description: Optional description of the role organizationId: type: string nullable: true description: | The owning organization id, or `null` for global, system-defined roles that are available to every organization. policies: type: array description: The policies attached to this role items: $ref: "#/components/schemas/NamedRef" required: - id - name CreateOrUpdateRole: type: object properties: name: type: string description: Name of the role (must be unique within the organization and cannot be a system role name) description: type: string nullable: true description: Optional description of the role policyIds: type: array description: The ids of the policies to attach to this role items: type: string required: - name - policyIds RoleResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: role: $ref: "#/components/schemas/Role" required: - success - data ListRolesResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: roles: type: array description: List of roles available to the organization, including global system roles items: $ref: "#/components/schemas/Role" required: - success - data DeleteRoleResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: string description: ID of the deleted role deleted: type: boolean description: Always `true` when the role was deleted required: - success - data Policy: type: object properties: id: type: string description: Unique identifier for the policy name: type: string description: Name of the policy description: type: string nullable: true description: Optional description of the policy permissions: type: array description: The permissions granted by this policy items: $ref: "#/components/schemas/NamedRef" required: - id - name CreateOrUpdatePolicy: type: object properties: name: type: string description: Name of the policy description: type: string nullable: true description: Optional description of the policy permissionIds: type: array description: The ids of the permissions granted by this policy items: type: string required: - name - permissionIds PolicyResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: policy: $ref: "#/components/schemas/Policy" required: - success - data ListPoliciesResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: policies: type: array description: List of the organization's custom policies items: $ref: "#/components/schemas/Policy" required: - success - data DeletePolicyResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: id: type: string description: ID of the deleted policy deleted: type: boolean description: Always `true` when the policy was deleted required: - success - data # Permission Schemas Permission: type: object properties: id: type: string description: Unique identifier for the permission name: type: string description: "The permission name, formatted as `resource:action` (e.g. `billing:read`)" description: type: string nullable: true description: Optional human-readable description of the permission required: - id - name ListPermissionsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: permissions: type: array description: List of all available permissions items: $ref: "#/components/schemas/Permission" required: - success - data # Project Member Schemas ProjectMember: type: object properties: id: type: string description: Unique identifier for the member email: type: string description: The member's email address name: type: string nullable: true description: The member's display name image: type: string nullable: true description: URL of the member's avatar image projectRole: allOf: - $ref: "#/components/schemas/NamedRef" nullable: true description: The member's project role, or `null` if they have no role required: - id - email ListProjectMembersResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: members: type: array description: List of members in the project items: $ref: "#/components/schemas/ProjectMember" total: type: integer description: Total number of members in the project required: - success - data ProjectMemberResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: member: $ref: "#/components/schemas/ProjectMember" required: - success - data # Project Invitation Schemas ProjectInvitation: type: object properties: id: type: integer description: Unique identifier for the invitation email: type: string description: The invited email address status: type: string enum: [PENDING, ACCEPTED, DECLINED] description: The current status of the invitation created_at: type: string format: date-time description: ISO 8601 timestamp of when the invitation was created projectRoleId: type: string nullable: true description: The id of the project role the invitee will receive, or `null` for the default required: - id - email - status - created_at CreateProjectInvitations: type: object properties: emails: type: array description: One or more email addresses to invite items: type: string format: email projectRoleId: type: string format: uuid description: Optional project role id to assign to all invitees. The `Owner` role cannot be assigned. required: - emails ListProjectInvitationsResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: invitations: type: array description: List of project invitations items: $ref: "#/components/schemas/ProjectInvitation" required: - success - data # Project Role & Policy Schemas ProjectRole: type: object properties: id: type: string description: Unique identifier for the role name: type: string description: Name of the role description: type: string nullable: true description: Optional description of the role projectId: type: string nullable: true description: | The owning project id, or `null` for global, system-defined roles that are available to every project. policies: type: array description: The policies attached to this role items: $ref: "#/components/schemas/NamedRef" required: - id - name ProjectRoleResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: role: $ref: "#/components/schemas/ProjectRole" required: - success - data ListProjectRolesResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: roles: type: array description: List of roles available to the project, including global system roles items: $ref: "#/components/schemas/ProjectRole" required: - success - data ListProjectPoliciesResponse: type: object properties: success: type: boolean description: Indicates if the request was successful data: type: object properties: policies: type: array description: List of the project's custom policies items: $ref: "#/components/schemas/Policy" required: - success - data InvitationResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object properties: invitation: $ref: "#/components/schemas/Invitation" required: - invitation required: - success - data ProjectInvitationResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object properties: invitation: $ref: "#/components/schemas/ProjectInvitation" required: - invitation required: - success - data DashboardWidgetType: type: string description: The visualization type of a widget. enum: - LINE - AREA - BAR - STACKED_BAR - TABLE - BIG_NUMBER DashboardWidgetUnit: type: string description: The unit a widget's values are measured in. enum: - COUNT - PERCENT - SCORE - SECONDS - USD - MILLISECONDS DashboardWidgetMode: type: string description: >- How a widget aggregates its lines. `TIME_SERIES` plots each configured line over time; `DIMENSION_SERIES` takes a single metric and splits it into one series per value of the widget's dimension. This is the widget's saved configuration — it does not by itself describe the shape of a query response (use `kind` on the query result for that). enum: - TIME_SERIES - DIMENSION_SERIES DashboardWidgetBucketMode: type: string description: >- How a widget's data is bucketed over the query time range. `SERIES` splits the range into one bucket per `granularity` interval (a time series); `RANGE` aggregates the whole range into a single bucket (one total, as used by BIG_NUMBER widgets). Defaults to `SERIES` when omitted. enum: - SERIES - RANGE DashboardDataModel: type: string description: The entity a line aggregates over. enum: - TRACE - SPAN - LLM_SPAN - AGENT_SPAN - RETRIEVER_SPAN - TOOL_SPAN - CUSTOM_SPAN - THREAD - END_USER - METRIC_DATA - ANNOTATION DashboardDimension: type: string description: The dimension a widget breaks down by when mode is DIMENSION_SERIES. enum: - project - trace_name - span_name - model - type - thread_id - test_case_id - test_run_id - end_user - source - annotator - name - error - prompt_alias - tag - label - evaluation_model - prompt_version - prompt_label - prompt_commit_hash - metadata - classifier DashboardLineColor: type: string description: The color of a line. If omitted or unrecognized, a color is auto-assigned from the palette. enum: - AMBER - VIOLET - EMERALD - BLUE - PINK - CYAN - ROSE - LIME - TEAL - ORANGE DashboardAggregation: type: string description: The aggregation applied to a line. The set of valid values depends on the line's dataModel. enum: - COUNT - ERROR_RATE - PASS_RATE - UNIQUE_END_USERS - UNIQUE_THREADS - UNIQUE_USERS - UNIQUE_METADATA_VALUES - AVG_LATENCY - P50_LATENCY - P90_LATENCY - P99_LATENCY - TOTAL_COST - AVG_COST - INPUT_COST - OUTPUT_COST - AVG_COST_PER_USER - INPUT_TOKENS - OUTPUT_TOKENS - TOTAL_TOKENS - ERROR_COUNT - NEW_USERS - RETENTION - AVG_SCORE - FAILURE_RATE - AVG_RATING DashboardFilterOperator: type: string description: How filters or groups are combined. enum: - AND - OR DashboardFilterCategory: type: string description: The property a filter row matches on (e.g. "Name", "User Id", "Model", "Metadata"). The set of valid values depends on the line's dataModel. DashboardFilterCondition: type: string description: The comparison a filter row applies. Valid conditions depend on the category. enum: - "Is" - "Is not" - "Is equal to" - "Does not equal" - "Is less than" - "Is equal or less than" - "Is greater than" - "Is equal or greater than" - "Has" - "Has not" - "Contains" - "Contains only" - "Does not contain" - "Has increased by more than" - "Has increased by less than" - "Has decreased by more than" - "Has decreased by less than" - "Has changed from" DashboardTopKDirection: type: string description: The sort direction for topK results. enum: - asc - desc DashboardTopKOrderBy: type: string description: The metric or column to order topK results by. enum: - count - avg_latency - p50_latency - p90_latency - p99_latency - error_rate - pass_rate - failure_rate - input_cost - output_cost - total_cost - avg_cost - input_tokens - output_tokens - total_tokens - count_distinct_endUserId - count_distinct_threadId - count_distinct_model - count_distinct_projectId - count_distinct_error - count_distinct_metadata - error_count - avg_score - stddev_score - avg_rating - created_at - start_time - dimension DashboardTopK: type: object description: Limits a dimension breakdown to the top K series. properties: limit: type: integer description: Maximum number of series to return. Defaults to 10. orderBy: $ref: "#/components/schemas/DashboardTopKOrderBy" direction: $ref: "#/components/schemas/DashboardTopKDirection" DashboardFilterRow: type: object description: A single filter predicate. properties: category: $ref: "#/components/schemas/DashboardFilterCategory" condition: $ref: "#/components/schemas/DashboardFilterCondition" value: description: The value to match against. oneOf: - type: string - type: number - type: array items: type: string key: type: string description: The property key. Auto-populated from category when omitted; required for Metadata, Metric, and Classifier filters. required: - category - condition - value DashboardFilterGroup: type: object description: A group of filter rows combined by the group's operator. properties: operator: $ref: "#/components/schemas/DashboardFilterOperator" filters: type: array description: The filter rows in this group. items: $ref: "#/components/schemas/DashboardFilterRow" required: - operator - filters DashboardFilterSet: type: object description: A set of filter groups combined by a top-level operator. properties: operator: $ref: "#/components/schemas/DashboardFilterOperator" groups: type: array description: The filter groups. items: $ref: "#/components/schemas/DashboardFilterGroup" required: - operator - groups DashboardLineExtraQueryParams: type: object nullable: true description: >- Advanced, per-line query parameters. Which keys take effect depends on the line's `dataModel`, and unrecognized keys are ignored. All values are strings. Most lines leave this `null`. properties: spanType: type: string description: >- For a `SPAN` line, restricts aggregation to a single span type. Not needed for the typed span models (`LLM_SPAN`, `AGENT_SPAN`, `RETRIEVER_SPAN`, `TOOL_SPAN`, `CUSTOM_SPAN`), which already imply their span type. enum: - LLM - AGENT - RETRIEVER - TOOL - CUSTOM metricMetadataKey: type: string description: >- For span (`SPAN`, `LLM_SPAN`, `AGENT_SPAN`, `RETRIEVER_SPAN`, `TOOL_SPAN`, `CUSTOM_SPAN`), `TRACE`, and `THREAD` lines, the metadata field key whose numeric value is aggregated. category: type: string description: For a `METRIC_DATA` line, the entity category the metric is attached to. enum: - SINGLE_TURN - MULTI_TURN - TEST_RUN - TRACE - SPAN - LLM_SPAN - AGENT_SPAN - RETRIEVER_SPAN - TOOL_SPAN - CUSTOM_SPAN - THREAD metricName: type: string description: For a `METRIC_DATA` line, the name of the metric to aggregate. dataType: type: string description: For an `ANNOTATION` line, which annotated entity type to aggregate over. enum: - Traces - Spans - Threads source: type: string description: >- For an `ANNOTATION` line, whether to aggregate annotations left by end users or by reviewers. enum: - User - Reviewer additionalProperties: type: string example: metricMetadataKey: tokenCount DashboardLineInput: type: object description: A single series within a widget. properties: name: type: string description: The line's name. color: $ref: "#/components/schemas/DashboardLineColor" dataModel: $ref: "#/components/schemas/DashboardDataModel" aggregation: $ref: "#/components/schemas/DashboardAggregation" filters: $ref: "#/components/schemas/DashboardFilterSet" extraQueryParams: $ref: "#/components/schemas/DashboardLineExtraQueryParams" required: - name DashboardWidgetInput: type: object description: A widget configuration to create or update. If layout is omitted, the server assigns a default grid position. properties: name: type: string description: The widget's name. description: type: string nullable: true description: An optional description of the widget. type: $ref: "#/components/schemas/DashboardWidgetType" unit: $ref: "#/components/schemas/DashboardWidgetUnit" mode: $ref: "#/components/schemas/DashboardWidgetMode" bucketMode: $ref: "#/components/schemas/DashboardWidgetBucketMode" dimension: $ref: "#/components/schemas/DashboardDimension" topK: $ref: "#/components/schemas/DashboardTopK" startTime: type: string format: date-time nullable: true description: The start of the widget's custom time range, if set. endTime: type: string format: date-time nullable: true description: The end of the widget's custom time range, if set. layout: $ref: "#/components/schemas/DashboardLayout" lines: type: array nullable: true description: The series to show in the widget. items: $ref: "#/components/schemas/DashboardLineInput" required: - name DashboardLayout: type: object description: A widget's position on the dashboard grid. properties: x: type: number description: The widget's left edge as a column index on the 12-column grid. y: type: number description: The widget's top edge as a row index on the grid. w: type: number description: The widget's width in grid columns. h: type: number description: The widget's height in grid rows. DashboardLine: type: object description: A series within a widget. properties: id: type: string description: The unique id of the line. name: type: string description: The line's name. color: $ref: "#/components/schemas/DashboardLineColor" dataModel: $ref: "#/components/schemas/DashboardDataModel" aggregation: $ref: "#/components/schemas/DashboardAggregation" filters: $ref: "#/components/schemas/DashboardFilterSet" extraQueryParams: $ref: "#/components/schemas/DashboardLineExtraQueryParams" DashboardWidget: type: object description: A widget (one chart) with its full configuration. properties: id: type: string description: The unique id of the widget. name: type: string description: The widget's name. description: type: string nullable: true description: An optional description of the widget. type: $ref: "#/components/schemas/DashboardWidgetType" unit: $ref: "#/components/schemas/DashboardWidgetUnit" mode: $ref: "#/components/schemas/DashboardWidgetMode" bucketMode: $ref: "#/components/schemas/DashboardWidgetBucketMode" dimension: $ref: "#/components/schemas/DashboardDimension" topK: $ref: "#/components/schemas/DashboardTopK" startTime: type: string format: date-time nullable: true description: The start of the widget's custom time range, if set. endTime: type: string format: date-time nullable: true description: The end of the widget's custom time range, if set. layout: $ref: "#/components/schemas/DashboardLayout" lines: type: array description: The series shown in the widget. items: $ref: "#/components/schemas/DashboardLine" DashboardOverview: type: object description: A lightweight overview of a dashboard, returned in list responses. properties: id: type: string description: The unique id of the dashboard. name: type: string description: The dashboard's name. description: type: string nullable: true description: An optional description of the dashboard. private: type: boolean description: Whether the dashboard is private to its creator. projectId: type: string description: The id of the project that owns the dashboard. userId: type: string nullable: true description: The id of the dashboard creator, when available. widgetCount: type: integer description: The number of widgets on the dashboard. createdAt: type: string format: date-time description: When the dashboard was created. updatedAt: type: string format: date-time description: When the dashboard was last updated. Dashboard: type: object description: A dashboard with its widgets. properties: id: type: string description: The unique id of the dashboard. name: type: string description: The dashboard's name. description: type: string nullable: true description: An optional description of the dashboard. private: type: boolean description: Whether the dashboard is private to its creator. projectId: type: string description: The id of the project that owns the dashboard. userId: type: string nullable: true description: The id of the dashboard creator, when available. createdAt: type: string format: date-time description: When the dashboard was created. updatedAt: type: string format: date-time description: When the dashboard was last updated. widgets: type: array description: Full widget configurations embedded in the dashboard. items: $ref: "#/components/schemas/DashboardWidget" CreateDashboardRequest: type: object description: The dashboard to create, optionally with starter widgets. properties: name: type: string description: The dashboard's name. description: type: string nullable: true description: An optional description of the dashboard. private: type: boolean description: Whether the dashboard is private. Defaults to false. widgets: type: array description: Optional starter widgets to create alongside the dashboard. items: $ref: "#/components/schemas/DashboardWidgetInput" required: - name UpdateDashboardRequest: type: object description: Partial update. At least one field must be provided. properties: name: type: string description: The dashboard's new name. description: type: string nullable: true description: The dashboard's new description. private: type: boolean description: Whether the dashboard is private to its creator. DashboardListResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: The dashboards in your project. properties: dashboards: type: array description: The list of dashboard overviews. items: $ref: "#/components/schemas/DashboardOverview" required: - dashboards deprecated: type: boolean description: Indicates if this endpoint is deprecated. required: - success - data - deprecated DashboardObjectResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: The dashboard, including its widget configurations. properties: dashboard: $ref: "#/components/schemas/Dashboard" required: - dashboard deprecated: type: boolean description: Indicates if this endpoint is deprecated. required: - success - data - deprecated DashboardIdResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: The id of the affected dashboard or widget. properties: id: type: string description: The id of the affected dashboard or widget. required: - id deprecated: type: boolean description: Indicates if this endpoint is deprecated. required: - success - data - deprecated DashboardAggregateGranularity: type: string description: Bucket size for computed widget data. enum: - thirty_minutes - hour - day - week - month DashboardWidgetQueryRequest: type: object description: Query overrides for a single widget. `startTime` and `endTime` must be provided together. properties: startTime: type: string format: date-time description: ISO 8601 start time for the query range. Must be provided with `endTime`. endTime: type: string format: date-time description: ISO 8601 end time for the query range. Must be provided with `startTime`. granularity: $ref: "#/components/schemas/DashboardAggregateGranularity" description: Optional bucket granularity override for the query. DashboardQueryRequest: type: object description: Query overrides for dashboard widgets. `startTime` and `endTime` must be provided together. properties: startTime: type: string format: date-time description: ISO 8601 start time for the query range. Must be provided with `endTime`. endTime: type: string format: date-time description: ISO 8601 end time for the query range. Must be provided with `startTime`. granularity: $ref: "#/components/schemas/DashboardAggregateGranularity" description: Optional bucket granularity override for the query. widgetIds: type: array description: Optional subset of widget IDs to query. If omitted, all widgets on the dashboard are queried. items: type: string DashboardSeriesPoint: type: object description: A single point in a series — an x/y pair for one time bucket or category. properties: x: type: string description: Time bucket start or category label. y: type: number nullable: true description: Numeric value for the series at this point, or null when no data is available. required: - x - y DashboardSeries: type: object description: One plotted series in a TIME_SERIES or DIMENSION widget result. properties: key: type: string description: Stable key that uniquely identifies this series within the result. Use it to correlate series across queries or as a render key. name: type: string description: Display name for the series. color: type: string description: Display color for the series. lineId: type: string description: The line id that produced this series, when applicable. points: type: array description: Points in this series. items: $ref: "#/components/schemas/DashboardSeriesPoint" required: - key - name - color - points DashboardScalarValue: type: object description: A single headline value in a BIG_NUMBER widget result. properties: key: type: string description: Stable key that uniquely identifies this value within the result. name: type: string description: Display name for the value. color: type: string description: Display color for the value. lineId: type: string description: The line id that produced this value, when applicable. value: type: number nullable: true description: Scalar value, or null when no data is available. required: - key - name - color - value DashboardWidgetData: type: object description: Computed data for a dashboard widget. The `kind` field indicates which shape-specific fields are populated. properties: widgetId: type: string description: The id of the queried widget. type: $ref: "#/components/schemas/DashboardWidgetType" description: >- The widget's visualization (display) type, echoed from its configuration. This is how the widget is drawn and does not determine which data fields are populated — use `kind` for that. mode: $ref: "#/components/schemas/DashboardWidgetMode" description: >- The widget's configured aggregation mode, echoed from its configuration. When reading the response, branch on `kind` rather than `mode`. kind: type: string description: >- The shape of the data in this response, and the field you should branch on when reading it. `TIME_SERIES` and `DIMENSION` populate `series` (with `xAxis.type` `time` and `category` respectively); `BIG_NUMBER` populates `values`; `TABLE` populates `columns` and `rows`. It is derived from `type` and `mode`, so it can differ from `mode` — e.g. a `DIMENSION_SERIES` widget displayed as a `TABLE` returns `kind: TABLE`. enum: - TIME_SERIES - DIMENSION - BIG_NUMBER - TABLE unit: $ref: "#/components/schemas/DashboardWidgetUnit" description: Unit for the returned values, when applicable. xAxis: type: object description: Present for TIME_SERIES and DIMENSION data. properties: type: type: string description: Axis type for the returned data. enum: - time - category required: - type series: type: array description: Present for TIME_SERIES and DIMENSION data. items: $ref: "#/components/schemas/DashboardSeries" values: type: array description: Present for BIG_NUMBER data. items: $ref: "#/components/schemas/DashboardScalarValue" columns: type: array description: >- Column definitions for TABLE data. The first column is the dimension (key `dimension`); the remaining columns are one per line, keyed by the line's name. items: type: object properties: key: type: string description: Stable column key. Read each row's value for this column as `row[key]`. label: type: string description: Display label for the column. required: - key - label rows: type: array description: Present for TABLE data. items: type: object description: >- A table row. Each value is keyed by its column's `key` — the dimension value under `dimension`, and each line's value under the line's name. additionalProperties: true required: - widgetId - kind DashboardWidgetQueryResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: $ref: "#/components/schemas/DashboardWidgetData" description: The computed widget data. deprecated: type: boolean description: Indicates if this endpoint is deprecated. required: - success - data - deprecated WidgetQueryRequest: type: object description: An ad-hoc widget definition to compute, plus an optional query time range. `startTime` and `endTime` must be provided together and override the widget's own time fields for this query. properties: widget: $ref: "#/components/schemas/DashboardWidgetInput" description: The widget configuration to compute data for. startTime: type: string format: date-time description: ISO 8601 start time for the query range. Must be provided with `endTime`. endTime: type: string format: date-time description: ISO 8601 end time for the query range. Must be provided with `startTime`. granularity: $ref: "#/components/schemas/DashboardAggregateGranularity" description: Optional bucket granularity override for the query. required: - widget WidgetQueryData: type: object description: Computed data for an ad-hoc widget. The `kind` field indicates which shape-specific fields are populated. properties: type: $ref: "#/components/schemas/DashboardWidgetType" description: The widget's visualization (display) type, echoed from the request. mode: $ref: "#/components/schemas/DashboardWidgetMode" description: The widget's aggregation mode, echoed from the request. Branch on `kind` rather than `mode` when reading the response. kind: type: string description: >- The shape of the data in this response, and the field to branch on when reading it. `TIME_SERIES` and `DIMENSION` populate `series` (with `xAxis.type` `time` and `category` respectively); `BIG_NUMBER` populates `values`; `TABLE` populates `columns` and `rows`. enum: - TIME_SERIES - DIMENSION - BIG_NUMBER - TABLE unit: $ref: "#/components/schemas/DashboardWidgetUnit" description: Unit for the returned values, when applicable. xAxis: type: object description: Present for TIME_SERIES and DIMENSION data. properties: type: type: string description: Axis type for the returned data. enum: - time - category required: - type series: type: array description: Present for TIME_SERIES and DIMENSION data. items: $ref: "#/components/schemas/DashboardSeries" values: type: array description: Present for BIG_NUMBER data. items: $ref: "#/components/schemas/DashboardScalarValue" columns: type: array description: >- Column definitions for TABLE data. The first column is the dimension (key `dimension`); the remaining columns are one per line, keyed by the line's name. items: type: object properties: key: type: string description: Stable column key. Read each row's value for this column as `row[key]`. label: type: string description: Display label for the column. required: - key - label rows: type: array description: Present for TABLE data. items: type: object description: A table row. Each value is keyed by its column's `key`. additionalProperties: true required: - kind WidgetQueryResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: $ref: "#/components/schemas/WidgetQueryData" description: The computed widget data. deprecated: type: boolean description: Indicates if this endpoint is deprecated. required: - success - data - deprecated DashboardQueryResult: type: object description: One widget's batch query result. When `status` is `error`, only `widgetId`, `status`, and `error` are populated. properties: widgetId: type: string description: The id of the queried widget. status: type: string description: Whether this widget's batch query succeeded. enum: - ok - error type: $ref: "#/components/schemas/DashboardWidgetType" description: >- The widget's visualization (display) type, echoed from its configuration. Does not determine which data fields are populated — use `kind` for that. Present when `status` is `ok`. mode: $ref: "#/components/schemas/DashboardWidgetMode" description: >- The widget's configured aggregation mode, echoed from its configuration. Branch on `kind` rather than `mode` when reading the data. Present when `status` is `ok`. kind: type: string description: >- The shape of the data in this result, and the field you should branch on when reading it. `TIME_SERIES` and `DIMENSION` populate `series` (with `xAxis.type` `time` and `category` respectively); `BIG_NUMBER` populates `values`; `TABLE` populates `columns` and `rows`. It is derived from `type` and `mode`, so it can differ from `mode`. Present when `status` is `ok`. enum: - TIME_SERIES - DIMENSION - BIG_NUMBER - TABLE unit: $ref: "#/components/schemas/DashboardWidgetUnit" description: Unit for the returned values, when applicable. xAxis: type: object description: Present for TIME_SERIES and DIMENSION results. properties: type: type: string description: Axis type for the returned data. enum: - time - category required: - type series: type: array description: Present for TIME_SERIES and DIMENSION results. items: $ref: "#/components/schemas/DashboardSeries" values: type: array description: Present for BIG_NUMBER results. items: $ref: "#/components/schemas/DashboardScalarValue" columns: type: array description: >- Column definitions for TABLE results. The first column is the dimension (key `dimension`); the remaining columns are one per line, keyed by the line's name. items: type: object properties: key: type: string description: Stable column key. Read each row's value for this column as `row[key]`. label: type: string description: Display label for the column. required: - key - label rows: type: array description: Present for TABLE results. items: type: object description: >- A table row. Each value is keyed by its column's `key` — the dimension value under `dimension`, and each line's value under the line's name. additionalProperties: true error: type: object description: Present only when `status` is `error`. properties: code: type: string description: Machine-readable query error code. enum: - QUERY_FAILED message: type: string description: Human-readable query error message. required: - code - message required: - widgetId - status DashboardQueryResponse: type: object properties: success: type: boolean description: Indicates if the request was successful. data: type: object description: The batch query payload. properties: results: type: array description: Computed data or error details for each queried widget. items: $ref: "#/components/schemas/DashboardQueryResult" required: - results deprecated: type: boolean description: Indicates if this endpoint is deprecated. required: - success - data - deprecated