openapi: 3.0.3 info: title: PostHog actions LLM Analytics API version: 1.0.0 description: '' tags: - name: LLM Analytics paths: /api/environments/{project_id}/llm_analytics/evaluation_summary/: post: operationId: llm_analytics_evaluation_summary_create description: "\nGenerate an AI-powered summary of evaluation results.\n\nThis endpoint analyzes evaluation runs and identifies patterns in passing\nand failing evaluations, providing actionable recommendations.\n\nData is fetched server-side by evaluation ID to ensure data integrity.\n\n**Use Cases:**\n- Understand why evaluations are passing or failing\n- Identify systematic issues in LLM responses\n- Get recommendations for improving response quality\n- Review patterns across many evaluation runs at once\n " parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/EvaluationSummaryRequest' examples: EvaluationSummaryRequest: value: evaluation_id: 550e8400-e29b-41d4-a716-446655440000 filter: all force_refresh: false summary: Evaluation Summary Request description: Summarize evaluation results by ID application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/EvaluationSummaryRequest' multipart/form-data: schema: $ref: '#/components/schemas/EvaluationSummaryRequest' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/EvaluationSummaryResponse' examples: SuccessResponse: value: overall_assessment: Evaluations show generally good quality with some factual accuracy issues. pass_patterns: - title: Clear Communication description: Responses consistently provided well-structured information frequency: common example_generation_ids: - gen_abc123 - gen_ghi789 fail_patterns: - title: Factual Errors description: Some responses contained inaccurate information frequency: occasional example_generation_ids: - gen_def456 na_patterns: [] recommendations: - Implement fact-checking for critical claims - Add source citations where applicable statistics: total_analyzed: 3 pass_count: 2 fail_count: 1 na_count: 0 summary: Success Response description: '' '400': content: application/json: schema: type: object additionalProperties: true description: '' '403': content: application/json: schema: type: object additionalProperties: true description: '' '404': content: application/json: schema: type: object additionalProperties: true description: '' '500': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - LLM Analytics - llm_analytics /api/environments/{project_id}/llm_analytics/sentiment/: post: operationId: llm_analytics_sentiment_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/SentimentRequest' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/SentimentRequest' multipart/form-data: schema: $ref: '#/components/schemas/SentimentRequest' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/SentimentBatchResponse' description: '' '400': content: application/json: schema: type: object additionalProperties: true description: '' '500': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - LLM Analytics - llm_analytics /api/environments/{project_id}/llm_analytics/summarization/: post: operationId: llm_analytics_summarization_create description: "\nGenerate an AI-powered summary of an LLM trace or event.\n\nThis endpoint analyzes the provided trace/event, generates a line-numbered text\nrepresentation, and uses an LLM to create a concise summary with line references.\n\n**Two ways to use this endpoint:**\n\n1. **By ID (recommended):** Pass `trace_id` or `generation_id` with an optional `date_from`/`date_to`.\n The backend fetches the data automatically. `summarize_type` is inferred.\n2. **By data:** Pass the full trace/event data blob in `data` with `summarize_type`.\n This is how the frontend uses it.\n\n**Summary Format:**\n- Title (concise, max 10 words)\n- Mermaid flow diagram showing the main flow\n- 3-10 summary bullets with line references\n- \"Interesting Notes\" section for failures, successes, or unusual patterns\n- Line references in [L45] or [L45-52] format pointing to relevant sections\n\nThe response includes the structured summary, the text representation, and metadata.\n " parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/SummarizeRequest' examples: TraceSummarization: value: summarize_type: trace data: trace: id: trace_123 properties: $ai_span_name: ChatBot Interaction hierarchy: - event: id: gen_1 event: $ai_generation properties: $ai_input: - role: user content: Hello $ai_output_choices: - message: role: assistant content: Hi there! children: [] summary: Trace Summarization description: Summarize a full trace with hierarchy EventSummarization: value: summarize_type: event data: event: id: gen_456 event: $ai_generation properties: $ai_input: - role: user content: Explain Python $ai_output_choices: - message: role: assistant content: Python is... summary: Event Summarization description: Summarize a single generation event application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/SummarizeRequest' multipart/form-data: schema: $ref: '#/components/schemas/SummarizeRequest' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/SummarizeResponse' examples: SuccessResponse: value: summary: '## Summary - User initiated conversation with greeting [L5-8] - Assistant responded with friendly message [L12-15] ## Interesting Notes - Standard greeting pattern with no errors' metadata: text_repr_length: 450 model: gpt-4.1 summary: Success Response description: '' '400': content: application/json: schema: type: object additionalProperties: true description: '' '403': content: application/json: schema: type: object additionalProperties: true description: '' '500': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - LLM Analytics - llm_analytics /api/environments/{project_id}/llm_analytics/summarization/batch_check/: post: operationId: llm_analytics_summarization_batch_check_create description: "\nCheck which traces have cached summaries available.\n\nThis endpoint allows batch checking of multiple trace IDs to see which ones\nhave cached summaries. Returns only the traces that have cached summaries\nwith their titles.\n\n**Use Cases:**\n- Load cached summaries on session view load\n- Avoid unnecessary LLM calls for already-summarized traces\n- Display summary previews without generating new summaries\n " parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/BatchCheckRequest' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/BatchCheckRequest' multipart/form-data: schema: $ref: '#/components/schemas/BatchCheckRequest' required: true responses: '200': content: application/json: schema: $ref: '#/components/schemas/BatchCheckResponse' description: '' '400': content: application/json: schema: type: object additionalProperties: true description: '' '403': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - LLM Analytics - llm_analytics /api/environments/{project_id}/llm_analytics/text_repr/: post: operationId: llm_analytics_text_repr_create description: "\nGenerate a human-readable text representation of an LLM trace event.\n\nThis endpoint converts LLM analytics events ($ai_generation, $ai_span, $ai_embedding, or $ai_trace)\ninto formatted text representations suitable for display, logging, or analysis.\n\n**Supported Event Types:**\n- `$ai_generation`: Individual LLM API calls with input/output messages\n- `$ai_span`: Logical spans with state transitions\n- `$ai_embedding`: Embedding generation events (text input → vector)\n- `$ai_trace`: Full traces with hierarchical structure\n\n**Options:**\n- `max_length`: Maximum character count (default: 2000000)\n- `truncated`: Enable middle-content truncation within events (default: true)\n- `truncate_buffer`: Characters at start/end when truncating (default: 1000)\n- `include_markers`: Use interactive markers vs plain text indicators (default: true)\n - Frontend: set true for `<<>>` markers\n - Backend/LLM: set false for `... (X chars truncated) ...` text\n- `collapsed`: Show summary vs full trace tree (default: false)\n- `include_hierarchy`: Include tree structure for traces (default: true)\n- `max_depth`: Maximum depth for hierarchical rendering (default: unlimited)\n- `tools_collapse_threshold`: Number of tools before auto-collapsing list (default: 5)\n - Tool lists >5 items show `<<>>` marker for frontend\n - Or `[+] AVAILABLE TOOLS: N` for backend when `include_markers: false`\n- `include_line_numbers`: Prefix each line with line number like L001:, L010: (default: false)\n\n**Use Cases:**\n- Frontend display: `truncated: true, include_markers: true, include_line_numbers: true`\n- Backend LLM context (summary): `truncated: true, include_markers: false, collapsed: true`\n- Backend LLM context (full): `truncated: false`\n\nThe response includes the formatted text and metadata about the rendering.\n " parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/TextReprRequest' examples: GenerationExample: value: event_type: $ai_generation data: id: gen_123 properties: $ai_input: - role: user content: What is the capital of France? $ai_output_choices: - message: role: assistant content: The capital of France is Paris. options: max_length: 10000 summary: Generation Example description: Stringify an $ai_generation event TraceExample: value: event_type: $ai_trace data: trace: trace_id: trace_123 name: ChatBot Interaction hierarchy: - id: gen_1 event: $ai_generation children: [] summary: Trace Example description: Stringify a full trace with hierarchy application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/TextReprRequest' multipart/form-data: schema: $ref: '#/components/schemas/TextReprRequest' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/TextReprResponse' examples: SuccessResponse: value: text: 'INPUT: [1] USER What is the capital of France? ...' metadata: event_type: $ai_generation event_id: gen_123 rendering: detailed char_count: 150 truncated: false summary: Success Response description: '' '400': content: application/json: schema: type: object additionalProperties: true description: '' '500': content: application/json: schema: type: object additionalProperties: true description: '' '503': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - LLM Analytics - llm_analytics /api/environments/{environment_id}/dataset_items/: get: operationId: environments_dataset_items_list parameters: - in: query name: dataset schema: type: string description: Filter by dataset ID examples: SingleDatasetID: value: 695401fa-6f0e-4389-b186-c45a7f1273d3 summary: Single dataset ID - $ref: '#/components/parameters/EnvironmentIdPath' - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedDatasetItemList' description: '' deprecated: true x-explicit-tags: - llm_analytics post: operationId: environments_dataset_items_create parameters: - $ref: '#/components/parameters/EnvironmentIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/DatasetItem' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/DatasetItem' multipart/form-data: schema: $ref: '#/components/schemas/DatasetItem' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' deprecated: true x-explicit-tags: - llm_analytics /api/environments/{environment_id}/dataset_items/{id}/: get: operationId: environments_dataset_items_retrieve parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' deprecated: true x-explicit-tags: - llm_analytics put: operationId: environments_dataset_items_update parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/DatasetItem' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/DatasetItem' multipart/form-data: schema: $ref: '#/components/schemas/DatasetItem' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' deprecated: true x-explicit-tags: - llm_analytics patch: operationId: environments_dataset_items_partial_update parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedDatasetItem' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedDatasetItem' multipart/form-data: schema: $ref: '#/components/schemas/PatchedDatasetItem' security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' deprecated: true x-explicit-tags: - llm_analytics delete: operationId: environments_dataset_items_destroy description: Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:write responses: '405': description: No response body deprecated: true x-explicit-tags: - llm_analytics /api/environments/{environment_id}/datasets/: get: operationId: environments_datasets_list parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: query name: id__in schema: type: array items: type: string format: uuid description: Multiple values may be separated by commas. explode: false style: form - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: array items: type: string enum: - -created_at - -updated_at - created_at - updated_at description: 'Ordering * `created_at` - Created At * `-created_at` - Created At (descending) * `updated_at` - Updated At * `-updated_at` - Updated At (descending)' explode: false style: form - in: query name: search schema: type: string description: Search in name, description, or metadata tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedDatasetList' description: '' deprecated: true x-explicit-tags: - llm_analytics post: operationId: environments_datasets_create parameters: - $ref: '#/components/parameters/EnvironmentIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/Dataset' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/Dataset' multipart/form-data: schema: $ref: '#/components/schemas/Dataset' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' deprecated: true x-explicit-tags: - llm_analytics /api/environments/{environment_id}/datasets/{id}/: get: operationId: environments_datasets_retrieve parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' deprecated: true x-explicit-tags: - llm_analytics put: operationId: environments_datasets_update parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/Dataset' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/Dataset' multipart/form-data: schema: $ref: '#/components/schemas/Dataset' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' deprecated: true x-explicit-tags: - llm_analytics patch: operationId: environments_datasets_partial_update parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedDataset' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedDataset' multipart/form-data: schema: $ref: '#/components/schemas/PatchedDataset' security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' deprecated: true x-explicit-tags: - llm_analytics delete: operationId: environments_datasets_destroy description: Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true parameters: - $ref: '#/components/parameters/EnvironmentIdPath' - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:write responses: '405': description: No response body deprecated: true x-explicit-tags: - llm_analytics /api/environments/{project_id}/evaluation_runs/: post: operationId: evaluation_runs_create description: 'Create a new evaluation run. This endpoint validates the request and enqueues a Temporal workflow to asynchronously execute the evaluation.' parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/EvaluationRunRequest' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/EvaluationRunRequest' multipart/form-data: schema: $ref: '#/components/schemas/EvaluationRunRequest' required: true security: - PersonalAPIKeyAuth: - evaluation:write responses: '200': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/evaluations/: get: operationId: evaluations_list parameters: - in: query name: enabled schema: type: boolean description: Filter by enabled status - in: query name: id__in schema: type: array items: type: string format: uuid description: Multiple values may be separated by commas. explode: false style: form - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: array items: type: string enum: - -created_at - -name - -updated_at - created_at - name - updated_at description: 'Ordering * `created_at` - Created At * `-created_at` - Created At (descending) * `updated_at` - Updated At * `-updated_at` - Updated At (descending) * `name` - Name * `-name` - Name (descending)' explode: false style: form - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Search in name or description tags: - LLM Analytics security: - PersonalAPIKeyAuth: - evaluation:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedEvaluationList' description: '' x-explicit-tags: - llm_analytics post: operationId: evaluations_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/Evaluation' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/Evaluation' multipart/form-data: schema: $ref: '#/components/schemas/Evaluation' required: true security: - PersonalAPIKeyAuth: - evaluation:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/Evaluation' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/evaluations/{id}/: get: operationId: evaluations_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - evaluation:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/Evaluation' description: '' x-explicit-tags: - llm_analytics put: operationId: evaluations_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/Evaluation' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/Evaluation' multipart/form-data: schema: $ref: '#/components/schemas/Evaluation' required: true security: - PersonalAPIKeyAuth: - evaluation:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/Evaluation' description: '' x-explicit-tags: - llm_analytics patch: operationId: evaluations_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedEvaluation' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedEvaluation' multipart/form-data: schema: $ref: '#/components/schemas/PatchedEvaluation' security: - PersonalAPIKeyAuth: - evaluation:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/Evaluation' description: '' x-explicit-tags: - llm_analytics delete: operationId: evaluations_destroy description: Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - evaluation:write responses: '405': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/evaluations/test_hog/: post: operationId: evaluations_test_hog_create description: Test Hog evaluation code against sample events without saving. parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/TestHogRequest' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/TestHogRequest' multipart/form-data: schema: $ref: '#/components/schemas/TestHogRequest' required: true security: - PersonalAPIKeyAuth: - evaluation:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/TestHogResponse' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/clustering_config/: get: operationId: llm_analytics_clustering_config_retrieve description: Team-level clustering configuration (event filters for automated pipelines). parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/clustering_config/set_event_filters/: post: operationId: llm_analytics_clustering_config_set_event_filters_create description: Team-level clustering configuration (event filters for automated pipelines). parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/clustering_jobs/: get: operationId: llm_analytics_clustering_jobs_list description: CRUD for clustering job configurations (max 5 per team). parameters: - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedClusteringJobList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_clustering_jobs_create description: CRUD for clustering job configurations (max 5 per team). parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/ClusteringJob' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/ClusteringJob' multipart/form-data: schema: $ref: '#/components/schemas/ClusteringJob' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/ClusteringJob' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/clustering_jobs/{id}/: get: operationId: llm_analytics_clustering_jobs_retrieve description: CRUD for clustering job configurations (max 5 per team). parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this clustering job. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/ClusteringJob' description: '' x-explicit-tags: [] put: operationId: llm_analytics_clustering_jobs_update description: CRUD for clustering job configurations (max 5 per team). parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this clustering job. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/ClusteringJob' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/ClusteringJob' multipart/form-data: schema: $ref: '#/components/schemas/ClusteringJob' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/ClusteringJob' description: '' x-explicit-tags: [] patch: operationId: llm_analytics_clustering_jobs_partial_update description: CRUD for clustering job configurations (max 5 per team). parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this clustering job. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedClusteringJob' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedClusteringJob' multipart/form-data: schema: $ref: '#/components/schemas/PatchedClusteringJob' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/ClusteringJob' description: '' x-explicit-tags: [] delete: operationId: llm_analytics_clustering_jobs_destroy description: CRUD for clustering job configurations (max 5 per team). parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this clustering job. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '204': description: No response body x-explicit-tags: [] /api/environments/{project_id}/llm_analytics/evaluation_config/: get: operationId: llm_analytics_evaluation_config_retrieve description: Get the evaluation config for this team parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - evaluation:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/EvaluationConfig' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/evaluation_config/set_active_key/: post: operationId: llm_analytics_evaluation_config_set_active_key_create description: Set the active provider key for evaluations parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/EvaluationConfigSetActiveKeyRequest' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/EvaluationConfigSetActiveKeyRequest' multipart/form-data: schema: $ref: '#/components/schemas/EvaluationConfigSetActiveKeyRequest' required: true security: - PersonalAPIKeyAuth: - evaluation:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/EvaluationConfig' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/evaluation_reports/: get: operationId: llm_analytics_evaluation_reports_list description: CRUD for evaluation report configurations + report run history. parameters: - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedEvaluationReportList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_evaluation_reports_create description: CRUD for evaluation report configurations + report run history. parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/EvaluationReport' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/EvaluationReport' multipart/form-data: schema: $ref: '#/components/schemas/EvaluationReport' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/EvaluationReport' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/evaluation_reports/{id}/: get: operationId: llm_analytics_evaluation_reports_retrieve description: CRUD for evaluation report configurations + report run history. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation report. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/EvaluationReport' description: '' x-explicit-tags: - llm_analytics put: operationId: llm_analytics_evaluation_reports_update description: CRUD for evaluation report configurations + report run history. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation report. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/EvaluationReport' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/EvaluationReport' multipart/form-data: schema: $ref: '#/components/schemas/EvaluationReport' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/EvaluationReport' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_analytics_evaluation_reports_partial_update description: CRUD for evaluation report configurations + report run history. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation report. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedEvaluationReport' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedEvaluationReport' multipart/form-data: schema: $ref: '#/components/schemas/PatchedEvaluationReport' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/EvaluationReport' description: '' x-explicit-tags: - llm_analytics delete: operationId: llm_analytics_evaluation_reports_destroy description: Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation report. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '405': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/evaluation_reports/{id}/generate/: post: operationId: llm_analytics_evaluation_reports_generate_create description: Trigger immediate report generation. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation report. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '202': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/evaluation_reports/{id}/runs/: get: operationId: llm_analytics_evaluation_reports_runs_list description: List report runs (history) for this report. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this evaluation report. required: true - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedEvaluationReportRunList' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/models/: get: operationId: llm_analytics_models_retrieve description: List available models for a provider. parameters: - in: query name: key_id schema: type: string format: uuid description: Optional provider key UUID. When supplied, models reachable with that specific key are returned (useful for Azure OpenAI, where the deployment list depends on the configured endpoint). Must belong to the same provider as the `provider` parameter. - $ref: '#/components/parameters/ProjectIdPath' - in: query name: provider schema: type: string enum: - anthropic - azure_openai - fireworks - gemini - openai - openrouter - together_ai description: LLM provider to list models for. Must be one of the supported providers. required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - evaluation:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMModelsListResponse' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/provider_key_validations/: post: operationId: llm_analytics_provider_key_validations_create description: Validate LLM provider API keys without persisting them parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_provider_key:write responses: '200': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/provider_keys/: get: operationId: llm_analytics_provider_keys_list parameters: - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_provider_key:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedLLMProviderKeyList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_provider_keys_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMProviderKey' multipart/form-data: schema: $ref: '#/components/schemas/LLMProviderKey' required: true security: - PersonalAPIKeyAuth: - llm_provider_key:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/provider_keys/{id}/: get: operationId: llm_analytics_provider_keys_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_provider_key:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: [] put: operationId: llm_analytics_provider_keys_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMProviderKey' multipart/form-data: schema: $ref: '#/components/schemas/LLMProviderKey' required: true security: - PersonalAPIKeyAuth: - llm_provider_key:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: [] patch: operationId: llm_analytics_provider_keys_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedLLMProviderKey' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedLLMProviderKey' multipart/form-data: schema: $ref: '#/components/schemas/PatchedLLMProviderKey' security: - PersonalAPIKeyAuth: - llm_provider_key:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: [] delete: operationId: llm_analytics_provider_keys_destroy parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_provider_key:write responses: '204': description: No response body x-explicit-tags: [] /api/environments/{project_id}/llm_analytics/provider_keys/{id}/assign/: post: operationId: llm_analytics_provider_keys_assign_create description: Assign this key to evaluations and optionally re-enable them. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMProviderKey' multipart/form-data: schema: $ref: '#/components/schemas/LLMProviderKey' required: true responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: [] /api/environments/{project_id}/llm_analytics/provider_keys/{id}/dependent_configs/: get: operationId: llm_analytics_provider_keys_dependent_configs_retrieve description: Get evaluations using this key and alternative keys for replacement. parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: [] /api/environments/{project_id}/llm_analytics/provider_keys/{id}/validate/: post: operationId: llm_analytics_provider_keys_validate_create parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this llm provider key. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMProviderKey' multipart/form-data: schema: $ref: '#/components/schemas/LLMProviderKey' required: true responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: [] /api/environments/{project_id}/llm_analytics/provider_keys/trial_evaluations/: get: operationId: llm_analytics_provider_keys_trial_evaluations_retrieve description: List enabled evaluations currently using trial credits for a given provider. parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMProviderKey' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/review_queue_items/: get: operationId: llm_analytics_review_queue_items_list parameters: - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: string description: Order by `created_at` or `updated_at`. examples: NewestFirst: value: -created_at summary: Newest first - $ref: '#/components/parameters/ProjectIdPath' - in: query name: queue_id schema: type: string format: uuid description: Filter by a specific review queue ID. - in: query name: search schema: type: string description: Search pending trace IDs. examples: Search: value: trace_ - in: query name: trace_id schema: type: string description: Filter by an exact trace ID. examples: TraceID: value: trace_123 summary: Trace ID - in: query name: trace_id__in schema: type: string description: Filter by multiple trace IDs separated by commas. examples: TraceIDs: value: trace_123,trace_456 summary: Trace IDs tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedReviewQueueItemList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_review_queue_items_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/ReviewQueueItemCreate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/ReviewQueueItemCreate' multipart/form-data: schema: $ref: '#/components/schemas/ReviewQueueItemCreate' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/ReviewQueueItem' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/review_queue_items/{id}/: get: operationId: llm_analytics_review_queue_items_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this review queue item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/ReviewQueueItem' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_analytics_review_queue_items_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this review queue item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedReviewQueueItemUpdate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedReviewQueueItemUpdate' multipart/form-data: schema: $ref: '#/components/schemas/PatchedReviewQueueItemUpdate' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/ReviewQueueItem' description: '' x-explicit-tags: - llm_analytics delete: operationId: llm_analytics_review_queue_items_destroy parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this review queue item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '204': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/review_queues/: get: operationId: llm_analytics_review_queues_list parameters: - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - in: query name: name schema: type: string - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: string description: Order by `name`, `updated_at`, or `created_at`. examples: Alphabetical: value: name - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Search review queue names. examples: QueueSearch: value: support summary: Queue search tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedReviewQueueList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_review_queues_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/ReviewQueueCreate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/ReviewQueueCreate' multipart/form-data: schema: $ref: '#/components/schemas/ReviewQueueCreate' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/ReviewQueue' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/review_queues/{id}/: get: operationId: llm_analytics_review_queues_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this review queue. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/ReviewQueue' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_analytics_review_queues_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this review queue. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedReviewQueueUpdate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedReviewQueueUpdate' multipart/form-data: schema: $ref: '#/components/schemas/PatchedReviewQueueUpdate' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/ReviewQueue' description: '' x-explicit-tags: - llm_analytics delete: operationId: llm_analytics_review_queues_destroy parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this review queue. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '204': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/score_definitions/: get: operationId: llm_analytics_score_definitions_list parameters: - in: query name: archived schema: type: boolean description: Filter by archived state. examples: ActiveScorers: value: false summary: Active scorers - in: query name: kind schema: type: string description: Filter by scorer kind. examples: CategoricalScorer: value: categorical summary: Categorical scorer - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: string description: Sort by name, kind, created_at, updated_at, or current_version. examples: SortByName: value: name summary: Sort by name - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Search scorers by name or description. examples: SearchScorer: value: quality summary: Search scorer tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedScoreDefinitionList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_score_definitions_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/ScoreDefinitionCreate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/ScoreDefinitionCreate' multipart/form-data: schema: $ref: '#/components/schemas/ScoreDefinitionCreate' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/ScoreDefinition' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/score_definitions/{id}/: get: operationId: llm_analytics_score_definitions_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this score definition. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/ScoreDefinition' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_analytics_score_definitions_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this score definition. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedScoreDefinitionMetadata' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedScoreDefinitionMetadata' multipart/form-data: schema: $ref: '#/components/schemas/PatchedScoreDefinitionMetadata' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/ScoreDefinition' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/score_definitions/{id}/new_version/: post: operationId: llm_analytics_score_definitions_new_version_create parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this score definition. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/ScoreDefinitionNewVersion' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/ScoreDefinitionNewVersion' multipart/form-data: schema: $ref: '#/components/schemas/ScoreDefinitionNewVersion' required: true responses: '200': content: application/json: schema: $ref: '#/components/schemas/ScoreDefinition' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/trace_reviews/: get: operationId: llm_analytics_trace_reviews_list parameters: - in: query name: definition_id schema: type: string format: uuid description: Filter by a stable scorer definition ID. - in: query name: definition_id__in schema: type: string description: Filter by multiple scorer definition IDs separated by commas. - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: string description: Order by `updated_at` or `created_at`. examples: NewestFirst: value: -updated_at summary: Newest first - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Search trace IDs and comments. examples: CommentSearch: value: hallucination summary: Comment search - in: query name: trace_id schema: type: string description: Filter by an exact trace ID. examples: TraceID: value: trace_123 summary: Trace ID - in: query name: trace_id__in schema: type: string description: Filter by multiple trace IDs separated by commas. examples: TraceIDs: value: trace_123,trace_456 summary: Trace IDs tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedTraceReviewList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_analytics_trace_reviews_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/TraceReviewCreate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/TraceReviewCreate' multipart/form-data: schema: $ref: '#/components/schemas/TraceReviewCreate' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/TraceReview' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/trace_reviews/{id}/: get: operationId: llm_analytics_trace_reviews_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this trace review. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/TraceReview' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_analytics_trace_reviews_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this trace review. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedTraceReviewUpdate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedTraceReviewUpdate' multipart/form-data: schema: $ref: '#/components/schemas/PatchedTraceReviewUpdate' security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/TraceReview' description: '' x-explicit-tags: - llm_analytics delete: operationId: llm_analytics_trace_reviews_destroy parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this trace review. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '204': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_analytics/translate/: post: operationId: llm_analytics_translate_create description: Translate text to target language. parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/TranslateRequest' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/TranslateRequest' multipart/form-data: schema: $ref: '#/components/schemas/TranslateRequest' required: true security: - PersonalAPIKeyAuth: - llm_analytics:write responses: '200': content: application/json: schema: type: object additionalProperties: true description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_prompts/: get: operationId: llm_prompts_list parameters: - in: query name: content schema: enum: - full - preview - none type: string default: full minLength: 1 description: 'Controls how much prompt content is included in the response. ''full'' includes the full prompt, ''preview'' includes a short prompt_preview, and ''none'' omits prompt content entirely. The outline field is always included. * `full` - full * `preview` - preview * `none` - none' - in: query name: created_by_id schema: type: integer description: Filter prompts by the ID of the user who created them. - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Optional substring filter applied to prompt names and prompt content. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_prompt:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedLLMPromptListList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_prompts_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMPrompt' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMPrompt' multipart/form-data: schema: $ref: '#/components/schemas/LLMPrompt' required: true security: - PersonalAPIKeyAuth: - llm_prompt:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/LLMPrompt' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_prompts/name/{prompt_name}/: get: operationId: llm_prompts_name_retrieve parameters: - in: query name: content schema: enum: - full - preview - none type: string default: full minLength: 1 description: 'Controls how much prompt content is included in the response. ''full'' includes the full prompt, ''preview'' includes a short prompt_preview, and ''none'' omits prompt content entirely. The outline field is always included. * `full` - full * `preview` - preview * `none` - none' - $ref: '#/components/parameters/ProjectIdPath' - in: path name: prompt_name schema: type: string pattern: ^[^/]+$ required: true - in: query name: version schema: type: integer minimum: 1 description: Specific prompt version to fetch. If omitted, the latest version is returned. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_prompt:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMPromptPublic' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_prompts_name_partial_update parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: prompt_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedLLMPromptPublish' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedLLMPromptPublish' multipart/form-data: schema: $ref: '#/components/schemas/PatchedLLMPromptPublish' security: - PersonalAPIKeyAuth: - llm_prompt:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMPrompt' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_prompts/name/{prompt_name}/archive/: post: operationId: llm_prompts_name_archive_create parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: prompt_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMPrompt' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMPrompt' multipart/form-data: schema: $ref: '#/components/schemas/LLMPrompt' required: true security: - PersonalAPIKeyAuth: - llm_prompt:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMPrompt' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_prompts/name/{prompt_name}/duplicate/: post: operationId: llm_prompts_name_duplicate_create parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: prompt_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMPromptDuplicate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMPromptDuplicate' multipart/form-data: schema: $ref: '#/components/schemas/LLMPromptDuplicate' required: true security: - PersonalAPIKeyAuth: - llm_prompt:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/LLMPrompt' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_prompts/resolve/name/{prompt_name}/: get: operationId: llm_prompts_resolve_name_retrieve parameters: - in: query name: before_version schema: type: integer minimum: 1 description: Return versions older than this version number. Mutually exclusive with offset. - in: query name: limit schema: type: integer maximum: 100 minimum: 1 default: 50 description: Maximum number of versions to return per page (1-100). - in: query name: offset schema: type: integer minimum: 0 description: Zero-based offset into version history for pagination. Mutually exclusive with before_version. - $ref: '#/components/parameters/ProjectIdPath' - in: path name: prompt_name schema: type: string pattern: ^[^/]+$ required: true - in: query name: version schema: type: integer minimum: 1 description: Specific prompt version to fetch. If omitted, the latest version is returned. - in: query name: version_id schema: type: string format: uuid description: Exact prompt version UUID to resolve. Can be used together with version for extra safety. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_prompt:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMPromptResolveResponse' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/: get: operationId: llm_skills_list parameters: - in: query name: created_by_id schema: type: integer description: Filter skills by the ID of the user who created them. - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Optional substring filter applied to skill names and descriptions. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_skill:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedLLMSkillListList' description: '' x-explicit-tags: - llm_analytics post: operationId: llm_skills_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMSkillCreate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMSkillCreate' multipart/form-data: schema: $ref: '#/components/schemas/LLMSkillCreate' required: true security: - PersonalAPIKeyAuth: - llm_skill:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/LLMSkillCreate' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/name/{skill_name}/: get: operationId: llm_skills_name_retrieve parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true - in: query name: version schema: type: integer minimum: 1 description: Specific skill version to fetch. If omitted, the latest version is returned. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_skill:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMSkill' description: '' x-explicit-tags: - llm_analytics patch: operationId: llm_skills_name_partial_update parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedLLMSkillPublish' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedLLMSkillPublish' multipart/form-data: schema: $ref: '#/components/schemas/PatchedLLMSkillPublish' security: - PersonalAPIKeyAuth: - llm_skill:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMSkill' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/name/{skill_name}/archive/: post: operationId: llm_skills_name_archive_create parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_skill:write responses: '204': description: No response body x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/name/{skill_name}/duplicate/: post: operationId: llm_skills_name_duplicate_create parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMSkillDuplicate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMSkillDuplicate' multipart/form-data: schema: $ref: '#/components/schemas/LLMSkillDuplicate' required: true security: - PersonalAPIKeyAuth: - llm_skill:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/LLMSkill' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/name/{skill_name}/files/: post: operationId: llm_skills_name_files_create parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMSkillFileCreate' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMSkillFileCreate' multipart/form-data: schema: $ref: '#/components/schemas/LLMSkillFileCreate' required: true security: - PersonalAPIKeyAuth: - llm_skill:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/LLMSkill' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/name/{skill_name}/files-rename/: post: operationId: llm_skills_name_files_rename_create parameters: - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/LLMSkillFileRename' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/LLMSkillFileRename' multipart/form-data: schema: $ref: '#/components/schemas/LLMSkillFileRename' required: true security: - PersonalAPIKeyAuth: - llm_skill:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMSkill' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/name/{skill_name}/files/{file_path}/: get: operationId: llm_skills_name_files_retrieve parameters: - in: path name: file_path schema: type: string pattern: ^.+$ required: true - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true - in: query name: version schema: type: integer minimum: 1 description: Specific skill version to fetch. If omitted, the latest version is returned. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_skill:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMSkillFile' description: '' x-explicit-tags: - llm_analytics delete: operationId: llm_skills_name_files_destroy parameters: - in: query name: base_version schema: type: integer minimum: 1 description: Latest version you are editing from. If provided, the request fails with 409 when another write has landed in the meantime. - in: path name: file_path schema: type: string pattern: ^.+$ required: true - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_skill:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMSkill' description: '' x-explicit-tags: - llm_analytics /api/environments/{project_id}/llm_skills/resolve/name/{skill_name}/: get: operationId: llm_skills_resolve_name_retrieve parameters: - in: query name: before_version schema: type: integer minimum: 1 description: Return versions older than this version number. Mutually exclusive with offset. - in: query name: limit schema: type: integer maximum: 100 minimum: 1 default: 50 description: Maximum number of versions to return per page (1-100). - in: query name: offset schema: type: integer minimum: 0 description: Zero-based offset into version history for pagination. Mutually exclusive with before_version. - $ref: '#/components/parameters/ProjectIdPath' - in: path name: skill_name schema: type: string pattern: ^[^/]+$ required: true - in: query name: version schema: type: integer minimum: 1 description: Specific skill version to fetch. If omitted, the latest version is returned. - in: query name: version_id schema: type: string format: uuid description: Exact skill version UUID to resolve. tags: - LLM Analytics security: - PersonalAPIKeyAuth: - llm_skill:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/LLMSkillResolveResponse' description: '' x-explicit-tags: - llm_analytics /api/projects/{project_id}/dataset_items/: get: operationId: dataset_items_list parameters: - in: query name: dataset schema: type: string description: Filter by dataset ID examples: SingleDatasetID: value: 695401fa-6f0e-4389-b186-c45a7f1273d3 summary: Single dataset ID - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedDatasetItemList' description: '' x-explicit-tags: - llm_analytics post: operationId: dataset_items_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/DatasetItem' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/DatasetItem' multipart/form-data: schema: $ref: '#/components/schemas/DatasetItem' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' x-explicit-tags: - llm_analytics /api/projects/{project_id}/dataset_items/{id}/: get: operationId: dataset_items_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' x-explicit-tags: - llm_analytics put: operationId: dataset_items_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/DatasetItem' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/DatasetItem' multipart/form-data: schema: $ref: '#/components/schemas/DatasetItem' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' x-explicit-tags: - llm_analytics patch: operationId: dataset_items_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedDatasetItem' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedDatasetItem' multipart/form-data: schema: $ref: '#/components/schemas/PatchedDatasetItem' security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/DatasetItem' description: '' x-explicit-tags: - llm_analytics delete: operationId: dataset_items_destroy description: Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset item. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:write responses: '405': description: No response body x-explicit-tags: - llm_analytics /api/projects/{project_id}/datasets/: get: operationId: datasets_list parameters: - in: query name: id__in schema: type: array items: type: string format: uuid description: Multiple values may be separated by commas. explode: false style: form - name: limit required: false in: query description: Number of results to return per page. schema: type: integer - name: offset required: false in: query description: The initial index from which to return the results. schema: type: integer - in: query name: order_by schema: type: array items: type: string enum: - -created_at - -updated_at - created_at - updated_at description: 'Ordering * `created_at` - Created At * `-created_at` - Created At (descending) * `updated_at` - Updated At * `-updated_at` - Updated At (descending)' explode: false style: form - $ref: '#/components/parameters/ProjectIdPath' - in: query name: search schema: type: string description: Search in name, description, or metadata tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/PaginatedDatasetList' description: '' x-explicit-tags: - llm_analytics post: operationId: datasets_create parameters: - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/Dataset' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/Dataset' multipart/form-data: schema: $ref: '#/components/schemas/Dataset' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '201': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' x-explicit-tags: - llm_analytics /api/projects/{project_id}/datasets/{id}/: get: operationId: datasets_retrieve parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:read responses: '200': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' x-explicit-tags: - llm_analytics put: operationId: datasets_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/Dataset' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/Dataset' multipart/form-data: schema: $ref: '#/components/schemas/Dataset' required: true security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' x-explicit-tags: - llm_analytics patch: operationId: datasets_partial_update parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchedDataset' application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/PatchedDataset' multipart/form-data: schema: $ref: '#/components/schemas/PatchedDataset' security: - PersonalAPIKeyAuth: - dataset:write responses: '200': content: application/json: schema: $ref: '#/components/schemas/Dataset' description: '' x-explicit-tags: - llm_analytics delete: operationId: datasets_destroy description: Hard delete of this model is not allowed. Use a patch API call to set "deleted" to true parameters: - in: path name: id schema: type: string format: uuid description: A UUID string identifying this dataset. required: true - $ref: '#/components/parameters/ProjectIdPath' tags: - LLM Analytics security: - PersonalAPIKeyAuth: - dataset:write responses: '405': description: No response body x-explicit-tags: - llm_analytics components: schemas: TestHogResponse: type: object properties: results: type: array items: $ref: '#/components/schemas/TestHogResultItem' message: type: string description: Optional message, e.g. when no recent events were found. required: - results TranslateRequest: type: object properties: text: type: string description: The text to translate maxLength: 10000 target_language: type: string default: en description: 'Target language code (default: ''en'' for English)' maxLength: 10 required: - text RoleAtOrganizationEnum: enum: - engineering - data - product - founder - leadership - marketing - sales - other type: string description: '* `engineering` - Engineering * `data` - Data * `product` - Product Management * `founder` - Founder * `leadership` - Leadership * `marketing` - Marketing * `sales` - Sales / Success * `other` - Other' LLMProviderKey: type: object properties: id: type: string format: uuid readOnly: true provider: $ref: '#/components/schemas/LLMProviderEnum' name: type: string maxLength: 255 state: allOf: - $ref: '#/components/schemas/LLMProviderKeyStateEnum' readOnly: true error_message: type: string readOnly: true nullable: true api_key: type: string writeOnly: true api_key_masked: type: string readOnly: true azure_endpoint: type: string format: uri writeOnly: true description: Azure OpenAI endpoint URL api_version: type: string writeOnly: true description: Azure OpenAI API version maxLength: 20 azure_endpoint_display: type: string nullable: true readOnly: true description: Azure endpoint (read-only, for display) api_version_display: type: string nullable: true readOnly: true description: Azure API version (read-only, for display) set_as_active: type: boolean writeOnly: true default: false created_at: type: string format: date-time readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true last_used_at: type: string format: date-time readOnly: true nullable: true required: - api_key_masked - api_version_display - azure_endpoint_display - created_at - created_by - error_message - id - last_used_at - name - provider - state OutputTypeEnum: enum: - boolean type: string description: '* `boolean` - Boolean (Pass/Fail)' PaginatedLLMSkillListList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/LLMSkillList' PaginatedEvaluationList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/Evaluation' LLMSkillVersionSummary: type: object properties: id: type: string format: uuid readOnly: true version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true is_latest: type: boolean readOnly: true required: - created_at - created_by - id - is_latest - version EvaluationConfig: type: object properties: trial_eval_limit: type: integer readOnly: true description: Maximum number of llm_judge runs the team may execute on PostHog trial credits. trial_evals_used: type: integer readOnly: true description: Number of llm_judge runs already consumed against the trial credit pool. trial_evals_remaining: type: integer readOnly: true description: Number of trial evaluation runs remaining before the team must supply its own provider key. active_provider_key: allOf: - $ref: '#/components/schemas/LLMProviderKey' readOnly: true nullable: true description: Provider key currently used to run llm_judge evaluations. Null when the team is on trial credits. created_at: type: string format: date-time readOnly: true description: Timestamp when the evaluation config row was created. updated_at: type: string format: date-time readOnly: true description: Timestamp when the evaluation config row was last modified. required: - active_provider_key - created_at - trial_eval_limit - trial_evals_remaining - trial_evals_used - updated_at TextReprResponse: type: object properties: text: type: string description: Generated text representation of the event metadata: allOf: - $ref: '#/components/schemas/TextReprMetadata' description: Metadata about the text representation required: - metadata - text NullEnum: enum: - null PatchedScoreDefinitionMetadata: type: object properties: name: type: string description: Updated scorer name. maxLength: 255 description: type: string nullable: true description: Updated scorer description. archived: type: boolean description: Whether the scorer is archived. PaginatedReviewQueueList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/ReviewQueue' LLMPromptList: type: object properties: id: type: string format: uuid readOnly: true name: type: string readOnly: true description: Unique prompt name using letters, numbers, hyphens, and underscores only. prompt: readOnly: true description: Prompt payload as JSON or string data. version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true deleted: type: boolean readOnly: true is_latest: type: boolean readOnly: true latest_version: type: integer readOnly: true version_count: type: integer readOnly: true first_version_created_at: type: string readOnly: true outline: type: array items: $ref: '#/components/schemas/LLMPromptOutlineEntry' readOnly: true prompt_preview: type: string readOnly: true prompt_size_bytes: type: integer readOnly: true required: - created_at - created_by - deleted - first_version_created_at - id - is_latest - latest_version - name - outline - prompt - prompt_preview - prompt_size_bytes - updated_at - version - version_count LLMSkillResolveResponse: type: object properties: skill: $ref: '#/components/schemas/LLMSkill' versions: type: array items: $ref: '#/components/schemas/LLMSkillVersionSummary' has_more: type: boolean required: - has_more - skill - versions BooleanScoreDefinitionConfig: type: object properties: true_label: type: string description: Optional label for a true value. false_label: type: string description: Optional label for a false value. SentimentBatchResponse: type: object properties: results: type: object additionalProperties: $ref: '#/components/schemas/SentimentResult' required: - results LLMSkill: type: object properties: id: type: string format: uuid readOnly: true name: type: string description: Unique skill name. Lowercase letters, numbers, and hyphens only. Max 64 characters. maxLength: 64 description: type: string description: What this skill does and when to use it. Max 4096 characters. maxLength: 4096 body: type: string description: The SKILL.md instruction content (markdown). license: type: string description: License name or reference to a bundled license file. maxLength: 255 compatibility: type: string description: Environment requirements (intended product, system packages, network access, etc.). maxLength: 500 allowed_tools: type: array items: type: string description: List of pre-approved tools the skill may use. metadata: type: object additionalProperties: true description: Arbitrary key-value metadata. files: type: array items: $ref: '#/components/schemas/LLMSkillFileManifest' readOnly: true description: Bundled files manifest. Each entry is path + content_type only; fetch content via /llm_skills/name/{name}/files/{path}/. outline: type: array items: $ref: '#/components/schemas/LLMSkillOutlineEntry' readOnly: true description: Flat list of markdown headings parsed from the skill body. Useful as a lightweight table of contents. version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true deleted: type: boolean readOnly: true is_latest: type: boolean readOnly: true latest_version: type: integer readOnly: true version_count: type: integer readOnly: true first_version_created_at: type: string readOnly: true required: - body - created_at - created_by - deleted - description - files - first_version_created_at - id - is_latest - latest_version - name - outline - updated_at - version - version_count DeliveryStatusEnum: enum: - pending - delivered - partial_failure - failed type: string description: '* `pending` - Pending * `delivered` - Delivered * `partial_failure` - Partial Failure * `failed` - Failed' LLMSkillEditOperation: type: object properties: old: type: string description: Text to find in the target content. Must match exactly once. new: type: string description: Replacement text. required: - new - old StructuredSummary: type: object properties: title: type: string description: Concise title (no longer than 10 words) summarizing the trace/event flow_diagram: type: string description: Mermaid flowchart code showing the main flow summary_bullets: type: array items: $ref: '#/components/schemas/SummaryBullet' description: Main summary bullets interesting_notes: type: array items: $ref: '#/components/schemas/InterestingNote' description: Interesting notes (0-2 for minimal, more for detailed) required: - flow_diagram - interesting_notes - summary_bullets - title LLMSkillFileManifest: type: object properties: path: type: string maxLength: 500 content_type: type: string maxLength: 100 required: - path EvaluationTypeEnum: enum: - llm_judge - hog type: string description: '* `llm_judge` - LLM as a judge * `hog` - Hog' FilterEnum: enum: - all - pass - fail - na type: string description: '* `all` - all * `pass` - pass * `fail` - fail * `na` - na' EvaluationReportFrequencyEnum: enum: - scheduled - every_n type: string description: '* `scheduled` - Scheduled * `every_n` - Every N' BlankEnum: enum: - '' TestHogRequest: type: object properties: source: type: string description: Hog source code to test. Must return a boolean (true = pass, false = fail) or null for N/A. minLength: 1 sample_count: type: integer maximum: 10 minimum: 1 default: 5 description: Number of recent $ai_generation events to test against (1–10, default 5). allows_na: type: boolean default: false description: Whether the evaluation can return N/A for non-applicable generations. conditions: type: array items: type: object additionalProperties: true description: Optional trigger conditions to filter which events are sampled. required: - source EvaluationRunRequest: type: object properties: evaluation_id: type: string format: uuid description: UUID of the evaluation to run. target_event_id: type: string format: uuid description: UUID of the $ai_generation event to evaluate. timestamp: type: string format: date-time description: ISO 8601 timestamp of the target event (needed for efficient ClickHouse lookup). event: type: string default: $ai_generation description: Event name. Defaults to '$ai_generation'. distinct_id: type: string nullable: true description: Distinct ID of the event (optional, improves lookup performance). required: - evaluation_id - target_event_id - timestamp LLMSkillDuplicate: type: object properties: new_name: type: string description: Name for the duplicated skill. Must be unique. maxLength: 64 required: - new_name LLMProviderKeyStateEnum: enum: - unknown - ok - invalid - error type: string description: '* `unknown` - Unknown * `ok` - Ok * `invalid` - Invalid * `error` - Error' SummaryBullet: type: object properties: text: type: string line_refs: type: string required: - line_refs - text PaginatedTraceReviewList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/TraceReview' PatchedEvaluationReport: type: object properties: id: type: string format: uuid readOnly: true evaluation: type: string format: uuid description: UUID of the evaluation this report config belongs to. frequency: allOf: - $ref: '#/components/schemas/EvaluationReportFrequencyEnum' description: 'How report generation is triggered. ''every_n'' fires once N new evaluation results have accumulated (subject to cooldown_minutes and daily_run_cap). ''scheduled'' fires on the cadence defined by rrule + starts_at + timezone_name. * `scheduled` - Scheduled * `every_n` - Every N' rrule: type: string description: RFC 5545 recurrence rule string (e.g. 'FREQ=WEEKLY;BYDAY=MO'). Must not contain DTSTART — the anchor is set via starts_at. Required when frequency is 'scheduled'; ignored otherwise. starts_at: type: string format: date-time nullable: true description: Anchor datetime for the rrule (ISO 8601, UTC — must end in 'Z'). Local-time interpretation is controlled by timezone_name. Required when frequency is 'scheduled'; ignored otherwise. timezone_name: type: string description: IANA timezone name used to expand the rrule in local time so e.g. '9am' stays at 9am across DST transitions (e.g. 'America/New_York'). Defaults to 'UTC'. maxLength: 64 next_delivery_date: type: string format: date-time readOnly: true nullable: true delivery_targets: description: 'List of delivery targets. Each entry is either {type: ''email'', value: ''user@example.com''} or {type: ''slack'', integration_id: , channel: ''''}. Slack integration_id must belong to this team.' max_sample_size: type: integer maximum: 2147483647 minimum: -2147483648 description: Maximum number of evaluation runs included in each report. Defaults to 200. enabled: type: boolean description: Whether report delivery is active. Disabled configs do not fire. deleted: type: boolean description: Set to true to soft-delete this report config. last_delivered_at: type: string format: date-time readOnly: true nullable: true report_prompt_guidance: type: string description: Optional custom instructions appended to the AI report prompt to steer focus, scope, or section choices without modifying the base prompt. trigger_threshold: type: integer maximum: 10000 minimum: 10 nullable: true description: Number of new evaluation results that triggers a report (every_n mode only). Min 10, max 10000. Defaults to 100. Required when frequency is 'every_n'. cooldown_minutes: type: integer maximum: 1440 minimum: 60 description: Minimum minutes between count-triggered reports to prevent spam (every_n mode only). Min 60, max 1440 (24 hours). Defaults to 60. daily_run_cap: type: integer maximum: 24 minimum: 1 description: Maximum count-triggered report runs per calendar day (UTC). Min 1, max 24 (one per cooldown window). Defaults to 10. created_by: type: integer readOnly: true nullable: true created_at: type: string format: date-time readOnly: true PaginatedScoreDefinitionList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/ScoreDefinition' LLMSkillFileEdit: type: object properties: path: type: string description: Path of the bundled file to edit. Must match an existing file on the current skill version. maxLength: 500 edits: type: array items: $ref: '#/components/schemas/LLMSkillEditOperation' description: Sequential find/replace operations to apply to this file's content. required: - edits - path EvaluationConfigSetActiveKeyRequest: type: object properties: key_id: type: string format: uuid description: UUID of an existing LLM provider key (state must be 'ok') to mark as the active key for running llm_judge evaluations team-wide. required: - key_id LLMPromptOutlineEntry: type: object properties: level: type: integer maximum: 6 minimum: 1 description: Markdown heading level (1-6). text: type: string description: Heading text with markdown link syntax preserved. required: - level - text BatchCheckRequest: type: object properties: trace_ids: type: array items: type: string description: List of trace IDs to check for cached summaries maxItems: 100 mode: allOf: - $ref: '#/components/schemas/DetailModeValueEnum' default: minimal description: 'Summary detail level to check for * `minimal` - minimal * `detailed` - detailed' model: type: string nullable: true description: LLM model used for cached summaries required: - trace_ids EvaluationReportRun: type: object properties: id: type: string format: uuid readOnly: true description: UUID of this report run. report: type: string format: uuid readOnly: true description: UUID of the report config that generated this run. content: readOnly: true description: Generated report content (markdown or structured text). metadata: readOnly: true description: Run metadata including model used, token counts, and generation stats. period_start: type: string format: date-time readOnly: true description: Start of the evaluation window covered by this report. period_end: type: string format: date-time readOnly: true description: End of the evaluation window covered by this report. delivery_status: allOf: - $ref: '#/components/schemas/DeliveryStatusEnum' readOnly: true description: '''pending'', ''delivered'', or ''failed''. * `pending` - Pending * `delivered` - Delivered * `partial_failure` - Partial Failure * `failed` - Failed' delivery_errors: readOnly: true description: List of delivery error messages if delivery failed. created_at: type: string format: date-time readOnly: true required: - content - created_at - delivery_errors - delivery_status - id - metadata - period_end - period_start - report TraceReviewScore: type: object properties: id: type: string format: uuid readOnly: true definition_id: type: string format: uuid readOnly: true description: Stable scorer definition ID. definition_name: type: string readOnly: true description: Human-readable scorer name. definition_kind: type: string readOnly: true description: Scorer kind for this saved score. definition_archived: type: boolean readOnly: true description: Whether the scorer is currently archived. definition_version_id: type: string format: uuid readOnly: true description: Immutable scorer version ID used to validate this score. definition_version: type: integer readOnly: true description: Immutable scorer version number used to validate this score. definition_config: allOf: - $ref: '#/components/schemas/ScoreDefinitionConfig' readOnly: true description: Immutable scorer configuration snapshot used to validate this score. categorical_values: type: array items: type: string readOnly: true nullable: true description: Categorical option keys selected for this score. numeric_value: type: string format: decimal pattern: ^-?\d{0,6}(?:\.\d{0,6})?$ readOnly: true nullable: true boolean_value: type: boolean readOnly: true nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true required: - boolean_value - categorical_values - created_at - definition_archived - definition_config - definition_id - definition_kind - definition_name - definition_version - definition_version_id - id - numeric_value - updated_at StatusReasonEnum: enum: - trial_limit_reached - model_not_allowed - provider_key_deleted type: string description: '* `trial_limit_reached` - Trial evaluation limit reached * `model_not_allowed` - Model not available on the trial plan * `provider_key_deleted` - Provider API key was deleted' TextReprRequest: type: object properties: event_type: allOf: - $ref: '#/components/schemas/EventTypeEnum' description: 'Type of LLM event to stringify * `$ai_generation` - $ai_generation * `$ai_span` - $ai_span * `$ai_embedding` - $ai_embedding * `$ai_trace` - $ai_trace' data: description: Event data to stringify. For traces, should include 'trace' and 'hierarchy' fields. options: allOf: - $ref: '#/components/schemas/TextReprOptions' description: Optional configuration for text generation required: - data - event_type InterestingNote: type: object properties: text: type: string line_refs: type: string required: - line_refs - text LLMPrompt: type: object properties: id: type: string format: uuid readOnly: true name: type: string description: Unique prompt name using letters, numbers, hyphens, and underscores only. maxLength: 255 prompt: description: Prompt payload as JSON or string data. version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true deleted: type: boolean readOnly: true is_latest: type: boolean readOnly: true latest_version: type: integer readOnly: true version_count: type: integer readOnly: true first_version_created_at: type: string readOnly: true outline: type: array items: $ref: '#/components/schemas/LLMPromptOutlineEntry' readOnly: true required: - created_at - created_by - deleted - first_version_created_at - id - is_latest - latest_version - name - outline - prompt - updated_at - version - version_count PatchedClusteringJob: type: object properties: id: type: string format: uuid readOnly: true name: type: string maxLength: 100 analysis_level: $ref: '#/components/schemas/ClusteringJobAnalysisLevelEnum' event_filters: {} enabled: type: boolean created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true PaginatedDatasetList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/Dataset' PatchedLLMProviderKey: type: object properties: id: type: string format: uuid readOnly: true provider: $ref: '#/components/schemas/LLMProviderEnum' name: type: string maxLength: 255 state: allOf: - $ref: '#/components/schemas/LLMProviderKeyStateEnum' readOnly: true error_message: type: string readOnly: true nullable: true api_key: type: string writeOnly: true api_key_masked: type: string readOnly: true azure_endpoint: type: string format: uri writeOnly: true description: Azure OpenAI endpoint URL api_version: type: string writeOnly: true description: Azure OpenAI API version maxLength: 20 azure_endpoint_display: type: string nullable: true readOnly: true description: Azure endpoint (read-only, for display) api_version_display: type: string nullable: true readOnly: true description: Azure API version (read-only, for display) set_as_active: type: boolean writeOnly: true default: false created_at: type: string format: date-time readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true last_used_at: type: string format: date-time readOnly: true nullable: true PaginatedLLMProviderKeyList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/LLMProviderKey' LLMProviderEnum: enum: - openai - anthropic - gemini - openrouter - fireworks - azure_openai - together_ai type: string description: '* `openai` - Openai * `anthropic` - Anthropic * `gemini` - Gemini * `openrouter` - Openrouter * `fireworks` - Fireworks * `azure_openai` - Azure OpenAI * `together_ai` - Together AI' PatchedDatasetItem: type: object properties: id: type: string format: uuid readOnly: true dataset: type: string format: uuid input: nullable: true output: nullable: true metadata: nullable: true ref_trace_id: type: string nullable: true maxLength: 255 ref_timestamp: type: string format: date-time nullable: true ref_source_id: type: string nullable: true maxLength: 255 deleted: type: boolean nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true team: type: integer readOnly: true LLMSkillFileRename: type: object properties: old_path: type: string description: Current file path to rename. maxLength: 500 new_path: type: string description: New file path. Must not already exist in the skill. maxLength: 500 base_version: type: integer minimum: 1 description: Latest version you are editing from. If provided, the request fails with 409 when another write has landed in the meantime. required: - new_path - old_path EvaluationReport: type: object properties: id: type: string format: uuid readOnly: true evaluation: type: string format: uuid description: UUID of the evaluation this report config belongs to. frequency: allOf: - $ref: '#/components/schemas/EvaluationReportFrequencyEnum' description: 'How report generation is triggered. ''every_n'' fires once N new evaluation results have accumulated (subject to cooldown_minutes and daily_run_cap). ''scheduled'' fires on the cadence defined by rrule + starts_at + timezone_name. * `scheduled` - Scheduled * `every_n` - Every N' rrule: type: string description: RFC 5545 recurrence rule string (e.g. 'FREQ=WEEKLY;BYDAY=MO'). Must not contain DTSTART — the anchor is set via starts_at. Required when frequency is 'scheduled'; ignored otherwise. starts_at: type: string format: date-time nullable: true description: Anchor datetime for the rrule (ISO 8601, UTC — must end in 'Z'). Local-time interpretation is controlled by timezone_name. Required when frequency is 'scheduled'; ignored otherwise. timezone_name: type: string description: IANA timezone name used to expand the rrule in local time so e.g. '9am' stays at 9am across DST transitions (e.g. 'America/New_York'). Defaults to 'UTC'. maxLength: 64 next_delivery_date: type: string format: date-time readOnly: true nullable: true delivery_targets: description: 'List of delivery targets. Each entry is either {type: ''email'', value: ''user@example.com''} or {type: ''slack'', integration_id: , channel: ''''}. Slack integration_id must belong to this team.' max_sample_size: type: integer maximum: 2147483647 minimum: -2147483648 description: Maximum number of evaluation runs included in each report. Defaults to 200. enabled: type: boolean description: Whether report delivery is active. Disabled configs do not fire. deleted: type: boolean description: Set to true to soft-delete this report config. last_delivered_at: type: string format: date-time readOnly: true nullable: true report_prompt_guidance: type: string description: Optional custom instructions appended to the AI report prompt to steer focus, scope, or section choices without modifying the base prompt. trigger_threshold: type: integer maximum: 10000 minimum: 10 nullable: true description: Number of new evaluation results that triggers a report (every_n mode only). Min 10, max 10000. Defaults to 100. Required when frequency is 'every_n'. cooldown_minutes: type: integer maximum: 1440 minimum: 60 description: Minimum minutes between count-triggered reports to prevent spam (every_n mode only). Min 60, max 1440 (24 hours). Defaults to 60. daily_run_cap: type: integer maximum: 24 minimum: 1 description: Maximum count-triggered report runs per calendar day (UTC). Min 1, max 24 (one per cooldown window). Defaults to 10. created_by: type: integer readOnly: true nullable: true created_at: type: string format: date-time readOnly: true required: - created_at - created_by - evaluation - id - last_delivered_at - next_delivery_date PatchedLLMPromptPublish: type: object properties: prompt: description: Full prompt payload to publish as a new version. Mutually exclusive with edits. edits: type: array items: $ref: '#/components/schemas/LLMPromptEditOperation' description: List of find/replace operations to apply to the current prompt version. Each edit's 'old' text must match exactly once. Edits are applied sequentially. Mutually exclusive with prompt. base_version: type: integer minimum: 1 description: Latest version you are editing from. Used for optimistic concurrency checks. PatchedReviewQueueUpdate: type: object properties: name: type: string description: Human-readable queue name. maxLength: 255 CategoricalScoreDefinitionConfig: type: object properties: options: type: array items: $ref: '#/components/schemas/CategoricalScoreOption' description: Ordered categorical options available to the scorer. selection_mode: allOf: - $ref: '#/components/schemas/SelectionModeEnum' description: 'Whether reviewers can select one option or multiple options. Defaults to `single`. * `single` - single * `multiple` - multiple' min_selections: type: integer minimum: 1 nullable: true description: Optional minimum number of options that can be selected when `selection_mode` is `multiple`. max_selections: type: integer minimum: 1 nullable: true description: Optional maximum number of options that can be selected when `selection_mode` is `multiple`. required: - options ScoreDefinitionNewVersion: type: object properties: config: allOf: - $ref: '#/components/schemas/ScoreDefinitionConfig' description: Next immutable scorer configuration. required: - config PaginatedDatasetItemList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/DatasetItem' LLMPromptEditOperation: type: object properties: old: type: string description: Text to find in the current prompt. Must match exactly once. new: type: string description: Replacement text. required: - new - old TestHogResultItem: type: object properties: event_uuid: type: string description: UUID of the $ai_generation event. trace_id: type: string nullable: true description: Trace ID if available. input_preview: type: string description: First 200 chars of the generation input. output_preview: type: string description: First 200 chars of the generation output. result: type: boolean nullable: true description: True = pass, False = fail, null = N/A or error. reasoning: type: string nullable: true description: Hog evaluation reasoning string, if any. error: type: string nullable: true description: Error message if the Hog code raised an exception. required: - error - event_uuid - input_preview - output_preview - reasoning - result LLMSkillFileCreate: type: object properties: path: type: string description: File path relative to skill root, e.g. 'scripts/setup.sh' or 'references/guide.md'. maxLength: 500 content: type: string description: Text content of the file. content_type: type: string default: text/plain description: MIME type of the file content. maxLength: 100 base_version: type: integer minimum: 1 description: Latest version you are editing from. If provided, the request fails with 409 when another write has landed in the meantime. required: - content - path LLMPromptDuplicate: type: object properties: new_name: type: string description: Name for the duplicated prompt. Must be unique and use only letters, numbers, hyphens, and underscores. maxLength: 255 required: - new_name PaginatedEvaluationReportList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/EvaluationReport' TraceReviewCreate: type: object properties: trace_id: type: string description: Trace ID for the review. Only one active review can exist per trace and team. maxLength: 255 comment: type: string nullable: true description: Optional comment or reasoning for the review. scores: type: array items: $ref: '#/components/schemas/TraceReviewScoreWrite' description: Full desired score set for this review. Omit scorers you want to leave blank. queue_id: type: string format: uuid nullable: true description: Optional review queue ID for queue-context saves. When provided, the matching pending queue item is cleared after the review is saved. If omitted, any pending queue item for the same trace is cleared. required: - trace_id PaginatedClusteringJobList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/ClusteringJob' EvaluationSummaryRequest: type: object description: Request serializer for evaluation summary - accepts IDs only, fetches data server-side. properties: evaluation_id: type: string format: uuid description: UUID of the evaluation config to summarize filter: allOf: - $ref: '#/components/schemas/FilterEnum' default: all description: 'Filter type to apply (''all'', ''pass'', ''fail'', or ''na'') * `all` - all * `pass` - pass * `fail` - fail * `na` - na' generation_ids: type: array items: type: string format: uuid description: 'Optional: specific generation IDs to include in summary (max 250)' maxItems: 250 force_refresh: type: boolean default: false description: If true, bypass cache and generate a fresh summary required: - evaluation_id ReviewQueue: type: object properties: id: type: string format: uuid readOnly: true name: type: string readOnly: true description: Human-readable queue name. pending_item_count: type: integer readOnly: true description: Number of pending traces currently assigned to this queue. created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true description: User who created this review queue. team: type: integer readOnly: true required: - created_at - created_by - id - name - pending_item_count - team - updated_at SentimentRequestAnalysisLevelEnum: enum: - trace - generation type: string description: '* `trace` - trace * `generation` - generation' ScoreDefinition: type: object properties: id: type: string format: uuid readOnly: true name: type: string readOnly: true description: type: string readOnly: true kind: allOf: - $ref: '#/components/schemas/ExperimentMetricKindEnum' readOnly: true archived: type: boolean readOnly: true current_version: type: integer readOnly: true description: Current immutable configuration version number. config: allOf: - $ref: '#/components/schemas/ScoreDefinitionConfig' readOnly: true description: Current immutable scorer configuration. created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true nullable: true description: User who created the scorer. created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true team: type: integer readOnly: true required: - archived - config - created_at - created_by - current_version - description - id - kind - name - team - updated_at LLMSkillFile: type: object properties: path: type: string maxLength: 500 content: type: string content_type: type: string maxLength: 100 required: - content - path SentimentResult: type: object properties: label: type: string score: type: number format: double scores: type: object additionalProperties: type: number format: double messages: type: object additionalProperties: $ref: '#/components/schemas/MessageSentiment' message_count: type: integer required: - label - message_count - messages - score - scores ReviewQueueItem: type: object properties: id: type: string format: uuid readOnly: true queue_id: type: string format: uuid readOnly: true description: Review queue ID that currently owns this pending trace. queue_name: type: string readOnly: true description: Human-readable name of the queue that currently owns this pending trace. trace_id: type: string readOnly: true description: Trace ID currently pending review. created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true description: User who queued this trace. team: type: integer readOnly: true required: - created_at - created_by - id - queue_id - queue_name - team - trace_id - updated_at PatchedLLMSkillPublish: type: object properties: body: type: string description: Full skill body (SKILL.md instruction content) to publish as a new version. Mutually exclusive with edits. edits: type: array items: $ref: '#/components/schemas/LLMSkillEditOperation' description: List of find/replace operations to apply to the current skill body. Each edit's 'old' text must match exactly once. Edits are applied sequentially. Mutually exclusive with body. description: type: string description: Updated description for the new version. maxLength: 4096 license: type: string description: License name or reference. maxLength: 255 compatibility: type: string description: Environment requirements. maxLength: 500 allowed_tools: type: array items: type: string description: List of pre-approved tools the skill may use. metadata: type: object additionalProperties: true description: Arbitrary key-value metadata. files: type: array items: $ref: '#/components/schemas/LLMSkillFileInput' description: Bundled files to include with this version. Replaces all files from the previous version. Mutually exclusive with file_edits. file_edits: type: array items: $ref: '#/components/schemas/LLMSkillFileEdit' description: Per-file find/replace updates. Each entry targets one existing file by path and applies sequential edits to its content. Non-targeted files carry forward unchanged. Cannot add, remove, or rename files — use 'files' for that. Mutually exclusive with files. base_version: type: integer minimum: 1 description: Latest version you are editing from. Used for optimistic concurrency checks. ExperimentMetricKindEnum: enum: - categorical - numeric - boolean type: string description: '* `categorical` - categorical * `numeric` - numeric * `boolean` - boolean' SentimentRequest: type: object properties: ids: type: array items: type: string description: Trace IDs or generation IDs to classify, depending on analysis_level. maxItems: 5 minItems: 1 analysis_level: allOf: - $ref: '#/components/schemas/SentimentRequestAnalysisLevelEnum' default: trace description: 'Whether the IDs are ''trace'' IDs or ''generation'' IDs. * `trace` - trace * `generation` - generation' force_refresh: type: boolean default: false description: If true, bypass cache and reclassify. date_from: type: string nullable: true description: Start of date range for the lookup (e.g. '-7d' or '2026-01-01'). Defaults to -30d. date_to: type: string nullable: true description: End of date range for the lookup. Defaults to now. required: - ids ClusteringJob: type: object properties: id: type: string format: uuid readOnly: true name: type: string maxLength: 100 analysis_level: $ref: '#/components/schemas/ClusteringJobAnalysisLevelEnum' event_filters: {} enabled: type: boolean created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true required: - analysis_level - created_at - id - name - updated_at BatchCheckResponse: type: object properties: summaries: type: array items: $ref: '#/components/schemas/CachedSummary' required: - summaries SummarizeRequest: type: object properties: summarize_type: allOf: - $ref: '#/components/schemas/SummarizeTypeEnum' description: 'Type of entity to summarize. Inferred automatically when using trace_id or generation_id. * `trace` - trace * `event` - event' mode: allOf: - $ref: '#/components/schemas/DetailModeValueEnum' default: minimal description: 'Summary detail level: ''minimal'' for 3-5 points, ''detailed'' for 5-10 points * `minimal` - minimal * `detailed` - detailed' data: description: 'Data to summarize. For traces: {trace, hierarchy}. For events: {event}. Not required when using trace_id or generation_id.' force_refresh: type: boolean default: false description: Force regenerate summary, bypassing cache model: type: string nullable: true description: LLM model to use (defaults based on provider) trace_id: type: string description: Trace ID to summarize. The backend fetches the trace data automatically. Requires date_from for efficient lookup. generation_id: type: string description: Generation event UUID to summarize. The backend fetches the event data automatically. Requires date_from for efficient lookup. date_from: type: string nullable: true description: Start of date range for ID-based lookup (e.g. '-7d' or '2026-01-01'). Defaults to -30d. date_to: type: string nullable: true description: End of date range for ID-based lookup. Defaults to now. CategoricalScoreOption: type: object properties: key: type: string description: Stable option key. Use lowercase letters, numbers, underscores, or hyphens. maxLength: 128 label: type: string description: Human-readable option label. maxLength: 256 required: - key - label PaginatedEvaluationReportRunList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/EvaluationReportRun' DatasetItem: type: object properties: id: type: string format: uuid readOnly: true dataset: type: string format: uuid input: nullable: true output: nullable: true metadata: nullable: true ref_trace_id: type: string nullable: true maxLength: 255 ref_timestamp: type: string format: date-time nullable: true ref_source_id: type: string nullable: true maxLength: 255 deleted: type: boolean nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true team: type: integer readOnly: true required: - created_at - created_by - dataset - id - team - updated_at SummarizeTypeEnum: enum: - trace - event type: string description: '* `trace` - trace * `event` - event' TraceReview: type: object properties: id: type: string format: uuid readOnly: true trace_id: type: string readOnly: true description: Trace ID for the review. comment: type: string readOnly: true nullable: true description: Optional comment or reasoning for the review. created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true reviewed_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true description: User who last saved this review. scores: type: array items: $ref: '#/components/schemas/TraceReviewScore' readOnly: true description: Saved scorer values for this review. team: type: integer readOnly: true required: - comment - created_at - created_by - id - reviewed_by - scores - team - trace_id - updated_at CachedSummary: type: object properties: trace_id: type: string title: type: string cached: type: boolean default: true required: - title - trace_id EventTypeEnum: enum: - $ai_generation - $ai_span - $ai_embedding - $ai_trace type: string description: '* `$ai_generation` - $ai_generation * `$ai_span` - $ai_span * `$ai_embedding` - $ai_embedding * `$ai_trace` - $ai_trace' PaginatedLLMPromptListList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/LLMPromptList' MessageSentiment: type: object properties: label: type: string score: type: number format: double scores: type: object additionalProperties: type: number format: double required: - label - score - scores PaginatedReviewQueueItemList: type: object required: - count - results properties: count: type: integer example: 123 next: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=400&limit=100 previous: type: string nullable: true format: uri example: http://api.example.org/accounts/?offset=200&limit=100 results: type: array items: $ref: '#/components/schemas/ReviewQueueItem' LLMPromptResolveResponse: type: object properties: prompt: $ref: '#/components/schemas/LLMPrompt' versions: type: array items: $ref: '#/components/schemas/LLMPromptVersionSummary' has_more: type: boolean required: - has_more - prompt - versions SummarizeResponse: type: object properties: summary: allOf: - $ref: '#/components/schemas/StructuredSummary' description: Structured AI-generated summary with flow, bullets, and optional notes text_repr: type: string description: Line-numbered text representation that the summary references metadata: description: Metadata about the summarization required: - summary - text_repr LLMSkillList: type: object description: List serializer that omits body and file manifest — progressive disclosure (Level 1). properties: id: type: string format: uuid readOnly: true name: type: string description: Unique skill name. Lowercase letters, numbers, and hyphens only. Max 64 characters. maxLength: 64 description: type: string description: What this skill does and when to use it. Max 4096 characters. maxLength: 4096 license: type: string description: License name or reference to a bundled license file. maxLength: 255 compatibility: type: string description: Environment requirements (intended product, system packages, network access, etc.). maxLength: 500 allowed_tools: type: array items: type: string description: List of pre-approved tools the skill may use. metadata: type: object additionalProperties: true description: Arbitrary key-value metadata. outline: type: array items: $ref: '#/components/schemas/LLMSkillOutlineEntry' readOnly: true description: Flat list of markdown headings parsed from the skill body. Useful as a lightweight table of contents. version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true deleted: type: boolean readOnly: true is_latest: type: boolean readOnly: true latest_version: type: integer readOnly: true version_count: type: integer readOnly: true first_version_created_at: type: string readOnly: true required: - created_at - created_by - deleted - description - first_version_created_at - id - is_latest - latest_version - name - outline - updated_at - version - version_count ScoreDefinitionCreate: type: object properties: name: type: string description: Human-readable scorer name. maxLength: 255 description: type: string nullable: true description: Optional human-readable description. kind: allOf: - $ref: '#/components/schemas/ExperimentMetricKindEnum' description: 'Scorer kind. This cannot be changed after creation. * `categorical` - categorical * `numeric` - numeric * `boolean` - boolean' archived: type: boolean default: false description: New scorers are always created as active. config: allOf: - $ref: '#/components/schemas/ScoreDefinitionConfig' description: Initial immutable scorer configuration. required: - config - kind - name ModelConfiguration: type: object description: Nested serializer for model configuration. properties: provider: $ref: '#/components/schemas/LLMProviderEnum' model: type: string maxLength: 100 provider_key_id: type: string format: uuid nullable: true provider_key_name: type: string nullable: true readOnly: true required: - model - provider - provider_key_name ReviewQueueItemCreate: type: object properties: queue_id: type: string format: uuid description: Review queue ID that should own this pending trace. trace_id: type: string description: Trace ID to add to the selected review queue. maxLength: 255 required: - queue_id - trace_id PatchedTraceReviewUpdate: type: object properties: trace_id: type: string description: Trace ID for the review. Only one active review can exist per trace and team. maxLength: 255 comment: type: string nullable: true description: Optional comment or reasoning for the review. scores: type: array items: $ref: '#/components/schemas/TraceReviewScoreWrite' description: Full desired score set for this review. Omit scorers you want to leave blank. queue_id: type: string format: uuid nullable: true description: Optional review queue ID for queue-context saves. When provided, the matching pending queue item is cleared after the review is saved. If omitted, any pending queue item for the same trace is cleared. DetailModeValueEnum: enum: - minimal - detailed type: string description: '* `minimal` - minimal * `detailed` - detailed' ClusteringJobAnalysisLevelEnum: enum: - trace - generation - evaluation type: string description: '* `trace` - trace * `generation` - generation * `evaluation` - evaluation' LLMPromptVersionSummary: type: object properties: id: type: string format: uuid readOnly: true version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true is_latest: type: boolean readOnly: true required: - created_at - created_by - id - is_latest - version LLMPromptPublic: type: object properties: id: type: string format: uuid name: type: string prompt: description: Full prompt content. Omitted when 'content=preview' or 'content=none'. prompt_preview: type: string description: First 160 characters of the prompt. Only present when 'content=preview'. outline: type: array items: $ref: '#/components/schemas/LLMPromptOutlineEntry' description: Flat list of markdown headings parsed from the prompt. Useful as a lightweight table of contents. version: type: integer created_at: type: string format: date-time updated_at: type: string format: date-time deleted: type: boolean is_latest: type: boolean latest_version: type: integer version_count: type: integer first_version_created_at: type: string format: date-time required: - created_at - deleted - first_version_created_at - id - is_latest - latest_version - name - outline - updated_at - version - version_count LLMSkillOutlineEntry: type: object properties: level: type: integer maximum: 6 minimum: 1 description: Markdown heading level (1-6). text: type: string description: Heading text. required: - level - text ScoreDefinitionConfig: oneOf: - $ref: '#/components/schemas/CategoricalScoreDefinitionConfig' - $ref: '#/components/schemas/NumericScoreDefinitionConfig' - $ref: '#/components/schemas/BooleanScoreDefinitionConfig' UserBasic: type: object properties: id: type: integer readOnly: true uuid: type: string format: uuid readOnly: true distinct_id: type: string nullable: true maxLength: 200 first_name: type: string maxLength: 150 last_name: type: string maxLength: 150 email: type: string format: email title: Email address maxLength: 254 is_email_verified: type: boolean nullable: true hedgehog_config: type: object additionalProperties: true nullable: true readOnly: true role_at_organization: nullable: true oneOf: - $ref: '#/components/schemas/RoleAtOrganizationEnum' - $ref: '#/components/schemas/BlankEnum' - $ref: '#/components/schemas/NullEnum' required: - email - hedgehog_config - id - uuid SelectionModeEnum: enum: - single - multiple type: string description: '* `single` - single * `multiple` - multiple' LLMSkillFileInput: type: object properties: path: type: string description: File path relative to skill root, e.g. 'scripts/setup.sh' or 'references/guide.md'. maxLength: 500 content: type: string description: Text content of the file. content_type: type: string default: text/plain description: MIME type of the file content. maxLength: 100 required: - content - path EvaluationStatusEnum: enum: - active - paused - error type: string description: '* `active` - Active * `paused` - Paused * `error` - Error' LLMModelsListResponse: type: object properties: models: type: array items: $ref: '#/components/schemas/LLMModelInfo' description: Models supported for the requested provider. required: - models EvaluationPattern: type: object properties: title: type: string description: type: string frequency: type: string example_generation_ids: type: array items: type: string required: - description - example_generation_ids - frequency - title PatchedDataset: type: object properties: id: type: string format: uuid readOnly: true name: type: string maxLength: 400 description: type: string nullable: true metadata: nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true deleted: type: boolean nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true team: type: integer readOnly: true ReviewQueueCreate: type: object properties: name: type: string description: Human-readable queue name. maxLength: 255 required: - name LLMModelInfo: type: object properties: id: type: string description: Provider-specific model identifier (e.g. 'gpt-4o-mini', 'claude-3-5-sonnet-20241022'). posthog_available: type: boolean description: Whether this model is available on PostHog's trial credits without bringing a provider key. required: - id - posthog_available Dataset: type: object properties: id: type: string format: uuid readOnly: true name: type: string maxLength: 400 description: type: string nullable: true metadata: nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true nullable: true deleted: type: boolean nullable: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true team: type: integer readOnly: true required: - created_at - created_by - id - name - team - updated_at EvaluationSummaryStatistics: type: object properties: total_analyzed: type: integer pass_count: type: integer fail_count: type: integer na_count: type: integer required: - fail_count - na_count - pass_count - total_analyzed TextReprOptions: type: object properties: max_length: type: integer description: 'Maximum length of generated text (default: 2000000)' truncated: type: boolean description: 'Use truncation for long content within events (default: true)' truncate_buffer: type: integer description: 'Characters to show at start/end when truncating (default: 1000)' include_markers: type: boolean description: 'Use interactive markers for frontend vs plain text for backend/LLM (default: true)' collapsed: type: boolean description: 'Show summary vs full tree hierarchy for traces (default: false)' include_metadata: type: boolean description: Include metadata in response include_hierarchy: type: boolean description: Include hierarchy information (for traces) max_depth: type: integer description: Maximum depth for hierarchical rendering tools_collapse_threshold: type: integer description: 'Number of tools before collapsing the list (default: 5)' include_line_numbers: type: boolean description: 'Prefix each line with line number (default: false)' EvaluationSummaryResponse: type: object properties: overall_assessment: type: string pass_patterns: type: array items: $ref: '#/components/schemas/EvaluationPattern' fail_patterns: type: array items: $ref: '#/components/schemas/EvaluationPattern' na_patterns: type: array items: $ref: '#/components/schemas/EvaluationPattern' recommendations: type: array items: type: string statistics: $ref: '#/components/schemas/EvaluationSummaryStatistics' required: - fail_patterns - na_patterns - overall_assessment - pass_patterns - recommendations - statistics NumericScoreDefinitionConfig: type: object properties: min: type: number format: double nullable: true description: Optional inclusive minimum score. max: type: number format: double nullable: true description: Optional inclusive maximum score. step: type: number format: double nullable: true description: Optional increment step for numeric input, for example 1 or 0.5. TextReprMetadata: type: object properties: event_type: type: string event_id: type: string trace_id: type: string rendering: type: string char_count: type: integer truncated: type: boolean error: type: string required: - char_count - rendering - truncated Evaluation: type: object properties: id: type: string format: uuid readOnly: true name: type: string description: Name of the evaluation. maxLength: 400 description: type: string description: Optional description of what this evaluation checks. enabled: type: boolean description: Whether the evaluation runs automatically on new $ai_generation events. status: allOf: - $ref: '#/components/schemas/EvaluationStatusEnum' readOnly: true status_reason: readOnly: true nullable: true oneOf: - $ref: '#/components/schemas/StatusReasonEnum' - $ref: '#/components/schemas/NullEnum' evaluation_type: allOf: - $ref: '#/components/schemas/EvaluationTypeEnum' description: '''llm_judge'' uses an LLM to score outputs against a prompt; ''hog'' runs deterministic Hog code. * `llm_judge` - LLM as a judge * `hog` - Hog' evaluation_config: oneOf: - type: object title: LLM judge config required: - prompt properties: prompt: type: string description: Evaluation criteria for the LLM judge. Describe what makes a good vs bad response. minLength: 1 additionalProperties: false - type: object title: Hog config required: - source properties: source: type: string description: Hog source code. Must return true (pass), false (fail), or null for N/A. minLength: 1 additionalProperties: false description: 'Configuration dict. For ''llm_judge'': {prompt}. For ''hog'': {source}.' output_type: allOf: - $ref: '#/components/schemas/OutputTypeEnum' description: 'Output format. Currently only ''boolean'' is supported. * `boolean` - Boolean (Pass/Fail)' output_config: type: object properties: allows_na: type: boolean description: Whether the evaluation can return N/A for non-applicable generations. default: false additionalProperties: false description: 'Output config. For ''boolean'' output_type: {allows_na} to permit N/A results.' conditions: description: Optional trigger conditions to filter which events are evaluated. OR between condition sets, AND within each. model_configuration: allOf: - $ref: '#/components/schemas/ModelConfiguration' nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true deleted: type: boolean description: Set to true to soft-delete the evaluation. required: - created_at - created_by - evaluation_type - id - name - output_type - status - status_reason - updated_at LLMSkillCreate: type: object description: Create serializer — accepts bundled files as write-only input on POST. properties: id: type: string format: uuid readOnly: true name: type: string description: Unique skill name. Lowercase letters, numbers, and hyphens only. Max 64 characters. maxLength: 64 description: type: string description: What this skill does and when to use it. Max 4096 characters. maxLength: 4096 body: type: string description: The SKILL.md instruction content (markdown). license: type: string description: License name or reference to a bundled license file. maxLength: 255 compatibility: type: string description: Environment requirements (intended product, system packages, network access, etc.). maxLength: 500 allowed_tools: type: array items: type: string description: List of pre-approved tools the skill may use. metadata: type: object additionalProperties: true description: Arbitrary key-value metadata. files: type: array items: $ref: '#/components/schemas/LLMSkillFileInput' writeOnly: true description: Bundled files to include with the initial version (scripts, references, assets). outline: type: array items: $ref: '#/components/schemas/LLMSkillOutlineEntry' readOnly: true description: Flat list of markdown headings parsed from the skill body. Useful as a lightweight table of contents. version: type: integer readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true deleted: type: boolean readOnly: true is_latest: type: boolean readOnly: true latest_version: type: integer readOnly: true version_count: type: integer readOnly: true first_version_created_at: type: string readOnly: true required: - body - created_at - created_by - deleted - description - first_version_created_at - id - is_latest - latest_version - name - outline - updated_at - version - version_count PatchedEvaluation: type: object properties: id: type: string format: uuid readOnly: true name: type: string description: Name of the evaluation. maxLength: 400 description: type: string description: Optional description of what this evaluation checks. enabled: type: boolean description: Whether the evaluation runs automatically on new $ai_generation events. status: allOf: - $ref: '#/components/schemas/EvaluationStatusEnum' readOnly: true status_reason: readOnly: true nullable: true oneOf: - $ref: '#/components/schemas/StatusReasonEnum' - $ref: '#/components/schemas/NullEnum' evaluation_type: allOf: - $ref: '#/components/schemas/EvaluationTypeEnum' description: '''llm_judge'' uses an LLM to score outputs against a prompt; ''hog'' runs deterministic Hog code. * `llm_judge` - LLM as a judge * `hog` - Hog' evaluation_config: oneOf: - type: object title: LLM judge config required: - prompt properties: prompt: type: string description: Evaluation criteria for the LLM judge. Describe what makes a good vs bad response. minLength: 1 additionalProperties: false - type: object title: Hog config required: - source properties: source: type: string description: Hog source code. Must return true (pass), false (fail), or null for N/A. minLength: 1 additionalProperties: false description: 'Configuration dict. For ''llm_judge'': {prompt}. For ''hog'': {source}.' output_type: allOf: - $ref: '#/components/schemas/OutputTypeEnum' description: 'Output format. Currently only ''boolean'' is supported. * `boolean` - Boolean (Pass/Fail)' output_config: type: object properties: allows_na: type: boolean description: Whether the evaluation can return N/A for non-applicable generations. default: false additionalProperties: false description: 'Output config. For ''boolean'' output_type: {allows_na} to permit N/A results.' conditions: description: Optional trigger conditions to filter which events are evaluated. OR between condition sets, AND within each. model_configuration: allOf: - $ref: '#/components/schemas/ModelConfiguration' nullable: true created_at: type: string format: date-time readOnly: true updated_at: type: string format: date-time readOnly: true created_by: allOf: - $ref: '#/components/schemas/UserBasic' readOnly: true deleted: type: boolean description: Set to true to soft-delete the evaluation. PatchedReviewQueueItemUpdate: type: object properties: queue_id: type: string format: uuid description: Review queue ID that should own this pending trace. TraceReviewScoreWrite: type: object properties: definition_id: type: string format: uuid description: Stable scorer definition ID. definition_version_id: type: string format: uuid nullable: true description: Optional immutable scorer version ID. Defaults to the scorer's current version. categorical_values: type: array items: type: string maxLength: 128 nullable: true description: Categorical option keys selected for this score. minItems: 1 numeric_value: type: string format: decimal pattern: ^-?\d{0,6}(?:\.\d{0,6})?$ nullable: true description: Numeric value selected for this score. boolean_value: type: boolean nullable: true description: Boolean value selected for this score. required: - definition_id parameters: ProjectIdPath: in: path name: project_id required: true schema: type: string description: Project ID of the project you're trying to access. To find the ID of the project, make a call to /api/projects/. EnvironmentIdPath: in: path name: environment_id required: true schema: type: string description: Deprecated. Use /api/projects/{project_id}/ instead. securitySchemes: PersonalAPIKeyAuth: type: http scheme: bearer x-tagGroups: - name: All endpoints tags: - LLM Analytics - actions - activity_log - activity_logs - advanced_activity_logs - alerts - annotations - approval_policies - batch_exports - cdp - change_requests - code - code-invites - cohorts - comments - conversations - core - customer_analytics - customer_journeys - customer_profile_configs - dashboard_templates - dashboards - data_color_themes - data_modeling_jobs - data_warehouse - dataset_items - datasets - desktop_recordings - domains - early_access_feature - early_access_features - elements - endpoints - environments - error_tracking - evaluation_runs - evaluations - event_definitions - event_filter - event_schemas - events - experiment_holdouts - experiment_saved_metrics - experiments - exports - external_data_schemas - external_data_sources - feature_flags - file_system - file_system_shortcut - flag_value - groups - groups_types - health_issues - heatmap_screenshots - heatmaps - hog_flows - hog_function_templates - hog_functions - insight_variables - insights - integrations - invites - js-snippet - legal_documents - lineage - live_debugger_breakpoints - llm_analytics - llm_prompts - llm_skills - logs - managed_viewsets - max - max_tools - mcp_server_installations - mcp_servers - mcp_store - mcp_tools - members - notebooks - oauth_applications - object_media_previews - organizations - persisted_folder - persons - platform_features - plugin_configs - product_analytics - product_tours - project_secret_api_keys - projects - property_definitions - proxy_records - public_hog_function_templates - query - replay - reverse_proxy - role_external_references - roles - sandbox-environments - sandbox_environments - saved - schema_property_groups - sdk_doctor - session_group_summaries - session_recording_playlists - session_recordings - session_summaries - sessions - signals - subscriptions - surveys - taggers - task-automations - task-runs - task_automations - tasks - uploaded_media - user_home_settings - user_interviews - users - visual_review - warehouse_dag - warehouse_model_paths - warehouse_saved_queries - warehouse_saved_query_folders - warehouse_tables - warehouse_view_link - warehouse_view_links - web_analytics - web_experiments - web_vitals - welcome - workflows