openapi: 3.1.0 info: title: Galileo API Server annotation scorer-feedback API version: 1.1085.0 servers: - url: https://api.galileo.ai description: Galileo API Server - galileo-v2 tags: - name: scorer-feedback paths: /scorer-feedbacks: post: tags: - scorer-feedback summary: Create Scorer Feedback description: 'Creates a new feedback item in the global queue for the specified scorer. **Validation:** - Validates both original_value and annotated_value against scorer''s output_type - Validates scorer_version_id exists and user has access - Validates user has access to the specified project - Auto-creates new queue if no active queue exists for this scorer - If active queue exists, adds feedback to that queue **Errors:** - 422 - Invalid annotated_value/original_value format or validation failure - 404 - Project, run, scorer, or scorer_version not found - 409 - Queue is currently locked (generating, reviewing, or completed)' operationId: create_scorer_feedback_scorer_feedbacks_post requestBody: content: application/json: schema: $ref: '#/components/schemas/CreateScorerFeedbackRequest' required: true responses: '201': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ScorerFeedbackResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] delete: tags: - scorer-feedback summary: Delete Scorer Feedback description: 'Deletes multiple feedback items by their IDs. **Validation:** - All feedback items must exist and user must have access - Feedback items must belong to queues that are not locked (not generating, reviewing, or completed) **Errors:** - 404 - One or more feedback items not found - 409 - One or more feedback items belong to a locked queue' operationId: delete_scorer_feedback_scorer_feedbacks_delete requestBody: content: application/json: schema: $ref: '#/components/schemas/DeleteScorerFeedbackRequest' required: true responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/DeleteScorerFeedbackResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] /scorer-feedbacks/{feedback_id}: patch: tags: - scorer-feedback summary: Update Scorer Feedback description: 'Updates an existing feedback item in the queue. Useful for refining values or rationale while reviewing feedback before applying. **Validation:** - Queue must be in pending state (not locked or completed) - annotated_value (if provided) must be valid for the scorer''s output_type - At least one field must be provided **Errors:** - 422 - Invalid annotated_value for scorer output_type, or no fields provided - 404 - Feedback item not found - 409 - Queue is currently processing or finished (locked, cannot modify)' operationId: update_scorer_feedback_scorer_feedbacks__feedback_id__patch security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: feedback_id in: path required: true schema: type: string format: uuid title: Feedback Id requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/UpdateScorerFeedbackRequest' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ScorerFeedbackResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /scorer-feedbacks/scorer/{scorer_id}: get: tags: - scorer-feedback summary: Get Scorer Feedback description: 'Gets all feedback items for the active queue of a scorer. Returns enriched feedback items including: - Basic feedback data (original/annotated values, rationale) - Project and run names - ClickHouse record data (input/output) - Queue metadata (status, id) If no active queue exists for the scorer, returns empty list with null queue_id/status. **Validation:** - User has access to the scorer **Errors:** - 404 - Scorer not found or user doesn''t have access' operationId: get_scorer_feedback_scorer_feedbacks_scorer__scorer_id__get security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: scorer_id in: path required: true schema: type: string format: uuid4 title: Scorer Id responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ScorerFeedbackListResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /scorer-feedbacks/scorer_version/{scorer_version_id}/applied: get: tags: - scorer-feedback summary: Get Applied Scorer Version Feedback description: 'Gets all feedback items that were applied to create a specific scorer version. Returns the historical feedback from the completed queue that produced this version, with enriched data including: - Basic feedback data (original/annotated values, rationale) - Project and run names - ClickHouse record data (input/output) - Queue metadata (status=completed, queue_id) **Validation:** - User has access to the scorer version **Errors:** - 404 - Scorer version not found or no feedback was applied to create this version' operationId: get_applied_scorer_version_feedback_scorer_feedbacks_scorer_version__scorer_version_id__applied_get security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: scorer_version_id in: path required: true schema: type: string format: uuid4 title: Scorer Version Id responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ScorerFeedbackListResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /scorer-feedbacks/feedback-counts: post: tags: - scorer-feedback summary: Get Feedback Counts description: 'Get the count of feedback items in each scorer''s active feedback queue. Returns a map of scorer_id to feedback count. Scorers without an active queue (or with no feedback items) return 0. **Errors:** - 422 - Empty scorer_ids list or invalid UUIDs' operationId: get_feedback_counts_scorer_feedbacks_feedback_counts_post requestBody: content: application/json: schema: $ref: '#/components/schemas/FeedbackCountsRequest' required: true responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/FeedbackCountsResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] /feedback-queues/scorer/{scorer_id}/generate-prompt: post: tags: - scorer-feedback summary: Generate Prompt description: 'Generate improved scorer prompt using Autotune (CLHF). This endpoint: 1. Fetches all feedback items from the active queue with ClickHouse context 2. Assembles the current metric system prompt 3. Constructs AutotuneInput with annotated examples 4. Queues a background job to the runners service 5. Returns task_id for polling The API prepares all data internally - no request body needed. **Validation:** - Scorer exists and user has access - Active queue exists in ''pending'' state - Queue has at least one feedback item **Response:** - 202 Accepted with task_id for polling **Errors:** - 404 - Scorer not found or no active pending queue exists - 409 - Queue is already locked (generating or reviewing) - 422 - Scorer missing required configuration or queue has no feedback items' operationId: generate_prompt_feedback_queues_scorer__scorer_id__generate_prompt_post security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: scorer_id in: path required: true schema: type: string format: uuid4 title: Scorer Id responses: '202': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/GeneratePromptResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /feedback-queues/scorer/{scorer_id}/status: get: tags: - scorer-feedback summary: Get Queue Status description: 'Lightweight endpoint for polling queue generation status. Returns only queue metadata (status, error message, draft prompt) without fetching feedback items or ClickHouse data. Designed for efficient polling after triggering prompt generation. **Errors:** - 404 - Scorer not found or no active queue exists' operationId: get_queue_status_feedback_queues_scorer__scorer_id__status_get security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: scorer_id in: path required: true schema: type: string format: uuid4 title: Scorer Id responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ScorerFeedbackQueueStatusResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /feedback-queues/{queue_id}: patch: tags: - scorer-feedback summary: Update Queue description: "Update a feedback queue by performing an action.\n\n**Supported actions:**\n- \"abort\": Abort the feedback application process and return the queue to pending state.\n Used when user clicks \"Back\" or \"Cancel\" during generation or review.\n Clears draft_prompt and generation_task_id. Same queue is reused.\n Can be called during 'generating' or 'reviewing' state.\n\n- \"complete\": Complete the queue after saving a new scorer version.\n Used when user saves a new version from the review flow.\n Sets status to completed (immutable) and links queue to the resulting version.\n Requires target_scorer_version_id, result_scorer_id, and result_scorer_version_id.\n Can only be called during 'reviewing' state.\n\n**Request body:**\n- action: The action to perform (\"abort\" or \"complete\")\n- target_scorer_version_id: Required for \"complete\" action - ID of the version feedback was applied to\n- result_scorer_id: Required for \"complete\" action - ID of the scorer that owns the result version.\n For custom LLM scorers this is the same as the queue's scorer. For preset scorers this is the\n new custom scorer duplicated from the preset.\n- result_scorer_version_id: Required for \"complete\" action - ID of the scorer version created\n\n**Response:**\n- 200 OK with queue details showing updated status\n\n**Errors:**\n- 404 - Queue not found or user doesn't have access\n- 409 - Queue is not in a valid state for the requested action\n- 422 - Invalid action or missing required parameters" operationId: update_queue_feedback_queues__queue_id__patch security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: queue_id in: path required: true schema: type: string format: uuid title: Queue Id requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/UpdateQueueRequest' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/UpdateQueueResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /feedback-queues/scorer/{scorer_id}/validation-runs: post: tags: - scorer-feedback summary: Validate Autotune Queue description: 'Test a draft prompt against feedback records in the active queue. Creates a metrics testing run by copying the feedback records and running generated scorer validation with the provided prompt. Used in the AutoTune Data tab for 3-column comparison: Original | Expected (Annotated) | New Result. Works in any active queue state (pending, generating, or reviewing). **Response:** - 202 Accepted with metrics_testing_run_id for viewing results **Errors:** - 404 - Scorer not found or no active queue exists - 422 - Queue has no feedback items or scorer has no version - 500 - Record processing failure' operationId: validate_autotune_queue_feedback_queues_scorer__scorer_id__validation_runs_post security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: scorer_id in: path required: true schema: type: string format: uuid4 title: Scorer Id requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ValidateAutotuneQueueRequest' responses: '202': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/ValidateAutotuneQueueResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /feedback-queues/{queue_id}/validation-results: get: tags: - scorer-feedback summary: Get Validation Results description: "Poll autotune validation results for a feedback queue.\n\nReturns one row per feedback item with the new scorer output joined in from\nthe metrics testing run. Used by the UI for the 3-column comparison table:\nOriginal Value | Expected (Annotated) | New Result.\n\n**Pollable:** Returns status=pending with null scores while validation is running.\nTransitions through pending → in_progress → completed as scores arrive.\n\n**Args:**\n- metrics_testing_run_id: Optional metrics testing run ID (from validate response).\n When provided, fetches scored records and joins with feedback items.\n\n**Errors:**\n- 404 - Queue not found or user doesn't have access" operationId: get_validation_results_feedback_queues__queue_id__validation_results_get security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: queue_id in: path required: true schema: type: string format: uuid title: Queue Id - name: metrics_testing_run_id in: query required: false schema: anyOf: - type: string format: uuid4 - type: 'null' description: Metrics testing run ID for fetching validation scores title: Metrics Testing Run Id description: Metrics testing run ID for fetching validation scores responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/AutotuneValidationResultsResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /feedback-queues/{queue_id}/validation-columns: get: tags: - scorer-feedback summary: Get Validation Columns description: 'Get column definitions for the autotune validation results table. Returns the column metadata needed to render the validation results table, including input/output columns, feedback columns, and scorer metric columns. **Errors:** - 404 - Queue not found or user doesn''t have access' operationId: get_validation_columns_feedback_queues__queue_id__validation_columns_get security: - ClassicAPIKeyHeader: [] - APIKeyHeader: [] - OAuth2PasswordBearer: [] - HTTPBasic: [] parameters: - name: queue_id in: path required: true schema: type: string format: uuid title: Queue Id responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/AutotuneValidationColumnsResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' components: schemas: MetricThreshold: properties: inverted: type: boolean title: Inverted description: Whether the column should be inverted for thresholds, i.e. if True, lower is better. default: false buckets: items: anyOf: - type: integer - type: number type: array title: Buckets description: Threshold buckets for the column. If the column is a metric, these are the thresholds for the column. display_value_levels: items: type: string type: array title: Display Value Levels description: Ordered list of strings that raw values get transformed to for displaying. type: object title: MetricThreshold description: 'Threshold configuration for metrics. Defines how metric values are bucketed and displayed, including whether lower or higher values are considered better.' ScorerFeedbackQueueStatusResponse: properties: queue_id: type: string format: uuid4 title: Queue Id description: Queue ID queue_status: $ref: '#/components/schemas/ScorerFeedbackQueueStatus' description: Current queue status generation_error_message: anyOf: - type: string - type: 'null' title: Generation Error Message description: Error message from the last failed generation attempt draft_prompt: anyOf: - type: string - type: 'null' title: Draft Prompt description: Draft prompt if generation completed type: object required: - queue_id - queue_status title: ScorerFeedbackQueueStatusResponse description: Lightweight response for polling queue generation status. FeedbackCountsResponse: properties: feedback_counts: additionalProperties: type: integer propertyNames: format: uuid4 type: object title: Feedback Counts description: Map of scorer_id to count of feedback items in the scorer's active queue. Returns 0 for scorers with no active queue. type: object required: - feedback_counts title: FeedbackCountsResponse description: Response schema mapping scorer IDs to their active queue feedback counts. FeedbackCountsRequest: properties: scorer_ids: items: type: string format: uuid4 type: array minItems: 1 title: Scorer Ids description: List of scorer IDs to query feedback counts for type: object required: - scorer_ids title: FeedbackCountsRequest description: Request schema for querying feedback counts by scorer IDs. AutotuneValidationResultsResponse: properties: queue_id: type: string format: uuid4 title: Queue Id description: Queue ID metrics_testing_run_id: anyOf: - type: string format: uuid4 - type: 'null' title: Metrics Testing Run Id description: Metrics testing run ID, null if validation not started status: $ref: '#/components/schemas/AutotuneValidationStatus' description: Overall validation scoring status total_count: type: integer title: Total Count description: Total number of feedback items scored_count: type: integer title: Scored Count description: Number of feedback items that have been scored results: items: $ref: '#/components/schemas/AutotuneValidationResultItem' type: array title: Results description: One row per feedback item type: object required: - queue_id - status - total_count - scored_count title: AutotuneValidationResultsResponse description: Response for the autotune validation results endpoint (pollable). GeneratePromptResponse: properties: task_id: type: string title: Task Id description: Celery task ID for polling queue_id: type: string format: uuid4 title: Queue Id description: Queue ID being processed scorer_id: type: string format: uuid4 title: Scorer Id description: Scorer ID feedback_count: type: integer title: Feedback Count description: Number of feedback items status: type: string title: Status description: Current status (generating) message: type: string title: Message description: Message with polling instructions type: object required: - task_id - queue_id - scorer_id - feedback_count - status - message title: GeneratePromptResponse description: Response schema for generate prompt request (202 Accepted). InputTypeEnum: type: string enum: - basic - llm_spans - retriever_spans - sessions_normalized - sessions_trace_io_only - tool_spans - trace_input_only - trace_io_only - trace_normalized - trace_output_only - agent_spans - workflow_spans title: InputTypeEnum description: Enumeration of input types. BaseGeneratedScorerDB: properties: id: type: string format: uuid4 title: Id name: type: string title: Name instructions: anyOf: - type: string - type: 'null' title: Instructions chain_poll_template: $ref: '#/components/schemas/ChainPollTemplate' user_prompt: anyOf: - type: string - type: 'null' title: User Prompt type: object required: - id - name - chain_poll_template title: BaseGeneratedScorerDB BaseScorerVersionDB: properties: id: type: string format: uuid4 title: Id version: type: integer title: Version scorer_id: type: string format: uuid4 title: Scorer Id generated_scorer: anyOf: - $ref: '#/components/schemas/BaseGeneratedScorerDB' - type: 'null' registered_scorer: anyOf: - $ref: '#/components/schemas/BaseRegisteredScorerDB' - type: 'null' finetuned_scorer: anyOf: - $ref: '#/components/schemas/BaseFinetunedScorerDB' - type: 'null' model_name: anyOf: - type: string - type: 'null' title: Model Name num_judges: anyOf: - type: integer - type: 'null' title: Num Judges scoreable_node_types: anyOf: - items: type: string type: array - type: 'null' title: Scoreable Node Types description: List of node types that can be scored by this scorer. Defaults to llm/chat. cot_enabled: anyOf: - type: boolean - type: 'null' title: Cot Enabled description: Whether to enable chain of thought for this scorer. Defaults to False for llm scorers. output_type: anyOf: - $ref: '#/components/schemas/OutputTypeEnum' - type: 'null' description: What type of output to use for model-based scorers (sessions_normalized, trace_io_only, etc.). input_type: anyOf: - $ref: '#/components/schemas/InputTypeEnum' - type: 'null' description: What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc.). type: object required: - id - version - scorer_id title: BaseScorerVersionDB description: Scorer version from the scorer_versions table. OutputTypeEnum: type: string enum: - boolean - categorical - count - discrete - freeform - percentage - multilabel - retrieved_chunk_list_boolean - boolean_multilabel title: OutputTypeEnum description: Enumeration of output types. LunaInputTypeEnum: type: string enum: - span - trace_object - trace_input_output_only title: LunaInputTypeEnum ScorerFeedbackListResponse: properties: queue_id: anyOf: - type: string format: uuid4 - type: 'null' title: Queue Id description: ID of the active queue, null if no active queue queue_status: anyOf: - $ref: '#/components/schemas/ScorerFeedbackQueueStatus' - type: 'null' description: Status of the active queue, null if no active queue generation_task_id: anyOf: - type: string - type: 'null' title: Generation Task Id description: Task ID for polling prompt generation, present when status is 'generating' generation_error_message: anyOf: - type: string - type: 'null' title: Generation Error Message description: Error message from the last failed generation attempt total_count: type: integer title: Total Count description: Total number of feedback items in the active queue feedback_items: items: $ref: '#/components/schemas/EnrichedScorerFeedbackItem' type: array title: Feedback Items description: List of feedback items with enriched data restricted_feedback: type: boolean title: Restricted Feedback description: True when the active queue contains one or more feedback items the caller cannot view (hidden because they originate from a project the caller has no view access to) default: false type: object required: - total_count title: ScorerFeedbackListResponse description: Response schema for active queue feedback list. RollUpMethodDisplayOptions: type: string enum: - average - sum - max - min - category_count - percentage_true - percentage_false title: RollUpMethodDisplayOptions description: 'Display options for roll up methods when showing rolled up metrics in the UI. Separates display intent from computation methods. The computation methods (NumericRollUpMethod, CategoricalRollUpMethod) control what aggregations are available. This enum controls how the UI displays the selected roll-up value for a scorer.' HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError DataType: type: string enum: - uuid - text - integer - floating_point - boolean - timestamp - string_list - tag - dataset - prompt - playground - rank - category_count - score_rating_aggregate - star_rating_aggregate - thumb_rating_aggregate - tags_rating_aggregate - text_rating_aggregate - annotation_agreement - fully_annotated title: DataType QueueAction: type: string enum: - abort - complete title: QueueAction description: Actions that can be performed on a feedback queue. UpdateQueueRequest: properties: action: $ref: '#/components/schemas/QueueAction' description: Action to perform on the queue target_scorer_version_id: anyOf: - type: string format: uuid4 - type: 'null' title: Target Scorer Version Id description: 'Required for ''complete'' action: ID of the scorer version that feedback was applied to' result_scorer_id: anyOf: - type: string format: uuid4 - type: 'null' title: Result Scorer Id description: 'Required for ''complete'' action: ID of the scorer that owns the result version. For custom LLM scorers this is the same scorer. For preset scorers this is the new custom scorer.' result_scorer_version_id: anyOf: - type: string format: uuid4 - type: 'null' title: Result Scorer Version Id description: 'Required for ''complete'' action: ID of the scorer version created from this queue''s feedback' type: object required: - action title: UpdateQueueRequest description: 'Request schema for updating a feedback queue state via PATCH. Used to perform actions on a queue, such as aborting generation/review or completing the queue.' StepType: type: string enum: - llm - retriever - tool - workflow - agent - control - trace - session title: StepType NodeNameFilter: properties: name: type: string const: node_name title: Name default: node_name operator: type: string enum: - eq - ne - contains - one_of - not_in title: Operator value: anyOf: - type: string - items: type: string type: array title: Value case_sensitive: type: boolean title: Case Sensitive default: true type: object required: - operator - value title: NodeNameFilter description: Filters on node names in scorer jobs. DeleteScorerFeedbackResponse: properties: deleted_count: type: integer title: Deleted Count description: Number of feedback items successfully deleted deleted_ids: items: type: string format: uuid4 type: array title: Deleted Ids description: List of feedback IDs that were deleted type: object required: - deleted_count - deleted_ids title: DeleteScorerFeedbackResponse description: Response schema for bulk delete operation. FewShotExample: properties: generation_prompt_and_response: type: string title: Generation Prompt And Response evaluating_response: type: string title: Evaluating Response type: object required: - generation_prompt_and_response - evaluating_response title: FewShotExample description: Few-shot example for a chainpoll metric prompt. ModelType: type: string enum: - slm - llm - code title: ModelType MultimodalCapability: type: string enum: - vision - audio title: MultimodalCapability LunaOutputTypeEnum: type: string enum: - float - string - string_list - bool_list title: LunaOutputTypeEnum ColumnCategory: type: string enum: - standard - metric - user_metadata - metric_status - dataset_metadata - dataset - feedback - tags title: ColumnCategory ScorerFeedbackResponse: properties: id: type: string format: uuid4 title: Id project_id: type: string format: uuid4 title: Project Id run_id: type: string format: uuid4 title: Run Id scorer_id: type: string format: uuid4 title: Scorer Id source_scorer_version_id: type: string format: uuid4 title: Source Scorer Version Id record_id: type: string format: uuid4 title: Record Id original_value: type: string title: Original Value annotated_value: type: string title: Annotated Value feedback_text: anyOf: - type: string - type: 'null' title: Feedback Text rationale: anyOf: - type: string - type: 'null' title: Rationale queue_id: type: string format: uuid4 title: Queue Id created_by: anyOf: - type: string format: uuid4 - type: 'null' title: Created By created_by_name: anyOf: - type: string - type: 'null' title: Created By Name created_at: type: string format: date-time title: Created At updated_at: type: string format: date-time title: Updated At type: object required: - id - project_id - run_id - scorer_id - source_scorer_version_id - record_id - original_value - annotated_value - feedback_text - rationale - queue_id - created_by - created_at - updated_at title: ScorerFeedbackResponse description: Response schema for scorer feedback. LogRecordsColumnInfo: properties: id: type: string title: Id description: Column id. Must be universally unique. label: anyOf: - type: string - type: 'null' title: Label description: Display label of the column in the UI. category: $ref: '#/components/schemas/ColumnCategory' description: Category of the column. description: anyOf: - type: string - type: 'null' title: Description description: Description of the column. group_label: anyOf: - type: string - type: 'null' title: Group Label description: Display label of the column group. data_type: anyOf: - $ref: '#/components/schemas/DataType' - type: 'null' description: Data type of the column. This is used to determine how to format the data on the UI. data_unit: anyOf: - $ref: '#/components/schemas/DataUnit' - type: 'null' description: Data unit of the column (optional). multi_valued: type: boolean title: Multi Valued description: Whether the column is multi-valued. default: false allowed_values: anyOf: - items: {} type: array uniqueItems: true - type: 'null' title: Allowed Values description: Allowed values for this column. sortable: type: boolean title: Sortable description: Whether the column is sortable. filterable: type: boolean title: Filterable description: Whether the column is filterable. is_empty: type: boolean title: Is Empty description: Indicates whether the column is empty and should be hidden. default: false applicable_types: items: $ref: '#/components/schemas/StepType' type: array uniqueItems: true title: Applicable Types description: List of types applicable for this column. is_optional: type: boolean title: Is Optional description: Whether the column is optional. default: false roll_up_method: anyOf: - type: string - type: 'null' title: Roll Up Method description: Default roll-up aggregation method for this metric (e.g., 'sum', 'average'). metric_key_alias: anyOf: - type: string - type: 'null' title: Metric Key Alias description: Alternate metric key for this column. When scorer UUIDs are used as column IDs, this holds the legacy metric_name string for dual-key ClickHouse query fallback. scorer_config: anyOf: - $ref: '#/components/schemas/ScorerConfig' - type: 'null' description: 'For metric columns only: Scorer config that produced the metric.' scorer_id: anyOf: - type: string format: uuid4 - type: 'null' title: Scorer Id description: 'For metric columns only: Scorer id that produced the metric. This is deprecated and will be removed in future versions.' insight_type: anyOf: - $ref: '#/components/schemas/InsightType' - type: 'null' description: Insight type. filter_type: anyOf: - $ref: '#/components/schemas/LogRecordsFilterType' - type: 'null' description: Filter type. threshold: anyOf: - $ref: '#/components/schemas/MetricThreshold' - type: 'null' description: Thresholds for the column, if this is a metrics column. label_color: anyOf: - type: string enum: - positive - negative - type: 'null' title: Label Color description: Type of label color for the column, if this is a multilabel metric column. type: object required: - id - category - data_type title: LogRecordsColumnInfo galileo_core__schemas__shared__scorers__scorer_name__ScorerName: type: string enum: - action_completion_luna - action_advancement_luna - agentic_session_success - agentic_session_success - action_completion_vision - action_completion_audio - agentic_workflow_success - agentic_workflow_success - agent_efficiency - agent_flow - chunk_attribution_utilization_luna - chunk_attribution_utilization - chunk_relevance - chunk_relevance_luna - context_precision - precision_at_k - completeness_luna - completeness - context_adherence - context_adherence_luna - context_adherence_vision - context_adherence_audio - context_relevance - context_relevance_luna - conversation_quality - correctness - correctness_vision - correctness_audio - ground_truth_adherence - ground_truth_adherence_vision - ground_truth_adherence_audio - visual_fidelity - visual_quality - input_pii - input_pii_gpt - input_sexist - input_sexist - input_sexist_luna - input_sexist_luna - input_tone - input_tone_gpt - input_toxicity - input_toxicity_luna - input_toxicity_vision - input_toxicity_audio - instruction_adherence - output_pii - output_pii_gpt - output_sexist - output_sexist - output_sexist_luna - output_sexist_luna - output_tone - output_tone_gpt - output_toxicity - output_toxicity_luna - output_toxicity_vision - output_toxicity_audio - prompt_injection - prompt_injection_luna - reasoning_coherence - reasoning_coherence_vision - reasoning_coherence_audio - sql_efficiency - sql_adherence - sql_injection - sql_correctness - tool_error_rate - tool_error_rate_luna - tool_selection_quality - tool_selection_quality_luna - user_intent_change - user_intent_change_vision - user_intent_change_audio - interruption_detection title: ScorerName BaseFinetunedScorerDB: properties: id: type: string format: uuid4 title: Id name: type: string title: Name lora_task_id: type: integer title: Lora Task Id lora_weights_path: anyOf: - type: string - type: 'null' title: Lora Weights Path prompt: type: string title: Prompt luna_input_type: anyOf: - $ref: '#/components/schemas/LunaInputTypeEnum' - type: 'null' luna_output_type: anyOf: - $ref: '#/components/schemas/LunaOutputTypeEnum' - type: 'null' class_name_to_vocab_ix: anyOf: - additionalProperties: items: type: integer type: array uniqueItems: true type: object - additionalProperties: type: integer type: object - type: 'null' title: Class Name To Vocab Ix executor: anyOf: - $ref: '#/components/schemas/galileo_core__schemas__shared__scorers__scorer_name__ScorerName' - type: 'null' description: Executor pipeline. Defaults to finetuned scorer pipeline but can run custom galileo score pipelines. type: object required: - id - name - lora_task_id - prompt title: BaseFinetunedScorerDB ScorerFeedbackQueueStatus: type: string enum: - pending - generating - reviewing - completed title: ScorerFeedbackQueueStatus description: Status of a scorer feedback queue throughout its lifecycle. AutotuneValidationStatus: type: string enum: - pending - in_progress - completed title: AutotuneValidationStatus description: Status of autotune validation scoring. LogRecordsFilterType: type: string enum: - id - date - number - boolean - text - collection - fully_annotated title: LogRecordsFilterType UpdateScorerFeedbackRequest: properties: annotated_value: anyOf: - type: string minLength: 1 - type: 'null' title: Annotated Value description: Updated correct value feedback_text: anyOf: - type: string - type: 'null' title: Feedback Text description: Updated explanation (empty string or null clears) type: object title: UpdateScorerFeedbackRequest description: Request schema for updating scorer feedback. UpdateQueueResponse: properties: queue_id: type: string format: uuid4 title: Queue Id description: Queue ID that was updated scorer_id: type: string format: uuid4 title: Scorer Id description: Scorer ID target_scorer_version_id: type: string format: uuid4 title: Target Scorer Version Id description: Target scorer version for the queue result_scorer_id: anyOf: - type: string format: uuid4 - type: 'null' title: Result Scorer Id description: Scorer that owns the result version (set on complete) result_scorer_version_id: anyOf: - type: string format: uuid4 - type: 'null' title: Result Scorer Version Id description: Scorer version created from this queue (set on complete) status: type: string title: Status description: Current queue status after update message: type: string title: Message description: Confirmation message type: object required: - queue_id - scorer_id - target_scorer_version_id - status - message title: UpdateQueueResponse description: 'Response schema for queue update request (200 OK). For abort action: Clears draft_prompt and generation_task_id, returns to pending state. For complete action: Sets status to completed and links queue to the resulting scorer version.' ValidateAutotuneQueueResponse: properties: metrics_testing_run_id: type: string format: uuid4 title: Metrics Testing Run Id description: ID of the metrics testing run containing validation results project_id: type: string format: uuid4 title: Project Id description: Project where the metrics testing run was created queue_id: type: string format: uuid4 title: Queue Id description: Queue ID that was tested feedback_count: type: integer title: Feedback Count description: Number of feedback records tested type: object required: - metrics_testing_run_id - project_id - queue_id - feedback_count title: ValidateAutotuneQueueResponse description: Response for autotune queue validation (202 Accepted). InsightType: type: string enum: - vertical_bar - horizontal_bar title: InsightType ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError DataUnit: type: string enum: - percentage - nano_seconds - milli_seconds - dollars - count_and_total title: DataUnit ChainPollTemplate: properties: metric_system_prompt: anyOf: - type: string - type: 'null' title: Metric System Prompt description: System prompt for the metric. metric_description: anyOf: - type: string - type: 'null' title: Metric Description description: Description of what the metric should do. value_field_name: type: string title: Value Field Name description: Field name to look for in the chainpoll response, for the rating. default: rating explanation_field_name: type: string title: Explanation Field Name description: Field name to look for in the chainpoll response, for the explanation. default: explanation template: type: string title: Template description: Chainpoll prompt template. metric_few_shot_examples: items: $ref: '#/components/schemas/FewShotExample' type: array title: Metric Few Shot Examples description: Few-shot examples for the metric. response_schema: anyOf: - additionalProperties: true type: object - type: 'null' title: Response Schema description: Response schema for the output type: object required: - template title: ChainPollTemplate description: 'Template for a chainpoll metric prompt, containing all the info necessary to send a chainpoll prompt.' MetadataFilter: properties: name: type: string const: metadata title: Name default: metadata operator: type: string enum: - one_of - not_in - eq - ne title: Operator key: type: string title: Key value: anyOf: - type: string - items: type: string type: array title: Value type: object required: - operator - key - value title: MetadataFilter description: Filters on metadata key-value pairs in scorer jobs. BaseRegisteredScorerDB: properties: id: type: string format: uuid4 title: Id name: type: string title: Name score_type: anyOf: - type: string - type: 'null' title: Score Type type: object required: - id - name title: BaseRegisteredScorerDB ModalityFilter: properties: name: type: string const: modality title: Name default: modality operator: type: string enum: - eq - ne - one_of - not_in title: Operator value: anyOf: - type: string description: Single enum value - specific options depend on the concrete enum type used example: ENUM_VALUE - items: type: string example: ENUM_VALUE type: array description: Array of enum values example: - ENUM_VALUE_1 - ENUM_VALUE_2 title: Value type: object required: - operator - value title: ModalityFilter description: 'Filters on content modalities in scorer jobs. Matches if at least one of the specified modalities is present.' ScorerConfig: properties: model_name: anyOf: - type: string - type: 'null' title: Model Name num_judges: anyOf: - type: integer - type: 'null' title: Num Judges filters: anyOf: - items: oneOf: - $ref: '#/components/schemas/NodeNameFilter' - $ref: '#/components/schemas/MetadataFilter' - $ref: '#/components/schemas/ModalityFilter' discriminator: propertyName: name mapping: metadata: '#/components/schemas/MetadataFilter' modality: '#/components/schemas/ModalityFilter' node_name: '#/components/schemas/NodeNameFilter' type: array - type: 'null' title: Filters description: List of filters to apply to the scorer. scoreable_node_types: anyOf: - items: type: string type: array - type: 'null' title: Scoreable Node Types description: List of node types that can be scored by this scorer. Defaults to llm/chat. cot_enabled: anyOf: - type: boolean - type: 'null' title: Cot Enabled description: Whether to enable chain of thought for this scorer. Defaults to False for llm scorers. output_type: anyOf: - $ref: '#/components/schemas/OutputTypeEnum' - type: 'null' description: What type of output to use for model-based scorers (boolean, categorical, etc.). input_type: anyOf: - $ref: '#/components/schemas/InputTypeEnum' - type: 'null' description: What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc..). id: type: string format: uuid4 title: Id name: anyOf: - type: string - type: 'null' title: Name scorer_type: $ref: '#/components/schemas/ScorerTypes' model_type: anyOf: - $ref: '#/components/schemas/ModelType' - type: 'null' description: Type of model to use for this scorer. slm maps to luna, and llm maps to plus scorer_version: anyOf: - $ref: '#/components/schemas/BaseScorerVersionDB' - type: 'null' description: ScorerVersion to use for this scorer. If not provided, the latest version will be used. multimodal_capabilities: anyOf: - items: $ref: '#/components/schemas/MultimodalCapability' type: array - type: 'null' title: Multimodal Capabilities description: Multimodal capabilities which this scorer can utilize in its evaluation. roll_up_method: anyOf: - $ref: '#/components/schemas/RollUpMethodDisplayOptions' - type: 'null' score_type: anyOf: - type: string - type: 'null' title: Score Type description: Return type of code scorers (e.g., 'bool', 'int', 'float', 'str'). type: object required: - id - scorer_type title: ScorerConfig description: Used for configuring a scorer for a scorer job. AutotuneValidationResultItem: properties: feedback_id: type: string format: uuid4 title: Feedback Id description: Feedback item ID record_id: type: string format: uuid4 title: Record Id description: Original ClickHouse record ID input: anyOf: - type: string - type: 'null' title: Input description: Input text of the record output: anyOf: - type: string - type: 'null' title: Output description: Output text of the record original_value: type: string title: Original Value description: Original scorer output that was incorrect annotated_value: type: string title: Annotated Value description: User-corrected expected value feedback_text: anyOf: - type: string - type: 'null' title: Feedback Text description: User's explanation for the correction new_score: anyOf: - type: string - type: 'null' title: New Score description: New scorer output from the validation run, null if not yet scored new_score_rationale: anyOf: - type: string - type: 'null' title: New Score Rationale description: Rationale from the new scorer (if CoT enabled), null if not scored or CoT disabled project_name: anyOf: - type: string - type: 'null' title: Project Name description: Name of the source project run_name: anyOf: - type: string - type: 'null' title: Run Name description: Name of the source run type: object required: - feedback_id - record_id - original_value - annotated_value title: AutotuneValidationResultItem description: One row in the autotune validation results — one per feedback item. ValidateAutotuneQueueRequest: properties: user_prompt: type: string minLength: 1 title: User Prompt description: The draft/edited prompt to test against feedback records type: object required: - user_prompt title: ValidateAutotuneQueueRequest description: Request to test a draft prompt against autotune feedback records. DeleteScorerFeedbackRequest: properties: feedback_ids: items: type: string format: uuid4 type: array minItems: 1 title: Feedback Ids description: List of feedback IDs to delete type: object required: - feedback_ids title: DeleteScorerFeedbackRequest description: Request schema for deleting scorer feedback items. AutotuneValidationColumnsResponse: properties: columns: items: $ref: '#/components/schemas/LogRecordsColumnInfo' type: array title: Columns description: Column definitions for the validation results table type: object title: AutotuneValidationColumnsResponse description: Response for the autotune validation columns endpoint. CreateScorerFeedbackRequest: properties: project_id: type: string format: uuid4 title: Project Id description: Project context for audit trail run_id: type: string format: uuid4 title: Run Id description: Run context scorer_id: type: string format: uuid4 title: Scorer Id description: Scorer being corrected scorer_version_id: type: string format: uuid4 title: Scorer Version Id description: Version that produced the incorrect output record_id: type: string format: uuid4 title: Record Id description: ID of the record that was scored original_value: type: string minLength: 1 title: Original Value description: The original incorrect scorer output annotated_value: type: string minLength: 1 title: Annotated Value description: Corrected value (validated against scorer output_type) feedback_text: anyOf: - type: string minLength: 1 - type: 'null' title: Feedback Text description: Rationale/explanation for the correction rationale: anyOf: - type: string - type: 'null' title: Rationale description: Original scorer response (value + rationale) for CLHF context type: object required: - project_id - run_id - scorer_id - scorer_version_id - record_id - original_value - annotated_value title: CreateScorerFeedbackRequest description: Request schema for creating scorer feedback. ScorerTypes: type: string enum: - llm - code - luna - preset title: ScorerTypes EnrichedScorerFeedbackItem: properties: id: type: string format: uuid4 title: Id project_id: type: string format: uuid4 title: Project Id run_id: type: string format: uuid4 title: Run Id scorer_id: type: string format: uuid4 title: Scorer Id source_scorer_version_id: type: string format: uuid4 title: Source Scorer Version Id record_id: type: string format: uuid4 title: Record Id original_value: type: string title: Original Value annotated_value: type: string title: Annotated Value feedback_text: anyOf: - type: string - type: 'null' title: Feedback Text rationale: anyOf: - type: string - type: 'null' title: Rationale queue_id: type: string format: uuid4 title: Queue Id created_by: anyOf: - type: string format: uuid4 - type: 'null' title: Created By created_by_name: anyOf: - type: string - type: 'null' title: Created By Name created_at: type: string format: date-time title: Created At updated_at: type: string format: date-time title: Updated At project_name: type: string title: Project Name description: Name of the project run_name: type: string title: Run Name description: Name of the run input: anyOf: - type: string - type: 'null' title: Input description: Input data for the record that was scored output: anyOf: - type: string - type: 'null' title: Output description: Output data for the record that was scored can_submit: type: boolean title: Can Submit description: Whether the caller may submit this feedback item into an autotune run default: true type: object required: - id - project_id - run_id - scorer_id - source_scorer_version_id - record_id - original_value - annotated_value - feedback_text - rationale - queue_id - created_by - created_at - updated_at - project_name - run_name title: EnrichedScorerFeedbackItem description: Enriched feedback item with record data and entity names. securitySchemes: ClassicAPIKeyHeader: type: apiKey in: header name: Galileo-API-Key APIKeyHeader: type: apiKey in: header name: Splunk-AO-API-Key OAuth2PasswordBearer: type: oauth2 flows: password: scopes: {} tokenUrl: https://api.galileo.ai/login HTTPBasic: type: http scheme: basic