openapi: 3.0.1
info:
  title: langfuse AnnotationQueues UnstableEvaluators API
  version: ''
  description: '## Authentication


    Authenticate with the API using [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication), get API keys in the project settings:


    - username: Langfuse Public Key

    - password: Langfuse Secret Key


    ## Exports


    - OpenAPI spec: https://cloud.langfuse.com/generated/api/openapi.yml'
tags:
- name: UnstableEvaluators
paths:
  /api/public/unstable/evaluators:
    post:
      description: 'Create an evaluator in the authenticated project.


        Use evaluators to define **how** Langfuse should score data: the prompt, the expected structured output, and the optional model configuration.


        Naming behavior:

        - If this is a new evaluator name in your project, Langfuse creates version `1`.

        - If the name already exists in your project, Langfuse creates the next version and returns it.

        - When a new project version is created, existing evaluation rules in that project automatically move to the newest version for that evaluator name.


        Recommended workflow:

        1. Create the evaluator.

        2. Read the returned `variables` array.

        3. Read the returned `outputDefinition.dataType` so the client knows whether future scores will be numeric, boolean, or categorical.

        4. Create one or more evaluation rules that reference the returned evaluator family using `name` and `scope`.


        Recovery guidance:

        - `422` with `code=evaluator_preflight_failed`: the evaluator cannot run with the resolved model configuration. Add a valid explicit `modelConfig`, or configure the project''s default evaluation model, then retry the same request.

        - `400` with `code=invalid_body`: the request shape is malformed. Use the structured `details.issues` array to fix the specific fields and retry.

        - `400` with `code=invalid_body` on `outputDefinition`: send `dataType`, `reasoning.description`, and `score.description`. Do not send `version`; it is not part of the public request shape.


        Unstable API note:

        - This surface may evolve while the underlying evaluation data model is being redesigned.'
      operationId: unstable_evaluators_create
      tags:
      - UnstableEvaluators
      parameters: []
      responses:
        '200':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstableEvaluator'
              examples:
                CreateEvaluatorVersion:
                  value:
                    id: evaltmpl_123
                    name: answer-correctness
                    version: 2
                    scope: project
                    type: llm_as_judge
                    prompt: 'You are grading an answer.


                      Input:

                      {{input}}


                      Output:

                      {{output}}


                      Return a score between 0 and 1.

                      '
                    variables:
                    - input
                    - output
                    outputDefinition:
                      dataType: NUMERIC
                      reasoning:
                        description: Explain why the score was assigned.
                      score:
                        description: Correctness score between 0 and 1.
                    modelConfig:
                      provider: openai
                      model: gpt-4.1-mini
                    evaluationRuleCount: 0
                    createdAt: '2026-03-30T09:00:00.000Z'
                    updatedAt: '2026-03-30T09:00:00.000Z'
        '400':
          description: ''
          content:
            application/json:
              schema: {}
        '401':
          description: ''
          content:
            application/json:
              schema: {}
        '403':
          description: ''
          content:
            application/json:
              schema: {}
        '404':
          description: ''
          content:
            application/json:
              schema: {}
        '405':
          description: ''
          content:
            application/json:
              schema: {}
        '409':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
        '422':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
        '429':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
        '500':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
      security:
      - BasicAuth: []
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/unstableCreateEvaluatorRequest'
            examples:
              CreateEvaluatorVersion:
                value:
                  name: answer-correctness
                  prompt: 'You are grading an answer.


                    Input:

                    {{input}}


                    Output:

                    {{output}}


                    Return a score between 0 and 1.

                    '
                  outputDefinition:
                    dataType: NUMERIC
                    reasoning:
                      description: Explain why the score was assigned.
                    score:
                      description: Correctness score between 0 and 1.
                  modelConfig:
                    provider: openai
                    model: gpt-4.1-mini
    get:
      description: 'List the evaluators available to the authenticated project.


        Important behavior:

        - This endpoint returns the latest version of each available evaluator.

        - Results can include evaluators from your project and Langfuse-managed evaluators.

        - If the same evaluator name exists in both places, both are returned as separate items with different `scope` values.'
      operationId: unstable_evaluators_list
      tags:
      - UnstableEvaluators
      parameters:
      - name: page
        in: query
        description: 1-based page number. Defaults to `1`.
        required: false
        schema:
          type: integer
          nullable: true
      - name: limit
        in: query
        description: Maximum number of items per page. Defaults to `50`.
        required: false
        schema:
          type: integer
          nullable: true
      responses:
        '200':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstableEvaluators'
        '400':
          description: ''
          content:
            application/json:
              schema: {}
        '401':
          description: ''
          content:
            application/json:
              schema: {}
        '403':
          description: ''
          content:
            application/json:
              schema: {}
        '404':
          description: ''
          content:
            application/json:
              schema: {}
        '405':
          description: ''
          content:
            application/json:
              schema: {}
        '429':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
        '500':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
      security:
      - BasicAuth: []
  /api/public/unstable/evaluators/{evaluatorId}:
    get:
      description: 'Get one evaluator by `id`.


        Use this endpoint when you want the prompt, output definition, model configuration, and derived variables for the evaluator you plan to use in an evaluation rule.'
      operationId: unstable_evaluators_get
      tags:
      - UnstableEvaluators
      parameters:
      - name: evaluatorId
        in: path
        description: Evaluator identifier returned by the evaluator endpoints.
        required: true
        schema:
          type: string
      responses:
        '200':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstableEvaluator'
        '400':
          description: ''
          content:
            application/json:
              schema: {}
        '401':
          description: ''
          content:
            application/json:
              schema: {}
        '403':
          description: ''
          content:
            application/json:
              schema: {}
        '404':
          description: ''
          content:
            application/json:
              schema: {}
        '405':
          description: ''
          content:
            application/json:
              schema: {}
        '429':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
        '500':
          description: ''
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/unstablePublicApiError'
      security:
      - BasicAuth: []
components:
  schemas:
    utilsMetaResponse:
      title: utilsMetaResponse
      type: object
      properties:
        page:
          type: integer
          description: current page number
        limit:
          type: integer
          description: number of items per page
        totalItems:
          type: integer
          description: number of total items given the current filters/selection (if any)
        totalPages:
          type: integer
          description: number of total pages given the current limit
      required:
      - page
      - limit
      - totalItems
      - totalPages
    unstableEvaluatorModelConfig:
      title: unstableEvaluatorModelConfig
      type: object
      description: 'Optional explicit model configuration for an evaluator.


        If omitted, Langfuse uses the project''s default evaluation model.

        If provided, the model must be available to the project when the evaluator or evaluation rule is enabled.


        To discover valid configured `provider` values for a project, call `GET /api/public/llm-connections` and read the `provider` field from the returned connections.

        Use a `provider` value that matches one of the connections already configured in the same project.


        Recovery guidance:

        - If evaluator creation returns `422` with `code=evaluator_preflight_failed`, either provide a valid explicit `modelConfig` here or configure the project''s default evaluation model, then retry the same request.'
      properties:
        provider:
          type: string
          description: 'Provider identifier to use for this evaluator, for example `openai` or `anthropic`.


            To discover valid values for the current project, call `GET /api/public/llm-connections` and use one of the returned `provider` values.'
          example: openai
        model:
          type: string
          description: Model identifier exposed by the provider, for example `gpt-4.1-mini`.
          example: gpt-4.1-mini
      required:
      - provider
      - model
    unstablePublicCategoricalEvaluatorOutputDefinition:
      title: unstablePublicCategoricalEvaluatorOutputDefinition
      type: object
      properties:
        dataType:
          $ref: '#/components/schemas/unstableEvaluatorOutputDataType'
          description: Always `CATEGORICAL`.
        reasoning:
          $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
        score:
          $ref: '#/components/schemas/unstablePublicCategoricalEvaluatorOutputScoreDefinition'
      required:
      - dataType
      - reasoning
      - score
    unstablePublicCategoricalEvaluatorOutputScoreDefinition:
      title: unstablePublicCategoricalEvaluatorOutputScoreDefinition
      type: object
      properties:
        description:
          type: string
        categories:
          type: array
          items:
            type: string
        shouldAllowMultipleMatches:
          type: boolean
      required:
      - description
      - categories
      - shouldAllowMultipleMatches
    unstablePublicBooleanEvaluatorOutputDefinition:
      title: unstablePublicBooleanEvaluatorOutputDefinition
      type: object
      properties:
        dataType:
          $ref: '#/components/schemas/unstableEvaluatorOutputDataType'
          description: Always `BOOLEAN`.
        reasoning:
          $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
        score:
          $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
      required:
      - dataType
      - reasoning
      - score
    unstableEvaluatorOutputDataType:
      title: unstableEvaluatorOutputDataType
      type: string
      enum:
      - NUMERIC
      - BOOLEAN
      - CATEGORICAL
      description: 'Structured score type returned by an evaluator.


        This controls the type of score value Langfuse stores for evaluation results:

        - `NUMERIC`: a numeric score such as `0.82`

        - `BOOLEAN`: a boolean score such as `true`

        - `CATEGORICAL`: one or more category labels from a fixed list'
    unstableCreateEvaluatorRequest:
      title: unstableCreateEvaluatorRequest
      type: object
      description: 'Request body for creating an evaluator.


        If the same `name` already exists in your project, Langfuse creates the next version and returns it.

        Existing evaluation rules in the same project are then moved to that new latest version automatically.'
      properties:
        name:
          type: string
          description: Evaluator name within the authenticated project.
          example: answer-correctness
        prompt:
          type: string
          description: Prompt template used by the evaluator.
          example: 'You are grading an answer.


            Input:

            {{input}}


            Output:

            {{output}}


            Return a score between 0 and 1.

            '
        outputDefinition:
          $ref: '#/components/schemas/unstableEvaluatorOutputDefinition'
          description: 'Structured output schema the evaluator must return.


            Always send `dataType`.

            Do not send `version`; it is an internal storage detail and not part of the public request contract.'
        modelConfig:
          $ref: '#/components/schemas/unstableEvaluatorModelConfig'
          nullable: true
          description: Optional explicit model configuration. Omit or set to `null` to use the project default evaluation model.
      required:
      - name
      - prompt
      - outputDefinition
    unstableEvaluator:
      title: unstableEvaluator
      type: object
      description: 'One evaluator that can be used for scoring.


        An evaluator describes **how** to score data:

        - prompt

        - extracted prompt variables

        - output schema

        - optional explicit model configuration


        It does not define **which** live objects are evaluated. That is the job of `evaluation-rules`.


        For agent clients, the most important fields are:

        - `variables`: use these exact names when building the evaluation-rule `mapping` array

        - `outputDefinition`: tells you the expected score type and the evaluator''s response instructions

        - `modelConfig`: tells you whether the evaluator uses the project default model (`null`) or an explicit provider/model


        Versioning behavior:

        - `GET /evaluators` returns the latest version of each available evaluator.

        - `GET /evaluators/{id}` can return an older version.

        - Evaluation rules always run against the latest version for the selected evaluator name within the same source (`project` or `managed`).'
      properties:
        id:
          type: string
          description: Identifier of this evaluator.
          example: evaltmpl_123
        name:
          type: string
          description: Evaluator name.
          example: answer-correctness
        version:
          type: integer
          description: Version number of this evaluator.
          example: 2
        scope:
          $ref: '#/components/schemas/unstableEvaluatorScope'
          description: 'Where this evaluator comes from: your project or Langfuse-managed defaults.'
        type:
          $ref: '#/components/schemas/unstableEvaluatorType'
          description: Evaluator engine type. Currently always `llm_as_judge`.
        prompt:
          type: string
          description: Prompt template used during evaluation.
          example: 'You are grading an answer.


            Input:

            {{input}}


            Output:

            {{output}}


            Return a score between 0 and 1.

            '
        variables:
          type: array
          items:
            type: string
          description: 'Variables extracted from the evaluator prompt.


            Every variable in this list must be mapped exactly once when creating an evaluation rule.'
          example:
          - input
          - output
        outputDefinition:
          $ref: '#/components/schemas/unstablePublicEvaluatorOutputDefinition'
          description: 'Structured output schema returned by this evaluator.


            Responses always include `dataType` and omit the internal output-definition `version`.

            Use `dataType` to decide how future scores should be interpreted.'
        modelConfig:
          $ref: '#/components/schemas/unstableEvaluatorModelConfig'
          nullable: true
          description: Explicit model configuration, or `null` when the project default evaluation model is used.
        evaluationRuleCount:
          type: integer
          description: Number of evaluation rules in the project that currently use this evaluator version.
          example: 0
        createdAt:
          type: string
          format: date-time
          description: Timestamp when this evaluator was created.
          example: '2026-03-30T09:00:00.000Z'
        updatedAt:
          type: string
          format: date-time
          description: Timestamp when this evaluator was last updated.
          example: '2026-03-30T09:00:00.000Z'
      required:
      - id
      - name
      - version
      - scope
      - type
      - prompt
      - variables
      - outputDefinition
      - evaluationRuleCount
      - createdAt
      - updatedAt
    unstablePublicApiErrorCode:
      title: unstablePublicApiErrorCode
      type: string
      enum:
      - authentication_failed
      - access_denied
      - invalid_request
      - invalid_query
      - invalid_body
      - invalid_filter_value
      - invalid_json_path
      - invalid_variable_mapping
      - missing_variable_mapping
      - duplicate_variable_mapping
      - resource_not_found
      - name_conflict
      - evaluator_preflight_failed
      - conflict
      - unprocessable_content
      - rate_limited
      - method_not_allowed
      - internal_error
      description: 'Machine-readable error code returned by the unstable evaluators API.


        SDKs, CLIs, and agents should branch on `code` rather than parsing the human-readable `message`.

        The HTTP status still indicates the broad error class, while `code` gives the specific failure reason.'
    unstablePublicApiError:
      title: unstablePublicApiError
      type: object
      description: 'Standard error envelope for the unstable evaluators API.


        Response handling guidance:

        - Use the HTTP status code for the broad class of failure.

        - Use `code` for precise branching in SDKs, CLIs, or agents.

        - Inspect `details` for field-level validation context such as invalid filter values, malformed JSONPath expressions, or missing variable mappings.

        - Retry only after fixing the specific issue described by `code` and `details`.'
      properties:
        message:
          type: string
          description: Human-readable description of the failure.
          example: 'Filter column "type" contains unsupported value(s): INVALID'
        code:
          $ref: '#/components/schemas/unstablePublicApiErrorCode'
          description: Stable machine-readable error code.
        details:
          $ref: '#/components/schemas/unstablePublicApiErrorDetails'
          nullable: true
          description: Optional structured error context. Inspect the populated fields based on `code`.
      required:
      - message
      - code
    unstablePublicApiErrorDetails:
      title: unstablePublicApiErrorDetails
      type: object
      description: 'Optional structured context attached to an unstable-evals error.


        The populated fields depend on the error `code`:

        - request parsing failures populate `issues`

        - filter validation failures populate `field`, `column`, `invalidValues`, and `allowedValues`

        - variable mapping failures populate `field`, `variable`, or `variables`

        - JSONPath validation failures populate `field`, `variable`, and `value`

        - evaluator preflight failures populate `evaluatorName`, `provider`, and `model`

        - rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and `resetAt`'
      properties:
        issues:
          type: array
          items:
            $ref: '#/components/schemas/unstablePublicApiValidationIssue'
          nullable: true
          description: Validation issues for malformed request bodies or query parameters.
        field:
          type: string
          nullable: true
          description: Path-like reference to the failing field, for example `mapping[1].jsonPath`.
        column:
          type: string
          nullable: true
          description: Filter column that failed validation.
        invalidValues:
          type: array
          items:
            type: string
          nullable: true
          description: Unsupported values supplied by the caller.
        allowedValues:
          type: array
          items:
            type: string
          nullable: true
          description: Allowed values for the failing filter column.
        variable:
          type: string
          nullable: true
          description: Evaluator variable involved in the failure.
        variables:
          type: array
          items:
            type: string
          nullable: true
          description: Multiple evaluator variables involved in the failure, for example missing mappings.
        value:
          type: string
          nullable: true
          description: Raw invalid value supplied by the caller.
        evaluatorName:
          type: string
          nullable: true
          description: Evaluator name used during preflight validation.
        provider:
          type: string
          nullable: true
          description: Provider resolved during evaluator preflight, if any.
        model:
          type: string
          nullable: true
          description: Model resolved during evaluator preflight, if any.
        retryAfterSeconds:
          type: integer
          nullable: true
          description: Suggested retry delay for rate-limited requests.
        limit:
          type: integer
          nullable: true
          description: Numeric limit associated with the failure, for example the active evaluation-rule cap or the current rate-limit window.
        remaining:
          type: integer
          nullable: true
          description: Remaining requests in the current rate-limit window.
        resetAt:
          type: string
          nullable: true
          description: ISO-8601 timestamp when the current rate-limit window resets.
    unstablePublicNumericEvaluatorOutputDefinition:
      title: unstablePublicNumericEvaluatorOutputDefinition
      type: object
      properties:
        dataType:
          $ref: '#/components/schemas/unstableEvaluatorOutputDataType'
          description: Always `NUMERIC`.
        reasoning:
          $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
        score:
          $ref: '#/components/schemas/unstableEvaluatorOutputFieldDefinition'
      required:
      - dataType
      - reasoning
      - score
    unstablePublicApiValidationIssue:
      title: unstablePublicApiValidationIssue
      type: object
      description: 'One validation issue returned for malformed request bodies or query parameters.


        This mirrors the most important parts of a Zod issue: a machine-readable `code`,

        a human-readable `message`, and a structured `path`.'
      properties:
        code:
          type: string
          description: Machine-readable validation issue code emitted by the server validator.
        message:
          type: string
          description: Human-readable explanation of the validation failure.
        path:
          type: array
          items: {}
          description: Path to the invalid field, for example `["mapping", 0, "jsonPath"]`.
      required:
      - code
      - message
      - path
    unstableEvaluators:
      title: unstableEvaluators
      type: object
      properties:
        data:
          type: array
          items:
            $ref: '#/components/schemas/unstableEvaluator'
        meta:
          $ref: '#/components/schemas/utilsMetaResponse'
      required:
      - data
      - meta
    unstableEvaluatorScope:
      title: unstableEvaluatorScope
      type: string
      enum:
      - project
      - managed
      description: 'Where an evaluator comes from.


        - `project`: created in your project

        - `managed`: provided by Langfuse'
    unstableEvaluatorType:
      title: unstableEvaluatorType
      type: string
      enum:
      - llm_as_judge
      description: 'The evaluator engine type.


        The unstable public API currently supports only LLM-as-a-judge evaluators.'
    unstableEvaluatorOutputDefinition:
      title: unstableEvaluatorOutputDefinition
      oneOf:
      - type: object
        allOf:
        - type: object
          properties:
            dataType:
              type: string
              enum:
              - NUMERIC
        - $ref: '#/components/schemas/unstablePublicNumericEvaluatorOutputDefinition'
        required:
        - dataType
      - type: object
        allOf:
        - type: object
          properties:
            dataType:
              type: string
              enum:
              - BOOLEAN
        - $ref: '#/components/schemas/unstablePublicBooleanEvaluatorOutputDefinition'
        required:
        - dataType
      - type: object
        allOf:
        - type: object
          properties:
            dataType:
              type: string
              enum:
              - CATEGORICAL
        - $ref: '#/components/schemas/unstablePublicCategoricalEvaluatorOutputDefinition'
        required:
        - dataType
      description: 'Structured output definition to send when creating an evaluator.


        Agent guidance:

        - `dataType` is required.

        - Do not send `version`; that is an internal storage detail and is not part of the public request contract.

        - For `NUMERIC` and `BOOLEAN`, provide `reasoning.description` and `score.description`.

        - For `CATEGORICAL`, also provide `score.categories` and `score.shouldAllowMultipleMatches`.'
    unstableEvaluatorOutputFieldDefinition:
      title: unstableEvaluatorOutputFieldDefinition
      type: object
      properties:
        description:
          type: string
          description: Human-readable instructions for what the evaluator should return in this field.
      required:
      - description
    unstablePublicEvaluatorOutputDefinition:
      title: unstablePublicEvaluatorOutputDefinition
      oneOf:
      - type: object
        allOf:
        - type: object
          properties:
            dataType:
              type: string
              enum:
              - NUMERIC
        - $ref: '#/components/schemas/unstablePublicNumericEvaluatorOutputDefinition'
        required:
        - dataType
      - type: object
        allOf:
        - type: object
          properties:
            dataType:
              type: string
              enum:
              - BOOLEAN
        - $ref: '#/components/schemas/unstablePublicBooleanEvaluatorOutputDefinition'
        required:
        - dataType
      - type: object
        allOf:
        - type: object
          properties:
            dataType:
              type: string
              enum:
              - CATEGORICAL
        - $ref: '#/components/schemas/unstablePublicCategoricalEvaluatorOutputDefinition'
        required:
        - dataType
      description: 'Evaluator output definition returned by the public API.


        This response always includes `dataType` and never includes an internal output-definition `version`.

        Legacy stored evaluator definitions are normalized into this shape before they are returned.


        Use this response shape when deciding how to interpret future evaluation scores:

        - `NUMERIC`: expect numeric score values

        - `BOOLEAN`: expect `true` / `false`

        - `CATEGORICAL`: expect one or more values from `score.categories`'
  securitySchemes:
    BasicAuth:
      type: http
      scheme: basic