openapi: 3.2.0 info: title: Scorecard Testsets API description: REST API for Scorecard version: 1.0.0 servers: - url: https://api2.scorecard.io/api/v2 security: - ApiKeyAuth: [] tags: - name: Test Sets paths: /testsets/{testsetId}: get: operationId: getTestset summary: Get Testset parameters: - in: path name: testsetId description: The ID of the Testset. schema: type: string example: '246' required: true responses: '200': description: Successfully retrieved Testset content: application/json: schema: $ref: '#/components/schemas/Testset' examples: Complete Testset details: value: id: '246' name: Long Context Q&A description: Testset for long context Q&A chatbot. jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string geo: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: - provenance - geo summary: Complete Testset details description: Example response showing all details of a Testset including its schema definition and field mappings. '401': $ref: '#/components/responses/UnauthenticatedError' '500': $ref: '#/components/responses/ServiceError' x-codeSamples: - lang: JavaScript source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testset = await client.testsets.get('246');\n\nconsole.log(testset.id);" - lang: Python source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestset = client.testsets.get(\n \"246\",\n)\nprint(testset.id)" - lang: cURL source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\"" tags: - Test Sets patch: operationId: updateTestset summary: Update Testset description: 'Update a Testset. Only the fields provided in the request body will be updated. If a field is provided, the new content will replace the existing content. If a field is not provided, the existing content will remain unchanged. When updating the schema: - If field mappings are not provided and existing mappings reference fields that no longer exist, those mappings will be automatically removed - To preserve all existing mappings, ensure all referenced fields remain in the updated schema - For complete control, provide both schema and fieldMapping when updating the schema' parameters: - in: path name: testsetId description: The ID of the Testset to update. schema: type: string example: '246' required: true requestBody: content: application/json: schema: type: object properties: name: type: string description: The name of the Testset. description: type: string description: The description of the Testset. jsonSchema: type: object description: The JSON schema for each Testcase in the Testset. additionalProperties: true fieldMapping: type: object properties: inputs: type: array items: type: string description: Fields that represent inputs to the AI system. expected: type: array items: type: string description: Fields that represent expected outputs. metadata: type: array items: type: string description: Fields that are not inputs or expected outputs. required: - inputs - expected - metadata description: Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata. examples: Update metadata only: value: name: Updated Q&A Testset description: Updated description for the Q&A Testset. summary: Update metadata only description: Simple metadata update without changing schema or mappings. Updates only the name and description fields while preserving the existing schema and field mappings. Remove schema field: value: jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string summary: Remove schema field description: This request removes the 'geo' field that existed in the original schema, but doesn't explicitly update the field mappings. Scorecard will automatically remove any field mappings that reference deleted fields. Full schema revision: value: jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: [] summary: Full schema revision description: Explicit update of both schema and field mappings, allowing complete control over the Testset structure. This example removes the 'geo' field and explicitly updates the field mappings to exclude 'provenance' from metadata. responses: '200': description: Testset updated successfully content: application/json: schema: $ref: '#/components/schemas/Testset' examples: Updated metadata only: value: id: '246' name: Updated Q&A Testset description: Updated description for the Q&A Testset. jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string geo: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: - provenance - geo summary: Updated metadata only description: Result after updating only the Testset's name and description. All schema fields and mappings remain unchanged. Auto-updated mappings: value: id: '246' name: Long Context Q&A description: Testset for long context Q&A chatbot. jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: - provenance summary: Auto-updated mappings description: Result after schema update with automatic field mapping cleanup. The 'geo' field has been automatically removed from the metadata mapping since it no longer exists in the schema. Custom field mapping: value: id: '246' name: Long Context Q&A description: Testset for long context Q&A chatbot. jsonSchema: type: object properties: question: type: string idealAnswer: type: string provenance: type: string fieldMapping: inputs: - question expected: - idealAnswer metadata: [] summary: Custom field mapping description: Result after explicit schema and field mapping update. Note that 'provenance' is not included in metadata since it wasn't specified in the request's field mapping. '401': $ref: '#/components/responses/UnauthenticatedError' '500': $ref: '#/components/responses/ServiceError' x-codeSamples: - lang: JavaScript source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testset = await client.testsets.update('246', {\n description: 'Updated description for the Q&A Testset.',\n name: 'Updated Q&A Testset',\n});\n\nconsole.log(testset.id);" - lang: Python source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestset = client.testsets.update(\n testset_id=\"246\",\n description=\"Updated description for the Q&A Testset.\",\n name=\"Updated Q&A Testset\",\n)\nprint(testset.id)" - lang: cURL source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID \\\n -X PATCH \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\"" tags: - Test Sets delete: operationId: deleteTestset summary: Delete Testset parameters: - in: path name: testsetId description: The ID of the Testset to delete. schema: type: string example: '246' required: true responses: '200': description: Testset deleted successfully content: application/json: schema: type: object properties: success: type: boolean description: Whether the deletion was successful. required: - success '401': $ref: '#/components/responses/UnauthenticatedError' '500': $ref: '#/components/responses/ServiceError' x-codeSamples: - lang: JavaScript source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testset = await client.testsets.delete('246');\n\nconsole.log(testset.success);" - lang: Python source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestset = client.testsets.delete(\n \"246\",\n)\nprint(testset.success)" - lang: cURL source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID \\\n -X DELETE \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\"" tags: - Test Sets /testsets/{testsetId}/testcases: get: operationId: listTestcases summary: List Testcases in Testset description: Retrieve a paginated list of Testcases belonging to a Testset. parameters: - in: path name: testsetId description: The ID of the Testset to list Testcases from. schema: type: string example: '246' required: true - in: query name: limit description: Maximum number of items to return (1-100). Use with `cursor` for pagination through large sets. schema: type: integer exclusiveMinimum: 0 default: 20 example: 20 - in: query name: cursor description: Cursor for pagination. Pass the `nextCursor` from the previous response to get the next page of results. schema: type: string example: '123' responses: '200': description: Successfully retrieved list of Testcases. content: application/json: schema: type: object properties: data: type: array items: $ref: '#/components/schemas/Testcase' nextCursor: type: - string - 'null' hasMore: type: boolean total: type: integer minimum: 0 required: - data - nextCursor - hasMore examples: Testcase list with pagination: value: data: - id: '123' testsetId: '246' jsonData: question: What is the capital of France? idealAnswer: Paris provenance: hand_curated inputs: question: What is the capital of France? expected: idealAnswer: Paris - id: '124' testsetId: '246' jsonData: question: What is the largest planet in our solar system? idealAnswer: Jupiter provenance: synthetic inputs: question: What is the largest planet in our solar system? expected: idealAnswer: Jupiter - id: '125' testsetId: '246' jsonData: question: What is the deepest ocean on Earth? provenance: user_feedback inputs: question: What is the deepest ocean on Earth? expected: {} validationErrors: - path: /data message: Required field 'idealAnswer' is missing nextCursor: '126' hasMore: true summary: Testcase list with pagination description: Example response showing a paginated list of Testcases. '401': $ref: '#/components/responses/UnauthenticatedError' '500': $ref: '#/components/responses/ServiceError' x-codeSamples: - lang: JavaScript source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\n// Automatically fetches more pages as needed.\nfor await (const testcase of client.testcases.list('246')) {\n console.log(testcase.id);\n}" - lang: Python source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\npage = client.testcases.list(\n testset_id=\"246\",\n)\npage = page.data[0]\nprint(page.id)" - lang: cURL source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID/testcases \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\"" tags: - Test Sets post: operationId: createTestcases summary: Create multiple Testcases description: Create multiple Testcases in the specified Testset. parameters: - in: path name: testsetId description: The ID of the Testset to add the Testcases to. schema: type: string example: '246' required: true requestBody: content: application/json: schema: type: object properties: items: type: array items: type: object properties: jsonData: type: object additionalProperties: true description: The JSON data of the Testcase, which is validated against the Testset's schema. required: - jsonData minItems: 1 maxItems: 100 description: Testcases to create (max 100). required: - items examples: Create multiple Testcases: value: items: - jsonData: question: What is the capital of France? idealAnswer: Paris provenance: hand_curated - jsonData: question: What is the largest planet in our solar system? idealAnswer: Jupiter provenance: synthetic - jsonData: question: How many planets are in our solar system? idealAnswer: 8 provenance: user_feedback summary: Create multiple Testcases description: Create multiple Testcases in a single request. responses: '201': description: Testcases created successfully content: application/json: schema: type: object properties: items: type: array items: $ref: '#/components/schemas/Testcase' required: - items examples: Created Testcases response: value: items: - id: '123' testsetId: '246' jsonData: question: What is the capital of France? idealAnswer: Paris provenance: hand_curated inputs: question: What is the capital of France? expected: idealAnswer: Paris - id: '124' testsetId: '246' jsonData: question: What is the largest planet in our solar system? idealAnswer: Jupiter provenance: synthetic inputs: question: What is the largest planet in our solar system? expected: idealAnswer: Jupiter - id: '125' testsetId: '246' jsonData: question: How many planets are in our solar system? idealAnswer: 8 provenance: user_feedback inputs: question: How many planets are in our solar system? expected: idealAnswer: 8 validationErrors: - path: /data/idealAnswer message: Expected string, received number summary: Created Testcases description: Example response showing successfully created Testcases. '401': $ref: '#/components/responses/UnauthenticatedError' '500': $ref: '#/components/responses/ServiceError' x-codeSamples: - lang: JavaScript source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testcase = await client.testcases.create('246', {\n items: [\n {\n jsonData: {\n question: 'What is the capital of France?',\n idealAnswer: 'Paris',\n provenance: 'hand_curated',\n },\n },\n {\n jsonData: {\n question: 'What is the largest planet in our solar system?',\n idealAnswer: 'Jupiter',\n provenance: 'synthetic',\n },\n },\n {\n jsonData: {\n question: 'How many planets are in our solar system?',\n idealAnswer: 8,\n provenance: 'user_feedback',\n },\n },\n ],\n});\n\nconsole.log(testcase.items);" - lang: Python source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestcase = client.testcases.create(\n testset_id=\"246\",\n items=[{\n \"json_data\": {\n \"question\": \"What is the capital of France?\",\n \"idealAnswer\": \"Paris\",\n \"provenance\": \"hand_curated\",\n }\n }, {\n \"json_data\": {\n \"question\": \"What is the largest planet in our solar system?\",\n \"idealAnswer\": \"Jupiter\",\n \"provenance\": \"synthetic\",\n }\n }, {\n \"json_data\": {\n \"question\": \"How many planets are in our solar system?\",\n \"idealAnswer\": 8,\n \"provenance\": \"user_feedback\",\n }\n }],\n)\nprint(testcase.items)" - lang: cURL source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID/testcases \\\n -H 'Content-Type: application/json' \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\" \\\n -d '{\n \"items\": [\n {\n \"jsonData\": {\n \"question\": \"bar\",\n \"idealAnswer\": \"bar\",\n \"provenance\": \"bar\"\n }\n },\n {\n \"jsonData\": {\n \"question\": \"bar\",\n \"idealAnswer\": \"bar\",\n \"provenance\": \"bar\"\n }\n },\n {\n \"jsonData\": {\n \"question\": \"bar\",\n \"idealAnswer\": \"bar\",\n \"provenance\": \"bar\"\n }\n }\n ]\n }'" tags: - Test Sets components: responses: ServiceError: description: An internal service error indicating an issue with the Scorecard service. content: application/json: schema: $ref: '#/components/schemas/ApiError' examples: Internal error: value: code: INTERNAL_ERROR message: An unexpected error occurred while processing your request. details: {} summary: Internal error description: Generic error when an unexpected internal issue occurs. UnauthenticatedError: description: Error indicating that the request is not authenticated. content: application/json: schema: $ref: '#/components/schemas/ApiError' examples: Authentication failure: value: code: UNAUTHORIZED message: Invalid or missing authentication token details: {} summary: Authentication failure description: Error returned when authentication credentials are invalid or missing. schemas: Testset: type: object properties: id: type: string description: The ID of the Testset. name: type: string description: The name of the Testset. description: type: string description: The description of the Testset. jsonSchema: type: object description: The JSON schema for each Testcase in the Testset. additionalProperties: true fieldMapping: type: object properties: inputs: type: array items: type: string description: Fields that represent inputs to the AI system. expected: type: array items: type: string description: Fields that represent expected outputs. metadata: type: array items: type: string description: Fields that are not inputs or expected outputs. required: - inputs - expected - metadata description: Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata. required: - id - name - description - jsonSchema - fieldMapping description: 'A collection of Testcases that share the same schema. Each Testset defines the structure of its Testcases through a JSON schema. The `fieldMapping` object maps top-level keys of the Testcase schema to their roles (input/expected output). Fields not mentioned in the `fieldMapping` during creation or update are treated as metadata. ## JSON Schema validation constraints supported: - **Required fields** - Fields listed in the schema''s `required` array must be present in Testcases. - **Type validation** - Values must match the specified type (string, number, boolean, null, integer, object, array). - **Enum validation** - Values must be one of the options specified in the `enum` array. - **Object property validation** - Properties of objects must conform to their defined schemas. - **Array item validation** - Items in arrays must conform to the `items` schema. - **Logical composition** - Values must conform to at least one schema in the `anyOf` array. Testcases that fail validation will still be stored, but will include `validationErrors` detailing the issues. Extra fields in the Testcase data that are not in the schema will be stored but are ignored during validation.' Testcase: type: object properties: id: type: string description: The ID of the Testcase. testsetId: type: string description: The ID of the Testset this Testcase belongs to. jsonData: type: object additionalProperties: true description: The JSON data of the Testcase, which is validated against the Testset's schema. inputs: type: object additionalProperties: true description: Derived from data based on the Testset's fieldMapping. Contains all fields marked as inputs, including those with validation errors. expected: type: object additionalProperties: true description: Derived from data based on the Testset's fieldMapping. Contains all fields marked as expected outputs, including those with validation errors. validationErrors: type: array items: type: object properties: path: type: string description: JSON Pointer to the field with the validation error. example: /data/question message: type: string description: Human-readable error description. example: Required field missing required: - path - message description: Validation errors found in the Testcase data. If present, the Testcase doesn't fully conform to its Testset's schema. required: - id - testsetId - jsonData - inputs - expected description: 'A test case in the Scorecard system. Contains JSON data that is validated against the schema defined by its Testset. The `inputs` and `expected` fields are derived from the `data` field based on the Testset''s `fieldMapping`, and include all mapped fields, including those with validation errors. Testcases are stored regardless of validation results, with any validation errors included in the `validationErrors` field.' ApiError: type: object properties: code: type: string message: type: string details: type: object additionalProperties: true x-stainless-any: true required: - code - message - details description: An API error. securitySchemes: ApiKeyAuth: type: http scheme: bearer bearerFormat: starts with ak_