openapi: 3.2.0 info: title: Appen AI Data Annotation Platform (ADAP) Test Questions API description: '# Introduction The Appen AI Data Annotation Platform (ADAP) APIs offer a structured, automated approach to ensure high-quality data labeling on a scale.' servers: - url: https://api.appen.com/ description: Production environment security: - TokenAuth: [] tags: - name: Test Questions description: Test question management operations for evaluating contributor performance and ensuring quality assurance in data annotation workflows. UpdateTestQuestionRequest: type: object required: - unitId - jobId - enableDisableMode - hideAnswerMode - quizWorkMode - minimumMatchingAccuracy - questions properties: unitId: type: string format: uuid description: Unique identifier of the unit containing the test question to update example: f46c1c2f-e312-431b-9456-fbb18655e054 jobId: type: string format: uuid description: Unique identifier of the job that contains the test question example: 968bf9a6-f091-4804-b3ca-b5d0a4abf89b enableDisableMode: type: boolean description: Whether the test question is enabled for contributor evaluation example: true hideAnswerMode: type: boolean description: Whether to hide the test question answers from contributors example: false quizWorkMode: type: string enum: - QUIZ - WORK - QUIZ_WORK description: The mode in which the test question operates example: QUIZ minimumMatchingAccuracy: type: number format: double minimum: 0 maximum: 100 description: Minimum matching accuracy threshold as a percentage (0-100) example: 0 questions: type: array items: $ref: '#/components/schemas/QuestionAnswerMapRequest' description: Array of test questions with their answers and explanations minItems: 1 QuestionAnswerMapRequest: type: object required: - name - answer - reason properties: name: type: string description: The test question text/content example: ask_question_here answer: type: array items: type: string description: Array of correct answers for the test question example: - first_option - second_option reason: type: array items: type: string description: Array of explanations for each answer example: - '' paths: /v3/project-flow/projects/add-test-question: post: tags: - Test Questions summary: Add Test Questions description: 'Add test questions from units in the dataset to a specific job within a project. **Important Notes**: 1. Only units already present in the project dataset can be added as test questions. 2. Contributor scores report can be downloaded from the "Download a job level report" tab using the job level report cURL with `type` parameter set to `contributor_scores_report`.' operationId: addTestQuestionsToJob parameters: - name: teamId in: query required: true description: The unique identifier of the team that owns the project and job. schema: type: string format: uuid example: cf22e84f-06c8-4955-b626-1c47d245a176 requestBody: required: true description: Test question configuration specifying the project, job, and units to be added as test questions. content: application/json: schema: type: object required: - projectId - jobId - addTestQuestionVO properties: projectId: type: string format: uuid description: The unique identifier of the project. example: 6c680788-be5c-4b5f-8776-d5e63b97bca7 jobId: type: string format: uuid description: The unique identifier of the job. example: 968bf9a6-f091-4804-b3ca-b5d0a4abf89b addTestQuestionVO: type: array description: Array of test question objects specifying which units to add as test questions. items: type: object required: - unitId properties: unitId: type: string format: uuid description: The unique identifier of the unit to be added as a test question. example: 92d60008-a942-4084-9038-713984710102 minItems: 1 example: - unitId: 92d60008-a942-4084-9038-713984710102 - unitId: 47cd1658-58dd-469b-8939-a2a3e110d701 x-codeSamples: - lang: curl label: Add Test Questions cURL source: "curl -L -X POST 'https://api.appen.com/v3/project-flow/projects/add-test-question?teamId=' \\\n-H 'Authorization: Token token=' \\\n-H 'Content-Type: application/json' \\\n-d '{\n \"projectId\": \"6c680788-be5c-4b5f-8776-d5e63b97bca7\",\n \"jobId\": \"968bf9a6-f091-4804-b3ca-b5d0a4abf89b\",\n \"addTestQuestionVO\": [\n {\n \"unitId\": \"92d60008-a942-4084-9038-713984710102\"\n },\n {\n \"unitId\": \"47cd1658-58dd-469b-8939-a2a3e110d701\"\n }\n ]\n}'\n" responses: '200': description: Test questions added successfully content: application/json: schema: type: object properties: status: type: integer example: 200 message: type: string example: Test questions added successfully data: type: object properties: addTestQuestionDTO: type: object properties: projectId: type: string format: uuid example: 22067303-5817-4f0e-9528-656c56713f60 jobId: type: string format: uuid example: 2b42bc7f-18cf-4e2e-b83f-f49762124ff8 teamId: type: string format: uuid example: cf22e84f-06c8-4955-b626-1c47d245a176 addTestQuestionVO: type: array items: type: object properties: unitId: type: string format: uuid example: d41f913a-d482-4f15-a35c-33f9634372c1 testId: type: string format: uuid example: 408ce1c6-ad17-47e6-8006-e4fa39e3cef7 total: type: integer example: 2 message: type: string example: Successfully added 2 test questions traceId: type: string example: 7fd9a66fb4f0064af4bdd24360f8d112 timestamp: type: string format: date-time example: '2025-09-10T22:43:11.996519Z' required: - status - message - data - traceId - timestamp '400': description: Bad request - Duplicate data for unitId content: application/json: schema: type: object properties: status: type: integer example: 400 errorCode: type: integer example: 2017 message: type: string example: 'Bad Request:Duplicate data for unitId: abcb4175-843d-4db2-a596-61e39e60d1cc This unit has already added as a test question in another job for this project' traceId: type: string example: 8cc863ec3c0e0b06 timestamp: type: string format: date-time example: '2025-09-09T06:16:29.631635Z' '403': description: Forbidden - Access denied content: application/json: schema: $ref: '#/components/schemas/Error' example: timestamp: '2025-07-01T10:28:59.893+00:00' status: 403 error: Forbidden message: Access denied. Check your privileges. path: /v3/project-flow/projects/add-test-question '404': description: Project, job, or unit not found content: application/json: schema: type: object properties: status: type: integer example: 404 error: type: string example: Not Found message: type: string example: Project, job, or unit not found '500': description: Internal server error content: application/json: schema: type: object properties: status: type: integer example: 500 error: type: string example: Internal Server Error message: type: string example: An unexpected error occurred security: - TokenAuth: [] /v3/project-flow/projects/upload-test-question: post: tags: - Test Questions summary: Upload Test Questions description: 'Upload test questions from a CSV file to a specific job within a project. This endpoint accepts a CSV file containing test question data with required columns: {column1, _golden, *_gold, *_gold_reason , quiz_work_mode}. Use the returned status poll URL to check the upload status. **Sample CSV File**: Download the sample CSV file: sample-test-questions.csv **Important Notes**: 1. Only units already present in the project dataset can be uploaded as test questions. Non-matching units are automatically filtered out. For example, uploading 10 units when only 5 exist in the dataset will result in only 5 test questions being added. Already uploaded test questions will not be uploaded again. 2. Contributor scores report can be downloaded from the "Download a job level report" tab using the job level report cURL with `type` parameter set to `contributor_scores_report`.' operationId: uploadTestQuestions parameters: - name: fileName in: query required: true description: The name of the CSV file being uploaded. schema: type: string example: testquestions.csv - name: projectId in: query required: true description: The unique identifier of the project. schema: type: string format: uuid example: f2d8e3f3-bab0-4bfb-9529-21d74748d0be - name: jobId in: query required: true description: The unique identifier of the job. schema: type: string format: uuid example: 81167fe9-d274-4b71-a02b-cb54c0cf6aa7 requestBody: required: true description: CSV file content containing test question data. Required columns:- Problem (question content), _golden (TRUE/FALSE for enabled status), example_gold (gold standard answer), example_gold_reason (explanation for the answer), quiz_work_mode (work mode type, values can be QUIZ, WORK, QUIZ_WORK). content: text/csv: schema: type: string format: binary example: 'column1,_golden,*_gold,*_gold_reason,quiz_work_mode "In a royal palace, a queen hires four royal advisors, and each advisor has five knights under their command. How many knights are there if we get the total number of them all?","TRUE","20","We will use Multiplication operation to solve this problem:\nif each of the four royal advisors has five knights under their command, then the total number of knights in the royal palace can be found by multiplying the number of advisors by the number of knights each advisor has","QUIZ_WORK" "The moon is currently 384,400 kilometer away from Earth. If a space probe is sent to the moon and it travels 345,000 kilometer from Earth, what is the remaining distance to reach the moon?","TRUE","39400","To solve this problem, we need to use the operation of subtraction. We know that the current distance between the moon and Earth is 384,400 km. We also know that the space probe has traveled 345,000 kilometer from Earth. To find the remaining distance to reach the moon, we need to subtract the distance traveled by the QUIZ","WORK" "In a flower garden, there are 8 sunflowers. If 2 more sunflowers bloom, how many sunflowers will there be total?","TRUE","10","We solve this problem using addition. We know that there are 8 sunflowers initially, and 2 more sunflowers bloom, so we can set up an equation","WORK" ' x-codeSamples: - lang: curl label: Upload Test Questions cURL source: 'curl -L -X POST ''https://api.appen.com/v3/project-flow/projects/upload-test-question?fileName=&projectId=&jobId='' \ -H ''Authorization: Token token='' \ -H ''Content-Type: text/csv'' \ -d '''' ' responses: '201': description: Test questions upload initiated successfully content: application/json: schema: type: object properties: statusCode: type: integer example: 201 projectId: type: string format: uuid example: d103d9a9-7b96-47d2-9fb3-a4d32a500b60 teamId: type: string format: uuid example: cf22e84f-06c8-4955-b626-1c47d245a176 batchNum: type: integer example: 2 fileName: type: string example: testquestion.csv fileId: type: string format: uuid example: 455d10e4-5e3a-4ab2-8805-11d78cf7295c statusPollUrl: type: string example: https://api.appen.com/v3/jobs/upload/status?fileId=455d10e4-5e3a-4ab2-8805-11d78cf7295c&teamId=cf22e84f-06c8-4955-b626-1c47d245a176 details: type: string example: Test Question File UPLOAD IS TRIGGERED. Please use Status Poll URL to check the upload status updatedAt: type: string format: date-time example: '2025-09-11T06:53:07.549348Z' createdAt: type: string format: date-time example: '2025-09-11T06:53:07.549348Z' required: - statusCode - projectId - teamId - batchNum - fileName - updatedAt - createdAt - details - fileId - statusPollUrl '400': description: Bad request - Invalid file, parameters, or project not found content: application/json: schema: type: object properties: errorMessage: type: string example: File content is required httpStatus: type: integer example: 400 code: type: string example: FILE_REQUIRED examples: file_required: summary: File content required value: errorMessage: File content is required httpStatus: 400 code: FILE_REQUIRED project_not_found: summary: Project not found value: message: Project with ID 'f2d8e3f3-bab0-4bfb-9529-21d74748d0be' does not exist. Please create the project first. httpStatus: 400 errorCode: PROJECT_ID_REQUIRED '403': description: Forbidden - Access denied content: application/json: schema: $ref: '#/components/schemas/Error' example: timestamp: '2025-07-01T10:28:59.893+00:00' status: 403 error: Forbidden message: Access denied. Check your privileges. path: /v3/project-flow/projects/upload-test-question '404': description: Project, job, or unit not found content: application/json: schema: type: object properties: status: type: integer example: 404 error: type: string example: Not Found message: type: string example: Project, job, or unit not found '409': description: Conflict - Upload already in progress content: application/json: schema: type: object properties: errorMessage: type: string example: A file is already being uploaded to this project f2d8e3f3-bab0-4bfb-9529-21d74748d0be. Please try again later. code: type: string example: TEST_QUESTION_UPLOAD_FAILED '500': description: Internal server error content: application/json: schema: type: object properties: status: type: integer example: 500 error: type: string example: Internal Server Error message: type: string example: An unexpected error occurred security: - TokenAuth: [] /v3/project-flow/projects/update-tq-by-project: post: tags: - Test Questions summary: Update Test Questions description: 'Update existing test questions and their settings within a job. This endpoint allows you to modify test question content, answers, explanations, and operational modes. **Important Notes**: 1. Only existing test questions can be updated. The unitId must correspond to an existing test question in the job. 2. All jobIds must belong to the specified project. 3. Changes are applied immediately and affect ongoing contributor evaluations. 4. **To identify which units are configured as test questions, use the List Units in Project (With Golden Units filter) under Route Units page to retrieve test question unitIds and status before calling this update endpoint**. **Valid Values:** - `quizWorkMode`: QUIZ, WORK, QUIZ_WORK - `enableDisableMode`: true, false - `hideAnswerMode`: true, false - `minimumMatchingAccuracy`: 0-100 (percentage) (This parameter should be used when there is TextArea in the CML design and Levenshtein''s distance is used to evaluate contributors on the test questions) **You can get the name attribute to be configured in the request from the CML design code editor.** ![](https://qualityflow-api-docs.s3.us-east-1.amazonaws.com/adap/cmdesign.png)' operationId: updateTestQuestions parameters: - name: projectId in: query required: true description: Unique identifier of the project containing the test questions to update schema: type: string format: uuid example: 6c680788-be5c-4b5f-8776-d5e63b97bca7 requestBody: required: true content: application/json: schema: type: array items: $ref: '#/components/schemas/UpdateTestQuestionRequest' examples: quiz_mode_example: summary: Quiz Mode Test Question Update value: - unitId: f46c1c2f-e312-431b-9456-fbb18655e054 enableDisableMode: true hideAnswerMode: false quizWorkMode: QUIZ minimumMatchingAccuracy: 0 questions: - name: ask_question_here answer: - first_option - second_option reason: - '' jobId: 968bf9a6-f091-4804-b3ca-b5d0a4abf89b work_mode_example: summary: Work Mode Test Question Update value: - unitId: 76fc54f5-c2ba-41d9-a8de-7fbfb02f6483 enableDisableMode: true hideAnswerMode: false quizWorkMode: WORK minimumMatchingAccuracy: 0 questions: - name: ask_question_here answer: - first_option reason: - jhuaga d jobId: 968bf9a6-f091-4804-b3ca-b5d0a4abf89b mixed_mode_example: summary: Mixed Mode Test Question Update value: - unitId: f46c1c2f-e312-431b-9456-fbb18655e054 enableDisableMode: true hideAnswerMode: true quizWorkMode: QUIZ_WORK minimumMatchingAccuracy: 85.5 questions: - name: What is the capital of France? answer: - Paris - Lyon - Marseille reason: - Paris is the capital and largest city of France - Lyon is a major city but not the capital - Marseille is a port city but not the capital jobId: 968bf9a6-f091-4804-b3ca-b5d0a4abf89b - unitId: 76fc54f5-c2ba-41d9-a8de-7fbfb02f6483 enableDisableMode: false hideAnswerMode: false quizWorkMode: QUIZ minimumMatchingAccuracy: 90.0 questions: - name: Identify the programming language answer: - Java - Python reason: - Both are popular programming languages jobId: 968bf9a6-f091-4804-b3ca-b5d0a4abf89b x-codeSamples: - lang: curl label: Update Test Questions cURL source: "curl -L -X POST 'https://api.appen.com/v3/project-flow/projects/update-tq-by-project?projectId=' \\\n-H 'Authorization: Token token=' \\\n-H 'Content-Type: application/json' \\\n-d '[\n {\n \"unitId\": \"f46c1c2f-e312-431b-9456-fbb18655e054\",\n \"enableDisableMode\": true,\n \"hideAnswerMode\": false,\n \"quizWorkMode\": \"QUIZ\",\n \"minimumMatchingAccuracy\": 0,\n \"questions\": [\n {\n \"name\": \"ask_question_here\",\n \"answer\": [\n \"first_option\",\n \"second_option\"\n ],\n \"reason\": [\n \"\"\n ]\n }\n ],\n \"jobId\": \"968bf9a6-f091-4804-b3ca-b5d0a4abf89b\"\n },\n {\n \"unitId\": \"76fc54f5-c2ba-41d9-a8de-7fbfb02f6483\",\n \"enableDisableMode\": true,\n \"hideAnswerMode\": false,\n \"quizWorkMode\": \"WORK\",\n \"minimumMatchingAccuracy\": 0,\n \"questions\": [\n {\n \"name\": \"ask_question_here\",\n \"answer\": [\n \"first_option\"\n ],\n \"reason\": [\n \"jhuaga d\"\n ]\n }\n ],\n \"jobId\": \"968bf9a6-f091-4804-b3ca-b5d0a4abf89b\"\n }\n]'\n" - lang: curl label: Update Test Question for Text Area cURL source: "curl -L -X POST 'https://api.appen.com/v3/project-flow/projects/update-tq-by-project?projectId=' \\\n-H 'Authorization: Token token=' \\\n-H 'Content-Type: application/json' \\\n-d '[\n {\n \"unitId\": \"b3b57b77-9850-487c-a043-6543692925f5\",\n \"enableDisableMode\": true,\n \"hideAnswerMode\": true,\n \"quizWorkMode\": \"QUIZ_WORK\",\n \"minimumMatchingAccuracy\": 50,\n \"questions\": [\n {\n \"name\": \"enter_some_text_1\",\n \"answer\": [\n \"Hi I am Elon Musk\"\n ],\n \"reason\": [\n \"Richest person in the world\"\n ]\n }\n ],\n \"jobId\": \"c91a91b4-52e9-4270-a41d-db9e264ae438\"\n }\n]'\n" - lang: curl label: List Test Questions in the Project source: "curl -X POST \"https://api.appen.com/v3/project-flow/projects/units?projectId=faecd4fa-683f-402e-b7ea-8a904ae92021\" \\\n-H \"Authorization: Token token=\" \\\n-H \"Content-Type: application/json\" \\\n-d '{\n \"startRow\": 0,\n \"endRow\": 29,\n \"filterModel\": {\n \"datasetRowType\": {\n \"filterType\": \"SET\",\n \"values\": [\"UNIT\", \"\"]\n },\n \"source._golden\": {\n \"filterType\": \"SET\",\n \"values\": [\"TRUE\"]\n }\n },\n \"sortModel\": [],\n \"queryString\": \"\"\n }'\n" responses: '200': description: Test questions updated successfully content: application/json: schema: type: object properties: status: type: integer example: 200 message: type: string example: Test questions updated successfully data: type: string example: BULK_UPDATE_SUCCESSFUL traceId: type: string format: uuid example: 123e4567-e89b-12d3-a456-426614174000 timestamp: type: string format: date-time example: '2025-01-27T10:30:00.000Z' examples: success_response: summary: Successful Update value: status: 200 message: Test questions updated successfully data: BULK_UPDATE_SUCCESSFUL traceId: 123e4567-e89b-12d3-a456-426614174000 timestamp: '2025-01-27T10:30:00.000Z' '400': description: Bad request - validation failed content: application/json: schema: $ref: '#/components/schemas/Error' examples: validation_failed: summary: Validation Failed value: status: 400 errorCode: 4002 message: Payload Validation Failed details: - unitId is required - jobId is required - questions cannot be empty - 'quizWorkMode must be one of: QUIZ, WORK, QUIZ_WORK' - minimumMatchingAccuracy must be between 0 and 100 traceId: 123e4567-e89b-12d3-a456-426614174000 timestamp: '2025-01-27T10:30:00.000Z' empty_request: summary: Empty Request Body value: status: 400 errorCode: 4001 message: Request body cannot be empty traceId: 123e4567-e89b-12d3-a456-426614174000 timestamp: '2025-01-27T10:30:00.000Z' '403': description: Forbidden - insufficient permissions content: application/json: schema: $ref: '#/components/schemas/Error' example: timestamp: '2025-01-27T10:30:00.000Z' status: 403 error: Forbidden message: Access denied. Check your privileges . path: /v3/project-flow/projects/update-tq-by-project '404': description: Project or job not found content: application/json: schema: $ref: '#/components/schemas/Error' examples: project_not_found: summary: Project Not Found value: status: 404 errorCode: 4004 message: 'Project not found: could not extract teamId' traceId: 123e4567-e89b-12d3-a456-426614174000 timestamp: '2025-01-27T10:30:00.000Z' job_not_found: summary: Job Not Found value: status: 404 errorCode: 4004 message: Job with ID 968bf9a6-f091-4804-b3ca-b5d0a4abf89b not found traceId: 123e4567-e89b-12d3-a456-426614174000 timestamp: '2025-01-27T10:30:00.000Z' '500': description: Internal server error content: application/json: schema: $ref: '#/components/schemas/Error' examples: internal_error: summary: Internal Server Error value: status: 500 errorCode: 2012 message: 'Internal server error: Unexpected error occurred' traceId: 123e4567-e89b-12d3-a456-426614174000 timestamp: '2025-01-27T10:30:00.000Z' security: - TokenAuth: [] components: schemas: Error: type: object required: - timestamp - status - error - message - path properties: timestamp: type: string format: date-time description: ISO 8601 timestamp when the error occurred status: type: integer description: HTTP status code error: type: string description: HTTP status text message: type: string description: Human-readable error message path: type: string description: API endpoint path that generated the error trace: type: string description: Stack trace (only in development) example: timestamp: '2025-07-01T10:28:59.893+00:00' status: 500 error: Internal Server Error message: Failed to update job path: /publicapi/work-service/jobs securitySchemes: TokenAuth: type: http scheme: bearer bearerFormat: Token description: 'API Token authentication. Pass your API token in the Authorization header: `Authorization: Token token=your-api-token-here` '