openapi: 3.0.3 info: title: Keeper Data Bridge API description: | REST API for the Keeper Data Bridge backend service. Provides endpoints for data import, OData-style querying, cleanse analysis, issue management, ad-hoc report export, holdings lookup, throttle policy management, and MongoDB performance benchmarking with noisy-neighbour diagnostics. version: 1.0.0 contact: name: DEFRA servers: - url: /api description: Base API path tags: - name: Import description: Bulk data import, reporting, lineage, collection management, and storage operations. - name: Query description: OData-style querying of MongoDB collections. - name: Cleanse description: Cleanse analysis operations, issue management, and report export. - name: Cleanse Export description: | Ad-hoc full cleanse report export operations. Exports all active issues to CSV/ZIP on S3 with progress tracking. Runs independently of the analysis pipeline — does not send notifications or update the incremental-export timestamp. - name: Holdings description: CTS and SAM CPH holding lookups. - name: Throttle Policies description: Throttle policy CRUD and activation. - name: External Catalogue description: External file catalogue browsing and file uploads. - name: Benchmark description: | Self-contained MongoDB benchmark for diagnosing environment-level performance differences. Creates temporary collections, runs read/write scenarios, and produces a detailed report with noisy-neighbour analysis. Does not touch production data collections. paths: # ═══════════════════════════════════════════════════════════════════ # Import # ═══════════════════════════════════════════════════════════════════ /import/start: post: tags: [Import] operationId: startBulkImport summary: Start a bulk file import description: | Starts a bulk file import process asynchronously. Returns immediately with an import ID once the lock has been acquired. The import continues running in the background. parameters: - name: sourceType in: query description: "The source type for the import ('internal' or 'external')" schema: type: string default: external enum: [internal, external] responses: '202': description: Import started successfully content: application/json: schema: $ref: '#/components/schemas/StartBulkImportResponse' '400': description: Invalid sourceType parameter content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '409': description: Another import is already running content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import: get: tags: [Import] operationId: getImportSummaries summary: Get paginated import summaries description: Returns import summaries in reverse chronological order. parameters: - $ref: '#/components/parameters/Skip' - $ref: '#/components/parameters/Top' responses: '200': description: Paginated list of import summaries content: application/json: schema: $ref: '#/components/schemas/ImportSummariesResponse' '400': description: Invalid pagination parameters content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/{importId}: get: tags: [Import] operationId: getImportReport summary: Get an import report description: Gets the full import report for a specific import ID. parameters: - name: importId in: path required: true schema: type: string format: uuid responses: '200': description: Import report content: application/json: schema: $ref: '#/components/schemas/ImportReport' '404': description: Import not found /import/{importId}/files: get: tags: [Import] operationId: getFileReports summary: Get file reports for an import description: Returns file processing reports for all files in a specific import. parameters: - name: importId in: path required: true schema: type: string format: uuid responses: '200': description: File reports content: application/json: schema: $ref: '#/components/schemas/FileReportsResponse' '404': description: Import not found /import/lineage/{collectionName}/{recordId}: get: tags: [Import] operationId: getRecordLineageEvents summary: Get record lineage events description: | Gets paginated lineage events for a specific record in chronological order. The recordId is a URL-safe SHA256 hash generated from composite key parts. parameters: - name: collectionName in: path required: true description: "The collection name (e.g., 'sam_cph_holdings')" schema: type: string - name: recordId in: path required: true description: URL-safe record ID (SHA256 hash) schema: type: string - $ref: '#/components/parameters/Skip' - $ref: '#/components/parameters/Top' responses: '200': description: Paginated lineage events content: application/json: schema: $ref: '#/components/schemas/PaginatedLineageEvents' '400': description: Invalid parameters content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Record lineage not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/generate-record-id: post: tags: [Import] operationId: generateRecordId summary: Generate a record ID from key parts description: Generates a URL-safe record ID from composite key parts using SHA256 hashing. requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/GenerateRecordIdRequest' responses: '200': description: Generated record ID content: application/json: schema: $ref: '#/components/schemas/GenerateRecordIdResponse' '400': description: Invalid key parts content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/collections/{collectionName}: delete: tags: [Import] operationId: deleteCollection summary: Delete a MongoDB collection description: Deletes a specific MongoDB collection by name. Must be defined in DataSetDefinitions. parameters: - name: collectionName in: path required: true schema: type: string responses: '200': description: Collection deleted content: application/json: schema: $ref: '#/components/schemas/DeleteCollectionResponse' '404': description: Collection not found in definitions content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/collections: delete: tags: [Import] operationId: deleteAllCollections summary: Delete all MongoDB collections description: Deletes all MongoDB collections defined in DataSetDefinitions. responses: '200': description: Collections deleted content: application/json: schema: $ref: '#/components/schemas/DeleteCollectionsResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/reporting-collections/{collectionName}: delete: tags: [Import] operationId: deleteReportingCollection summary: Delete a reporting collection description: | Deletes a specific reporting/lineage collection. Valid collections: import_reports, import_files, record_lineage, record_lineage_events. parameters: - name: collectionName in: path required: true schema: type: string responses: '200': description: Reporting collection deleted content: application/json: schema: $ref: '#/components/schemas/DeleteCollectionResponse' '404': description: Collection not valid content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/reporting-collections: delete: tags: [Import] operationId: deleteAllReportingCollections summary: Delete all reporting collections description: | Deletes all reporting/lineage collections. Includes: import_reports, import_files, record_lineage, record_lineage_events. responses: '200': description: Reporting collections deleted content: application/json: schema: $ref: '#/components/schemas/DeleteCollectionsResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /import/internal-storage: delete: tags: [Import] operationId: clearDownInternalStorage summary: Clear down internal storage description: Deletes all objects from internal target storage under the configured prefix. parameters: - name: sourceType in: query schema: type: string default: internal enum: [internal, external] responses: '200': description: Storage cleared content: application/json: schema: $ref: '#/components/schemas/ClearDownStorageResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' # ═══════════════════════════════════════════════════════════════════ # Query # ═══════════════════════════════════════════════════════════════════ /query/{collectionName}: get: tags: [Query] operationId: queryCollection summary: Query a MongoDB collection description: | Query a MongoDB collection with OData-style parameters. Examples: - `GET /api/query/sam_cph_holdings?$filter=CPH eq 'ABC123'&$top=50` - `GET /api/query/sam_cph_holdings?$filter=contains(CPH,'ABC') and IsDeleted eq false&$orderby=UpdatedAtUtc desc` - `GET /api/query/sam_cph_holdings?$select=CPH,UpdatedAtUtc&$top=20` parameters: - name: collectionName in: path required: true description: "Name of the collection (e.g., 'sam_cph_holdings')" schema: type: string - name: $filter in: query description: OData $filter expression schema: type: string - name: $orderby in: query description: "OData $orderby expression (e.g., 'UpdatedAtUtc desc')" schema: type: string - name: $select in: query description: "OData $select expression (e.g., 'CPH,UpdatedAtUtc')" schema: type: string - name: $skip in: query schema: type: integer - name: $top in: query description: Max 1000, default 100 schema: type: integer default: 100 maximum: 1000 - name: $count in: query description: Include total count in response schema: type: boolean default: true responses: '200': description: Query results content: application/json: schema: $ref: '#/components/schemas/QueryResult' '400': description: Invalid query parameters '404': description: Collection not found '499': description: Client closed request '500': description: Internal server error # ═══════════════════════════════════════════════════════════════════ # Cleanse — Operations # ═══════════════════════════════════════════════════════════════════ /cleanse/start-analysis: post: tags: [Cleanse] operationId: startAnalysis summary: Start a cleanse analysis description: | Starts a new cleanse analysis operation in the background. Progress can be tracked via GET /cleanse/run/{operationId}. responses: '202': description: Analysis started content: application/json: schema: $ref: '#/components/schemas/StartAnalysisResponse' '409': description: An analysis is already running content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/cancel-analysis: post: tags: [Cleanse] operationId: cancelAnalysis summary: Cancel the running analysis description: Requests cancellation of the currently running cleanse analysis operation. responses: '200': description: Cancellation requested content: application/json: schema: $ref: '#/components/schemas/CancelAnalysisResponse' '404': description: No running analysis found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/runs: get: tags: [Cleanse] operationId: getAnalysisRuns summary: Get paginated analysis run summaries parameters: - $ref: '#/components/parameters/Skip' - name: top in: query schema: type: integer default: 10 minimum: 1 maximum: 100 responses: '200': description: Analysis run summaries content: application/json: schema: $ref: '#/components/schemas/AnalysisRunsResponse' '400': description: Invalid pagination parameters content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/run/{operationId}: get: tags: [Cleanse] operationId: getAnalysisRun summary: Get full details of an analysis run description: | Returns the complete operation state including per-phase progress, live RPM statistics, and projected completion times. parameters: - name: operationId in: path required: true schema: type: string responses: '200': description: Analysis run details content: application/json: schema: $ref: '#/components/schemas/CleanseAnalysisOperationDto' '404': description: Operation not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/run/{operationId}/regenerate-url: post: tags: [Cleanse] operationId: regenerateReportUrl summary: Regenerate a report presigned URL description: Regenerates the presigned S3 URL for an operation's report when the original has expired. parameters: - name: operationId in: path required: true schema: type: string responses: '200': description: URL regenerated content: application/json: schema: $ref: '#/components/schemas/RegenerateUrlResponse' '400': description: Operation has no report content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Operation not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/delete-data: post: tags: [Cleanse] operationId: deleteCleanseData summary: Delete all cleanse issues responses: '200': description: Data deleted content: application/json: schema: $ref: '#/components/schemas/DeleteDataResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '500': description: Server error content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/delete-metadata: post: tags: [Cleanse] operationId: deleteCleanseMetadata summary: Delete all operation history responses: '200': description: Metadata deleted content: application/json: schema: $ref: '#/components/schemas/DeleteDataResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '500': description: Server error content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/test-notification: post: tags: [Cleanse] operationId: sendTestNotification summary: Send a test notification email description: Sends a test email via GOV.UK Notify to verify configuration. responses: '200': description: Test notification sent content: application/json: schema: $ref: '#/components/schemas/TestNotificationResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '500': description: Notification failed content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' # ═══════════════════════════════════════════════════════════════════ # Cleanse — Issues # ═══════════════════════════════════════════════════════════════════ /cleanse/issues: get: tags: [Cleanse] operationId: getIssues summary: Get paginated, filterable cleanse issues parameters: - $ref: '#/components/parameters/Skip' - $ref: '#/components/parameters/Top' - name: ctsLidFullIdentifier in: query description: Filter by CTS LID (contains, case-insensitive) schema: type: string - name: cph in: query description: Filter by CPH (contains, case-insensitive) schema: type: string - name: issueCode in: query description: Filter by exact issue code schema: type: string - name: ruleCode in: query description: Filter by exact rule code schema: type: string - name: errorCode in: query description: Filter by exact error code schema: type: string - name: isActive in: query schema: type: boolean - name: isIgnored in: query schema: type: boolean - name: resolutionStatus in: query description: "None, Todo, InProgress, Resolved" schema: type: string enum: [None, Todo, InProgress, Resolved] - name: assignedTo in: query schema: type: string - name: isUnassigned in: query schema: type: boolean - name: createdAfterUtc in: query schema: type: string format: date-time - name: createdBeforeUtc in: query schema: type: string format: date-time - name: updatedAfterUtc in: query schema: type: string format: date-time - name: updatedBeforeUtc in: query schema: type: string format: date-time - name: sortBy in: query description: Field to sort by (default LastUpdatedAtUtc) schema: type: string - name: sortDescending in: query schema: type: boolean default: true responses: '200': description: Paginated issues content: application/json: schema: $ref: '#/components/schemas/IssuesResponse' '400': description: Invalid parameters content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/issues/{issueId}/ignore: post: tags: [Cleanse] operationId: ignoreIssue summary: Ignore an issue parameters: - name: issueId in: path required: true schema: type: string requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PerformedByRequest' responses: '200': description: Issue ignored content: application/json: schema: $ref: '#/components/schemas/IssueCommandResponse' '404': description: Issue not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/issues/{issueId}/unignore: post: tags: [Cleanse] operationId: unignoreIssue summary: Remove ignored flag from an issue parameters: - name: issueId in: path required: true schema: type: string requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PerformedByRequest' responses: '200': description: Issue unignored content: application/json: schema: $ref: '#/components/schemas/IssueCommandResponse' '404': description: Issue not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/issues/{issueId}/resolution-status: post: tags: [Cleanse] operationId: updateResolutionStatus summary: Update resolution status parameters: - name: issueId in: path required: true schema: type: string requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/UpdateResolutionStatusRequest' responses: '200': description: Status updated content: application/json: schema: $ref: '#/components/schemas/IssueCommandResponse' '400': description: Invalid status value content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Issue not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/issues/{issueId}/assign: post: tags: [Cleanse] operationId: assignIssue summary: Assign an issue to a user parameters: - name: issueId in: path required: true schema: type: string requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/AssignIssueRequest' responses: '200': description: Issue assigned content: application/json: schema: $ref: '#/components/schemas/IssueCommandResponse' '404': description: Issue not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/issues/{issueId}/unassign: post: tags: [Cleanse] operationId: unassignIssue summary: Unassign an issue parameters: - name: issueId in: path required: true schema: type: string requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PerformedByRequest' responses: '200': description: Issue unassigned content: application/json: schema: $ref: '#/components/schemas/IssueCommandResponse' '404': description: Issue not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /cleanse/issues/{issueId}/history: get: tags: [Cleanse] operationId: getIssueHistory summary: Get issue history description: Returns paginated history/lineage entries for a specific issue. parameters: - name: issueId in: path required: true schema: type: string - $ref: '#/components/parameters/Skip' - name: top in: query schema: type: integer default: 50 minimum: 1 maximum: 100 responses: '200': description: Issue history content: application/json: schema: $ref: '#/components/schemas/IssueHistoryResponse' '400': description: Invalid pagination parameters content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' # ═══════════════════════════════════════════════════════════════════ # Cleanse Export — Ad-hoc full export # ═══════════════════════════════════════════════════════════════════ /cleanse-export/start: post: tags: [Cleanse Export] operationId: startFullExport summary: Start an ad-hoc full export of all active issues description: | The export runs in a long-running background task with a distributed lock to prevent concurrent exports. Progress can be tracked via the returned export ID. Unlike the export phase of a cleanse analysis (which is incremental), this always exports all active issues. It does NOT update the incremental-export timestamp and does NOT send email notifications. responses: '202': description: Export started successfully content: application/json: schema: $ref: '#/components/schemas/StartExportResponse' '409': description: An export is already running content: application/json: schema: $ref: '#/components/schemas/ExportErrorResponse' /cleanse-export/{exportId}: get: tags: [Cleanse Export] operationId: getExportOperation summary: Get progress and details of a specific export operation description: | Poll this endpoint to track export progress. When the export completes successfully, the response includes reportUrl (a presigned S3 download URL) and reportObjectKey. If the export fails, the error field contains the failure details. parameters: - name: exportId in: path required: true description: The export operation ID returned from the start endpoint schema: type: string responses: '200': description: Export operation details content: application/json: schema: $ref: '#/components/schemas/CleanseExportOperationDto' '404': description: Export operation not found content: application/json: schema: $ref: '#/components/schemas/ExportErrorResponse' /cleanse-export/{exportId}/regenerate-url: post: tags: [Cleanse Export] operationId: regenerateExportUrl summary: Regenerate the presigned URL for an export report description: | Use this when the original presigned URL has expired. The S3 object key must exist (i.e., the export must have completed successfully). The newly generated URL is also persisted on the export operation document. parameters: - name: exportId in: path required: true description: The export operation ID schema: type: string responses: '200': description: URL regenerated successfully content: application/json: schema: $ref: '#/components/schemas/RegenerateExportUrlResponse' '400': description: Export has no report file (may not have completed successfully) content: application/json: schema: $ref: '#/components/schemas/ExportErrorResponse' '404': description: Export operation not found content: application/json: schema: $ref: '#/components/schemas/ExportErrorResponse' /cleanse-export: get: tags: [Cleanse Export] operationId: getExportOperations summary: Get a paginated list of export operations description: Returns export operations in reverse chronological order (most recent first). parameters: - $ref: '#/components/parameters/Skip' - $ref: '#/components/parameters/Top' responses: '200': description: Paginated list of export operations content: application/json: schema: $ref: '#/components/schemas/ExportOperationsResponse' '400': description: Invalid parameters content: application/json: schema: $ref: '#/components/schemas/ExportErrorResponse' # ═══════════════════════════════════════════════════════════════════ # Holdings # ═══════════════════════════════════════════════════════════════════ /holdings/cts/{lidFullIdentifier}: get: tags: [Holdings] operationId: getCtsCphHolding summary: Get a CTS CPH holding by LID description: "Looks up a CTS CPH holding by its LID full identifier (format: XX-CC/PPP/HHHH)." parameters: - name: lidFullIdentifier in: path required: true description: "LID full identifier (e.g., 'AB-12/345/6789')" schema: type: string responses: '200': description: CTS holding details content: application/json: schema: $ref: '#/components/schemas/CtsCphHoldingResponse' '400': description: Invalid LID format content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Holding not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /holdings/sam/{cph}: get: tags: [Holdings] operationId: getSamCphHolding summary: Get a SAM CPH holding description: "Looks up a SAM CPH holding by its CPH value (format: CC/PPP/HHHH)." parameters: - name: cph in: path required: true description: "CPH value (e.g., '12/345/6789')" schema: type: string responses: '200': description: SAM holding details content: application/json: schema: $ref: '#/components/schemas/SamCphHoldingResponse' '400': description: Invalid CPH format content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Holding not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '499': description: Client closed request content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' # ═══════════════════════════════════════════════════════════════════ # Throttle Policies # ═══════════════════════════════════════════════════════════════════ /throttle-policies: get: tags: [Throttle Policies] operationId: getAllThrottlePolicies summary: Get all throttle policies responses: '200': description: List of throttle policies content: application/json: schema: type: array items: $ref: '#/components/schemas/ThrottlePolicy' post: tags: [Throttle Policies] operationId: createThrottlePolicy summary: Create a throttle policy requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateThrottlePolicyCommand' responses: '201': description: Policy created content: application/json: schema: $ref: '#/components/schemas/ThrottlePolicy' '400': description: Validation error '409': description: Duplicate slug /throttle-policies/active: get: tags: [Throttle Policies] operationId: getActiveThrottlePolicy summary: Get the active throttle policy responses: '200': description: Active policy (or Normal fallback) content: application/json: schema: $ref: '#/components/schemas/ThrottlePolicy' /throttle-policies/{slug}: get: tags: [Throttle Policies] operationId: getThrottlePolicyBySlug summary: Get a throttle policy by slug parameters: - name: slug in: path required: true schema: type: string responses: '200': description: Throttle policy content: application/json: schema: $ref: '#/components/schemas/ThrottlePolicy' '404': description: Policy not found put: tags: [Throttle Policies] operationId: updateThrottlePolicy summary: Update a throttle policy parameters: - name: slug in: path required: true schema: type: string requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/UpdateThrottlePolicyCommand' responses: '200': description: Policy updated content: application/json: schema: $ref: '#/components/schemas/ThrottlePolicy' '400': description: Validation error '404': description: Policy not found delete: tags: [Throttle Policies] operationId: deleteThrottlePolicy summary: Delete a throttle policy parameters: - name: slug in: path required: true schema: type: string responses: '204': description: Policy deleted '400': description: Cannot delete (e.g., Normal policy) '404': description: Policy not found '409': description: Cannot delete active policy /throttle-policies/{slug}/activate: post: tags: [Throttle Policies] operationId: activateThrottlePolicy summary: Activate a throttle policy parameters: - name: slug in: path required: true schema: type: string responses: '200': description: Policy activated content: application/json: schema: $ref: '#/components/schemas/ThrottlePolicy' '400': description: Cannot activate (e.g., Normal policy) '404': description: Policy not found /throttle-policies/deactivate: post: tags: [Throttle Policies] operationId: deactivateAllThrottlePolicies summary: Deactivate all policies description: Reverts to Normal fallback policy. responses: '204': description: All policies deactivated # ═══════════════════════════════════════════════════════════════════ # External Catalogue # ═══════════════════════════════════════════════════════════════════ /externalcatalogue/files: get: tags: [External Catalogue] operationId: getFilesReport summary: Get file catalogue report description: Returns a plain text report of available files for the specified source type. parameters: - name: sourceType in: query required: true schema: type: string enum: [internal, external] - name: days in: query required: true description: Number of days to look back schema: type: integer responses: '200': description: Plain text file report content: text/plain: schema: type: string '400': description: Invalid parameters /externalcatalogue/upload: post: tags: [External Catalogue] operationId: uploadFile summary: Upload a CSV file description: Uploads a file to internal S3 storage. Filename must match a dataset definition pattern. parameters: - name: objectKey in: query required: true description: Filename (object key) — no path separators schema: type: string requestBody: required: true content: multipart/form-data: schema: type: object properties: file: type: string format: binary responses: '200': description: File uploaded content: application/json: schema: type: object properties: message: type: string objectKey: type: string size: type: integer contentType: type: string '422': description: Validation error /externalcatalogue/upload-raw: post: tags: [External Catalogue] operationId: uploadRawFile summary: Upload raw CSV content description: Uploads raw file content to internal S3 storage. Alternative for testing. parameters: - name: objectKey in: query required: true schema: type: string requestBody: required: true content: text/csv: schema: type: string format: binary application/csv: schema: type: string format: binary text/plain: schema: type: string format: binary responses: '200': description: File uploaded content: application/json: schema: type: object properties: message: type: string objectKey: type: string size: type: integer contentType: type: string '422': description: Validation error # ═══════════════════════════════════════════════════════════════════ # Benchmark # ═══════════════════════════════════════════════════════════════════ /benchmark/start: post: tags: [Benchmark] operationId: startBenchmark summary: Start a benchmark run description: | Starts a benchmark run with the supplied (or default) configuration. The run executes in the background; poll GET /api/benchmark/status and GET /api/benchmark/report for progress and results. Creates temporary `_benchmark_*` collections that are cleaned up automatically on completion or cancellation. requestBody: required: false content: application/json: schema: $ref: '#/components/schemas/BenchmarkConfig' responses: '202': description: Benchmark started content: application/json: schema: $ref: '#/components/schemas/StartBenchmarkResponse' '409': description: A benchmark is already running content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /benchmark/cancel: post: tags: [Benchmark] operationId: cancelBenchmark summary: Cancel the running benchmark description: | Requests cancellation of the currently running benchmark. Temporary collections are cleaned up automatically. responses: '200': description: Cancellation requested content: application/json: schema: type: object properties: message: type: string '404': description: No benchmark is currently running content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /benchmark/report: get: tags: [Benchmark] operationId: getBenchmarkReport summary: Get the last benchmark report description: | Returns the full benchmark report from the most recent completed (or cancelled) run. Includes scenario results, driver metrics, explain plans, dataset/index fingerprints, and noisy-neighbour analysis with severity levels, remediation guidance, and a cross-correlated probable cause. responses: '200': description: Benchmark report content: application/json: schema: $ref: '#/components/schemas/BenchmarkReport' '404': description: No benchmark report available content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /benchmark/status: get: tags: [Benchmark] operationId: getBenchmarkStatus summary: Get benchmark subsystem status description: Returns whether a benchmark is running and summary info about the last report. responses: '200': description: Benchmark status content: application/json: schema: $ref: '#/components/schemas/BenchmarkStatus' # ═════════════════════════════════════════════════════════════════════ # Components # ═════════════════════════════════════════════════════════════════════ components: parameters: Skip: name: skip in: query schema: type: integer default: 0 minimum: 0 Top: name: top in: query schema: type: integer default: 10 minimum: 1 maximum: 100 schemas: # ── Shared ──────────────────────────────────────────────────── ErrorResponse: type: object required: [message] properties: message: type: string timestamp: type: string format: date-time # ── Import ──────────────────────────────────────────────────── StartBulkImportResponse: type: object properties: importId: type: string format: uuid sourceType: type: string message: type: string startedAt: type: string format: date-time ImportSummariesResponse: type: object properties: skip: type: integer top: type: integer count: type: integer imports: type: array items: type: object timestamp: type: string format: date-time ImportReport: type: object description: Full import report (structure varies). FileReportsResponse: type: object properties: importId: type: string format: uuid totalFiles: type: integer files: type: array items: type: object timestamp: type: string format: date-time PaginatedLineageEvents: type: object properties: collectionName: type: string recordId: type: string skip: type: integer top: type: integer count: type: integer totalEvents: type: integer events: type: array items: type: object GenerateRecordIdRequest: type: object required: [keyParts] properties: keyParts: type: array items: type: string example: ["NORTH", "F001"] GenerateRecordIdResponse: type: object properties: recordId: type: string description: URL-safe SHA256 hash (43 characters) keyParts: type: array items: type: string timestamp: type: string format: date-time DeleteCollectionResponse: type: object properties: collectionName: type: string success: type: boolean message: type: string deletedAtUtc: type: string format: date-time DeleteCollectionsResponse: type: object properties: deletedCollections: type: array items: type: string totalCount: type: integer success: type: boolean message: type: string deletedAtUtc: type: string format: date-time ClearDownStorageResponse: type: object properties: deletedKeys: type: array items: type: string totalDeleted: type: integer success: type: boolean message: type: string deletedAtUtc: type: string format: date-time # ── Query ───────────────────────────────────────────────────── QueryResult: type: object properties: collectionName: type: string count: type: integer totalCount: type: integer nullable: true data: type: array items: type: object executedAtUtc: type: string format: date-time # ── Cleanse — Operations ────────────────────────────────────── StartAnalysisResponse: type: object properties: operationId: type: string status: type: string message: type: string startedAtUtc: type: string format: date-time CancelAnalysisResponse: type: object properties: message: type: string requestedAtUtc: type: string format: date-time AnalysisRunsResponse: type: object properties: skip: type: integer top: type: integer count: type: integer runs: type: array items: $ref: '#/components/schemas/CleanseAnalysisOperationSummaryDto' timestamp: type: string format: date-time CleanseAnalysisOperationDto: type: object properties: id: type: string status: type: string enum: [Running, Completed, Failed, Cancelling, Cancelled] startedAtUtc: type: string format: date-time completedAtUtc: type: string format: date-time nullable: true progressPercentage: type: number format: double description: Weighted aggregate across all phases (0-100) statusDescription: type: string recordsAnalyzed: type: integer totalRecords: type: integer issuesFound: type: integer issuesResolved: type: integer error: type: string nullable: true durationMs: type: integer format: int64 nullable: true reportObjectKey: type: string nullable: true reportUrl: type: string nullable: true finalAverageRpm: type: number format: double nullable: true cancelledAtUtc: type: string format: date-time nullable: true progress: nullable: true description: Unified operation tree progress snapshot. Null until the first periodic flush. $ref: '#/components/schemas/OperationNode' CleanseAnalysisOperationSummaryDto: type: object properties: id: type: string status: type: string startedAtUtc: type: string format: date-time completedAtUtc: type: string format: date-time nullable: true progressPercentage: type: number format: double recordsAnalyzed: type: integer totalRecords: type: integer issuesFound: type: integer issuesResolved: type: integer durationMs: type: integer format: int64 nullable: true reportObjectKey: type: string nullable: true reportUrl: type: string nullable: true finalAverageRpm: type: number format: double nullable: true cancelledAtUtc: type: string format: date-time nullable: true RegenerateUrlResponse: type: object properties: operationId: type: string objectKey: type: string reportUrl: type: string regeneratedAtUtc: type: string format: date-time DeleteDataResponse: type: object properties: success: type: boolean collectionName: type: string deletedCount: type: integer format: int64 message: type: string deletedAtUtc: type: string format: date-time TestNotificationResponse: type: object properties: success: type: boolean recipient: type: string notificationId: type: string nullable: true message: type: string sentAtUtc: type: string format: date-time # ── Cleanse — Issues ────────────────────────────────────────── IssuesResponse: type: object properties: skip: type: integer top: type: integer count: type: integer totalCount: type: integer issues: type: array items: $ref: '#/components/schemas/IssueDto' timestamp: type: string format: date-time IssueDto: type: object properties: id: type: string issueCode: type: string ruleCode: type: string errorCode: type: string cph: type: string ctsLidFullIdentifier: type: string nullable: true description: type: string isActive: type: boolean isIgnored: type: boolean resolutionStatus: type: string assignedTo: type: string nullable: true operationId: type: string createdAtUtc: type: string format: date-time lastUpdatedAtUtc: type: string format: date-time IssueCommandResponse: type: object properties: issueId: type: string message: type: string timestamp: type: string format: date-time IssueHistoryResponse: type: object properties: issueId: type: string skip: type: integer top: type: integer count: type: integer entries: type: array items: $ref: '#/components/schemas/IssueHistoryEntryDto' timestamp: type: string format: date-time IssueHistoryEntryDto: type: object properties: id: type: string action: type: string performedBy: type: string nullable: true detail: type: string nullable: true timestamp: type: string format: date-time PerformedByRequest: type: object required: [performedBy] properties: performedBy: type: string UpdateResolutionStatusRequest: type: object required: [status, performedBy] properties: status: type: string description: "None, Todo, InProgress, Resolved" enum: [None, Todo, InProgress, Resolved] performedBy: type: string AssignIssueRequest: type: object required: [assignedTo, performedBy] properties: assignedTo: type: string performedBy: type: string # ── Cleanse Export ──────────────────────────────────────────── StartExportResponse: type: object required: [exportId, status, message] properties: exportId: type: string description: Unique identifier for the started export operation. Use this to poll progress. status: type: string description: Initial status of the export (Pending). message: type: string startedAtUtc: type: string format: date-time CleanseExportOperationDto: type: object description: Full details of an ad-hoc cleanse export operation. properties: id: type: string description: Unique export operation identifier. status: type: string enum: [Pending, Running, Completed, Failed] startedAtUtc: type: string format: date-time completedAtUtc: type: string format: date-time nullable: true progressPercentage: type: number format: double description: Progress percentage (0-100). statusDescription: type: string description: Human-readable status description. totalRecords: type: integer description: Total number of records to export. recordsExported: type: integer description: Number of records exported so far. reportObjectKey: type: string nullable: true description: S3 object key for the generated report. Populated on successful completion. reportUrl: type: string nullable: true description: Presigned URL to download the report. Use regenerate-url if expired. error: type: string nullable: true description: Error message with exception details if the export failed. durationMs: type: integer format: int64 nullable: true description: Total duration in milliseconds when complete. CleanseExportOperationSummaryDto: type: object description: Lightweight summary of an ad-hoc export operation for listing. properties: id: type: string status: type: string enum: [Pending, Running, Completed, Failed] startedAtUtc: type: string format: date-time completedAtUtc: type: string format: date-time nullable: true progressPercentage: type: number format: double totalRecords: type: integer recordsExported: type: integer reportObjectKey: type: string nullable: true reportUrl: type: string nullable: true durationMs: type: integer format: int64 nullable: true RegenerateExportUrlResponse: type: object required: [exportId, objectKey, reportUrl] properties: exportId: type: string description: The export operation ID. objectKey: type: string description: The S3 object key. reportUrl: type: string description: The new presigned URL. regeneratedAtUtc: type: string format: date-time ExportOperationsResponse: type: object required: [exports] properties: skip: type: integer top: type: integer count: type: integer description: Actual number of export operations returned. exports: type: array items: $ref: '#/components/schemas/CleanseExportOperationSummaryDto' timestamp: type: string format: date-time ExportErrorResponse: type: object required: [message] description: Error response for export operations. properties: message: type: string description: The error message with details. timestamp: type: string format: date-time # ── Holdings ────────────────────────────────────────────────── CtsCphHoldingResponse: type: object properties: lidFullIdentifier: type: string locationName: type: string nullable: true holding: type: object additionalProperties: true keepers: $ref: '#/components/schemas/QueryResult' timestamp: type: string format: date-time SamCphHoldingResponse: type: object properties: cph: type: string locationName: type: string nullable: true holding: type: object additionalProperties: true herd: $ref: '#/components/schemas/QueryResult' parties: $ref: '#/components/schemas/QueryResult' holders: $ref: '#/components/schemas/QueryResult' timestamp: type: string format: date-time # ── Throttle Policies ───────────────────────────────────────── ThrottlePolicy: type: object properties: name: type: string slug: type: string isActive: type: boolean isReadOnly: type: boolean settings: $ref: '#/components/schemas/ThrottlePolicySettings' ThrottlePolicySettings: type: object properties: ingestion: type: object cleanseAnalysis: type: object properties: pumpBatchSize: type: integer pumpDelayMs: type: integer recordIssueDelayMs: type: integer progressUpdateInterval: type: integer rpmWindowSeconds: type: integer cleanseExport: type: object properties: streamBatchSize: type: integer throttlingDelayMs: type: integer rpmWindowSeconds: type: integer issueDeactivation: type: object properties: batchSize: type: integer throttleDelayMs: type: integer rpmWindowSeconds: type: integer issueQuery: type: object CreateThrottlePolicyCommand: type: object required: [name] properties: name: type: string settings: $ref: '#/components/schemas/ThrottlePolicySettings' UpdateThrottlePolicyCommand: type: object properties: name: type: string settings: $ref: '#/components/schemas/ThrottlePolicySettings' # ── Benchmark ──────────────────────────────────────────────── BenchmarkConfig: type: object description: | Configuration for a benchmark run. All values have safe defaults designed to avoid overloading a shared production database. Omit the body entirely to use all defaults. properties: seedCount: type: integer default: 10000 description: Number of deterministic seed records to create. concurrency: type: integer default: 4 description: Maximum degree of parallelism for scenario execution. duration: type: string default: "00:03:00" description: "Total benchmark duration (TimeSpan format, e.g. '00:03:00'). Scenarios loop until this elapses." throttleDelay: type: string default: "00:00:00.0100000" description: "Delay injected between every Mongo operation (TimeSpan format, e.g. '00:00:00.010')." collectionPrefix: type: string default: "_benchmark_" description: Prefix applied to all temporary benchmark collections. StartBenchmarkResponse: type: object properties: message: type: string config: $ref: '#/components/schemas/BenchmarkConfig' BenchmarkStatus: type: object properties: isRunning: type: boolean hasReport: type: boolean lastReportStatus: type: string nullable: true description: "Status of the last report: Completed, Cancelled, or Failed." lastReportTimestamp: type: string format: date-time nullable: true BenchmarkReport: type: object description: | Full benchmark report. Designed to be JSON-serialised, written to disk, and compared across environments to detect noisy-neighbour impact. properties: environment: type: string description: Machine name where the benchmark ran. timestampUtc: type: string format: date-time config: $ref: '#/components/schemas/BenchmarkConfig' status: type: string description: "Completed, Cancelled, or Failed: ." totalElapsedSeconds: type: number format: double datasetFingerprints: type: array items: $ref: '#/components/schemas/DatasetFingerprint' indexFingerprints: type: array items: $ref: '#/components/schemas/IndexFingerprint' scenarioResults: type: array description: "Results for each scenario: PointLookup, RangeQuery, Aggregation, BulkWrite, MiniETL." items: $ref: '#/components/schemas/ScenarioResult' driverMetrics: $ref: '#/components/schemas/DriverMetrics' explainResults: type: array items: $ref: '#/components/schemas/ExplainResult' noisyNeighbourAnalysis: $ref: '#/components/schemas/NoisyNeighbourAnalysis' DatasetFingerprint: type: object description: Fingerprint of a benchmark collection's data distribution. properties: collectionName: type: string documentCount: type: integer format: int64 avgDocumentSizeBytes: type: number format: double p95DocumentSizeBytes: type: number format: double IndexFingerprint: type: object description: Fingerprint of an index on a benchmark collection. properties: collectionName: type: string indexName: type: string keyDefinition: type: object additionalProperties: true description: BSON key definition document. isUnique: type: boolean ScenarioResult: type: object description: Result of a single benchmark scenario. properties: scenarioName: type: string totalOperations: type: integer errorCount: type: integer elapsedSeconds: type: number format: double opsPerSecond: type: number format: double description: Wall-clock throughput. effectiveOpsPerSecond: type: number format: double description: Throughput excluding throttle wait time — pure Mongo performance. latency: $ref: '#/components/schemas/LatencyStats' LatencyStats: type: object description: Percentile latency statistics in milliseconds. properties: avgMs: type: number format: double p50Ms: type: number format: double p95Ms: type: number format: double p99Ms: type: number format: double minMs: type: number format: double maxMs: type: number format: double DriverMetrics: type: object description: Aggregated MongoDB driver command and connection-pool statistics. properties: commandLatency: type: object additionalProperties: $ref: '#/components/schemas/LatencyStats' description: "Per-command-type latency breakdown (e.g. find, update, aggregate)." commandFailures: type: object additionalProperties: type: integer description: Total count of failed commands by command name. connectionCheckoutWait: $ref: '#/components/schemas/LatencyStats' nullable: true description: Connection checkout wait-time statistics. checkoutFailures: type: integer description: Number of checkout failures (pool exhausted). connectionsCreated: type: integer connectionsClosed: type: integer poolClearedEvents: type: integer description: Number of times the driver was forced to reset the connection pool. ExplainResult: type: object description: Captured explain-plan output for a key query. properties: queryName: type: string description: "Name of the explained query: PointLookup, RangeQuery, or Aggregation." winningPlan: type: string description: JSON representation of the winning query plan. totalDocsExamined: type: integer format: int64 totalKeysExamined: type: integer format: int64 nReturned: type: integer format: int64 rawExplain: type: object additionalProperties: true nullable: true description: Full raw explain output (BSON document). NoisyNeighbourAnalysis: type: object description: | Diagnostic analysis for noisy-neighbour indicators. Cross-correlates connection pool health, command latency, and scenario errors to produce a probable root cause. properties: hasRedFlags: type: boolean description: True if any red flag fired. overallRisk: type: string enum: [None, Warning, Critical] description: Highest severity across all flags. probableCause: type: string nullable: true description: | Plain-language root-cause diagnosis based on cross-correlation of multiple flags. Null when no flags are present. flags: type: array items: $ref: '#/components/schemas/RedFlag' RedFlag: type: object description: A single noisy-neighbour diagnostic flag. properties: category: type: string description: Machine-readable category for comparison tooling. severity: type: string enum: [None, Warning, Critical] description: type: string description: Human-readable description of the problem observed. remediation: type: string description: Actionable guidance on what to investigate next. observedValue: type: number format: double description: The observed value that triggered the flag. threshold: type: number format: double description: The threshold that was exceeded. # ── Timings ────────────────────────────────────────────────── OperationNode: type: object nullable: true description: | Immutable snapshot of a single node in the operation tree. Combines timing, progress, and rate metrics into one self-describing recursive structure. The root node is named "total" with phase children (Analysis, Deactivation, Export) that may contain further sub-scopes. properties: name: type: string description: "Node name (e.g. \"total\", \"Analysis\", \"CTS Pump\", \"fetching\")." status: type: string enum: [not-started, in-progress, completed, failed, cancelled] description: Current status of this node. description: type: string nullable: true description: Human-readable description of what this node is doing. percentComplete: type: number format: double nullable: true description: "Progress percentage (0-100). Leaf nodes compute from processedCount/totalRecords; parent nodes use weighted average of children." processedCount: type: integer nullable: true description: Number of records processed so far in this scope. totalRecords: type: integer nullable: true description: Total records expected in this scope. elapsedMs: type: integer format: int64 description: Cumulative elapsed time in milliseconds. elapsed: type: string description: "Formatted elapsed time (e.g. \"00:07:42.3\")." projectedRemainingMs: type: integer format: int64 nullable: true description: Estimated remaining time in milliseconds based on current throughput. projectedEndTimeUtc: type: string format: date-time nullable: true description: Projected completion time based on current throughput. currentRecordsPerMinute: type: number format: double nullable: true description: Windowed records-per-minute rate (last 60 seconds). averageRecordsPerMinute: type: number format: double nullable: true description: Average records-per-minute rate since the scope started. children: type: array nullable: true description: Child nodes forming a recursive tree. items: $ref: '#/components/schemas/OperationNode' securitySchemes: ApiKeyAuth: type: apiKey in: header name: X-API-Key description: API key authentication. security: - ApiKeyAuth: []