openapi: 3.0.3 info: title: Feldera Input Connectors API description: "\nWith Feldera, users create data pipelines out of SQL programs.\nA SQL program comprises tables and views, and includes as well the definition of\ninput and output connectors for each respectively. A connector defines a data\nsource or data sink to feed input data into tables or receive output data\ncomputed by the views respectively.\n\n## Pipeline\n\nThe API is centered around the **pipeline**, which most importantly consists\nout of the SQL program, but also has accompanying metadata and configuration parameters\n(e.g., compilation profile, number of workers, etc.).\n\n* A pipeline is identified and referred to by its user-provided unique name.\n* The pipeline program is asynchronously compiled when the pipeline is first created or\n when its program is subsequently updated.\n* Pipeline deployment start is only able to proceed to provisioning once the program is successfully\n compiled.\n* A pipeline cannot be updated while it is deployed.\n\n## Concurrency\n\nEach pipeline has a version, which is incremented each time its core fields are updated.\nThe version is monotonically increasing. There is additionally a program version which covers\nonly the program-related core fields, and is used by the compiler to discern when to recompile.\n\n## Client request handling\n\n### Request outcome expectations\n\nThe outcome of a request is that it either fails (e.g., DNS lookup failed) without any response\n(no status code nor body), or it succeeds and gets back a response status code and body.\n\nIn case of a response, usually it is the Feldera endpoint that generated it:\n- If it is success (2xx), it will return whichever body belongs to the success response.\n- Otherwise, if it is an error (4xx, 5xx), it will return a Feldera error response JSON body\n which will have an application-level `error_code`.\n\nHowever, there are two notable exceptions when the response is not generated by the Feldera\nendpoint:\n- If the HTTP server, to which the endpoint belongs, encountered an issue, it might return\n 4xx (e.g., for an unknown endpoint) or 5xx error codes by itself (e.g., when it is initializing).\n- If the Feldera API server is behind a (reverse) proxy, the proxy can return error codes by itself,\n for example BAD GATEWAY (502) or GATEWAY TIMEOUT (504).\n\nAs such, it is not guaranteed that the (4xx, 5xx) will have a Feldera error response JSON body\nin these latter cases.\n\n### Error handling and retrying\n\nThe error type returned by the client should distinguish between the error responses generated\nby Feldera endpoints themselves (which have a Feldera error response body) and those that are\ngenerated by other sources.\n\nIn order for a client operation (e.g., `pipeline.resume()`) to be robust (i.e., not fail due to\na single HTTP request not succeeding) the client should use a retry mechanism if the operation\nis idempotent. The retry mechanism must however have a time limit, after which it times out.\nThis guarantees that the client operation is eventually responsive, which enables the script\nit is a part of to not hang indefinitely on Feldera operations and instead be able to decide\nby itself whether and how to proceed. If no response is returned, the mechanism should generally\nretry. When a response is returned, the decision whether to retry can generally depend on the status\ncode: especially the status codes 408, 502, 503 and 504 should be considered as transient errors.\nFiner grained retry decisions should be made by taking into account the application-level\n`error_code` if the response body was indeed a Feldera error response body.\n\n## Feldera client errors (4xx)\n\n_Client behavior:_ clients should generally return with an error when they get back a 4xx status\ncode, as it usually means the request will likely not succeed even if it is sent again. Certain\nrequests might make use of a timed retry mechanism when the client error is transient without\nrequiring any user intervention to overcome, for instance a transaction already being in progress\nleading to a temporary CONFLICT (409) error.\n\n- **BAD REQUEST (400)**: invalid user request (general).\n - _Example:_ the new pipeline name `example1@~` contains invalid characters.\n\n- **UNAUTHORIZED (401)**: the user is not authorized to issue the request.\n - _Example:_ an invalid API key is provided.\n\n- **NOT FOUND (404)**: a resource required to exist in order to process the request was not found.\n - _Example:_ a pipeline named `example` does not exist when trying to update it.\n\n- **CONFLICT (409)**: there is a conflict between the request and a relevant resource.\n - _Example:_ a pipeline named `example` already exists.\n - _Example:_ another transaction is already in process.\n\n## Feldera server errors (5xx)\n\n- **INTERNAL SERVER ERROR (500)**: the server is unexpectedly unable to process the request\n (general).\n - _Example:_ unable to reach the database.\n - _Client behavior:_ immediately return with an error.\n\n- **NOT IMPLEMENTED (501)**: the server does not implement functionality required to process the\n request.\n - _Example:_ making a request to an enterprise-only endpoint in the OSS edition.\n - _Client behavior:_ immediately return with an error.\n\n- **SERVICE UNAVAILABLE (503)**: the server is not (yet) able to process the request.\n - _Example:_ pausing a pipeline which is still provisioning.\n - _Client behavior:_ depending on the type of request, client may use a timed retry mechanism.\n\n## Feldera error response body\n\nWhen the Feldera API returns an HTTP error status code (4xx, 5xx), the body will contain the\nfollowing JSON object:\n\n```json\n{\n \"message\": \"Human-readable explanation.\",\n \"error_code\": \"CodeSpecifyingError\",\n \"details\": {\n\n }\n}\n```\n\nIt contains the following fields:\n- **message (string)**: human-readable explanation of the error that occurred and potentially\n hinting what can be done about it.\n- **error_code (string)**: application-level code about the error that occurred, written in CamelCase.\n For example: `UnknownPipelineName`, `DuplicateName`, `PauseWhileNotProvisioned`, ... .\n- **details (object)**: JSON object corresponding to the `error_code` with fields that provide\n details relevant to it. For example: if a name is unknown, a field with the unknown name in\n question.\n" contact: name: Feldera Team email: dev@feldera.com license: name: MIT OR Apache-2.0 version: 0.323.0 tags: - name: Input Connectors paths: /v0/pipelines/{pipeline_name}/completion_status: get: tags: - Input Connectors summary: Check Completion Status description: 'Check the status of a completion token returned by the `/ingress` or `/completion_token` endpoint.' operationId: completion_status parameters: - name: pipeline_name in: path description: Unique pipeline name required: true schema: type: string - name: token in: query description: Completion token returned by the '/ingress' or '/completion_status' endpoint. required: true schema: type: string responses: '200': description: The pipeline has finished processing inputs associated with the provided completion token. content: application/json: schema: $ref: '#/components/schemas/CompletionStatusResponse' '202': description: The pipeline is still processing inputs associated with the provided completion token. content: application/json: schema: $ref: '#/components/schemas/CompletionStatusResponse' '400': description: An invalid completion token was provided content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Pipeline with that name does not exist content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: message: Unknown pipeline name 'non-existent-pipeline' error_code: UnknownPipelineName details: pipeline_name: non-existent-pipeline '410': description: Completion token was created by a previous incarnation of the pipeline and is not valid for the current incarnation. This indicates that the pipeline was suspended and resumed from a checkpoint or restarted after a failure. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '500': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '503': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Disconnected during response: value: message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Pipeline is currently unavailable: value: message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again Pipeline is not deployed: value: message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline" error_code: PipelineInteractionNotDeployed details: pipeline_name: my_pipeline status: Stopped desired_status: Provisioned Response timeout: value: message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)' security: - JSON web token (JWT) or API key: [] /v0/pipelines/{pipeline_name}/ingress/{table_name}: post: tags: - Input Connectors summary: Insert Data description: 'Push data to a SQL table. The client sends data encoded using the format specified in the `?format=` parameter as a body of the request. The contents of the data must match the SQL table schema specified in `table_name` The pipeline ingests data as it arrives without waiting for the end of the request. Successful HTTP response indicates that all data has been ingested successfully. On success, returns a completion token that can be passed to the ''/completion_status'' endpoint to check whether the pipeline has fully processed the data.' operationId: http_input parameters: - name: pipeline_name in: path description: Unique pipeline name required: true schema: type: string - name: table_name in: path description: SQL table name. Unquoted SQL names have to be capitalized. Quoted SQL names have to exactly match the case from the SQL program. required: true schema: type: string - name: force in: query description: When `true`, push data to the pipeline even if the pipeline is paused. The default value is `false` required: true schema: type: boolean - name: format in: query description: Input data format, either `csv' or 'json'. required: true schema: type: string - name: array in: query description: Set to `true` if updates in this stream are packaged into JSON arrays (used in conjunction with `format=json`). The default values is `false`. required: false schema: type: boolean nullable: true - name: update_format in: query description: JSON data change event format (used in conjunction with `format=json`). The default value is 'insert_delete'. required: false schema: allOf: - $ref: '#/components/schemas/JsonUpdateFormat' nullable: true requestBody: description: Input data in the specified format content: text/plain: schema: type: string required: true responses: '200': description: Data successfully delivered to the pipeline. The body of the response contains a completion token that can be passed to the '/completion_status' endpoint to check whether the pipeline has fully processed the data. content: application/json: schema: $ref: '#/components/schemas/CompletionTokenResponse' '400': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '404': description: Pipeline and/or table with that name does not exist content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Pipeline with that name does not exist: value: message: Unknown pipeline name 'non-existent-pipeline' error_code: UnknownPipelineName details: pipeline_name: non-existent-pipeline '500': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '503': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Disconnected during response: value: message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Pipeline is currently unavailable: value: message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again Pipeline is not deployed: value: message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline" error_code: PipelineInteractionNotDeployed details: pipeline_name: my_pipeline status: Stopped desired_status: Provisioned Response timeout: value: message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)' security: - JSON web token (JWT) or API key: [] /v0/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/completion_token: get: tags: - Input Connectors summary: Get Completion Token description: 'Generate a completion token for an input connector. Returns a token that can be passed to the `/completion_status` endpoint to check whether the pipeline has finished processing all inputs received from the connector before the token was generated.' operationId: completion_token parameters: - name: pipeline_name in: path description: Unique pipeline name required: true schema: type: string - name: table_name in: path description: SQL table name. Unquoted SQL names have to be capitalized. Quoted SQL names have to exactly match the case from the SQL program. required: true schema: type: string - name: connector_name in: path description: Unique input connector name required: true schema: type: string responses: '200': description: Completion token that can be passed to the '/completion_status' endpoint. content: application/json: schema: $ref: '#/components/schemas/CompletionTokenResponse' '404': description: Specified pipeline, table, or connector does not exist content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: message: Unknown pipeline name 'non-existent-pipeline' error_code: UnknownPipelineName details: pipeline_name: non-existent-pipeline '500': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '503': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Disconnected during response: value: message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Pipeline is currently unavailable: value: message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again Pipeline is not deployed: value: message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline" error_code: PipelineInteractionNotDeployed details: pipeline_name: my_pipeline status: Stopped desired_status: Provisioned Response timeout: value: message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)' security: - JSON web token (JWT) or API key: [] /v0/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/stats: get: tags: - Input Connectors summary: Get Input Status description: Retrieve the status of an input connector. operationId: get_pipeline_input_connector_status parameters: - name: pipeline_name in: path description: Unique pipeline name required: true schema: type: string - name: table_name in: path description: Unique table name required: true schema: type: string - name: connector_name in: path description: Unique input connector name required: true schema: type: string responses: '200': description: Input connector status retrieved successfully content: application/json: schema: $ref: '#/components/schemas/InputEndpointStatus' '404': description: Pipeline, table and/or input connector with that name does not exist content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Pipeline with that name does not exist: value: message: Unknown pipeline name 'non-existent-pipeline' error_code: UnknownPipelineName details: pipeline_name: non-existent-pipeline '500': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '503': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Disconnected during response: value: message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Pipeline is currently unavailable: value: message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again Pipeline is not deployed: value: message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline" error_code: PipelineInteractionNotDeployed details: pipeline_name: my_pipeline status: Stopped desired_status: Provisioned Response timeout: value: message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)' security: - JSON web token (JWT) or API key: [] /v0/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/{action}: post: tags: - Input Connectors summary: Control Input Connector description: 'Start (resume) or pause the input connector. The following values of the `action` argument are accepted: `start` and `pause`. Input connectors can be in either the `Running` or `Paused` state. By default, connectors are initialized in the `Running` state when a pipeline is deployed. In this state, the connector actively fetches data from its configured data source and forwards it to the pipeline. If needed, a connector can be created in the `Paused` state by setting its [`paused`](https://docs.feldera.com/connectors/#generic-attributes) property to `true`. When paused, the connector remains idle until reactivated using the `start` command. Conversely, a connector in the `Running` state can be paused at any time by issuing the `pause` command. The current connector state can be retrieved via the `GET /v0/pipelines/{pipeline_name}/stats` endpoint. Note that only if both the pipeline *and* the connector state is `Running`, is the input connector active. ```text Pipeline state Connector state Connector is active? -------------- --------------- -------------------- Paused Paused No Paused Running No Running Paused No Running Running Yes ```' operationId: post_pipeline_input_connector_action parameters: - name: pipeline_name in: path description: Unique pipeline name required: true schema: type: string - name: table_name in: path description: SQL table name required: true schema: type: string - name: connector_name in: path description: Input connector name required: true schema: type: string - name: action in: path required: true schema: type: string responses: '200': description: Action has been processed '404': description: Pipeline, table and/or input connector with that name does not exist content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Pipeline with that name does not exist: value: message: Unknown pipeline name 'non-existent-pipeline' error_code: UnknownPipelineName details: pipeline_name: non-existent-pipeline '500': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' '503': description: '' content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: Disconnected during response: value: message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Pipeline is currently unavailable: value: message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again Pipeline is not deployed: value: message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline" error_code: PipelineInteractionNotDeployed details: pipeline_name: my_pipeline status: Stopped desired_status: Provisioned Response timeout: value: message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"' error_code: PipelineInteractionUnreachable details: pipeline_name: my_pipeline request: /pause error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)' security: - JSON web token (JWT) or API key: [] components: schemas: ShortEndpointConfig: type: object description: Schema definition for endpoint config that only includes the stream field. required: - stream properties: stream: type: string description: The name of the stream. CompletionStatusResponse: type: object description: Response to a completion token status request. required: - status properties: status: $ref: '#/components/schemas/CompletionStatus' step: type: integer format: int64 description: 'If all of the data associated with the token has been processed through the pipeline, this is the final step that includes at least one record. When the pipeline''s `total_completed_steps` reaches this value, the token has been completed. This is `None` before the data associated with the token has been processed through the pipeline.' nullable: true minimum: 0 InputEndpointStatus: type: object description: Input endpoint status information. required: - endpoint_name - config - metrics - paused - barrier properties: barrier: type: boolean description: Endpoint is currently a barrier to checkpointing and suspend. completed_frontier: allOf: - $ref: '#/components/schemas/CompletedWatermark' nullable: true config: $ref: '#/components/schemas/ShortEndpointConfig' endpoint_name: type: string description: Endpoint name. fatal_error: type: string description: The first fatal error that occurred at the endpoint. nullable: true health: allOf: - $ref: '#/components/schemas/ConnectorHealth' nullable: true metrics: $ref: '#/components/schemas/InputEndpointMetrics' parse_errors: type: array items: $ref: '#/components/schemas/ConnectorError' description: Recent parse errors on this endpoint. nullable: true paused: type: boolean description: Endpoint has been paused by the user. transport_errors: type: array items: $ref: '#/components/schemas/ConnectorError' description: Recent transport errors on this endpoint. nullable: true CompletedWatermark: type: object description: A watermark that has been fully processed by the pipeline. required: - metadata - ingested_at - processed_at - completed_at properties: completed_at: type: string format: date-time description: Timestamp when all outputs produced from this input have been pushed to all output endpoints. ingested_at: type: string format: date-time description: Timestamp when the data was ingested from the wire. metadata: type: object description: Metadata that describes the position in the input stream (e.g., Kafka partition/offset pairs). processed_at: type: string format: date-time description: Timestamp when the data was processed by the circuit. CompletionTokenResponse: type: object description: Response to a completion token creation request. required: - token properties: token: type: string description: 'Completion token. An opaque string associated with the current position in the input stream generated by an input connector. Pass this string to the `/completion_status` endpoint to check whether all inputs associated with the token have been fully processed by the pipeline.' ConnectorHealthStatus: type: string enum: - Healthy - Unhealthy ErrorResponse: type: object description: Information returned by REST API endpoints on error. required: - message - error_code - details properties: details: description: 'Detailed error metadata. The contents of this field is determined by `error_code`.' error_code: type: string description: Error code is a string that specifies this error type. example: CodeSpecifyingErrorType message: type: string description: Human-readable error message. example: Explanation of the error that occurred. ConnectorHealth: type: object required: - status properties: description: type: string nullable: true status: $ref: '#/components/schemas/ConnectorHealthStatus' InputEndpointMetrics: type: object description: Performance metrics for an input endpoint. required: - total_bytes - total_records - buffered_records - buffered_bytes - num_transport_errors - num_parse_errors - end_of_input properties: buffered_bytes: type: integer format: int64 description: Number of bytes currently buffered by the endpoint (not yet consumed by the circuit). minimum: 0 buffered_records: type: integer format: int64 description: Number of records currently buffered by the endpoint (not yet consumed by the circuit). minimum: 0 end_of_input: type: boolean description: True if end-of-input has been signaled. num_parse_errors: type: integer format: int64 description: Number of parse errors. minimum: 0 num_transport_errors: type: integer format: int64 description: Number of transport errors. minimum: 0 total_bytes: type: integer format: int64 description: Total bytes pushed to the endpoint since it was created. minimum: 0 total_records: type: integer format: int64 description: Total records pushed to the endpoint since it was created. minimum: 0 ConnectorError: type: object required: - timestamp - index - message properties: index: type: integer format: int64 description: 'Sequence number of the error. The client can use this field to detect gaps in the error list reported by the pipeline. When the connector reports a large number of errors, the pipeline will only preserve and report the most recent errors of each kind.' minimum: 0 message: type: string description: Error message. tag: type: string description: 'Optional tag for the error. The tag is used to group errors by their type.' nullable: true timestamp: type: string format: date-time description: Timestamp when the error occurred, serialized as RFC3339 with microseconds. JsonUpdateFormat: type: string description: 'Supported JSON data change event formats. Each element in a JSON-formatted input stream specifies an update to one or more records in an input table. We support several different ways to represent such updates. ### `InsertDelete` Each element in the input stream consists of an "insert" or "delete" command and a record to be inserted to or deleted from the input table. ```json {"insert": {"column1": "hello, world!", "column2": 100}} ``` ### `Weighted` Each element in the input stream consists of a record and a weight which indicates how many times the row appears. ```json {"weight": 2, "data": {"column1": "hello, world!", "column2": 100}} ``` Note that the line above would be equivalent to the following input in the `InsertDelete` format: ```json {"insert": {"column1": "hello, world!", "column2": 100}} {"insert": {"column1": "hello, world!", "column2": 100}} ``` Similarly, negative weights are equivalent to deletions: ```json {"weight": -1, "data": {"column1": "hello, world!", "column2": 100}} ``` is equivalent to in the `InsertDelete` format: ```json {"delete": {"column1": "hello, world!", "column2": 100}} ``` ### `Debezium` Debezium CDC format. Refer to [Debezium input connector documentation](https://docs.feldera.com/connectors/sources/debezium) for details. ### `Snowflake` Uses flat structure so that fields can get parsed directly into SQL columns. Defines three metadata fields: * `__action` - "insert" or "delete" * `__stream_id` - unique 64-bit ID of the output stream (records within a stream are totally ordered) * `__seq_number` - monotonically increasing sequence number relative to the start of the stream. ```json {"PART":1,"VENDOR":2,"EFFECTIVE_SINCE":"2019-05-21","PRICE":"10000","__action":"insert","__stream_id":4523666124030717756,"__seq_number":1} ``` ### `Raw` This format is suitable for insert-only streams (no deletions). Each element in the input stream contains a record without any additional envelope that gets inserted in the input table.' enum: - insert_delete - weighted - debezium - snowflake - raw - redis CompletionStatus: type: string description: Completion token status returned by the `/completion_status` endpoint. enum: - complete - inprogress securitySchemes: JSON web token (JWT) or API key: type: http scheme: bearer bearerFormat: JWT description: "Use a JWT token obtained via an OAuth2/OIDC\n login workflow or an API key obtained via\n the `/v0/api-keys` endpoint."