# Copyright 2023-2025 Broadcom
# SPDX-License-Identifier: Apache-2.0

openapi: 3.0.3

servers:
  - url: http://localhost:8092
    description: quickstart-vdk default local setup

info:
  title: Versatile Data Kit Control Service API
  version: '1.0'
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
  description: "The Data Jobs API of Versatile Data Kit Control Service.
    Data Jobs allows Data Engineers to implement automated pull ingestion (E in ELT)
    and batch data transformation into a database (T in ELT).
    See also https://github.com/vmware/versatile-data-kit/wiki/Introduction

    The API has resource-oriented URLs, JSON-encoded responses, and uses standard HTTP response codes, authentication, and verbs.
    The API enables creating, deploying, managing and executing Data Jobs in the runtime environment.<br>
    <br>
    ![](https://github.com/vmware/versatile-data-kit/wiki/vdk-data-job-lifecycle-state-diagram.png)
    <br>
    The API reflects the usual Data Job Development lifecycle:<br>
    <li> Create a new data job (webhook to further configure the job, e.g authorize its creation, setup permissions, etc).
    <li> Download keytab. Develop and run the data job locally.
    <li> Deploy the data job in cloud runtime environment to run on a scheduled basis.
    <br><br>
    If Authentication is enabled, pass OAuth2 access token in HTTP header 'Authorization: Bearer [access-token-here]' (https://datatracker.ietf.org/doc/html/rfc6750).
    <br

    The API promotes some best practices (inspired by https://12factor.net):
    <li> Explicitly declare and isolate dependencies.
    <li> Strict separation of configurations from code. Configurations vary substantially across deploys, code does not.
    <li> Separation between the build, release/deploy, and run stages.
    <li> Data Jobs are stateless and share-nothing processes. Any data that needs to be persisted must be stored in a stateful backing service (e.g IProperties).
    <li> Implementation is assumed to be atomic and idempotent - should be OK for a job to fail somewhere in the middle; subsequent restart should not cause data corruption.
    <li> Keep development, staging, and production as similar as possible.
    <br><br>
    <b>API Evolution</b><br>
    In the following sections, there are some terms that have a special meaning in the context of the APIs.
    <br><br>
    <li> <i>Stable</i> - The implementation of the API has been battle-tested (has been in production for some time).
                         The API is a subject to semantic versioning model and will follow deprecation policy.
    <li> <i>Experimental</i> - May disappear without notice and is not a subject to semantic versioning.
                               Implementation of the API is not considered stable nor well tested.
                               Generally this is given to clients to experiment within testing environment. Must not be used in production.
    <li> <i>Deprecated</i> - API is expected to be removed within next one or two major version upgrade.
                             The deprecation notice/comment will say when the API will be removed and what alternatives should be used instead."

paths:
  '/data-jobs/for-team/{team_name}/info':
    get:
      tags:
        - Data Jobs Service
      responses:
        '200':
          $ref: '#/components/responses/Info'
      operationId: info
      summary: Get API and Data Jobs Service info, list of supported python versions
      description: <i>(Introduced in v1.0)</i> | Get API and Data Jobs service information
      parameters:
        -  name: team_name
           description: Team Name
           schema:
             type: string
           in: path
           required: true

  '/data-jobs/for-team/{team_name}/jobs':
    get:
      operationId: jobsQuery
      summary: Query Data Jobs details using GraphQL
      tags:
        - Data Jobs
      description: |
        List data Jobs with GraphQL like query. By choosing which field to be returned you can control the output.
        You can learn more about the GraphQL queries by visiting [GraphQL official website](https://graphql.org/learn/queries/)
        Query should be provided as GET parameter, not by POST body. Don't worry about the spaces and tabs
        Keep in mind that each aditional field <b>could make query response time slower, for instance deployments</b>,
        it's best if you request only what you need <br/><br/>

        The <b>pageNumber</b> and <b>pageSize</b> arguments are required! Page number should be a <b>number greater than 1</b>,
        and pageSize <b>should be greater than 1</b> (per page)
        Check the latest example for the full list of supported query fields.<br/><br/>

        Simplest query that you could make is to fetch the job names
        ```
        {
          jobs(pageNumber: 1, pageSize: 25) {
            content {
              jobName
            }
          }
        }
        ```

        You could also use filtering and sorting function. Filter object has <b>property, pattern and sort</b> fields.<br/>
        * <b>property</b> points out which field you want to filter, if you point out some other field that is not supported,
        an invalid response will be return.<br/>
        * <b>pattern</b> should be a non-empty string which the provided property should contains [ignoring cases], for instance: `starshot` pattern will match
        <b>import-starshot-sql, StarShot-servers and notify-starshot</b> job names, but it won't match <b>stars-shot-daily-prune`</b>
        If a pattern string is not provided, then you must atleast provide the property field<br/>
        * <b>sort</b> should be an enum value - ASC (ascending) or DESC (descending) option [not required, default is ASC]
        Multiple filters could be applied, but <b>maximum one should contain sorting</b>!
        ```
        {
          jobs(
            pageNumber: 1,
            pageSize: 25,
            filter: [{
              property: "jobName",
              pattern: "starshot",
              sort: DESC
            }],
          ) {
            content {
              jobName
            }
          }
        }
        ```

        You could also search for a string into the properties that you are requesting, for instance:
        This query will search for job names, team names and descriptions which contains the provided "starshot" string
        ```
        {
          jobs(
            pageNumber: 1,
            pageSize: 25,
            search: "starshot"
          ) {
            content {
              jobName,
              config {
                team
                description
              }
            }
          }
        }
        ```

        Data jobs execution could also be searched by providing arguments to the <b>execution</b> field.
        Same as parent query arguments, the <b>pageNumber</b> and <b>pageSize</b> arguments are required! Page number should be a <b>number greater than 1</b>,
        and pageSize <b>should be between 1 and 100 results</b> (per page). You can also <b>filter</b> using the similar object structure as the parent query,
        but currently <b>filtering is not supported</b>, you can only provide field for sorting.
        This query will search
        ```
        {
          jobs(
            pageNumber: 1,
            pageSize: 25,
          ) {
            content {
              jobName,
              deployments {
                id
                executions(
                  pageNumber: 1,
                  pageSize: 5,
                  filter: [{
                    teamNameIn: ["starshot"]
                  }],
                  order: {
                    property: "startTime",
                      direction: DESC
                  }
                ) {
                  id
                  status
                  startTime
                  endTime
                {
              }
            }
          }
        }
        ```

        Full example of currently available for fetching fields. Note that if you combine searching and filtering, first
        it will apply filters and then within filtered jobs it will apply the search, vice versa is currently not supported:
        ```
        {
          jobs(
            pageNumber: 1,
            pageSize: 25,
            search: "daily",
            filter: [{
              property: "jobName",
              pattern: "import-sql",
            },{
              property: "team",
              pattern: "starshot",
              sort: DESC
            },{
              property: "deployments.enabled",
              pattern: "enabled",
            }],
          ) {
            content {
              jobName
              config {
                team
                description
                sourceUrl
                schedule {
                  scheduleCron
                  nextRunEpochSeconds
                }
                contacts {
                  notifiedOnJobFailureUserError
                  notifiedOnJobFailurePlatformError
                  notifiedOnJobSuccess
                  notifiedOnJobDeploy
                }
              }
              deployments {
                id
                enabled
                jobVersion
                mode
                executions(
                  pageNumber: 1,
                  pageSize: 25,
                  filter: [{
                    teamNameIn: ["starshot"]
                  }],
                  order: {
                    property: "startTime",
                      direction: DESC
                  }
                ) {
                  id
                  type
                  status
                  message
                  startTime
                  endTime
                  opId
                  vkdVersion
                  jobVersion
                  jobSchedule
                  resourcesCpuRequest
                  resourcesCpuLimit
                  resourcesMemoryRequest
                  resourcesMemoryLimit
                  deployedDate
                  deployedBy
                  startedBy
                  logsUrl
                }
              }
            }
            totalPages
            totalItems
            }
          }
        }
        ```

      parameters:
        - name: team_name
          description: The Team which owns the Data Job
          schema:
            type: string
          in: path
          required: true
        - name: query
          description: Request a GraphQL-like query.
          schema:
            type: string
            example: "{
                jobs(pageNumber: 1, pageSize: 20, filter: []) {
                  content {
                    jobName
                    config {
                      team
                      description
                      schedule {
                        scheduleCron
                        nextRunEpochSeconds
                      }
                      sourceUrl
                      contacts {
                        notifiedOnJobFailureUserError
                        notifiedOnJobFailurePlatformError
                        notifiedOnJobSuccess
                        notifiedOnJobDeploy
                      }
                    }
                  }
                  totalPages
                  totalItems
                }
              }"
            default: "{
                jobs(pageNumber: 1, pageSize: 20, filter: []) {
                  content {
                    jobName
                    config {
                      team
                      description
                      schedule {
                        scheduleCron
                      }
                    }
                  }
                  totalPages
                  totalItems
                }
              }"
          required: false
          in: query
        - name: operation_name
          description: If the query is specified and contains several named operations, an operation_name query parameter can be used to control which one should be executed
          schema:
            type: string
            example: jobs
          required: false
          in: query
        - name: variables
          description: If query is specified then variables can be sent as a JSON-encoded string in an additional query parameter called variables
          schema:
            type: string
            example: |
              {
                "pageSize": "10",
                "pageNumber": "1"
              }
          required: false
          in: query
      responses:
        '200':
          $ref: '#/components/responses/DataJobQueryResponse'
        '400':
          $ref: '#/components/responses/DataJobQueryResponseWithError'
    post:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJob'
        required: true
      parameters:
        - name: team_name
          description: The Team which owns the Data Job
          schema:
            type: string
          in: path
          required: true
        - in: query
          name: name
          schema:
            type: string
          required: false
          description: The Name of the Data Job
      tags:
        - Data Jobs
      responses:
        '201':
          description: Created successfully.
        '400':
          $ref: '#/components/responses/Error'
        '409':
          $ref: '#/components/responses/Error'
      operationId: dataJobCreate
      summary: Creates a new Data Job | (Stable)
      description: |
        <i>(Introduced in v1.0)</i> | Creates a new Data Job.<br>
        The name of the Data job should comply to the following naming convention:<br>
        <li>should start with a letter
        <li>50 characters at most
        <li>5 characters at least
        <li>should contain only lowercase alphanumeric symbols and dash [a-z0-9\-]
        <br><br>
        When succesfully created clients can download the keytab associated with the Data Job to run it locally.<br>
        To deploy so that it can be executed reguarly in the runtime environment use /data-jobs/for-team/{team_name}/jobs/{job_name}/deployments API.

  '/data-jobs/for-team/{team_name}/jobs/{job_name}':
    summary: |
      A Data Job is a sequence of Python and/or SQL scripts and configurations, executed by Versatile Data Kit SDK.<br>
      See more in https://github.com/vmware/versatile-data-kit/wiki/Getting-Started
    get:
      tags:
        - Data Jobs
      responses:
        '200':
          $ref: '#/components/responses/DataJob'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobRead
      summary: Retrieves details of an existing Data Job by specifying the name of the Data Job. | (Stable)
      description: <i>(Introduced in v1.0)</i>
    put:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJob'
        required: true
      tags:
        - Data Jobs
      responses:
        '201':
          description: Created successfully.
        '204':
          description: Updated successfully.
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobUpdate
      summary: Update Data Job. | (Stable)
      description: <i>(Introduced in v1.0)</i>
    delete:
      tags:
        - Data Jobs
      responses:
        '200':
          description: Deleted successfully.
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobDelete
      summary: Delete Data Job | (Stable)
      description: |
        <i>(Introduced in v1.0)</i> | Delete Data Job including its state (properties) and deployments.
        Currently executing Data Jobs will be left to finish.
    parameters:
      - name: team_name
        description: Team Name
        schema:
          type: string
        in: path
        required: true
      - name: job_name
        description: Data Job Name
        schema:
          type: string
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/keytab':
    summary: |
      List of credentials to be used by a data job - for example kerberos key to authenticate to the database.
      Note: will be renamed to /credentials as it seems more clear/generic
    get:
      tags:
        - Data Jobs
      responses:
        '200':
          description: Success
          content:
            application/octet-stream:
              schema:
                type: string
                format: binary
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobKeytabDownload
      summary: Get data job keytab. | (Stable)
      description: <i>(Introduced in v1.0)</i>
    parameters:
      - name: team_name
        description: Team Name
        schema:
          type: string
        in: path
        required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/team/{new_team}':
    put:
      tags:
        - Data Jobs
      responses:
        '200':
          description: Team changed successfully.
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobTeamUpdate
      summary: Update API for Data Jobs team | (Stable)
      description: <i>(Introduced in v1.0)</i> | Update API for Data Jobs team
      parameters:
        -  name: team_name
           description: Current Team Name.
           schema:
             type: string
           in: path
           required: true
        -  name: new_team
           description: New Team Name.
           schema:
             type: string
           in: path
           required: true
        - name: job_name
          description: Data Job Name.
          schema:
            type: string
          in: path
          required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments':
    summary: |
      Deployments are requests to deploy a specific (code version of a Data Job to be executed in the runtime environment (Kubernetes).
      Users can deploy the same (code) version of a Data Job with different properties.<br>
      Note: Currently deployment_id and mode are not implemented. Only a single active deployment of a job is possible.
    get:
      tags:
        - "Data Jobs Deployment"
      parameters:
        - name: deployment_id   # Ignored. Only one deployment currently supported
          description: Get the Deployment with this id (e.g production, staging, dev)
          schema:
            type: string
          in: query
        - name: mode     # Ignored. Only one mode currently supported
          description: Get Deployments only for this mode.
          schema:
            $ref: '#/components/schemas/DataJobMode'
          in: query
      responses:
        '200':
          $ref: '#/components/responses/DeploymentArray'
        '404':
          $ref: '#/components/responses/Error'
      operationId: deploymentList
      summary: Get Data Job deployments. | (Stable)
      description: <i>(Introduced in v1.0)</i>
    post:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJobDeployment'
      parameters:
        - name: send_notification
          description: |
            Allows administrators to disable sending notification during maintance window for platfrom errors -
            e.g if all jobs are being re-deployed due to migration or after recovering from an outage.
            Monitoring data will still be popoulated so deployment status can be monitored by admins.
          schema:
            type: boolean
            default: True
          required: false
          in: query
      tags:
        - "Data Jobs Deployment"
      responses:
        '202':
          description: Update accepted successfully.
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: deploymentUpdate
      summary: Creates or updates a deployment of a Data Job. | (Stable)
      description: |
        <i>(Introduced in v1.0)</i> | Creates or updates a deployment of a Data Job.
        The API returns before the actual Job is deployed (the operation is asynchrounous).
        Depending on settings specified during Data Job creation, clients will get an e-mail in case of success or failure of the deployment.
        If you need to change only deployment settings without changing the job version, use PATCH deployments request which is synchronous.
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}':
    summary: Manage single deployment
    get:
      tags:
        - "Data Jobs Deployment"
      responses:
        '200':
          $ref: '#/components/responses/Deployment'
        '404':
          $ref: '#/components/responses/Error'
      operationId: deploymentRead
      summary: Get Data Job deployments. | (Stable)
      description: <i>(Introduced in v1.0)</i>
    delete:
      tags:
        - "Data Jobs Deployment"
      responses:
        '202':
          description: Deletion accepted successfully.
        '404':
          $ref: '#/components/responses/Error'
      operationId: deploymentDelete
      summary: |
        Delete Deployment of a Data Job.
        Currently executing Data Job will be left to finish.  | (Stable)
      description: <i>(Introduced in v1.0)</i>
    patch:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJobDeployment'
        required: true
      tags:
        - "Data Jobs Deployment"
      responses:
        '202':
          description: Update accepted successfully.
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: deploymentPatch
      summary: |
          Patch a deployment of a Data Job. Use it to change the configuration of a data job.
          For example: to enable or disable deployment, to change the vdk version.
          The operation is guranteed to be synchrounous so it cannot be used to deploy new version of a data job -
          job_version cannot be changed using PATCH. Use POST .../deployments for this. | (Stable)
      description: <i>(Introduced in v1.0)</i>
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data job Name.
        schema:
          type: string
        in: path
        required: true
      - name: deployment_id     # Only a single deployment. Can pass 'dummy' for now.
        description: Data job deployment id.
        schema:
          $ref: '#/components/schemas/DataJobDeploymentId'
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/executions':
    summary: Execution API is used to query recent executions of Data Jobs and to trigger new executions.
    get:
      tags:
        - Data Jobs Execution
      responses:
        '200':
          $ref: '#/components/responses/ExecutionArray'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobExecutionList
      summary: Get Data Jobs (recent) executions.
      description: <i>(Introduced in v1.0)</i>
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true
      - name: execution_status
        description: Execution status
        schema:
          type: array
          items:
            type: string
        example: submitted
        in: query

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/executions':
    summary: Execution API is used to query recent executions of Data Jobs and to trigger new executions.
    get:
      tags:
        - Data Jobs Execution
      responses:
        '200':
          $ref: '#/components/responses/ExecutionArray'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobDeploymentExecutionList
      summary: Get Data Jobs (recent) executions.
      description: <i>(Introduced in v1.0)</i>
      parameters:
        - name: execution_status
          description: Execution status
          schema:
            type: array
            items:
              type: string
          example: submitted
          in: query
    post:
      tags:
        - Data Jobs Execution
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJobExecutionRequest'
      responses:
        '202':
          description: Execution triggered successfully.
          headers:
            Location:
              description: URI of the execution
              schema:
                type: string
        '400':
          $ref: '#/components/responses/Error'
        '409':
          $ref: '#/components/responses/Error'
      operationId: dataJobExecutionStart
      summary: Trigger Data Job Execution.
      description: |
        <i>(Introduced in v1.0)</i> | Trigger new execution of a Data Job.
        If Data Job deployment is currently being executed it will return an error (409 Conflict).
        Only deployed Data Jobs can be executed. Currently the Data Job arguments ('args') are ignored.
        Note: in the future it will be possible to trigger different deployments.
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true
      - name: deployment_id
        description: Data job deployment ID. Currently it is being ignored as multiple deployments are not implemented.
        schema:
          $ref: '#/components/schemas/DataJobDeploymentId'
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/executions/{execution_id}':
    summary: Data Job Execution details.
    get:
      tags:
        - Data Jobs Execution
      responses:
        '200':
          $ref: '#/components/responses/Execution'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobExecutionRead
      summary: Get Data Job Execution details.
      description: <i>(Introduced in v1.0)</i>
    delete:
      tags:
        - Data Jobs Execution
      responses:
        '200':
          description: Canceled execution successfully.
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobExecutionCancel
      summary: Cancel (if running) Data Job Execution
      description: <i>(Introduced in v1.0)</i> | Cancel the execution of a Data Job with a given execution ID.
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true
      - name: execution_id
        description: Execution ID.
        schema:
          type: string
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/executions/{execution_id}/logs':
    summary: Data Job Execution Logs.
    get:
      tags:
        - Data Jobs Execution
      responses:
        '200':
          $ref: '#/components/responses/ExecutionLogs'
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobLogsDownload
      summary: |
          Download data job logs.
          This API is guranteed to provide logs only if the jobs is currently running.
          For logs from older job executions - use logsUrl field passed by GET execution API or jobsQuery API.
      description: <i>(Introduced in v1.2.9)</i>
      parameters:
        -  name: team_name
           description: Team Name
           schema:
             type: string
           in: path
           required: true
        - name: job_name
          description: Data Job Name.
          schema:
            type: string
          in: path
          required: true
        - name: execution_id
          description: Execution ID.
          schema:
            type: string
          in: path
          required: true
        - name: tail_lines
          description: tail_lines when set to positive value, the number of lines from the end of the logs to return.
          schema:
            type: integer
            default: 0
            example: 10
          required: false
          in: query

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/properties':
    summary: |
      Data Job properties API.
      See https://github.com/vmware/versatile-data-kit/blob/6726aa6771b8a47edc81168afcd13030127a7c4b/projects/vdk-core/src/taurus/api/job_input.py#L11 <br>
      Currently they are not implemented in Versatile Data Kit.
    get:
      tags:
        - Data Jobs Properties
      responses:
        '200':
          $ref: '#/components/responses/Properties'
      operationId: dataJobPropertiesRead
      summary: Get Data Job properties.
      description: <i>(Introduced in v1.0)</i>
    put:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJobProperties'
        required: true
      tags:
        - Data Jobs Properties
      responses:
        '201':
          description: Created successfully.
        '204':
          description: Updated successfully.
      operationId: dataJobPropertiesUpdate
      summary: Update Data Job properties.
      description: <i>(Introduced in v1.0)</i>
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true
      - name: deployment_id
        description: Data job deployment id.
        schema:
          $ref: '#/components/schemas/DataJobDeploymentId'
        in: path
        required: true

  '/data-jobs/for-team/{team_name}/jobs/{job_name}/deployments/{deployment_id}/secrets':
    summary: |
      Data Job Secrets API allows the management of secrets for specific data jobs. Secreats are any sensitive data:
      passwords/tokens/credentials which are need by a data job.
    get:
      tags:
        - Data Jobs Secrets
      responses:
        '200':
          $ref: '#/components/responses/Secrets'
      operationId: dataJobSecretsRead
      summary: Get Data Job secrets.
      description: Get the secrets associated with a data job.
    put:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DataJobSecrets'
        required: true
      tags:
        - Data Jobs Secrets
      responses:
        '202':
          description: Created successfully.
        '204':
          description: Updated successfully.
        '413':
          description: Payload too large. The total size of the secrets object is too large.
      operationId: dataJobSecretsUpdate
      summary: Update Data Job secrets.
      description: Update the secrets associated with a data job.
    parameters:
      - name: team_name
        description: Team Name
        schema:
          type: string
        in: path
        required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true
      - name: deployment_id
        description: Data job deployment id.
        schema:
          $ref: '#/components/schemas/DataJobDeploymentId'
        in: path
        required: true

  '/data-jobs/teams/{team_name}/oauth-credentials':
    summary: |
      API to store/retrieve the Team's OAuth Application Credentials
    get:
      tags:
        - Data Jobs Secrets
      responses:
        '200':
          $ref: '#/components/responses/OauthTeamCredentials'
        '404':
          $ref: '#/components/responses/Error'
      operationId: oauthCredentialsGet
      summary: Get the Team's Oauth Application Credentials. | (Stable)
      description: |
        Get the Team's Oauth Application Credentials.
    put:
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/OauthCredentials'
      tags:
        - Data Jobs Secrets
      responses:
        '202':
          description: Update accepted successfully.
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: oauthCredentialsPut
      summary: Creates or updates a Team's Oauth Application Credentials. | (Stable)
      description: |
        Creates or updates a Team's Oauth Application Credentials.
    parameters:
      - name: team_name
        description: Team Name
        schema:
          type: string
        in: path
        required: true

  '/data-jobs/teams/{team_name}/oauth-credentials/client-id':
    get:
      tags:
        - Data Jobs Secrets
      responses:
        '200':
          $ref: '#/components/responses/OauthTeamClientId'
        '404':
          $ref: '#/components/responses/Error'
      operationId: clientIdGet
      summary: Retrieve the OAuth client ID for a specific team identified by {team_name}. | (Stable)
      description: |
        Retrieve the OAuth client ID for a specific team identified by {team_name}.
    parameters:
      - name: team_name
        description: Team Name
        schema:
          type: string
        in: path
        required: true

  '/data-jobs/oauth-credentials/client-ids':
    post:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: array
              items:
                type: string
      tags:
        - Data Jobs Secrets
      responses:
        '200':
          $ref: '#/components/responses/OauthTeamClientIdsArray'
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: getTeamIdsForClientIds
      summary: for a given list of ClientID(s) return a list of Team Ids,e.g. list of [clientId:teamId,clientId1:teamID1]
      description: |
        for a given list of ClientID(s) return a list of Team Ids,e.g. list of [clientId:teamId,clientId1:teamID1]


  '/data-jobs/for-team/{team_name}/jobs/{job_name}/sources':
    get:
      tags:
        - Data Jobs Sources
      responses:
        '200':
          description: Success
          content:
            application/octet-stream:
              schema:
                type: string
                format: binary
        '400':
          $ref: '#/components/responses/Error'
        '404':
          $ref: '#/components/responses/Error'
      operationId: dataJobSourcesDownload
      summary: Download data job source code. | (Not Implemented)
      description: <i>(Introduced in v1.0)</i>
    post:
      requestBody:
        content:
          application/octet-stream:
            schema:
              type: string
              format: binary
        required: true
      tags:
        - Data Jobs Sources
      responses:
        '200':
          $ref: '#/components/responses/Source'
        '400':
          $ref: '#/components/responses/Error'
      operationId: sourcesUpload
      summary: Upload Data Job source code. | (Stable)
      description: |
        <i>(Introduced in v1.0)</i> | Uploads the Data Job source code and returns its version.
        Upload is persisted in source (Git) repository, so that all code deployed is versioned.
        The format of the payload is expected to be a ZIP archive, where the directory content can be found.
        <br>Example:<br>
        <br>
        If zip is <br>
        foo/file.txt<br>
        foo/nested_dir/file2.txt<br>
        <br>then data job dir uploaded will look like <br>
        job-name/file.txt<br>
        job-name/nested_dir/file2.txt<br>
      parameters:
        - in: query
          name: reason
          description: The reason for executing the request
          schema:
            type: string
          required: false
    delete:
      tags:
        - Data Jobs Sources
      responses:
        '200':
          description: Data Job Source deleted successfully.
        '400':
          $ref: '#/components/responses/Error'
      operationId: sourcesDelete
      summary: Delete Data Job source.
      description: |
        <i>(Introduced in v1.0)</i> | Data Job Source is persisted in version control.<br>
        Delete removes the Data Job Source code.
      parameters:
        - in: query
          name: reason
          description: The reason for executing the request
          schema:
            type: string
          required: false
    parameters:
      -  name: team_name
         description: Team Name
         schema:
           type: string
         in: path
         required: true
      - name: job_name
        description: Data Job Name.
        schema:
          type: string
        in: path
        required: true

components:
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT

  schemas:
    DataJobApiInfo:
      description: API and Data Job service information, list of supported python versions
      required:
        - api_version
        - supported_python_versions
      type: object
      properties:
        api_version:
          description: ''
          type: string
        supported_python_versions:
          description: 'list of supported python versions'
          type: array
          items:
            type: string
          example: [ "python3.7","python3.8","python3.9" ]
      example:
        api_version: 1.0.0
        supported_python_versions: [ "python3.7","python3.8","python3.9" ]

    DataJobSummary:
      description: Data Job summary
      required:
        - job_name
        - team
        - description
      type: object
      properties:
        job_name:
          description: Data Job name
          type: string
          example: starshot-processing-vmc-fact-daily
        team:
          description: Team name
          type: string
          example: starshot
        description:
          description: Description
          type: string
          example: Data Job responsible for transforming vmc related fact tables on daily basis.
        source_url:
          description: Link to source code.
          type: string

    DataJob:
      description: Data Job Details
      required:
        - job_name
        - description
        - config
      type: object
      properties:
        job_name:
          description: Data Job name
          type: string
          example: starshot-processing-vmc-fact-daily
        description:
          description: Short Data Job description
          type: string
          example: Data Job responsible for transforming vmc-related fact tables on a daily basis.
        config:
          $ref: '#/components/schemas/DataJobConfig'
        team:
          description: (Optional) Team name
          type: string
          example: starshot (Optional property)

    Error:
      description: Contains description for one or more errors detected.
      required:
        - messages
      type: object
      properties:
        messages:
          description: Error messages
          type: array
          items:
            type: string
          example:
            - Data Job starshot-processing-vmc-fact-daily not found

    DataJobDeployment:
      description: A deployment of the Data Job
      type: object
      properties:
        vdk_version:
          description: A specific VDK version to use
          type: string
          example: 2.1
        job_version:
          description: Job version (can be Git commit)
          type: string
          example: 11a403ba
        python_version:
          description: A python release version (supported by the service) to be used for job deployments.
          type: string
          example: '3.9'
        mode:
          $ref: '#/components/schemas/DataJobMode'
        id:
          $ref: '#/components/schemas/DataJobDeploymentId'
        enabled:
          description: Enable/disable flag
          type: boolean
          example: false
        deployed_by:
          description: User or service that deployed the Data Job
          type: string
          example: auserov@example.mail.com
        deployed_date:
          description: The Data Job deployment date
          type: string
          format: date-time
        schedule:
          $ref: '#/components/schemas/DataJobSchedule'
        resources:
          $ref: '#/components/schemas/DataJobResources'

    DataJobDeploymentStatus:
      description: A deployment status of the Data Job, which includes information such as last deployment date, and who deployed the Data Job.
      type: object
      properties:
        vdk_version:
          description: A specific VDK version to use
          type: string
          example: 2.1
        job_version:
          description: Job version (can be Git commit)
          type: string
          example: 11a403ba
        python_version:
          description: A python release version (supported by the service) to be used for job deployments.
          type: string
          example: '3.9'
        mode:
          $ref: '#/components/schemas/DataJobMode'
        id:
          $ref: '#/components/schemas/DataJobDeploymentId'
        enabled:
          description: Enable/disable flag
          type: boolean
          example: false
        contacts:
          $ref: '#/components/schemas/DataJobContacts'
        schedule:
          $ref: '#/components/schemas/DataJobSchedule'
        resources:
          $ref: '#/components/schemas/DataJobResources'
        last_deployed_date:
          description: The date and time in UTC of the last deployment of the data job
          type: string
          example: 2021-02-02T10:50:40.034Z
          default: null
        last_deployed_by:
          description: The username of the user who modified the data job last
          type: string
          example: jdoe
          default: null


    DataJobExecutionRequest:
      description: Request to start execution of Data Job. The job must have been deployed before that (see Deployment API) and will run its latest version.
      type: object
      properties:
        started_by:
          description: User or service that started the execution (e.g manual/auserov@example.mail.com or scheduled/runtime)
          type: string
          example: schedule/runtime
        args:
          description: Data Job arguments
          type: object
          additionalProperties: true
          example:
            key: value

    DataJobExecution:
      description: Executions of a Data Job
      type: object
      properties:
        id:
          description: Data Job Execution ID
          type: string
          example: starshot-processing-vmc-fact-daily-2018623174356
        job_name:
          description: Data Job name
          type: string
          example: starshot-processing-vmc-fact-daily
        status:
          description: The current status
          enum:
            - submitted
            - running
            - succeeded
            - cancelled
            - skipped
            - user_error
            - platform_error
          type: string
          example: submitted
        type:
          description: Execution type - manual or scheduled
          enum:
            - manual
            - scheduled
          type: string
          example: scheduled
        start_time:
          description: Start of execution
          type: string
          format: date-time
        end_time:
          description: Start of execution
          type: string
          format: date-time
        started_by:
          description: User or service that started the execution (e.g manual/auserov@example.mail.com or scheduled/runtime)
          type: string
          example: schedule/runtime
        logs_url:
          description: |
            URL link to persisted logs in central location. Logs generally should be available for longer time.
            The link is available only if operators have configured it during installation of Control Service.
            During install operators can configure logs to be persisted to log aggregator service whose link can be exposed here.
          type: string
          example: http://logs/jobs?filter=job-name
        message:
          description: Message (usually error) during execution
          type: string
          example:
        op_id:
          description: Operation id used for tracing calls between different services
          type: string
        deployment:
          $ref: '#/components/schemas/DataJobDeployment'

    DataJobExecutionLogs:
      description: Executions of a Data Job
      type: object
      properties:
        logs:
          description: The logs of the data job execution.
          type: string

    DataJobProperties:
      description: Properties of a Data Job.
      type: object
      additionalProperties: {}
      example:
        redshift-user: foo
        redshift-password: bar
        last-execution-time: 1585303338

    DataJobSecrets:
      description: Secrets of a Data Job. Used to store passwords/tokens/credentials/sensitive data.
      type: object
      additionalProperties: {}
      example:
        redshift-user: foo
        redshift-password: bar

    OauthCredentials:
      description: Oauth credentials
      required:
        - clientId
        - clientSecret
      type: object
      properties:
        clientId:
          description: Oauth Application Client ID
          type: string
          example: HDZe8w71KzzzRQkpRlicFM2UAdUsgStRGv
        clientSecret:
          description: Oauth Application Client Secret
          type: string
          example: OYXIXlHE2sRuPBCXHTBHxN3rZOiBtFkrbnCpAj1YSFE3gSr5mT

    OauthTeamCredentials:
      description: Oauth credentials for a team.
      type: object
      properties:
        teamName:
          type: string
          description: The name of the team
        clientId:
          type: string
          description: The clientID
        clientSecret:
          type: string
          description: The clientSecret
      required:
        - teamName
        - clientId
        - clientSecret

    OauthTeamClientId:
      description: Oauth credentials for a team.
      type: object
      properties:
        teamName:
          type: string
          description: The name of the team
        clientId:
          type: string
          description: The clientID
      required:
        - clientId
        - teamName

    DataJobMode:
      description: The execution mode that the data job is deployed in. Data Jobs used for development or testing purposes are marked as testing.
        This is used by Operations team on platform rollout and infrastructure changes adoption. For example, rollout and validation of testing jobs first, then proceeding with release data jobs. Also, testing and release jobs may have different limits and SLA targets.
      enum:
        - testing
        - release
      type: string
      example: release

    DataJobDeploymentId:
      description: |
        String that identifies a single deployment of a Data Job.
        Currently only one single deployment per Data Job is possible.<br>
        In the future:<br>
        It's recommended to use following ids - development, testing, production.
        However users are free to come up with their own.
        For example, this enables the creation of 3 different deployments, using the same Data Job code:<br>
        `development  deployment  --deployment-id development`<br>
        `testing deployment  --deployment-id testing`<br>
        `production deployment  --deployment-id prod`
      type: string
      example: release

    DataJobResources:
      description: Resource cofiguration of a data Data Job Deployment.
      type: object
      properties:
        cpu_request:
          format: float
          description: Initial CPU shares in deciCores (1 dCore = 0.1 Core = 100 mCores)
          type: number
          example: 10
        cpu_limit:
          format: float
          description: Max CPU shares in deciCores (1 dCore = 0.1 Core = 100 mCores)
          type: number
          example: 20
        memory_request:
          format: int32
          description: Initial Memory in MiB.
          type: integer
          example: 1024
        memory_limit:
          format: int32
          description: Max Memory in MiB.
          type: integer
          example: 2048

    DataJobConfig:
      description: Data Job configuration properties.
      type: object
      properties:
        db_default_type:
          example: TRINO
          description: |
            Default DB connection provided for the job, e.g. 'TRINO', 'IMPALA', 'REDSHIFT'.<br>
          type: string
        contacts:
          $ref: '#/components/schemas/DataJobContacts'
        schedule:
          $ref: '#/components/schemas/DataJobSchedule'
          deprecated: true
        generate_keytab:
          example: false
          description: Enable Disable flag for generating secret with keytab
          type: boolean
          default: true
        enable_execution_notifications:
          example: false
          description: (Optional) Specifies whether to dispatch email notifications per data job execution.
          type: boolean
          default: true
        notification_delay_period_minutes:
          format: int32
          description: |
            (Optional) Specifies the time (in minutes) a job execution is allowed to be delayed from
            its schedule before an alert is triggered.
            The value of this property for each data job is exposed as a Prometheus metric by the service
            at the /data-jobs/debug/prometheus endpoint. These metrics (taurus_datajob_notification_delay)
            are subsequently used in JobDelay Prometheus rules to generate the alerts.
          type: integer
          example: 60
          default: 240


    DataJobSchedule:
      description: Schedule configuration
      type: object
      properties:
        schedule_cron:
          example: 0 0 13 * 5
          description: |
            For format see https://en.wikipedia.org/wiki/Cron<br>
            The cron expression is evaluated in UTC time.
            If it is time for a new job run and the previous job run hasn't finished yet, the cron job kills and replaces the currently running job run with a new job run.
            Jobs configured to run more often than once per hour are not supported and their schedule may be overridden by the platform.
            To distribute load evenly, Administrators may override the minute you specified.
            Use https://crontab.guru for help.
          type: string

    DataJobContacts:
      description: Data Job contacts
      type: object
      properties:
        notified_on_job_failure_user_error:
          description: |
            List of email addresses to be notified on job execution failure caused by user code or user configuration problem.
            E.g. if the job contains a SQL script with a syntax error.
          type: array
          items:
            type: string
          example:
            - auserov@example.mail.com
        notified_on_job_failure_platform_error:
          description: List of email addresses to be notified on job execution failure caused by a platform problem
          type: array
          items:
            type: string
          example:
            - auserov@example.mail.com
        notified_on_job_success:
          description: List of email addresses to be notified on job execution success
          type: array
          items:
            type: string
          example:
            - auserov@example.mail.com
        notified_on_job_deploy:
          description: List of email addresses to be notified of job deployment outcome
          type: array
          items:
            type: string
          example:
            - auserov@example.mail.com

    DataJobVersion:
      description: Data Job version
      required:
        - version_sha
      type: object
      properties:
        version_sha:
          description: SHA hash which specifies the latest deployed version of the data job
          type: string
      example:
        version_sha: 17012900f60461778c01ab24728807e70a5f2c87
    DataJobQueryResponseWithError:
      description: Query response containing Data Jobs
      type: object
      properties:
        errors:
          type: array
          items:
            type: object
          description: Errors while making query (validation errors, exceptions, etc)
          example:
            application/json: [
               {
                 "message": "Validation error of type FieldUndefined",
                 "locations": [],
                 "description": "Field 'someField' in type 'DataJob' is undefined",
                 "validationErrorType": "FieldUndefined",
                 "queryPath": [
                   "jobs",
                   "content",
                   "someField"
                 ],
                 "extensions": null,
                 "errorType": "ValidationError",
                 "path": null
               }
             ]
        data:
          $ref: '#/components/schemas/DataJobPage'
    DataJobQueryResponse:
      description: Query response containing Data Jobs
      type: object
      properties:
        errors:
          type: array
          items:
            type: object
          description: Errors while making query (validation errors, exceptions, etc)
        data:
          $ref: '#/components/schemas/DataJobPage'
    DataJobPage:
      description: Page object containing Data Jobs list with information for total elements and pages
      type: object
      properties:
        content:
          type: array
          items:
            type: object
          example:
            application/json: [
              {
                "jobName": "starshot-processing-vmc-fact-daily",
                "config": {
                  "team": "starshot",
                  "description": "Data Job responsible for transforming vmc related fact tables on daily basis",
                  "schedule": {
                    "scheduleCron": "5 0 * 8 *",
                    "nextRun": 1618914371
                  },
                  "sourceUrl": "https://github.com/product-analytics/data-jobs/tree/master/starshot-processing-vmc-fact-daily",
                  "contacts": {
                    "notifiedOnJobFailureUserError": "[auser@example.mail.com]",
                    "notifiedOnJobFailurePlatformError": "[auser2@example.mail.com, auser@example.mail.com]",
                    "notifiedOnJobSuccess": "[auser@example.mail.com]",
                    "notifiedOnJobDeploy": "[auser2@example.mail.com, auser@example.mail.com]"
                  }
                }
              },
              ...
            ]
        totalItems:
          description: Number of elements which meet the given query requirement
          type: integer
          example: 100
        totalPages:
          description: Number of pages with elements which meet the given query requirement
          type: integer
          example: 5

  responses:
    Info:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobApiInfo'
      description: API and Data Jobs service information
    DataJob:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJob'
      description: Data Job details
    Error:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
      description: A response for any errors detected
    DeploymentArray:
      content:
        application/json:
          schema:
            type: array
            items:
              $ref: '#/components/schemas/DataJobDeploymentStatus'
      description: A list of data job deployments
    Deployment:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobDeploymentStatus'
      description: Deployment properties
    DataJobConfig:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobConfig'
      description: Data Job custom configuration
    ExecutionArray:
      content:
        application/json:
          schema:
            type: array
            items:
              $ref: '#/components/schemas/DataJobExecution'
      description: Data Job executions
    Execution:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobExecution'
      description: Execution details
    ExecutionLogs:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobExecutionLogs'
      description: Deployment properties
    Properties:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobProperties'
      description: Data Job properties
    Secrets:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobSecrets'
      description: Data Job Secrets
    OauthTeamCredentials:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/OauthTeamCredentials'
      description: Team Oauth Credentials
    OauthTeamClientId:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/OauthTeamClientId'
      description: Team Oauth ClientId
    OauthTeamClientIdsArray:
      content:
        application/json:
          schema:
            type: array
            items:
              $ref: '#/components/schemas/OauthTeamClientId'
      description: An array of Team Oauth ClientIds

    Source:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobVersion'
      description: Data Job version
    DataJobQueryResponse:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobQueryResponse'
      description: Data Job query response
    DataJobQueryResponseWithError:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DataJobQueryResponseWithError'
      description: Data Job query response with error

security:
  - bearerAuth: []

tags:
  - name: "Data Jobs"
    description: (Stable)
  - name: "Data Jobs Deployment"
    description: "(Stable) | Data Job Deployment takes the code/build and deploy-specific properties of the data job, then it is ready for immediate run in the execution environment. <br>
      Data job code/build - all python, SQL files, and requirements.txt of the data job.
      NOTE: deploymentID and mode are not implemented, so they are ignored."
  - name: "Data Jobs Execution"
    description:  API for managing Data Job Execution. An instance of a running data job deployment is called an execution.
  - name: "Data Jobs Properties"
    description: "Any saved state or configuration of a Data Job. Those are tracked per deployment.
      NOTE: Versioned properties, so all changes are tracked (when, who - opid or user), is not implemented yet."
  - name: "Data Jobs Secrets"
    description: "Secrets for a Team or Data Job. Example: passwords/tokens/credentials/sensitive data."
  - name: "Data Jobs Service"
    description: (Stable)
  - name: "Data Jobs Sources"
    description: (Stable)