openapi: 3.2.0 info: title: Crawler API description: A crawler allows you to retrieve a full list of assets contained in a connection, in a single operation. Manage crawlers to retrieve data at a large scale and enrich your inventory more efficiently. contact: {} version: 2021-03 servers: - url: https://api.eu.cloud.talend.com description: URL for the AWS Europe region x-talend: isPublished: true - url: https://api.ap.cloud.talend.com description: URL for the AWS Asia Pacific region x-talend: isPublished: true - url: https://api.us.cloud.talend.com description: URL for the AWS United States East region x-talend: isPublished: true - url: https://api.au.cloud.talend.com description: URL for the AWS Australia region x-talend: isPublished: true - url: https://api.us-west.cloud.talend.com description: URL for the Azure United States West region x-talend: isPublished: true security: - BearerAuthentication: [] tags: - name: Crawler paths: /connections/crawlers: get: tags: - Crawler summary: Retrieve all the crawlers of a tenant description: 'This endpoint lets you retrieve all the crawlers of a tenant. You can also include the crawlers that have been deleted by activating the dedicated option. You can retrieve the crawler only from a connection by indicating the connection ID.' operationId: getApiV1Crawlers parameters: - name: limit in: query description: not used schema: type: integer format: int32 description: not used - name: offset in: query description: not used schema: type: integer format: int32 description: not used - name: talendVersion in: query description: default version of the API schema: type: string description: default version of the API enum: - 2021-03 - name: includeDeleted in: query description: if true, it will also returns the crawlers that have been deleted. schema: type: boolean description: if true, it will also returns the crawlers that have been deleted. default: false - name: connectionId in: query description: Use this option if you want to retrieve the crawler linked to a connection schema: type: string description: Use this option if you want to retrieve the crawler linked to a connection - name: talend-version in: header description: default version of the API schema: type: string description: default version of the API enum: - 2021-03 responses: '200': description: Status 200 content: application/json: schema: $ref: '#/components/schemas/PaginatedResources_CrawlerModel' example: "{\n \"data\": [\n {\n \"id\": \"59451bf0-a81a-11eb-bcbc-0242ac130002\",\n \"connectionId\": \"d54a8f03-7906-4930-a7cc-4eb90e968f89\",\n \"name\": \"Crawler1\",\n \"description\": \"Description du crawler 1\",\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ],\n \"status\": {\n \"runStatus\": \"NotStarted\"\n },\n \"createdAt\": \"2021-01-08T15:41:29.263Z\",\n \"createdBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"crawledDatasets\": [\n \"Table1\",\n \"Table2\",\n \"Table3\",\n \"View1\"\n ]\n },\n {\n \"id\": \"3a45cb46-a81a-11eb-bcbc-0242ac130002\",\n \"connectionId\": \"165ea830-e003-11eb-ba80-0242ac130004\",\n \"name\": \"Crawler2\",\n \"description\": \"Description du crawler 2\",\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ],\n \"status\": {\n \"runStatus\": \"RetrievingProperties\",\n \"runStartedAt\": \"2021-01-08T15:41:29.263Z\",\n \"runBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\" \n },\n \"createdAt\": \"2021-01-08T15:41:29.263Z\",\n \"createdBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"crawledDatasets\": [\n \"Dataset 1\",\n \"Dataset 2\",\n \"Dataset 3\",\n \"Dataset 4\"\n ]\n },\n {\n \"id\": \"108fb1c2-a81a-11eb-bcbc-0242ac130002\",\n \"connectionId\": \"1db0db6c-e003-11eb-ba80-0242ac130004\",\n \"name\": \"Crawler3\",\n \"description\": \"Description du crawler 3\",\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ],\n \"status\": {\n \"runStatus\": \"PropertiesRetrievalFailed\",\n \"runStartedAt\": \"2021-01-08T15:41:29.263Z\",\n \"runBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"failure\": \"cannot generate dataset properties\"\n },\n \"createdAt\": \"2021-01-08T15:41:29.263Z\",\n \"createdBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"crawledDatasets\": [\n \"Dataset 1\",\n \"Dataset 2\",\n \"Dataset 3\",\n \"Dataset 4\"\n ]\n },\n {\n \"id\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"connectionId\": \"2695204e-e003-11eb-ba80-0242ac130004\",\n \"name\": \"Crawler4\",\n \"description\": \"Description du crawler 4\",\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ],\n \"status\": {\n \"runStatus\": \"CreatingDatasets\",\n \"runStartedAt\": \"2021-01-08T15:41:29.263Z\",\n \"runBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\"\n },\n \"createdAt\": \"2021-01-08T15:41:29.263Z\",\n \"createdBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"crawledDatasets\": [\n \"Dataset 1\",\n \"Dataset 2\",\n \"Dataset 3\",\n \"Dataset 4\"\n ]\n },\n {\n \"id\": \"7c7f7872-a81a-11eb-bcbc-0242ac130002\",\n \"connectionId\": \"2e46ed68-e003-11eb-ba80-0242ac130004\",\n \"name\": \"Crawler5\",\n \"description\": \"Description du crawler 5\",\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ],\n \"status\": {\n \"runStatus\": \"Finished\",\n \"runStartedAt\": \"2021-01-08T15:41:29.263Z\",\n \"runBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"runFinishedAt\": \"2021-01-08T15:41:29.263Z\"\n },\n \"createdAt\": \"2021-01-08T15:41:29.263Z\",\n \"createdBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"crawledDatasets\": [\n \"Dataset 1\",\n \"Dataset 2\",\n \"Dataset 3\",\n \"Dataset 4\"\n ]\n }\n ],\n \"offset\": 0,\n \"limit\": 0,\n \"total\": 5\n}" '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' post: tags: - Crawler summary: Create a new crawler description: 'At this time, a crawler can only be created on a JDBC connection. You can only have one active crawler per connection. An active crawler is a crawler that has not been deleted. When the user runs the crawler, datasets from the tables and views of the JDBC connection will be created. Known limitations: **Max objects limit** : We recommend selecting less than 1000 tables/views. Beyond this limit, you may encounter issues when launching the run endpoint. **Max datasets limit**: The maximum number of datasets a user can have is 1500. Beyond this limit, you may encounter timeouts when calling the dataset endpoint that list them all for a user. In consequence, when configuring a crawler, it is important to ensure that you will not exceed this limit after the crawler has run.' operationId: postApiV1Crawlers parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 requestBody: content: application/json: schema: $ref: '#/components/schemas/CreateCrawlerRequest' example: "{\n \"connectionId\": \"d54a8f03-7906-4930-a7cc-4eb90e968f89\",\n \"name\": \"Crawler - JDBC\",\n \"selectedDatasets\": [\n \"accounts\",\n \"orders\",\n \"items\"\n ],\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ]\n}" responses: '201': description: Status 201 content: application/json: schema: $ref: '#/components/schemas/CreateCrawlerResponse' example: "{\n \"id\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\"\n}" '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '409': description: Already Exist content: application/json: schema: $ref: '#/components/schemas/AlreadyExist' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' /connections/crawlers/{crawlerId}: description: 'Various operations on a crawler based on its ID: - get a crawler - update the name and the description - update the tables and views selection - delete a crawler' parameters: - name: crawlerId in: path required: true schema: type: string get: tags: - Crawler summary: Get a crawler by its ID description: 'Retrieve a crawler using its ID. The response payload contains the crawler itself.' operationId: getApiV1CrawlersCrawlerid parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: includeDeleted in: query description: if true, it will also returns the crawlers that have been deleted. schema: type: boolean description: if true, it will also returns the crawlers that have been deleted. default: false - name: talend-version in: header schema: type: string enum: - 2021-03 responses: '200': description: Status 200 content: application/json: schema: $ref: '#/components/schemas/CrawlerModel' example: "{\n \"id\": \"59451bf0-a81a-11eb-bcbc-0242ac130002\",\n \"connectionId\": \"d54a8f03-7906-4930-a7cc-4eb90e968f89\",\n \"name\": \"Crawler1\",\n \"description\": \"Description du crawler 1\",\n \"sharings\": [\n {\n \"scimType\": \"user\",\n \"scimId\": \"b8a78dcb-65b4-4823-ad76-88720fc6309e\",\n \"level\": \"OWNER\"\n },\n {\n \"scimType\": \"group\",\n \"scimId\": \"877f89dc-709b-4ef1-8d0e-a851f67a065a\",\n \"level\": \"READER\"\n },\n {\n \"scimType\": \"user\",\n \"scimId\": \"bd4c7ae4-a1df-4702-845e-11946fa07d85\",\n \"level\": \"WRITER\"\n }\n ],\n \"status\": {\n \"runStatus\": \"NotStarted\",\n \"nbDatasetsToCrawl\": 0,\n \"nbDatasetsFinished\": 0,\n \"nbDatasetsCreated\": 0,\n \"nbDatasetsFailed\": 0,\n \"nbSamplesFailed\": 0\n },\n \"createdAt\": \"2021-01-08T15:41:29.263Z\",\n \"createdBy\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"crawledDatasets\": [\n \"Dataset 1\",\n \"Dataset 2\",\n \"Dataset 3\",\n \"Dataset 4\"\n ]\n}" '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' put: tags: - Crawler summary: Update the tables and views selection of an existing crawler description: This endpoint allows you to add and remove some tables or views from the crawler configuration. operationId: putApiV1CrawlersCrawlerid parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 requestBody: content: application/json: schema: $ref: '#/components/schemas/UpdateCrawlerRequest' responses: '200': description: Status 200 content: application/json: schema: type: string description: The technical talend ID of the crawler that has been updated. '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '409': description: The crawler is running, action not available content: application/json: schema: $ref: '#/components/schemas/AlreadyRunning' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' delete: tags: - Crawler summary: Delete a crawler description: 'Use this method to delete a crawler. Because you can only have one crawler at a time on a JDBC connection, you may need to delete a crawler in order to create a new one. You can also edit an existing crawler and make some modifications. Deleting a crawler doesn''t physically remove it unless this crawler has no more related datasets. You can still find it with the endpoint that lists all the crawlers by including the deleted crawlers in the search.' operationId: deleteApiV1CrawlersCrawlerid parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 responses: '204': description: Status 204 '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '409': description: The crawler is running, action not available content: application/json: schema: $ref: '#/components/schemas/AlreadyRunning' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' patch: tags: - Crawler summary: Update the name and description of a crawler description: This endpoint allows you to update the name and the description of a crawler. operationId: patchApiV1CrawlersCrawlerid parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 requestBody: content: application/json: schema: $ref: '#/components/schemas/PatchCrawlerRequest' responses: '200': description: Status 200 content: application/json: schema: type: string description: The technical talend ID of the crawler that has been updated. '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '409': description: The crawler is running, action not available content: application/json: schema: $ref: '#/components/schemas/AlreadyRunning' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' /connections/crawlers/{crawlerId}/run: parameters: - name: crawlerId in: path required: true schema: type: string post: tags: - Crawler summary: Run a crawler description: 'This endpoint allows you to start the crawler. When calling this endpoint, the crawler will rely on its configuration in order to retrieve all the selected tables and views and turn them into datasets. Once the dataset will be created the crawler will also retrieve their samples. You can launch the crawler as many time as you want. Running a crawler once will create the datasets. Running a crawler again will only refresh the sample of the existing datasets. Known limitations: **Max objects limit** : We recommend selecting less than 1000 tables/views. Beyond this limit, you may encounter issues when launching the run endpoint. **Max datasets limit**: The maximum number of datasets a user can have is 1500. Beyond this limit, you may encounter timeouts when calling the dataset endpoint that list them all for a user. In consequence, when configuring a crawler, it is important to ensure that you will not exceed this limit after the crawler has run.' operationId: postApiV1CrawlersCrawleridRun parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 responses: '202': description: Status 202 '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' /connections/crawlers/{crawlerId}/end: parameters: - name: crawlerId in: path required: true schema: type: string post: tags: - Crawler summary: End a crawler while it is running description: 'This endpoint allows you to stop a crawler while it is running. After launching a crawler, the run can take up to a few hours to complete, according the number of objects you selected. You may want to stop the run for many reasons, for instance if you notice that the crawler was created on the wrong connection. Stopping a crawler does not mean cancelling the crawler. The datasets that have already been created will not be deleted. If you want to clean them, you can use the faceted search to retrieve the datasets created by a crawler and delete them.' operationId: postApiV1CrawlersCrawleridEnd parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 responses: '202': description: Status 202 '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' /connections/crawlers/{crawlerId}/datasets: parameters: - name: crawlerId in: path required: true schema: type: string get: tags: - Crawler summary: Retrieve all the statuses of the datasets related to a crawler description: This endpoint allows you to retrieve the statuses of all the datasets related to a crawler. operationId: getApiV1CrawlersCrawleridDatasets parameters: - name: limit in: query schema: type: integer format: int32 - name: offset in: query schema: type: integer format: int32 - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 responses: '200': description: Status 200 content: application/json: schema: $ref: '#/components/schemas/PaginatedResources_CrawledDataset' example: "{\n \"data\": [\n {\n \"id\": \"ac2c714e-33bb-4346-add5-acdab5c07d8f\",\n \"crawlerId\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"datasetId\": \"d54f8f03-7906-4930-a7cc-4eb90e968f89\",\n \"displayName\": \"Dataset 1\",\n \"technicalName\": \"Dataset1\",\n \"metadata\": {},\n \"exportStatus\": \"NotStarted\",\n \"lastUpdate\": \"2021-01-08T15:41:29.263Z\"\n },\n {\n \"id\": \"848adbdc-b7b6-11eb-8529-0242ac130003\",\n \"crawlerId\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"datasetId\": \"e3d7bd9e-b7b6-11eb-8529-0242ac130003\",\n \"displayName\": \"Dataset 2\",\n \"technicalName\": \"Dataset2\",\n \"metadata\": {},\n \"exportStatus\": \"Creating\",\n \"lastUpdate\": \"2021-01-08T15:41:29.263Z\"\n },\n {\n \"id\": \"39ebaa42-b7b7-11eb-8529-0242ac130003\",\n \"crawlerId\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"datasetId\": \"ed80cec6-b7b6-11eb-8529-0242ac130003\",\n \"displayName\": \"Dataset 3\",\n \"technicalName\": \"Dataset3\",\n \"metadata\": {},\n \"exportStatus\": \"Finished\",\n \"lastUpdate\": \"2021-01-08T15:41:29.263Z\"\n },\n {\n \"id\": \"4bf7c108-b7b7-11eb-8529-0242ac130003\",\n \"crawlerId\": \"ac6e2117-fbb5-442a-bb02-cefabbf04516\",\n \"datasetId\": \"6bb92356-b7b7-11eb-8529-0242ac130003\",\n \"displayName\": \"Dataset 4\",\n \"technicalName\": \"Dataset4\",\n \"metadata\": {},\n \"exportStatus\": \"CreationFailed\",\n \"failure\": \"Could not export the dataset because of ...\",\n \"lastUpdate\": \"2021-01-08T15:41:29.263Z\"\n }\n ],\n \"offset\": 0,\n \"limit\": 100,\n \"total\": 4\n}" '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' '404': description: Not found content: application/json: schema: $ref: '#/components/schemas/NotFound' '500': description: Internal Server Error content: application/json: schema: $ref: '#/components/schemas/ServerError' '503': description: Service unavailable content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' /connections/crawlers/{crawlerId}/errors.log: parameters: - name: crawlerId in: path required: true schema: type: string get: tags: - Crawler summary: Get the error log file description: This endpoint allows you to retrieve the error log file of a crawler run. Even if your crawler ends successfully, some samples may have not been fetched, so you might want to know the technical reasons by downloading the error logs. operationId: getApiV1CrawlersCrawleridErrors.log parameters: - name: talendVersion in: query schema: type: string enum: - 2021-03 - name: talend-version in: header schema: type: string enum: - 2021-03 responses: '200': description: Status 200 headers: Content-Disposition: required: true schema: type: string content: text/plain: schema: type: string '401': description: Not authenticated content: application/json: schema: $ref: '#/components/schemas/NotAuthenticated' '403': description: Not authorized content: application/json: schema: $ref: '#/components/schemas/NotAuthorized' components: schemas: NotAuthorized: type: object required: - message description: The user has been authenticated but doesn't have the entitlement for this service. properties: message: type: string cause: type: string Status: type: object required: - nbDatasetsCreated - nbDatasetsFailed - nbDatasetsFinished - nbDatasetsToCrawl - nbSamplesFailed - runStatus description: This object contains informations about the crawler status properties: runStatus: $ref: '#/components/schemas/RunStatus' runStartedAt: type: string format: date-time description: The date when the run has started runBy: type: string description: Technical ID of the talend user. runFinishedAt: type: string format: date-time description: The date when the run has finished failure: type: string nbDatasetsToCrawl: type: integer format: int32 description: Number of datasets to retrieve nbDatasetsFinished: type: integer format: int32 description: Number of datasets already retrieved nbDatasetsCreated: type: integer format: int32 description: Number of datasets tha has been created nbDatasetsFailed: type: integer format: int32 description: Number of datasets that has not been created nbSamplesFailed: type: integer format: int32 description: Number of samples that has not been created Metadata_infos: type: object description: 'This objects contains the metadata of an entity according to the context of the call. Exemple : indicate if an object is a table or a view.' CreateCrawlerResponse: type: object required: - id description: Contains the response following a crawler creation request. The object contains the ID of the crawler created. properties: id: type: string description: The technical talend ID of the crawler that has been created CrawlerModel: description: 'This object represents a whole crawler. It has two formats : either complete or light.' oneOf: - $ref: '#/components/schemas/CrawlerComplete' - $ref: '#/components/schemas/CrawlerLight' UpdateCrawlerRequest: type: object required: - name description: Describe the elements to update on the crawler as the name, description and the tables/views selection. properties: name: type: string description: New name of the crawler description: type: string description: New description of the crawler selectedDatasets: type: array description: Names of the tables and views that we want to retrieve with the crawler. items: type: string CrawlerLight: type: object required: - connectionId - id - name - runStatus description: This object represents a crawler in the light version (only contain the status). properties: id: type: string description: The technical talend ID of the crawler name: type: string description: The name of the crawler connectionId: type: string description: The technical talend ID of the connection runStatus: $ref: '#/components/schemas/RunStatus' deletedAt: type: string format: date-time description: The date when the crawler has been deleted AlreadyExist: type: object required: - connectionId description: A crawler already exists for this connection. You can only have one crawler per connection. properties: connectionId: type: string description: The technical talend ID of the connection i18nMsg: type: string EntityType: type: string description: Entity type used for the NotFound data type in order to indicate the type of the object that hasn't been found. enum: - Connection - ConnectionScan - CrawledDataset - Crawler - Dataset - Datastore - MassSampling - Tenant PaginatedResources_CrawledDataset: type: object required: - limit - offset - total description: This objects contains the datasets related to a crawler in a paginated context. properties: data: type: array description: Contains the list of the objects selected in the crawler configuration items: $ref: '#/components/schemas/CrawledDataset' offset: type: integer format: int32 description: Pagination offset limit: type: integer format: int32 description: Pagination limit total: type: integer format: int32 description: Total number of crawlers CrawlerComplete: type: object required: - connectionId - createdAt - createdBy - id - name - status description: This object represents a crawler in the complete version. properties: id: type: string description: The technical talend ID of the crawler connectionId: type: string description: The technical talend ID of the connection name: type: string description: Name of the crawler description: type: string description: Description of the crawler sharings: type: array description: Sharing policies items: $ref: '#/components/schemas/Sharing' status: $ref: '#/components/schemas/Status' createdAt: type: string format: date-time description: The date when the crawler has been created createdBy: type: string description: Technical ID of the talend user. crawledDatasets: type: array description: Names of the tables and views that we want to retrieve with the crawler. items: type: string updateAt: type: string format: date-time description: The date when the crawler has been updated updatedBy: type: string description: Technical ID of the talend user. deletedAt: type: string format: date-time description: The date when the crawler has been deleted deletedBy: type: string description: Technical ID of the talend user. RunStatus: type: string description: This object contains the different run status of the crawler. example: CreatingDatasets PatchCrawlerRequest: type: object description: 'This object is used when the user wants to update the name and the description of the crawler. Those fields will replace the current ones once the update will be applied.' properties: name: type: string description: New name of the crawler description: type: string description: New description of the crawler AlreadyRunning: type: object required: - crawlerId description: This entity indicates that the crawler is already running and provides the crawler ID. properties: crawlerId: type: string description: The technical talend ID of the crawler i18nMsg: type: string Sharing: type: object required: - level - scimId - scimType description: This object indicate the sharing informations. properties: scimType: type: string description: indicate USER or GROUP enum: - USER - GROUP scimId: type: string description: the scim id of the USER or GROUP level: type: string description: the level of the sharing enum: - READER - WRITER - OWNER PaginatedResources_CrawlerModel: type: object required: - limit - offset - total description: This object represents a whole crawler but in a paginated context. properties: data: type: array description: Contains the list of the crawlers items: $ref: '#/components/schemas/CrawlerModel' offset: type: integer format: int32 description: Pagination offset limit: type: integer format: int32 description: Pagination limit total: type: integer format: int32 description: Total number of crawlers NotAuthenticated: type: object required: - message description: The user has not been authenticated. properties: message: type: string cause: type: string ServerError: type: object required: - message description: The server encountered an error during the processing of the request. properties: message: type: string cause: type: string CreateCrawlerRequest: type: object required: - connectionId - name description: Describe the crawler about to be created with the name, description, tables and views selection, user and group for the sharing policies. properties: connectionId: type: string description: The technical talend ID of the connection name: type: string description: Name of the crawler description: type: string description: Description of the crawler selectedDatasets: type: array description: Names of the tables and views that we want to retrieve with the crawler items: type: string sharings: type: array description: Sharing policies items: $ref: '#/components/schemas/Sharing' NotFound: type: object required: - entityId - entityType description: Generic response payload when an object hasn't been found. The entityType gives more information about the object. properties: entityId: type: string description: The technical talend ID of the entity entityType: $ref: '#/components/schemas/EntityType' i18nMsg: type: string ExportStatus: type: string description: this object contains the different statuses of a dataset in a crawler context. enum: - Creating - CreationFailed - Finished - NotStarted - Sampling - SamplingFailed CrawledDataset: type: object required: - crawlerId - displayName - exportStatus - id - lastUpdate - metadata - technicalName description: This object represents a datasets related to a crawler. properties: id: type: string description: The technical ID of the object selected by the crawler crawlerId: type: string description: The technical talend ID of the crawler datasetId: type: string description: The technical ID of the dataset generated by the crawler displayName: type: string description: Names of the tables and views that we want to retrieve with the crawler technicalName: type: string description: Technical names of the tables and views that we want to retrieve with the crawler metadata: $ref: '#/components/schemas/Metadata_infos' exportStatus: $ref: '#/components/schemas/ExportStatus' failure: type: string description: Indicate if we encountered a failure during the crawling for this dataset lastUpdate: type: string format: date-time description: Date of the last time this dataset has been refreshed securitySchemes: BearerAuthentication: type: http scheme: bearer bearerFormat: Bearer