swagger: '2.0' info: version: v1 title: Logisland standard API description: >- **The Logisland REST API for third party applications.** The API should be implemented by a third party application and logisland will regularly poll this endpoint in order to: - Ask for configuration changes to be triggered. - Report the latest configuration applied (to ease up resynchronization and business continuity). *As a general rule, the changes will be triggered if the **lastUpdated** field of the object you are going to modify is fresher than the one known by logisland.* In terms of API, two degrees of freedom are possible: - **Dataflow**: A dataflow is a set of services and streams allowing a data flowing from one or more sources, being transformed and reach one or more destinations (sinks). Act at dataflow level if you want to: - Add/Remove any streaming endpoint - Change any active stream configuration (e.g. kafka topic) - Create/Remote/Modify any service - **Pipeline**: A pipeline is a processing chain acting on a data flowing point-to-point. The api gives you the possibility to have a finer-grained control of what is going of any stream pipeline without perturbing the stream itself. This means that the processor chain will be dynamically reconfigured without the need of stopping the stream and reconfigure the whole dataflow. Act at pipeline level if you want to: - Add/Remove processors in the pipeline - Change any processor configuration *Please note that if you need to add/remove controller services you must act your changes at dataflow level.* contact: name: Hurence email: support@hurence.com host: localhost:8081 schemes: - http - https consumes: - application/json produces: - application/json paths: /dataflows/{dataflowName}: parameters: - name: dataflowName in: path type: string required: true description: the dataflow name (aka the logisland job name) get: tags: - dataflow operationId: pollDataflowConfiguration summary: Retrieves the configuration for a specified dataflow description: >- A dataflow is a set of services and streams allowing a data flowing from one or more sources, being transformed and reach one or more destinations (sinks). Logisland will call this endpoint to know which configuration should be run. This endpoint also supports HTTP caching (Last-Updated, If-Modified-Since) as per RFC 7232, section 3.3 parameters: - name: If-Modified-Since in: header type: string description: Timestamp of last response required: false responses: "200": description: >- Return the dataflow configuration. On logisland side, the following will happen: - At dataflow level: - Fully reconfigure a dataflow (stop and then start) if nothing is running (initial state) or if lastUpdated is fresher than the one of the already running dataflow. In this case be aware that old stream and services will be destroyed and new ones will be created. - Do nothing otherwise (keep running the active dataflow) - At pipeline level: - The processor chain will be fully reconfigured if and only if the pipeline lastUpdated is fresher than the lastUpdated known by the system. In any case the stream is never stopped. headers: Last-Updated: type: string description: Should be used for subsequent requests as If-Modified-Since request header. schema: $ref: '#/definitions/DataFlow' examples: A single stream dataflow: lastModified: '1983-06-04T10:00.000Z' modificationReason: Index Apache Logs again services: - component: com.hurence.logisland.service.elasticsearch.Elasticsearch_5_4_0_ClientService documentation: elasticsearch service to sink records name: elasticsearch_service config: - key: hosts value: eshost:9300 - key: cluster.name value: escluster streams: - name: kafka_in component: com.hurence.logisland.stream.spark.KafkaRecordStreamParallelProcessing config: - key: kafka.input.topics value: logisland_raw - key: kafka.output.topics value: logisland_events - key: kafka.error.topics value: logisland_errors - key: kafka.input.topics.serializer value: none - key: kafka.output.topics.serializer value: com.hurence.logisland.serializer.KryoSerializer - key: kafka.error.topics.serializer value: com.hurence.logisland.serializer.JsonSerializer - key: kafka.metadata.broker.list value: sandbox:9092 - key: kafka.zookeeper.quorum value: sandbox:2181 - key: kafka.topic.autoCreate value: 'true' - key: kafka.topic.default.partitions value: '4' - key: kafka.topic.default.replicationFactor value: '1' pipeline: lastModified: '1983-06-04T10:00.000Z' modificationReason: Initial configuration processors: - component: com.hurence.logisland.processor.SplitText name: apache_parser documentation: parse apache logs with a regexp config: - key: record.type value: apache_log - key: value.regex value: (\S+)\s+(\S+)\s+(\S+)\s+\[([\w:\/]+\s[+\-]\d{4})\]\s+"(\S+)\s+(\S+)\s*(\S*)"\s+(\S+)\s+(\S+) - key: value.fields value: src_ip,identd,user,record_time,http_method,http_query,http_version,http_status,bytes_out - component: com.hurence.logisland.processor.elasticsearch.BulkAddElasticsearch documentation: a processor that indexes processed events in elasticsearch name: es_publisher config: - key: elasticsearch.client.service value: elasticsearch_service - key: default.index value: logisland - key: default.type value: event - key: timebased.index value: yesterday - key: es.index.field value: search_index - key: es.type.field value: record_type "304": description: | Nothing has been modified since the last call. In this case the body content will be completely ignored (hence the server can answer with an empty body to save network and resources). "404": description: Not found (the server probably does not handle this dataflow) default : description: Unexpected error post: tags: - dataflow operationId: notifyDataflowConfiguration summary: Push the configuration of running dataflows. description: >- In order to ensure business continuity, Logisland will contact the third party application in order to push a snapshot of the current configuration. The endpoint will be called: - On a regular basis (according to logisland configuration). - Each time the a dataflow or a pipeline configuration change has been applied. This service can be seen as well as a liveness ping. parameters: - name: jobId in: path type: string required: true description: logisland job id (aka the engine name) - in: body name: dataflow required: true schema: $ref: '#/definitions/DataFlow' responses: default : description: | The server should return HTTP 200 OK. By the way, the response is ignored by Logisland since the operation has a *fire and forget* nature. definitions: Property: type: object required: - key - value properties: key: type: string type: type: string default: "string" value: type: string Component: type: object required: - component - name properties: name: type: string component: type: string documentation: type: string config: type: array items: $ref: '#/definitions/Property' Service: type: object description: A logisland 'controller service'. allOf: - $ref: '#/definitions/Component' Processor: type: object description: A logisland 'processor'. allOf: - $ref: '#/definitions/Component' Versioned: type: object description: a versioned component properties: lastModified: type: string format: date-time description: the last modified timestamp of this pipeline (used to trigger changes). modificationReason: type: string description: Can be used to document latest changeset. required: - lastModified Pipeline: type: object description: Tracks stream processing pipeline configuration allOf: - $ref: '#/definitions/Versioned' - properties: processors: type: array items: $ref: '#/definitions/Processor' Stream: type: object allOf: - $ref: '#/definitions/Component' - properties: pipeline: $ref: '#/definitions/Pipeline' required: - pipeline DataFlow: type: object description: A streaming pipeline. allOf: - $ref: "#/definitions/Versioned" - properties: services: type: array description: The service controllers. items: $ref: '#/definitions/Service' streams: type: array description: The engine properties. items: $ref: '#/definitions/Stream'