== Getting started === Prerequisites A Kubernetes or OpenShift cluster is required to deploy Strimzi. Strimzi supports all kinds of clusters - from public and private clouds down to local deployments intended for development purposes. This guide expects that a Kubernetes or OpenShift cluster is available and the `kubectl` or `oc` command line tools are installed and configured to connect to the running cluster. When no existing Kubernetes or OpenShift cluster is available, `Minikube` or `Minishift` can be used to create a local cluster. More details can be found in <<_installing_kubernetes_and_openshift_cluster>> In order to execute the commands in this guide, your Kubernetes/OpenShift user needs to have the rights to create and manage RBAC resources (Roles and Role Bindings). === Cluster Controller Strimzi uses a component called the Cluster Controller to deploy and manage Kafka (including Zookeeper) and Kafka Connect clusters. The Cluster Controller is deployed as a process running inside your Kubernetes or OpenShift cluster. To deploy a Kafka cluster, a ConfigMap with the cluster configuration has to be created. Based on the information in that ConfigMap, the Cluster Controller will deploy a corresponding Kafka cluster. By default, the ConfigMap needs to be labeled with following labels: [source,yaml] strimzi.io/type: kafka strimzi.io/kind: cluster and contain the cluster configuration in a specific format. The ConfigMap format is described in <>. Strimzi contains example YAML files which make deploying a Cluster Controller easier. ==== Deploying to Kubernetes To deploy the Cluster Controller on Kubernetes, the following command should be executed: [source] kubectl create -f examples/install/cluster-controller To verify whether the Cluster Controller has been deployed successfully, the Kubernetes Dashboard or the following command can be used: [source] kubectl describe all ==== Deploying to OpenShift To deploy the Cluster Controller on OpenShift, the following commands should be executed: [source] oc create -f examples/install/cluster-controller oc create -f examples/templates/cluster-controller To verify whether the Cluster Controller has been deployed successfully, the OpenShift console or the following command can be used: [source] oc describe all === Kafka broker Strimzi uses StatefulSets feature of Kubernetes/OpenShift to deploy Kafka brokers. With StatefulSets, the pods receive a unique name and network identity and that makes it easier to identify the individual Kafka broker pods and set their identity (broker ID). The deployment uses **regular** and **headless** services: - regular services can be used as bootstrap servers for Kafka clients - headless services are needed to have DNS resolve the pods IP addresses directly As well as Kafka, Strimzi also installs a Zookeeper cluster and configures the Kafka brokers to connect to it. The Zookeeper cluster also uses StatefulSets. Strimzi provides two flavors of Kafka broker deployment: **ephemeral** and **persistent**. The **ephemeral** flavour is suitable only for development and testing purposes and not for production. The ephemeral flavour uses `emptyDir` volumes for storing broker information (Zookeeper) and topics/partitions (Kafka). Using `emptyDir` volume means that its content is strictly related to the pod life cycle (it is deleted when the pod goes down). This makes the in-memory deployment well-suited to development and testing because you don't have to provide persistent volumes. The **persistent** flavour uses PersistentVolumes to store Zookeeper and Kafka data. The PersistentVolume is acquired using a PersistentVolumeClaim – that makes it independent of the actual type of the PersistentVolume. For example, it can use HostPath volumes on Minikube or Amazon EBS volumes in Amazon AWS deployments without any changes in the YAML files. The PersistentVolumeClaim can use a StorageClass to trigger automatic volume provisioning. To deploy a Kafka cluster, a ConfigMap with the cluster configuration has to be created. The ConfigMap should have the following labels: [source,yaml] strimzi.io/type: kafka strimzi.io/kind: cluster Example ConfigMaps and the details about the ConfigMap format are in <>. ==== Deploying to Kubernetes To deploy a Kafka broker on Kubernetes, the corresponding ConfigMap has to be created. To create the ephemeral cluster using the provided example ConfigMap, the following command should be executed: [source] kubectl apply -f examples/configmaps/cluster-controller/kafka-ephemeral.yaml Another example ConfigMap is provided for persistent Kafka cluster. To deploy it, the following command should be run: [source] kubectl apply -f examples/configmaps/cluster-controller/kafka-persistent.yaml ==== Deploying to OpenShift For OpenShift, the Kafka broker is provided in the form of a template. The cluster can be deployed from the template either using the command line or using the OpenShift console. To create the ephemeral cluster, the following command should be executed: [source] oc new-app strimzi-ephemeral Similarly, to deploy a persistent Kafka cluster the following command should be run: [source] oc new-app strimzi-persistent === Kafka Connect The Cluster Controller can also deploy a https://kafka.apache.org/documentation/#connect[Kafka Connect] cluster which can be used with either of the Kafka broker deployments described above. It is implemented as a Deployment with a configurable number of workers. The default image currently contains only the Connectors distributed with Apache Kafka Connect: `FileStreamSinkConnector` and `FileStreamSourceConnector`. The REST interface for managing the Kafka Connect cluster is exposed internally within the Kubernetes/OpenShift cluster as `kafka-connect` service on port `8083`. Example ConfigMaps and the details about the ConfigMap format for deploying Kafka Connect can be found in <>. ==== Deploying to Kubernetes To deploy Kafka Connect on Kubernetes, the corresponding ConfigMap has to be created. An example ConfigMap can be created using the following command: [source] kubectl apply -f examples/configmaps/cluster-controller/kafka-connect.yaml ==== Deploying to OpenShift On OpenShift, Kafka Connect is provided in the form of a template. It can be deployed from the template either using the command line or using the OpenShift console. To create a Kafka Connect cluster from the command line, the following command should be run: [source] oc new-app strimzi-connect ==== Using Kafka Connect with additional plugins Strimzi Docker images for Kafka Connect contain, by default, only the `FileStreamSinkConnector` and `FileStreamSourceConnector` connectors which are part of Apache Kafka. To facilitate deployment with 3rd party connectors, Kafka Connect is configured to automatically load all plugins/connectors which are present in the `/opt/kafka/plugins` directory during startup. There are two ways of adding custom plugins into this directory: - Using a custom Docker image - Using the OpenShift build system with the Strimzi S2I image ===== Create a new image based on `strimzi/kafka-connect` Strimzi provides its own Docker image for running Kafka Connect which can be found on Docker Hub as https://hub.docker.com/r/strimzi/kafka-connect/[`strimzi/kafka-connect`]. This image could be used as a base image for building a new custom image with additional plugins. The following steps describe the process for creating such a custom image: 1. Create a new `Dockerfile` which uses `strimzi/kafka-connect` as the base image + [source,Dockerfile] ---- FROM strimzi/kafka-connect:latest USER root:root COPY ./my-plugin/ /opt/kafka/plugins/ USER kafka:kafka ---- 2. Build the Docker image and upload it to the appropriate Docker repository 3. Use the new Docker image in the Kafka Connect deployment: - On OpenShift, the template parameters `IMAGE_REPO_NAME`, `IMAGE_NAME` and `IMAGE_TAG` can be changed to point to the new image when the Kafka Connect cluster is being deployed - On Kubernetes, the Kafka Connect ConfigMap has to be modified to use the new image ===== Using OpenShift Build and S2I image OpenShift supports https://docs.openshift.org/3.6/dev_guide/builds/index.html[Builds] which can be used together with https://docs.openshift.org/3.6/creating_images/s2i.html#creating-images-s2i[Source-to-Image (S2I)] framework to create new Docker images. OpenShift Build takes a builder image with S2I support together with source code and/or binaries provided by the user and uses them to build a new Docker image. The newly created Docker Image will be stored in OpenShift's local Docker repository and can then be used in deployments. Strimzi provides a Kafka Connect builder image https://hub.docker.com/r/strimzi/kafka-connect-s2i/[`strimzi/kafka-connect-s2i`] with such S2I support. It takes user-provided binaries (with plugins and connectors) and creates a new Kafka Connect image. This enhanced Kafka Connect image can be used with our Kafka Connect deployment. The S2I deployment is again provided as an OpenShift template. It can be deployed from the template either using the command line or using the OpenShift console. To create Kafka Connect S2I cluster from the command line, the following command should be run: [source] oc new-app strimzi-connect-s2i Once the cluster is deployed, a new Build can be triggered from the command line: 1. A directory with Kafka Connect plugins has to be prepared first. For example: + [source,shell] ---- $ tree ./my-plugins/ ./my-plugins/ ├── debezium-connector-mongodb │   ├── bson-3.4.2.jar │   ├── CHANGELOG.md │   ├── CONTRIBUTE.md │   ├── COPYRIGHT.txt │   ├── debezium-connector-mongodb-0.7.1.jar │   ├── debezium-core-0.7.1.jar │   ├── LICENSE.txt │   ├── mongodb-driver-3.4.2.jar │   ├── mongodb-driver-core-3.4.2.jar │   └── README.md ├── debezium-connector-mysql │   ├── CHANGELOG.md │   ├── CONTRIBUTE.md │   ├── COPYRIGHT.txt │   ├── debezium-connector-mysql-0.7.1.jar │   ├── debezium-core-0.7.1.jar │   ├── LICENSE.txt │   ├── mysql-binlog-connector-java-0.13.0.jar │   ├── mysql-connector-java-5.1.40.jar │   ├── README.md │   └── wkb-1.0.2.jar └── debezium-connector-postgres ├── CHANGELOG.md ├── CONTRIBUTE.md ├── COPYRIGHT.txt ├── debezium-connector-postgres-0.7.1.jar ├── debezium-core-0.7.1.jar ├── LICENSE.txt ├── postgresql-42.0.0.jar ├── protobuf-java-2.6.1.jar └── README.md ---- 2. To start a new image build using the prepared directory, the following command has to be run: + [source] oc start-build my-connect-cluster-connect --from-dir ./my-plugins/ + _The name of the build should be changed according to the cluster name of the deployed Kafka Connect cluster._ 3. Once the build is finished, the new image will be used automatically by the Kafka Connect deployment. === Topic Controller Strimzi uses a component called the Topic Controller to manage topics in the Kafka cluster. The Topic Controller is deployed as a process running inside a Kubernetes/OpenShift cluster. To create a new Kafka topic, a ConfigMap with the related configuration (name, partitions, replication factor, ...) has to be created. Based on the information in that ConfigMap, the Topic Controller will create a corresponding Kafka topic in the cluster. Deleting a topic ConfigMap raises the deletion of the corresponding Kafka topic as well. The Cluster Controller is able to deploy a Topic Controller, which can be configured in the cluster ConfigMap. Alternatively, it is possible to deploy a Topic Controller manually, rather than having it deployed by the Cluster Controller. ==== Deploying through the Cluster Controller To deploy the Topic Controller through the Cluster Controller, its configuration needs to be provided in the cluster ConfigMap in the `topic-controller-config` field as a JSON string. For more information on the JSON configuration format see <>. ==== Deploying standalone Topic Controller If you are not going to deploy the Kafka cluster using the Cluster Controller but you already have a Kafka cluster deployed on Kubernetes or OpenShift, it could be useful to deploy the Topic Controller using the provided YAML files. In that case you can still leverage on the Topic Controller features of managing Kafka topics through related ConfigMaps. ===== Deploying to Kubernetes To deploy the Topic Controller on Kubernetes (not through the Cluster Controller), the following command should be executed: [source] kubectl create -f examples/install/topic-controller.yaml To verify whether the Topic Controller has been deployed successfully, the Kubernetes Dashboard or the following command can be used: [source] kubectl describe all ===== Deploying to OpenShift To deploy the Topic Controller on OpenShift (not through the Cluster Controller), the following command should be executed: [source] oc create -f examples/install/topic-controller To verify whether the Topic Controller has been deployed successfully, the OpenShift console or the following command can be used: [source] oc describe all ==== Topic ConfigMap When the Topic Controller is deployed by the Cluster Controller it will be configured to watch for "topic ConfigMaps" which are those with the following labels: [source,yaml] strimzi.io/cluster: strimzi.io/kind: topic When the Topic Controller is deployed manually the `strimzi.io/cluster` label is not necessary. The topic ConfigMap contains the topic configuration in a specific format. The ConfigMap format is described in <>.