# Deployment to Kubernetes

This document describes how to use the Carvel packages to deploy Spring Cloud Data Flow, and it also has a section on creating local development cluster using minikube or kind.

## Deployment using Carvel

Deployment of a carvel package requires the installation of tools and specific Kubernetes controllers. Then you will add the package repository to the cluster and install the application.

For local minikube or kind cluster you can use: Configure Kubernetes for local development or testing, and follow the instructions until the section Deploy Spring Cloud Data Flow

### Required Tools

- `kubectl` - Kubernetes CLI (Install with `brew install kubectl`)

- `carvel` - Packaging and Deployment tools

Carvel CLI can be installed using:

    wget -O- https://carvel.dev/install.sh | bash
    # or with curl...
    curl -L https://carvel.dev/install.sh | bash

Alternative following the instructions at the bottom of the home page at carvel.dev

The following tools are use by the scripts.

- `jq` - lightweight JSON parser

- `yq` - lightweight YAML parser

- `wget` - Invoke http requests.

- `dirname` provides the directory part of a filename.

- `readlink` provides absolute path of a relative link.

Note | Some of these utilities are not installed in macOS or *nix by default but will be available from MacPorts or HomeBrew.

### Scripts

These scripts assume you are connected to a Kubernetes cluster and `kubectl` is available.

Name | Arguments | Descriptions
--- | --- | ---
`start-deploy.sh` | <broker> [scdf-type] [namespace] [release|snapshot] | Configures environmental variables needs for the rest of the scripts. `BROKER`, `NS` and `SCDF_TYPE` are set. The default `NS` is `scdf`. The namespace will be created if it doesn’t exist by `setup-scdf-repo.sh`. The default `SCDF_TYPE` is `oss`. release|snapshot and scdf-type will determine the value of `PACKAGE_VERSION` set.
`prepare-cluster.sh` | N/A | Installs cert-manager, secretgen-controller and kapp-controller
`carvel-use-template.sh` | [scdf-type] (oss, pro) | Creates `scdf-values.yml` in current directory based on `scdf-pro-values.yml` or `scdf-oss-values.yml`
`setup-scdf-repo.sh` | [scdf-type] (oss, pro) | Creates the namespace and installs the relevant Carvel package and credentials. If the optional scdf-type is not provided the environmental variable `SCDF_TYPE` will be used.
`configure-prometheus-proxy.sh` | <host> <port> [step] | Configures Spring Boot Actuator properties for Data Flow, Skipper, Streams and Tasks. Default `step` is 10s
`configure-database.sh` | <app> <database> <url> <username/secret-name> [password/secret-username-key] [secret-password-key] | If only secret-name is provided then secret-username-key defaults to `username` and secret-password-key defaults to `password`. The following 3 combinations are allowed after the url: <secret-name> <secret-name> <username-key> <password-key> <username> <password>
`deploy-scdf.sh` | [app-name] | Deploys the application using the package and `scdf-values.yml` in the current directory. The default app-name is `scdf-${SCDF_TYPE}`.
`update-scdf.sh` | [app-name] | Updated the deployed application using a modified values file. The default app-name is `scdf-${SCDF_TYPE}`.
`export-dataflow-ip.sh` | N/A | Will print the URL to access dataflow. If you use `source ./export-dataflow-ip.sh` it will export `DATAFLOW_URL` to be used by `register-apps.sh`
`register-apps.sh` | <broker> [stream-application-version] | broker must be one of rabbit or kafka. stream-application-version is optional and will install the latest version. The latest version is 2021.1.2

Note | Take note that the registration of application in the pro version can take a few minutes since it retrieves all version information and metadata upfront.

### Preparation

You will need to prepare a values file named scdf-values.yml The following steps will provide help.

#### Prepare Configuration parameters

Executing the following script will configure the environmental variables needed.

    source ./carvel/start-deploy.sh <broker> <namespace> [scdf-type] [release|snapshot]

Where:

- `broker` is one of rabbitmq or kafka

- `namespace` A valid Kubernetes namespace other than `default`

- `scdf-type` One of oss or pro. oss is the default.

- `release|snapshot` and `scdf-type` will determine the value of `PACKAGE_VERSION`.

The environmental variables can also be configured manually to override the values.

Name | Description | Default
--- | --- | ---
`PACKAGE_VERSION` | Version of Carvel package. | Release version
`DATAFLOW_VERSION` | Version of Spring Cloud Data Flow | 2.10.3
`SKIPPER_VERSION` | Version of Spring Cloud Skipper | 2.9.3
`REGISTRY` | Url and repository of package registry. Format `<private-registry-host/repo-name>`. This will be used to prefix the carvel repo and package. | `docker.io/springcloud`
`BROKER` | One of `kafka` or `rabbitmq` | `rabbitmq`
`DATABASE` | One of `mariadb` or `postgresql`. The default is `postgresql`. This will only apply when you `deploy-local-database.sh` | `postgresql`
`NS` | A Kubernetes namespace other than `default`. | `scdf`
`SCDF_TYPE` | One of `oss` or `pro`. | `oss`

#### Prepare Configuration file

Create a file name `scdf-values.yml` by executing:

    ./carvel/carvel-use-template.sh

Edit the file as needed to configure the deployment. The `deploy-local-` scripts will

Uses scdf-type previously selected.

### Prepare cluster and add repository

Login to docker and optionally registry.pivotal.io for Spring Cloud Data Flow Pro.

    # When deploying SCDF Pro.
    export TANZU_DOCKER_USERNAME="<tanzu-net-username>"
    export TANZU_DOCKER_PASSWORD="<tanzu-net-password>"
    docker login --username $TANZU_DOCKER_USERNAME --password $TANZU_DOCKER_PASSWORD registry.pivotal.io

    # Always required to ensure you don't experience rate limiting with Docker HUB
    export DOCKER_HUB_USERNAME="<docker-hub-username>"
    export DOCKER_HUB_PASSWORD="<docker-hub-password>"
    docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD index.docker.io

Install carvel kapp-controller, secretgen-controller and certmanager

    ./carvel/prepare-cluster.sh

Load scdf repo package for the scdf-type

    ./carvel/setup-scdf-repo.sh

### Install supporting services

In a production environment you should be using supported database and broker services or operators along with shared observability tools.

For local development or demonstration the following can be used to install database, broker and prometheus.

#### Deploy local database.

    ./carvel/deploy-local-database.sh <database>  # (1)

1. `database` must be one of `postgresql` or `mariadb`. Default is postgresql or configure in `DATABASE` using `start-deploy.sh`.

Note | This script updates `scdf-values.yml` with the correct secret name.

#### Deploy local message broker.

    ./carvel/deploy-local-broker.sh

#### Deploy local Prometheus and proxy.

    ./carvel/deploy-local-prometheus.sh

This script also configures the Grafana endpoint in `scdf-values.yml`

### Configure Prometheus proxy

In the case where and existing prometheus and prometheus proxy is deployed the proxy can be configured using:

    ./carvel/configure-prometheus-proxy.sh <host> <port> [step]

### Deploy Spring Cloud Data Flow

    ./carvel/deploy-scdf.sh
    source ./carvel/export-dataflow-ip.sh
    # expected output: Dataflow URL: <url-to-access-dataflow>
    ./carvel/register-apps.sh

### Update deployed application.

You can modify the values file used during installation and then update the deployment using `./carvel/update-scdf.sh`

### Configure Kubernetes for local development or testing

#### Prerequisites

You will need to install kubectl and then kind or minikube for a local cluster.

All the examples assume you have cloned the `spring-cloud-dataflow` repository and are executing the scripts from `deploy/k8s`.

On macOS you may need to install `realpath` from Macports or `brew install realpath`

Note | The scripts require a shell like `bash` or `zsh` and should work on Linux, WSL 2 or macOS.

#### Steps

- Choose Kubernetes provider. Kind, Minikube or remote GKE or TMC.

- Decide the namespace to use for deployment if not `default`.

- Configure Kubernetes and loadbalancer.

- Choose Broker with `export BROKER=kafka|rabbitmq`

- Build or Pull container images for Skipper and Data Flow Server.

- Deploy and Launch Spring Cloud Data Flow.

- Export Data Flow Server address to env.

##### Kubernetes Provider

How do I choose between minikube and kind? kind will generally provide quicker setup and teardown time than Minikube. There is little to choose in terms of performance between the 2 apart from being able to configure limits on CPUs and memory when deploying minikube. So in the case where you have memory constraints or need to enforce memory limitations Minikube will be a better option.

##### Kubectl

You will need to install kubectl in order to configure the Kubernetes cluster

##### Kind

Kind is Kubernetes in docker and ideal for local development.

- Installation

- LoadBalancer

The LoadBalancer will be installed by the `configure-k8s.sh` script by will require an update to a yaml file to provide the address range available to the LoadBalancer.

##### Minikube

Minikube uses one of a selection of drivers to provide a virtualization environment.

- Installation

- LoadBalancer

Note | Delete existing Minikube installation if you have any. `minikube delete`

##### Remote TMC Cluster

Tanzu Mission Control

#### Building and loading containers.

For local development you need control of the containers used in the local environment.

In order to ensure to manage the specific versions of data flow and skipper containers you can set SKIPPER_VERSION and DATAFLOW_VERSION environmental variable and then invoke `./images/pull-dataflow.sh` and `./images/pull-skipper.sh` or if you want to use a locally built application you can invoke `./images/build-skipper-image.sh` and `./images/build-dataflow.sh`

#### Configure k8s environment

You can invoke one of the following scripts to choose the type of installation you are targeting:

    ./k8s/use-kind.sh [<namespace>] [<database>] [<broker>]
    ./k8s/use-mk-docker.sh [<namespace>] [<database>] [<broker>]
    ./k8s/use-mk-kvm2.sh [<namespace>] [<database>] [<broker>]
    ./k8s/use-mk.sh <driver> [<namespace>] [<database>] [<broker>] # (1)
    ./k8s/use-tmc.sh <cluster-name> [<namespace>] [<database>] [<broker>]
    ./k8s/use-gke.sh <cluster-name> [<namespace>] [<database>] [<broker>]

1. <driver> must be one of `kvm2`, `docker`, `vmware`, `virtualbox`, `vmwarefusion` or `hyperkit`. `docker` is the recommended option for local development.

Note | `<namespace>` will be `default` if not provided. The default `<database>` is `postgresql` and the default `<broker>` is `kafka`.

Since these scripts export environmental variable they need to be executes as in the following example:

    source ./k8s/use-mk-docker.sh test-ns postgresql rabbitmq

##### TMC or GKE Cluster in Cloud

The cluster must exist before use and you should use the relevant cli to login before executing `source ./k8s/use-gke.sh`

##### Create Local Cluster.

The following script will create the local cluster.

    # Optionally add to control cpu and memory allocation.
    export MK_ARGS="--cpus=8 --memory=12g"
    ./k8s/configure-k8s.sh

- For kind follow instruction to update `./k8s/yaml/metallb-configmap.yaml` and then apply using `kubectl apply -f ./k8s/yaml/metallb-configmap.yaml`

- For minikube launch a new shell and execute `minikube tunnel`

##### Deploy Spring Cloud Data Flow.

The `use-*` scripts will configure the values of BROKER and DATABASE.

###### Configure Broker

    export BROKER=<broker> # (1)

1. <broker> one of `kafka` or `rabbitmq`

###### Configure Database

    export DATABASE=<database> # (1)

1. <database> one of `mariadb` or `postgresql`

Docker credentials need to be configured for Kubernetes to pull the various container images.

For Docker Hub you can create a personal free account and use a personal access token as your password.

Test your docker login using `./k8s/docker-login.sh`

    export DOCKER_SERVER=https://docker.io
    export DOCKER_USER=<docker-userid>
    export DOCKER_PASSWORD=<docker-password>
    export DOCKER_EMAIL=<email-of-docker-use>

Before you can install SCDF you will need to pull the following images to ensure they are present for uploading to the k8s cluster.

Use:

    ./images/pull-app-images.sh
    ./images/pull-dataflow.sh
    ./images/pull-skipper.sh
    ./images/pull-composed-task-runner.sh

    ./k8s/install-scdf.sh
    source ./k8s/export-dataflow-ip.sh

##### Delete the deployment from the cluster.

    ./k8s/delete-scdf.sh

##### Delete the cluster

This script will also delete the TMC cluster if you have configured one.

    ./k8s/destroy-k8s.sh

#### Utilities

The following list of utilities may prove useful.

Name | Description
--- | ---
`k9s` | k9s is a text based monitor to explore the Kubernetes cluster.
`kail` | Extra and tail the logs of various pods based on various naming criteria.

##### `kail`

- Using kail to log activity related to a specific stream.

    kail --label=spring-group-id=<stream-name>

- Using kail to log all pods in specific namespace.

    kail --ns=<namespace>

#### Scripts

Some of the scripts apply to local containers as well and can be found in `src/local`, the Kubernetes specific scripts are in `deploy/k8s`

Script | Description
--- | ---
`./images/build-app-images.sh` | Build all images of Restaurant Sample Stream Apps
`./images/pull-app-images.sh` | Pull all images of Restaurant Sample Stream Apps from Docker Hub
`./images/pull-dataflow.sh` | Pull dataflow from DockerHub based on `DATAFLOW_VERSION`.
`./images/pull-scdf-pro.sh` | Pull Dataflow Pro from Tanzu Network based on `SCDF_PRO_VERSION`.
`./images/pull-skipper.sh` | Pull Skipper from DockerHub base on the `SKIPPER_VERSION`.
`./images/build-dataflow-image.sh` | Build a docker image from the local repo of Dataflow
`./images/build-scdf-pro-image.sh` | Build a docker image from the local repo of Dataflow Pro. Set `USE_PRO=true` in environment to use Dataflow Pro
`./images/build-skipper-image.sh` | Build a docker image from the local repo of Skipper.
`./k8s/configure-k8s.sh` | Configure the Kubernetes environment based on your configuration of K8S_DRIVER.
`./k8s/delete-scdf.sh` | Delete all Kubernetes resources create by the deployment.
`./k8s/destroy-k8s.sh` | Delete cluster, kind or minikube.
`./k8s/export-dataflow-ip.sh` | Export the url of the data flow server to `DATAFLOW_IP`
`./k8s/export-http-url.sh` | Export the url of an http source of a specific flow by name to `HTTP_APP_URL`
`./k8s/install-scdf.sh` | Configure and deploy all the containers for Spring Cloud Dataflow
`./k8s/load-images.sh` | Load all container images required by tests into kind or minikube to ensure you have control over what is used.
`./k8s/load-image.sh` | Load a specific container image into local kind or minikube.
`src/local/local-k8s-acceptance-tests.sh` | Execute acceptance tests against cluster where `DATAFLOW_IP` is pointing.
`./k8s/register-apps.sh` | Register the Task and Stream apps used by the unit tests.

Important | Please report any errors with the scripts along with detail information about the relevant environment.