= Spring Cloud Data Flow Server

The Data Flow Server deploys Streams by delegating to Spring Cloud Skipper and deploys Tasks directly.
 Both Skipper and the Data Flow server can deploy to the local machine, Cloud Foundry, and Kubernetes.

NOTE: The local version of the Data Flow server should only be used for Stream development but can be used in a production environment for Task deployment as a replacement for the Spring Cloud Batch Admin Server.


== Getting Started

This shows how to start the Data Flow Server and Shell to create the `time | log` stream.

=== Start the Data Flow Server and Shell

. Start Kafka locally (e.g. `kafka-server-start.sh /usr/local/etc/kafka/server.properties`)
. Build from the spring-cloud-dataflow root directory:
+
----
./mvnw clean install
----
+
. Start the Data Flow Server application:
+
----
java -jar spring-cloud-dataflow-server/target/spring-cloud-dataflow-server-<version>.jar
----
+
. Start the shell
+
----
$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-<version>.jar
----

=== Creating the `time | log` stream:

. Create the 'ticktock' stream:
+
----
dataflow:>stream create --name ticktock --definition "time | log"
Created new stream 'ticktock'
----
+
This is equivalent to the following HTTP POST request
+
----
$ curl -X POST -d "name=ticktock&definition=time | log" http://localhost:9393/streams/definitions?deploy=false
----
+
. List all streams available in the repository:
+
----
dataflow:>stream list
╔═══════════╤═════════════════╤══════════╗
║Stream Name│Stream Definition│  Status  ║
╠═══════════╪═════════════════╪══════════╣
║ticktock   │time | log       │undeployed║
╚═══════════╧═════════════════╧══════════╝
----
+
This is equivalent to the following HTTP get request
+
----
$ curl http://localhost:9393/streams/definitions
----
+
. Deploy the 'ticktock' stream:
+
----
dataflow:>stream deploy --name ticktock
Deployed stream 'ticktock'
----
+
This is equivalent to the following HTTP get request
+
----
$ curl -X POST http://localhost:9393/streams/deployments/ticktock
----

If successful you should see output similar to the following in the `Data Flow Server` console:

----
...o.s.c.d.spi.local.LocalAppDeployer    : deploying app ticktock.log instance 0
   Logs will be in /some/path/ticktock.log
----

If you tail the `stdout_0.log` file from the directory mentioned, you should see output similar to the following:

----
2016-04-26 15:10:18.320  INFO 59890 --- [pool-1-thread-1] log.sink    : 04/26/16 15:10:18
2016-04-26 15:10:19.322  INFO 59890 --- [pool-1-thread-1] log.sink    : 04/26/16 15:10:19
2016-04-26 15:10:20.322  INFO 59890 --- [pool-1-thread-1] log.sink    : 04/26/16 15:10:20
----

== Configuration

=== Default
To configure the Data Flow Server you can follow the configuration setup guidelines specified in the boot documentation found https://docs.spring.io/spring-boot/docs/1.5.13.RELEASE/reference/html/boot-features-external-config.html[here]

Note: The `dataflow-server.yml` containing the defaults can be found https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-starter-dataflow-server/src/main/resources/dataflow-server.yml[here]

=== Spring Cloud Configuration
The Spring Cloud Data Flow Server offers the user the ability to configure properties via
https://cloud.spring.io/spring-cloud-config/spring-cloud-config.html[spring-cloud-config].
All configurations retrieved from the cloud config will take precedence over Boot's
defaults enumerated above. The Spring Cloud Data Flow Server will look for the server at
`localhost:8888`, however this can be overwritten by setting the `spring.cloud.config.uri`
property to the desired url.

==== Cloud-Config-Server configuration

To specify a repository in the cloud config server configuration.yml for the Data Flow Server,
setup a repo profile with the pattern `spring-cloud-dataflow-server`. For example:

[source,yml]
----
spring:
  cloud:
     config:
       server:
         git:
           uri: https://github.com/myrepo/configurations
           repos:
            spring-cloud-dataflow-server:
              pattern: spring-cloud-dataflow-server
              uri: https://github.com/myrepo/configurations
              searchPaths: dataFlowServer
----

==== Fail Fast
In some cases, it may be desirable to fail startup of a service if it cannot connect to
the Config Server. If this is the desired behavior, set the bootstrap configuration
property `spring.cloud.config.failFast=true` and the client will halt with an Exception.

==== Note
If the Data Flow Server cannot connect to the cloud config server, the
following warning message will be logged:
----
`WARN 42924 --- [main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/spring-cloud-dataflow-server/default":Connection refused; nested exception is java.net.ConnectException: Connection refused`
----
To disable the cloud config server set the `spring.cloud.config.enabled` property to `false`.

== Docker-Compose Integration Tests

NOTE: The docker-compose files have been decomposed so that you have to specify the database and messaging engine separately. Previously mysql and kafka was the default.

To run the docker compose integration, enable the `-Pfailsafe` maven profile and the `-Dgroups="docker-compose"` JUnit tag.

. You can run just the integration tests like this:

----
./mvnw clean test-compile failsafe:integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose"
----

With the help of the `TestProperties` properties one can change the docker-compose files used, the exact versions of the
SCDF, Skipper servers installed, or the versions Stream and Task apps:

----
./mvnw clean test-compile failsafe:integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.paths="../src/docker-compose/docker-compose.yml,../src/docker-compose/docker-compose-influxdb.yml,../src/docker-compose/docker-compose-postgres.yml,../src/docker-compose/docker-compose-rabbitmq.yml" \
   -Dtest.docker.compose.stream.apps.uri=https://dataflow.spring.io/rabbitmq-maven-latest \
   -Dtest.docker.compose.dataflow.version=2.8.0-SNAPSHOT \
   -Dtest.docker.compose.skipper.version=2.7.0-SNAPSHOT \
----

The `test.docker.compose.paths` property accepts comma separated list of docker compose file names and supports `file:`, `classpath:`, and `http:`/`https:` URI schemas. If the schema prefix is not explicitly set, the file is treated as local one.

----
./mvnw clean test-compile integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.paths=""../src/docker-compose/docker-compose.yml,\
    ../src/docker-compose/docker-compose-mysql.yml, \
    ../src/docker-compose/docker-compose-kafka.yml, \
    ../src/docker-compose/docker-compose-dood.yml, \
    ../src/docker-compose/docker-compose-prometheus.yml" \
   -Dtest.docker.compose.stream.apps.uri=https://dataflow.spring.io/kafka-docker-latest \
   -Dtest.docker.compose.task.apps.uri=https://dataflow.spring.io/task-docker-latest \
   -Dtest.docker.compose.dataflow.version=2.8.0-SNAPSHOT \
   -Dtest.docker.compose.skipper.version=2.7.0-SNAPSHOT \
   -Dtest.docker.compose.apps.port.range=80 \
   -Dtest.docker.compose.pullOnStartup=false
----

Use the `-Dtest.docker.compose.dood=true` property to activate the DooD (Docker-Out-Of-Docker) shortcut that implicitly will add `docker-compose-dood.yml` to your configuration :

----
./mvnw clean test-compile integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.dood=true \
   -Dtest.docker.compose.pullOnStartup=false
----

Use the `-Dit.test=class-name#method-name` to filter only selected tests.

=== Run Integration Tests with HTTPS connections

----
./mvnw clean test-compile integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.paths=""../src/docker-compose/docker-compose.yml,\
    ../src/docker-compose/docker-compose-mysql.yml, \
    ../src/docker-compose/docker-compose-kafka.yml, \
    ../src/docker-compose/docker-compose-ssl.yml" \
   -Dtest.docker.compose.dataflow.uri=https://dataflow-server:9393 \
   -Dtest.docker.compose.skipper.uri=https://skipper-server:7577 \
   -Dtest.docker.compose.waiting.for.service.format=https://$HOST:$EXTERNAL_PORT \
   -Dit.test=DataFlowIT#streamPartitioning \
   -Dspring.cloud.dataflow.client.server-uri=https://localhost:9393 \
   -Dspring.cloud.dataflow.client.skip-ssl-validation=true \
   -Dtest.platform.connection.application-over-https=true
----

Use the following env. variables to run an IT test from IDE:
----
TEST_DOCKER_COMPOSE_PATHS=../src/docker-compose/docker-compose.yml,\
    ../src/docker-compose/docker-compose-mysql.yml, \
    ../src/docker-compose/docker-compose-kafka.yml, \
    ../src/docker-compose/docker-compose-ssl.yml
TEST_DOCKER_COMPOSE_SKIPPER_URI=https://skipper-server:7577
TEST_DOCKER_COMPOSE_DATAFLOW_URI=https://dataflow-server:9393
TEST_DOCKER_COMPOSE_WAITING_FOR_SERVICE_FORMAT=https://$HOST:$EXTERNAL_PORT
TEST_PLATFORM_CONNECTION_APPLICATIONOVERHTTPS=true
SPRING_CLOUD_DATAFLOW_CLIENT_SKIPSSLVALIDATION=true
SPRING_CLOUD_DATAFLOW_CLIENT_SERVERURI=https://localhost:9393
COMPOSE_HTTP_TIMEOUT=300
----

Use the following env. variables to run an IT test from IDE with existing docker-compose cluster:
----
TEST_DOCKER_COMPOSE_DISABLE_EXTENSION=true
SPRING_CLOUD_DATAFLOW_CLIENT_SERVERURI=https://localhost:9393
SPRING_CLOUD_DATAFLOW_CLIENT_SKIPSSLVALIDATION=true
TEST_PLATFORM_CONNECTION_APPLICATIONOVERHTTPS=true
----

Create HTTPS docker-compose from shell:
----
export DATAFLOW_URI=https://dataflow-server:9393
export SKIPPER_URI=https://skipper-server:7577
docker-compose -f ./src/docker-compose/docker-compose.yml \
               -f ./src/docker-compose/docker-compose-mysql.yml, \
               -f ./src/docker-compose/docker-compose-kafka.yml, \
               -f ./src/docker-compose/docker-compose-ssl.yml
----


== Run Integration Tests against k8s

For this you need a pre-installed Data Flow on GKE or minikube.
Register the OOTB kafka/docker and task/docker apps and the run:

----
./mvnw integration-test -pl spring-cloud-dataflow-server -Dtest=foo -DfailIfNoTests=false -Dgroups="docker-compose" \
    -Dtest.docker.compose.disable.extension=true \
    -Dspring.cloud.dataflow.client.server-uri=http://your-k8s-scdf-server \
    -Pfailsafe
----

* replace `http://your-k8s-scdf-server` with the url of your Data Flow server.
* the `test.docker.compose.disable.extension=true` property disables the docker-compose fixture.
* the `spring.cloud.dataflow.client.server-uri=` property points to the pre-installed DataFlow REST API.

For the analytic tests you need to pre-install Prometheus according to the installation instructions and forward the prometheus server's 9090 port
----
kubectl port-forward prometheus-67896dcc8-7wm4h 9090:9090
----
Replace the `prometheus-67896dcc8-7wm4h` with your pod name.

Or use the `test.platform.connection.prometheusUrl` property to set an alternative Prometheus url.

=== Integration Tests with Minikube

The Integration Tests require that the Kubernetes cluster running Data Flow provides a https://kubernetes.io/docs/concepts/services-networking/[Load Balancer] service.

Most cloud providers offer the Load Balancer service by default. For Minikube, you would need to install one yourself.
The https://metallb.universe.tf/[MetalLB] is a lightweight load balancer implementation for Kubernetes and is one option to provide the `LoadBalancer` service on Minikube.

* Install MetalLB
----
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml
----

* Create a ConfigMap with the appropriate settings

----
cat <<EOF >metallbcm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.99.116/28
EOF
----

NOTE: The IP in the addresses section is based on running Minikube via the default VirtualBox driver with default Minikube IP  is `192.168.99.100`.
For different Minikube configurations, you might need to customize the address.

* Apply the ConfigMap:

----
kubectl apply -f metallbcm.yaml
----

At this point MetalLB is setup and ready to hand out IP's to any service of type LoadBalancer, just as things would work in say GKE.

* If a change is made to the ConfigMap and re-applied, bounce the pods in the metallb-system namespace to pick-up the new changes, ie:

----
kubectl delete pod --all -n metallb-system
----

== Testcontainers Integration Tests

Some database and security tests are executed in containers. There are a set
of junit tags which can be used to disect which tests to run. Here are some
examples how to run those locally.

Run oauth security tests:
----
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=oauth -Pfailsafe
----

Run database tests:
----
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=database -Pfailsafe
----

Run mariadb, postgres or mssql tests:
----
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=mariadb -Pfailsafe
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=postgres -Pfailsafe
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=mssql -Pfailsafe
----

Run mariadb tests with shared database and with latest dataflow:
----
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups="mariadb&database-shared&dataflow_main" -Pfailsafe
----

Run database and oauth tests:
----
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups="database|oauth" -Pfailsafe
----
