# Run k0s in Docker You can create a k0s cluster on top of Docker. ## Prerequisites You will require a [Docker environment](https://docs.docker.com/get-docker/) running on a Mac, Windows, or Linux system. ## Container images The k0s OCI images are published to both Docker Hub and GitHub Container registry. For simplicity, the examples given here use Docker Hub (GitHub requires separate authentication, which is not covered here). The image names are as follows: - `docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}` - `ghcr.io/k0sproject/k0s:{{{ k0s_docker_version }}}` **Note:** Due to Docker's tag validation scheme, `-` is used as the k0s version separator instead of the usual `+`. For example, k0s version `{{{ k0s_version }}}` is tagged as `docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}`. ## Start k0s ### 1. Run a controller By default, running the k0s OCI image will launch a controller with workloads enabled (i.e. a controller with the `--enable-worker` flag) to provide an easy local testing "cluster": ```sh docker run -d --name k0s-controller --hostname k0s-controller \ -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \ --tmpfs /run `# this is where k0s stores runtime data` \ --privileged `# this is the easiest way to enable container-in-container workloads` \ -p 6443:6443 `# publish the Kubernetes API server port` \ docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} ``` Explanation of command line arguments: - `-d` runs the container in detached mode, i.e. in the background. - `--name k0s-controller` names the container "k0s-controller". - `--hostname k0s-controller` sets the host name of the container to "k0s-controller". - `-v /var/lib/k0s -v /var/log/pods` creates two Docker volumes and mounts them to `/var/lib/k0s` and `/var/log/pods` respectively inside the container, ensuring that cluster data persists across container restarts. - `--tmpfs /run` **TODO** - `--privileged` gives the container the elevated privileges that k0s needs to function properly within Docker. See the section on [adding additional workers](#2-optional-add-additional-workers) for a more detailed discussion of privileges. - `-p 6443:6443` exposes the container's Kubernetes API server port 6443 to the host, allowing you to interact with the cluster externally. - `docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}` is the name of the k0s image to run. By default, the k0s image starts a k0s controller with worker components enabled within the same container, creating a cluster with a single controller-and-worker node using the following command: ```Dockerfile {% include "../Dockerfile" start="# Start CMD" end="# End CMD" %} ``` Alternatively, a controller-only node can be run like this: ```sh docker run -d --name k0s-controller --hostname k0s-controller \ --read-only `# k0s won't write any data outside the below paths` \ -v /var/lib/k0s `# this is where k0s stores its data` \ --tmpfs /run `# this is where k0s stores runtime data` \ --tmpfs /tmp `# allow writing temporary files` \ -p 6443:6443 `# publish the Kubernetes API server port` \ docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} \ k0s controller ``` Note the addition of `k0s controller` to override the image's default command. Also note that a controller-only node requires fewer privileges. ### 2. (Optional) Add additional workers You can add multiple worker nodes to the cluster and then distribute your application containers to separate workers. 1. Acquire a [join token](k0s-multi-node.md#3-create-a-join-token) for the worker: ```sh token=$(docker exec k0s-controller k0s token create --role=worker) ``` 2. Run the container to create and join the new worker: ```sh docker run -d --name k0s-worker1 --hostname k0s-worker1 \ -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \ --tmpfs /run `# this is where k0s stores runtime data` \ --privileged `# this is the easiest way to enable container-in-container workloads` \ docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} \ k0s worker $token ``` Alternatively, with fine-grained privileges: ```sh docker run -d --name k0s-worker1 --hostname k0s-worker1 \ -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \ --tmpfs /run `# this is where k0s stores runtime data` \ --security-opt seccomp=unconfined \ --device /dev/kmsg \ --cap-add sys_admin \ --cap-add net_admin \ --cap-add sys_ptrace \ --cap-add sys_resource \ --cap-add syslog \ docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} \ k0s worker "$token" ``` Notes on the security-related flags: - `--security-opt seccomp=unconfined` is required for `runc` to access the [session keyring]. - `--device /dev/kmsg` makes `/dev/kmsg` visible from inside the container. The kubelet's OOM watcher uses this. Notes on [Linux capabilities]: - `CAP_SYS_ADMIN` allows for a variety of administrative tasks, including mounting file systems and managing namespaces, which are necessary for creating and configuring nested containers. - `CAP_NET_ADMIN` allows manipulation of network settings such as interfaces and routes, allowing containers to create isolated or bridged networks, and so on. - `CAP_SYS_PTRACE` allows to inspect and modify processes, used to monitor other containers in a nested environment. - `CAP_SYS_RESOURCE` allows containers to override resource limits for things like memory or file descriptors, used to manage and adjust resource allocation in nested container environments. - `CAP_SYSLOG` allows containers to perform privileged syslog operations. This is required in order to read `/dev/kmsg`. Note that more privileges may be required depending on your cluster configuration and workloads. Repeat this step for each additional worker node and adjust the container and host names accordingly. Make sure that the workers can reach the controller on the [required ports]. If you are using Docker's default bridged network, this should be the case. [session keyring]: https://www.man7.org/linux/man-pages/man7/session-keyring.7.html [Linux capabilities]: https://www.man7.org/linux/man-pages/man7/capabilities.7.html [required ports]: networking.md#controller-worker-communication ### 3. Access your cluster #### a) Using kubectl within the container To check cluster status and list nodes, use: ```sh docker exec k0s-controller k0s kubectl get nodes ``` #### b) Using kubectl locally To configure local access to your k0s cluster, follow these steps: 1. Generate the kubeconfig: ```sh docker exec k0s-controller k0s kubeconfig admin > ~/.kube/k0s.config ``` 2. Update kubeconfig with Localhost Access: To automatically replace the server IP with localhost dynamically in `~/.kube/k0s.config`, use the following command: ```sh sed -i '' -e "$(awk '/server:/ {print NR; exit}' ~/.kube/k0s.config)s|https://.*:6443|https://localhost:6443|" ~/.kube/k0s.config ``` This command updates the kubeconfig to point to localhost, allowing access to the API server from your host machine 3. Set the KUBECONFIG Environment Variable: ```sh export KUBECONFIG=~/.kube/k0s.config ``` 4. Verify Cluster Access: ```sh kubectl get nodes ``` #### c) Use [Lens] Access the k0s cluster using Lens by following the instructions on [how to add a cluster]. [Lens]: https://k8slens.dev/ [how to add a cluster]: https://docs.k8slens.dev/getting-started/add-cluster/ ## Use Docker Compose (alternative) As an alternative you can run k0s using Docker Compose: ```yaml {% include "compose.yaml" %} ``` Below is a more complex example, using Traefik as a load balancer, along with three controller and three worker nodes: ```yaml {% include "compose-cluster.yaml" %} ``` Running the above: ```console ❯ docker compose up -d [+] Running 11/11 ✔ Network compose-cluster_k0s-net Created 0.1s ✔ Volume "compose-cluster_k0s-token-secrets" Created 0.0s ✔ Volume "compose-cluster_k0s-controller-token" Created 0.0s ✔ Volume "compose-cluster_k0s-worker-token" Created 0.0s ✔ Container k0s-lb Started 0.5s ✔ Container k0s-controller-1 Started 11.8s ✔ Container k0s-controller-2 Started 12.2s ✔ Container k0s-worker-1 Started 12.4s ✔ Container k0s-worker-2 Started 12.3s ✔ Container k0s-worker-3 Started 12.1s ✔ Container k0s-controller-3 Started 12.5s ``` After a short while: ```console $ docker exec k0s-controller-1 k0s kc get node,po -A NAME STATUS ROLES AGE VERSION node/k0s-worker-1 Ready 1m36s {{{ k8s_version }}}+k0s node/k0s-worker-2 Ready 1m36s {{{ k8s_version }}}+k0s node/k0s-worker-3 Ready 1m36s {{{ k8s_version }}}+k0s NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/coredns-7d4f7fbd5c-54lxp 1/1 Running 0 1m27s kube-system pod/coredns-7d4f7fbd5c-pwbck 1/1 Running 0 1m27s kube-system pod/konnectivity-agent-5g8pn 1/1 Running 0 1m22s kube-system pod/konnectivity-agent-6rp7r 1/1 Running 0 1m22s kube-system pod/konnectivity-agent-zx9fn 1/1 Running 0 1m22s kube-system pod/kube-proxy-9m77t 1/1 Running 0 1m36s kube-system pod/kube-proxy-v5vs6 1/1 Running 0 1m36s kube-system pod/kube-proxy-xfw2h 1/1 Running 0 1m36s kube-system pod/kube-router-6c62v 1/1 Running 0 1m36s kube-system pod/kube-router-98ss8 1/1 Running 0 1m36s kube-system pod/kube-router-lr46f 1/1 Running 0 1m36s kube-system pod/metrics-server-7778865875-fzhx6 1/1 Running 0 1m37s ``` ## Known limitations ### No custom Docker networks Currently, k0s nodes cannot be run if the containers are configured to use custom networks (for example, with `--net my-net`). This is because Docker sets up a custom DNS service within the network which creates issues with CoreDNS. No completely reliable workarounds are available, however no issues should arise from running k0s cluster(s) on a bridge network. ## Next Steps - [Install using k0sctl](k0sctl-install.md): Deploy multi-node clusters using just one command - [Control plane configuration options](configuration.md): Networking and data store configuration - [Worker node configuration options](worker-node-config.md): Node labels and kubelet arguments - [Support for cloud providers](cloud-providers.md): Load balancer or storage configuration - [Installing the Traefik Ingress Controller](examples/traefik-ingress.md): Ingress deployment information