<!--
SPDX-FileCopyrightText: 2020 k0s authors
SPDX-License-Identifier: CC-BY-SA-4.0
-->

# Configuration options

## Using a configuration file

k0s can be installed without a config file. In that case the default configuration will be used. You can, though, create and run your own non-default configuration (used by the k0s controller nodes).

k0s supports providing only partial configurations. In case of partial configuration is provided, k0s will use the defaults for any missing values.

1. Generate a YAML config file that uses the default settings.

    ```shell
    mkdir -p /etc/k0s
    k0s config create > /etc/k0s/k0s.yaml
    ```

2. Modify the new YAML config file according to your needs, refer to [Configuration file reference](#configuration-file-reference) below. You can remove the default values if wanted as k0s supports partial configs too.

3. Install k0s with your new config file.

    ```shell
    sudo k0s install controller -c /etc/k0s/k0s.yaml
    ```

4. If you need to modify your existing configuration later on, you can change your config file also when k0s is running, but remember to restart k0s to apply your configuration changes.

    ```shell
    sudo k0s stop
    sudo k0s start
    ```

## Configuring k0s via k0sctl

k0sctl can deploy your configuration options at cluster creation time. Your
options should be placed in the `spec.k0s.config` section of the k0sctl
configuration file. See the section on how to install [k0s via
k0sctl][k0sctl-install] and the [k0sctl README] for more information.

[k0sctl-install]: k0sctl-install.md
[k0sctl README]: https://github.com/k0sproject/k0sctl/blob/main/README.md

## Configuration file reference

**CAUTION**: As many of the available options affect items deep in the stack, you should fully understand the correlation between the configuration file components and your specific environment before making any changes.

A YAML config file follows, with defaults as generated by the `k0s config create` command:

```yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  api:
    address: 192.168.68.104
    k0sApiPort: 9443
    port: 6443
    sans:
    - 192.168.68.104
    ca:
      expiresAfter: 87600h
      certificatesExpireAfter: 8760h
  controllerManager: {}
  extensions:
    helm:
      concurrencyLevel: 5
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    clusterDomain: cluster.local
    dualStack:
      enabled: false
    kubeProxy:
      iptables:
        minSyncPeriod: 0s
        syncPeriod: 0s
      ipvs:
        minSyncPeriod: 0s
        syncPeriod: 0s
        tcpFinTimeout: 0s
        tcpTimeout: 0s
        udpTimeout: 0s
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
    kuberouter:
      autoMTU: true
      hairpin: Enabled
      metricsPort: 8080
    nodeLocalLoadBalancing:
      enabled: false
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
    podCIDR: 10.244.0.0/16
    provider: kuberouter
    serviceCIDR: 10.96.0.0/12
  scheduler: {}
  storage:
    etcd:
      peerAddress: 192.168.68.104
      ca:
        expiresAfter: 87600h
        certificatesExpireAfter: 8760h
    type: etcd
  telemetry:
    enabled: true
```

## `spec` Key Detail

### `spec.api`

| Element                      | Description                                                                                                                                                                                                                                                                   |
|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `address`                    | IP Address used by cluster components to talk to the API server. Also serves as one of the addresses pushed on the k0s create service certificate on the API. Defaults to first non-local address found on the node.                                                          |
| `onlyBindToAddress`          | The API server binds to all interfaces by default. With this option set to `true`, the API server will only listen on the IP address configured by the `address` option (first non-local address by default). This can be necessary with multi-homed control plane nodes.     |
| `externalAddress`            | The load balancer address (for k0s controllers running behind a load balancer). Configures all cluster components to connect to this address and configures this address for use when joining new nodes to the cluster.                                                       |
| `sans`                       | List of additional addresses to push to API servers serving the certificate.                                                                                                                                                                                                  |
| `ca.expiresAfter`            | The expiration duration of the CA certificate (default: 87600h)                                                                                                                                                                                                               |
| `ca.certificatesExpireAfter` | The expiration duration of the server certificate (default: 8760h)                                                                                                                                                                                                            |
| `extraArgs`                  | Map of key-values (strings) for any extra arguments to pass down to Kubernetes API server process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)                  |
| `rawArgs`                    | Slice of strings for any raw arguments to pass down to the kube-apiserver process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |
| `port`¹                      | Custom port for the Kubernetes API server to listen on (default: 6443)                                                                                                                                                                                                        |
| `k0sApiPort`¹                | Custom port for k0s API server to listen on (default: 9443)                                                                                                                                                                                                                   |

¹ If `port` and `k0sApiPort` are used with the `externalAddress` element, the load balancer serving at `externalAddress` must listen on the same ports.

### `spec.storage`

| Element                           | Description                                                                                                                                                                                                                                        |
|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `type`                            | Type of the data store (valid values:`etcd` or `kine`). **Note**: Type `etcd` will cause k0s to create and manage an elastic etcd cluster within the controller nodes.                                                                             |
| `etcd.peerAddress`                | Node address used for etcd cluster peering.                                                                                                                                                                                                        |
| `etcd.extraArgs`                  | Map of key-values (strings) for any extra arguments to pass down to etcd process. `extraArgs` are recommeneded over `rawrArgs` if the use case allows it.  Any behavior triggered by these parameters is outside k0s support.                      |
| `etcd.rawArgs`                    | Slice of strings for any raw arguments to pass down to the etcd process. These are appeneded after `extraArg`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. |
| `etcd.ca.expiresAfter`            | The expiration duration of the CA certificate (default: 87600h)                                                                                                                                                                                    |
| `etcd.ca.certificatesExpireAfter` | The expiration duration of the server certificate (default: 8760h)                                                                                                                                                                                 |
| `kine.dataSource`                 | [kine](https://github.com/k3s-io/kine) data source URL.                                                                                                                                                                                            |
| `etcd.externalCluster`            | Configuration when etcd is externally managed, i.e. running on dedicated nodes. See [`spec.storage.etcd.externalCluster`](#specstorageetcdexternalcluster)                                                                                         |

#### `spec.storage.etcd.externalCluster`

k0s can also work with an externally managed Etcd cluster. If this is configured, k0s will NOT set up etcd, it has to be managed manually.

| Element          | Description                                                                                                                                                 |
|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `endpoints`      | Array of Etcd endpoints to use.                                                                                                                             |
| `etcdPrefix`     | Prefix to use for this cluster. The same external Etcd cluster can be used for several k0s clusters, each prefixed with a unique prefix to store data with. |
| `caFile`         | CaFile is the host path to a file with the Etcd cluster CA certificate.                                                                                     |
| `clientCertFile` | ClientCertFile is the host path to a file with the TLS certificate for etcd client.                                                                         |
| `clientKeyFile`  | ClientKeyFile is the host path to a file with the TLS key for etcd client.                                                                                  |

### `spec.network`

| Element                | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `provider`             | Network provider (valid values: `calico`, `kuberouter`, or `custom`). For `custom`, you can push any network provider (default: `kuberouter`). Be aware that it is your responsibility to configure all the CNI-related setups, including the CNI provider itself and all necessary host levels setups (for example, CNI binaries). **Note:** Once you initialize the cluster with a network provider the only way to change providers is through a full cluster redeployment. |
| `podCIDR`              | Pod network CIDR to use in the cluster. Defaults to `10.244.0.0/16`.                                                                                                                                                                                                                                                                                                                                                                                                           |
| `serviceCIDR`          | Network CIDR to use for cluster VIP services. Defaults to `10.96.0.0/12`.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `primaryAddressFamily` | Defines the primary family for the cluster. Valid values are empty, `IPv4`, `IPv6`. If empty, K0s determines it based on `.spec.API.ExternalAddress`, if this isn't present it will use `.spec.API.Address.`. If both addresses are empty or the chosen address is a host name, defaults to `IPv4`.                                                                                                                                                                            |
| `clusterDomain`        | Cluster domain to be passed to the [kubelet](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) and the CoreDNS configuration. Defaults to `cluster.local`.                                                                                                                                                                                                                                           |

#### `spec.network.calico`

| Element                 | Description                                                                                                                                                                                                                                                                                                                                                                                                     |
|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `mode`                  | Indicates the Calico backend to use. Either `bird` or `vxlan` (default). The deprecated legacy value `ipip` is also accepted.                                                                                                                                                                                                                                                                                   |
| `overlay`               | Overlay mode: `Always` (default), `CrossSubnet` or `Never` (requires `mode=vxlan` to disable Calico overlay-network).                                                                                                                                                                                                                                                                                           |
| `vxlanPort`             | The UDP port for VXLAN (default: `4789`).                                                                                                                                                                                                                                                                                                                                                                       |
| `vxlanVNI`              | The virtual network ID for VXLAN (default: `4096`).                                                                                                                                                                                                                                                                                                                                                             |
| `mtu`                   | MTU for overlay network (default: `0`, which causes Calico to detect optimal MTU during bootstrap).                                                                                                                                                                                                                                                                                                             |
| `wireguard`             | Enable WireGuard-based encryption (default: `false`). Your host system must be WireGuard ready (refer to the [Calico documentation](https://docs.projectcalico.org/security/encrypt-cluster-pod-traffic) for details).                                                                                                                                                                                          |
| `flexVolumeDriverPath`  | The host path for Calico's flex-volume-driver(default: `/usr/libexec/k0s/kubelet-plugins/volume/exec/nodeagent~uds`). Change this path only if the default path cannot be used (refer to [Project Calico Issue #2712](https://github.com/projectcalico/calico/issues/2712) for details). Ideally, you will pair this option with a custom `volumePluginDir` in the profile you use for your worker nodes.       |
| `ipAutodetectionMethod` | Used to force Calico to pick up the interface for pod network inter-node routing (default: `""`, meaning not set, so that Calico will instead use its defaults). For more information, refer to the [Calico documentation](https://docs.projectcalico.org/reference/node/configuration#ip-autodetection-methods).                                                                                               |
| `envVars`               | Map of key-values (strings) for any calico-node [environment variable](https://docs.projectcalico.org/reference/node/configuration#ip-autodetection-methods).                                                                                                                                                                                                                                                   |

#### `spec.network.calico.envVars`

Environment variable's value must be string, e.g.:

```yaml
spec:
  network:
    provider: calico
    calico:
      envVars:
        TEST_BOOL_VAR: "true"
        TEST_INT_VAR: "42"
        TEST_STRING_VAR: test
```

K0s runs Calico with some predefined vars, which can be overwritten by setting new value in `spec.network.calico.envVars`:

```shell
CALICO_IPV4POOL_CIDR: "{{ spec.network.podCIDR }}"
CALICO_DISABLE_FILE_LOGGING: "true"
FELIX_DEFAULTENDPOINTTOHOSTACTION: "ACCEPT"
FELIX_LOGSEVERITYSCREEN: "info"
FELIX_HEALTHENABLED: "true"
FELIX_PROMETHEUSMETRICSENABLED: "true"
FELIX_FEATUREDETECTOVERRIDE: "ChecksumOffloadBroken=true"
```

`FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true` disables VXLAN offloading because of [projectcalico/calico#4727](https://github.com/projectcalico/calico/issues/4727).

In single-stack mode, there are additional vars:

```shell
FELIX_IPV6SUPPORT: "false"
```

In dual-stack mode, there are additional vars:

```shell
CALICO_IPV6POOL_NAT_OUTGOING: "true"
FELIX_IPV6SUPPORT: "true"
IP6: "autodetect"
CALICO_IPV6POOL_CIDR: "{{ spec.network.dualStack.IPv6podCIDR }}"
```

#### `spec.network.kuberouter`

| Element          | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
|------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `autoMTU`        | Auto-detection of used MTU (default: `true`).                                                                                                                                                                                                                                                                                                                                                                 |
| `mtu`            | Override MTU setting, if `autoMTU` must be set to `false`.                                                                                                                                                                                                                                                                                                                                                    |
| `metricsPort`    | Kube-router metrics server port. Set to 0 to disable metrics (default: `8080`).                                                                                                                                                                                                                                                                                                                               |
| `peerRouterIPs`  | DEPRECATED: Use extraArgs with peerRouterIPs instead. Comma-separated list of [global peer addresses](https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md#global-external-bgp-peers).                                                                                                                                                                                                      |
| `peerRouterASNs` | DEPRECATED: Use extraArgs with peerRouterASNs instead. Comma-separated list of [global peer ASNs](https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md#global-external-bgp-peers).                                                                                                                                                                                                          |
| `hairpin`        | Configure [hairpin mode](https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode). Supported modes are `Enabled`: enabled cluster wide, `Allowed`: must be allowed per service using annotations, `Disabled`: doesn't work at all (default: `Enabled`)                                                                                                                     |
| `hairpinMode`    | **Deprecated** Use `hairpin` instead. If both `hairpin` and `hairpinMode` are defined, this is ignored. If only hairpinMode is configured explicitly activates hairpinMode.                                                                                                                                                                                                                                   |
| `ipMasq`         | IP masquerade for traffic originating from the pod network, and destined outside of it (default: false)                                                                                                                                                                                                                                                                                                       |
| `extraArgs`      | Extra arguments to pass to Kube-router. `extraArgs` are recommended over `rawArgs` if the use case allows it. Can be also used to override any k0s managed arguments. For reference, see Kube-router [documentation](https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#command-line-options). Any behavior triggered by these parameters is outside k0s support. (default: empty) |
| `rawArgs`        | Slice of strings for any raw arguments to pass down to the kube-router process. These are appended after `extraArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty)                                                                                                                                                                                                     |

**Note**: Kube-router allows many networking aspects to be configured per node, service, and pod (for more information, refer to the [Kube-router user guide](https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md)).

#### `spec.network.kubeProxy`

| Element             | Description                                                                                                                                                                                                                                                               |
|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `disabled`          | Disable kube-proxy altogether (default: `false`).                                                                                                                                                                                                                         |
| `mode`              | Kube proxy operating mode, supported modes `iptables`, `ipvs`, `nftables`, `userspace` (default: `iptables`)                                                                                                                                                              |
| `iptables`          | Kube proxy iptables settings                                                                                                                                                                                                                                              |
| `ipvs`              | Kube proxy IPVS settings                                                                                                                                                                                                                                                  |
| `nftables`          | Kube proxy nftables settings                                                                                                                                                                                                                                              |
| `nodePortAddresses` | Kube proxy [nodePortAddresses](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/)                                                                                                                                                             |
| `extraArgs`         | Map of key-values (strings) for any extra arguments to pass down to kube-proxy process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)                         |
| `rawArgs`           | Slice of strings for any raw arguments to pass down to the kube-proxy process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |

Default kube-proxy iptables settings:

```yaml
iptables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
```

Default kube-proxy IPVS settings:

```yaml
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
```

Default kube-proxy nftables settings:

```yaml
nftables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
```

#### `spec.network.nodeLocalLoadBalancing`

Configuration options related to k0s's [node-local load balancing] feature.

| Element          | Description                                                                                                                   |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `enabled`        | Indicates if node-local load balancing should be used to access Kubernetes API servers from worker nodes. Default: `false`.   |
| `type`           | The type of the node-local load balancer to deploy on worker nodes. Default: `EnvoyProxy`. (This is the only option for now.) |
| `envoyProxy`     | Configuration options related to the "EnvoyProxy" type of load balancing.                                                     |

[node-local load balancing]: nllb.md

##### `spec.network.nodeLocalLoadBalancing.envoyProxy`

Configuration options required for using Envoy as the backing implementation for
node-local load balancing.

**Note:** This type of load balancing is not supported on ARMv7 workers.

| Element                      | Description                                                                                                                               |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `image`                      | The OCI image that's being used for the Envoy Pod.                                                                                        |
| `imagePullPolicy`            | The pull policy being used used for the Envoy Pod. Defaults to `spec.images.default_pull_policy` if omitted.                              |
| `apiServerBindPort`          | Port number on which to bind the Envoy load balancer for the Kubernetes API server to on a worker's loopback interface. Default: `7443`.  |
| `konnectivityServerBindPort` | Port number on which to bind the Envoy load balancer for the konnectivity server to on a worker's loopback interface. Default: `7132`.    |

##### `spec.network.controlPlaneLoadBalancing`

Configuration options related to k0s's [control plane load balancing] feature

| Element      | Description                                                                                                                    |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| `enabled`    | Indicates if control plane load balancing should be enabled. Default: `false`.                                                 |
| `type`       | The type of the control plane load balancer to deploy on controller nodes. Currently, the only supported type is `Keepalived`. |
| `keepalived` | Contains the keepalived configuration.                                                                                         |

[control plane load balancing]: cplb.md

##### `spec.network.controlPlaneLoadBalancing.Keepalived`

Configuration options related to keepalived in [control plane load balancing]

| Element                  | Description                                                                                                                     |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| `vrrpInstances`          | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs.                     |
| `virtualServers`         | Configuration options related to load balancing. This is an array which allows to configure multiple load balancers.            |
| `userSpaceProxyBindPort` | The port the user space proxy will bind to. This port is for internal use only, but listens on every interface. Default: `6444` |
| `disableLoadBalancer`    | Disables the load balancer. Default: `false`                                                                                    |

##### `spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances`

Configuration options required for using VRRP to configure VIPs in control plane load balancing.

| Element                 | Description                                                                                                                                                                                                                                                                                          |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `virtualIPs`            | List of virtual IP address used by the VRRP instance. Each virtual IP must be a CIDR as defined in RFC 4632 and RFC 4291.A list of the CIDRs handled by the VRRP instance.                                                                                                                           |
| `interface`             | The NIC used by the virtual router. K0s allows to use either a device name or a MAC address. If not specified, k0s will use the interface that owns the default route.                                                                                                                               |
| `virtualRouterID`       | The VRRP router ID. If not specified, k0s will automatically number the IDs for each VRRP instance, starting with 51. It must be in the range of 1-255, all the control plane nodes must use the same `virtualRouterID`. Other clusters in the same network must not use the same `virtualRouterID`. |
| `advertIntervalSeconds` | Advertisement interval in seconds. Defaults to 1 second.                                                                                                                                                                                                                                             |
| `authPass`              | The password for accessing VRRPD. This is not a security feature but a way to prevent accidental misconfigurations. It must be in the range of 1-8 characters                                                                                                                                        |
| `unicastPeers`          | A list of IP addresses to connect using unicast. If this field is specified, `unicastSourceIP` is mandatory, and this list must not contain the IP address specified in `unicastSourceIP`.                                                                                                           |
| `unicastSourceIP`       | The source IP address when using unicast. If `unicastPeers` isn't defined this field is ignored.                                                                                                                                                                                                     |
| `addressLabel`          | The label for the VRRP instance for IPv6 VIPs.  This value is ignored for IPv4 VIPs. This is used to set the routing preference as per RFC 6724. If set to `0`, uses default value. Default: `10000`                                                                                                 |

##### `spec.network.controlPlaneLoadBalancing.keepalived.virtualServers`

Configuration options required for using VRRP to configure VIPs in control plane load balancing.

| Element                     | Description                                                                                                                                |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `ipAddress`                 | The load balancer's listen address.                                                                                                        |
| `delayLoop`                 | Delay timer for check polling. Accepts microsecond precision. Further precision will be truncated without warnings. Defaults to `1m`.      |
| `lbAlgo`                    | Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`.              |
| `lbKind`                    | Kind of IPVS load balancer. Supported values: `NAT`, `DR`, `TUN`. Default: `DR`.                                                           |
| `persistenceTimeoutSeconds` | Timeout for persistent connections in seconds. Must be in the range of 1-2678400 (31 days). If not specified, defaults to 360 (6 minutes). |

### `spec.controllerManager`

| Element     | Description                                                                                                                                                                                                                                                                            |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `extraArgs` | Map of key-values (strings) for any extra arguments you want to pass down to the Kubernetes controller manager process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)      |
| `rawArgs`   | Slice of strings for any raw arguments to pass down to the kube-controller-manager process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |

### `spec.scheduler`

| Element     | Description                                                                                                                                                                                                                                                                   |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `extraArgs` | Map of key-values (strings) for any extra arguments you want to pass down to Kubernetes scheduler process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)          |
| `rawArgs`   | Slice of strings for any raw arguments to pass down to the kube-scheduler process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |

### `spec.workerProfiles`

Worker profiles are used to manage worker-specific configuration in a
centralized manner. A ConfigMap is generated for each worker profile. Based on
the `--profile` argument given to the `k0s worker`, the configuration in the
corresponding ConfigMap is picked up during start-up.

The worker profiles are defined as an array. Each element has following
properties:

| Property | Description                                                                      |
| -------- | -------------------------------------------------------------------------------- |
| `name`   | String; name to use as profile selector for the worker process                   |
| `values` | Object; [Kubelet configuration][kubelet-config] overrides, see below for details |

#### `spec.workerProfiles[].values` (Kubelet configuration overrides)

The Kubelet configuration overrides of a profile override the defaults defined
by k0s.

Note that there are several fields that cannot be overridden:

- `clusterDNS`
- `clusterDomain`
- `apiVersion`
- `kind`
- `staticPodURL`

[kubelet-config]: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

#### Configuration examples

##### Custom volumePluginDir

```yaml
spec:
  workerProfiles:
    - name: custom-pluginDir
      values:
         volumePluginDir: /var/libexec/k0s/kubelet-plugins/volume/exec
```

##### Eviction Policy

```yaml
spec:
  workerProfiles:
    - name: custom-eviction
      values:
        evictionHard:
          memory.available: "500Mi"
          nodefs.available: "1Gi"
          imagefs.available: "100Gi"
        evictionMinimumReclaim:
          memory.available: "0Mi"
          nodefs.available: "500Mi"
          imagefs.available: "2Gi"
```

##### Unsafe Sysctls

```yaml
spec:
  workerProfiles:
    - name: custom-eviction
      values:
        allowedUnsafeSysctls:
          - fs.inotify.max_user_instances
```

### `spec.featureGates`

Available components are:

- `kube-apiserver`
- `kube-controller-manager`
- `kubelet`
- `kube-scheduler`
- `kube-proxy`

If `components` is omitted, propagates to all Kubernetes components.

Modifies `extraArgs`.

#### Examples

##### Generic feature gates example

```yaml
spec:
    featureGates:
      - name: feature-gate-0
        enabled: true
        components: ["kube-apiserver", "kube-controller-manager", "kubelet", "kube-scheduler"]
      - name: feature-gate-1
        enabled: true
      - name: feature-gate-2
        enabled: false
```

##### Kubelet feature gates example

The below is an example of a k0s config with feature gates enabled:

```yaml
spec:
    featureGates:
      - name: DevicePlugins
        enabled: true
        components: ["kubelet"]
      - name: Accelerators
        enabled: true
        components: ["kubelet"]
      - name: AllowExtTrafficLocalEndpoints
        enabled: false
```

### `spec.images`

Nodes under the `images` key all have the same basic structure:

```yaml
spec:
  images:
    coredns:
      image: {{{ src_var('CoreDNSImage') }}}
      version: {{{ src_var('CoreDNSImageVersion') }}}
```

If you want the list of default images and their versions to be included, use `k0s config create --include-images`.

#### Available keys

- `spec.images.konnectivity`
- `spec.images.metricsserver`
- `spec.images.kubeproxy`
- `spec.images.coredns`
- `spec.images.pause`
- `spec.images.calico.cni`
- `spec.images.calico.node`
- `spec.images.calico.kubecontrollers`
- `spec.images.kuberouter.cni`
- `spec.images.kuberouter.cniInstaller`
- `spec.images.repository`¹
- `spec.images.default_pull_policy`²

¹ If `spec.images.repository` is set and not empty, every image will be pulled from there.

² If `spec.images.default_pull_policy` is set and not empty, it will be used as a pull policy for each bundled image.

#### Image example

{% set cali_ver = src_var('CalicoComponentImagesVersion') -%}
{% set metrics_ver = src_var('MetricsImageVersion') -%}

```yaml
images:
  repository: airgap-repo.local
  calico:
    kubecontrollers:
      image: repo.acme.corp/k0sproject/calico-kube-controllers
      version: {{{ cali_ver }}}
  metricsserver:
    image: repo.acme.corp/k0sproject/metrics-server
    version: {{{ metrics_ver }}}
```

In the runtime the image names are calculated as
`airgap-repo.local/k0sproject/calico-kube-controllers:{{{ cali_ver }}}` and
`airgap-repo.local/k0sproject/metrics-server:{{{ metrics_ver }}}`. This only
affects the image pull location, and thus omitting an image specification here
will not disable component deployment.

### `spec.extensions.helm`

`spec.extensions.helm` is the config file key in which you configure the list of [Helm](https://helm.sh) repositories and charts to deploy during cluster bootstrap (for more information, refer to [Helm Charts](helm-charts.md)).

### `spec.konnectivity`

The `spec.konnectivity` key is the config file key in which you configure Konnectivity-related settings.

- `agentPort` agent port to listen on (default 8132)
- `adminPort` admin port to listen on (default 8133)

### `spec.telemetry`

To improve the end-user experience k0s is configured by default to collect telemetry data from clusters and send it to the k0s development team. To disable the telemetry function, change the `enabled` setting to `false`.

The telemetry interval is ten minutes.

```yaml
spec:
  telemetry:
    enabled: true
```

## Disabling controller components

k0s allows to completely disable some of the system components. This allows
users to build a minimal Kubernetes control plane and use what ever components
they need to fulfill their need for the control plane. Disabling the system
components happens through a command line flag for the controller process:

```text
--disable-components strings                     disable components (valid items: applier-manager,autopilot,control-api,coredns,csr-approver,endpoint-reconciler,helm,konnectivity-server,kube-controller-manager,kube-proxy,kube-scheduler,metrics-server,network-provider,node-role,system-rbac,update-prober,windows-node,worker-config)
```

If you use k0sctl, just add the flag when installing the cluster for the first
controller at `spec.hosts.installFlags` in the config file like e.g.:

```yaml
spec:
  hosts:
  - role: controller
    installFlags:
    - --disable-components=metrics-server
```

As seen from the component list, the only always-on component is the Kubernetes
API server, without that k0s serves no purpose.

## Kubelet root directory

Unlike vanilla Kubernetes, k0s by default deploys kubelet's root directory
inside `--data-dir`, which defaults to `/var/lib/k0s/` and kubelet ultimately
runs in `/var/lib/k0s/kubelet`. This can result in incompatibilities on external
software that mounts the kubelet working directory, such as CSI plugins.

This can be changed using the flag `--kubelet-root-dir`, for instance, to use
Kubernetes' default value it can be set up as:

```shell
sudo k0s install worker --token-file /var/lib/k0s/join-token --kubelet-root-dir=/var/lib/kubelet
```

Keep in mind that changing the flag in an existing node will not remove existing directories.
Equally importantly, some pods such as CSI plugins are likely to mount this directory,
having inconsistent values across nodes is very likely to cause problems on Deployments and
DaemonSets.