---
categories:
- kubernetes
- jetstream2
date: '2024-07-31'
aliases:
- 2024-05-07-ubuntu22-minimal-image-jetstream
layout: post
slug: ubuntu22-minimal-image-jetstream
title: Ubuntu 22.04 Minimal on Jetstream
---
**Update July 2024**: Ubuntu 22.04 Minimal image with Generic Kernel
Virtual Machine images provided by the Jetstream team are fully fledged and therefore quite large.
Julien Chastang proposed the shift to a minimalist distribution to save disk space, resources and shorten deployment time, see [the relevant issue discussion on the Jetstream Gitlab organization](https://gitlab.com/jetstream-cloud/image-build-pipeline/-/issues/33).
We are mostly interested in replacing Ubuntu as the default image for our Kubespray developments, so we are [looking only at OS supported by Kubespray](https://github.com/kubernetes-sigs/kubespray?tab=readme-ov-file#supported-linux-distributions)
I previously tested the [container-specific Flatcar image](./2024-04-30-flatcar-image-jetstream.md), now testing Ubuntu 22.04 Minimal.
## Ubuntu 22.04 Minimal cloud image
Best features:
* Minimalist, image is less than `200 MB`
* It's Ubuntu, everyone is familiar with it
* Wide support (for example GPU virtualization used in Jetstream should work)
* Automatic updates
Downloaded from
The name of the image is `Ubuntu2204Minimal`, it is set as a Community image ([script to load image to Openstack](https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/blob/master/vm_image/upload_image_ubuntu.sh):
```
openstack image list --community | grep Minimal
| 0fa9f3b4-d29f-4f68-a8a7-16bf44ffae69 | Ubuntu2204Minimal | active |
```
## Ubuntu minimal version with Generic kernel
By default Ubuntu Minimal ships with the `kvm` kernel which is a smaller kernel, it does not offer some modules, for example `nfsd` is not available, this prevents running NFS servers on the node or even in containers inside the node.
To circumvent this issue, I created a new version of the Minimal image with the only modification being the `generic` kernel instead of the `kvm` one.
The name of the image is `Ubuntu2204MinimalGenKernel`, ID `671f03f1-4d86-4363-b03c-f5d54818693a`
See
This image a bit larger image:
```
Filesystem Size Used Avail Use% Mounted on
tmpfs 297M 664K 297M 1% /run
/dev/sda1 20G 2.7G 17G 14% /
tmpfs 1.5G 0 1.5G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 105M 6.1M 99M 6% /boot/efi
tmpfs 297M 4.0K 297M 1% /run/user/1000
```
### GPU support
While Ubuntu potentially supports GPUs, the default image from Ubuntu does not.
We need [some specific customization](https://gitlab.com/jetstream-cloud/image-build-pipeline/-/blob/main/ansible/roles/js2-specific/tasks/Ubuntu-22.yml?ref_type=heads#L120-167) on the image.
Therefore we would need to create a GPU-specific minimal version that also includes the GPU drivers.
See [the dedicated tutorial about GPU support](./2024-05-13-ubuntu22-minimal-gpu-image-jetstream.md).
### Exosphere
The instance boots correctly on Exosphere, I can ssh into the instance as `exouser`.
It looks like Exosphere installs another 1.4GB of packages, so disk usage is higher compared to Horizon, anyway still tiny compared to a full-featured Ubuntu.
```
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 20G 2.5G 17G 13% /
tmpfs 1.5G 0 1.5G 0% /dev/shm
tmpfs 600M 672K 599M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 105M 6.1M 99M 6% /boot/efi
tmpfs 300M 4.0K 300M 1% /run/user/1000
```
### Horizon
All normal, boots fine, usual login with the `ubuntu` user and the Openstack keypair.
Occupies minimal space:
```
Filesystem Size Used Avail Use% Mounted on
/dev/root 20G 1.1G 19G 6% /
tmpfs 1.5G 0 1.5G 0% /dev/shm
tmpfs 600M 460K 599M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 105M 6.1M 99M 6% /boot/efi
tmpfs 300M 4.0K 300M 1% /run/user/1000
```
No warnings in `dmesg`.
### Kubespray
The real objective of this test is to deploy Kubernetes on Jetstream with this smaller image, following the [latest tutorial](https://www.zonca.dev/posts/2023-07-19-jetstream2_kubernetes_kubespray) with minor modifications.
We can replace the official Ubuntu with the minimal image on Terraform by modifying in `cluster.tfvars`:
image = "Ubuntu2204Minimal"
With the minimal image, Kubespray gives the error:
modprobe: FATAL: Module ip_vs_sh not found in directory /lib/modules/5.15.0-1058-kvm\n"
the simple workaround is to set:
kube_proxy_mode: iptables
instead of
kube_proxy_mode: ipvs
in `inventory/kubejetstream/group_vars/k8s_cluster/k8s-cluster.yml`
CPU Nodes are fine, but GPU nodes as expected do not work.
```
k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubejetstream-1 Ready control-plane 20m v1.25.6 10.1.90.167 149.165.174.34 Ubuntu 22.04.4 LTS 5.15.0-1058-kvm containerd://1.6.15
kubejetstream-k8s-node-nf-cpu-1 Ready 18m v1.25.6 10.1.90.186 Ubuntu 22.04.4 LTS 5.15.0-1058-kvm containerd://1.6.15
kubejetstream-k8s-node-nf-cpu-2 Ready 18m v1.25.6 10.1.90.5 Ubuntu 22.04.4 LTS 5.15.0-1058-kvm containerd://1.6.15
kubejetstream-k8s-node-nf-gpu-1 NotReady 18m v1.25.6 10.1.90.230 Ubuntu 22.04.4 LTS 5.15.0-1058-kvm containerd://1.6.15
```
Did not feel it was necessary to test JupyterHub, I'm confident it will work.
Let's review the disk space occupied on a `m3.medium` worker with Ubuntu Minimal before running Ansible:
```
ubuntu@kubejetstream-k8s-node-1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 58G 831M 58G 2% /
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 5.9G 484K 5.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 105M 6.1M 99M 6% /boot/efi
tmpfs 3.0G 4.0K 3.0G 1% /run/user/1000
```
and after Kubernetes has being installed (from 800 MB to 6.8 GB):
```
Filesystem Size Used Avail Use% Mounted on
/dev/root 58G 6.8G 52G 12% /
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 5.9G 2.1M 5.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 105M 6.1M 99M 6% /boot/efi
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1a777fcf86f2e982969b83f78302bb19427470995262032dabe0384af26d202b/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1331d819ef46d1d4ee94cd1faa58c0965e6c63b25d5fd69bb86e5b1769c9041e/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/6c6a245269c66d8a7ba69d31e590e9e807d17d19acfe9d2a251daa6b73127b78/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/2ed3c7de52d4c63e3a4df8e00486f4fcde91634891708e88aaf506256c2637c7/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/03b1e64d75fd951f7aefa050746717baa3811636636254132d4d11758c1a239c/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/bda38bc2b547cd4bdeca1aa7653fdf6483da2fbb8b69abbda2a35a3d93b982a8/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/3288593c52f5a21fec4be1222326d22f65612c61135e289addab1fe11b423334/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/e9fd081509ca17913118ae4c69a80dc7cd50ea3a14469a90bc7e63ff5ff36c6f/shm
tmpfs 3.0G 4.0K 3.0G 1% /run/user/1000
```
Here is instead the comparison with the default Jetstream Ubuntu 22 image:
```
Filesystem Size Used Avail Use% Mounted on
udev 15G 0 15G 0% /dev
tmpfs 3.0G 1.2M 3.0G 1% /run
/dev/sda1 58G 8.8G 50G 16% /
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 15G 0 15G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/2264
/dev/loop1 92M 92M 0 100% /snap/lxd/24061
/dev/sda15 105M 6.1M 99M 6% /boot/efi
/dev/loop2 39M 39M 0 100% /snap/snapd/21465
tmpfs 3.0G 4.0K 3.0G 1% /run/user/1000
xxxxxxxxxxxx 9.8T 493G 9.3T 5% /software
```
After having installed Kubernetes (from 8.8 GB to 13):
```
Size Used Avail Use% Mounted on
udev 15G 0 15G 0% /dev
tmpfs 3.0G 3.0M 3.0G 1% /run
/dev/sda1 58G 13G 46G 23% /
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 15G 0 15G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/2264
/dev/loop1 92M 92M 0 100% /snap/lxd/24061
/dev/sda15 105M 6.1M 99M 6% /boot/efi
/dev/loop2 39M 39M 0 100% /snap/snapd/21465
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c0fc748f6469ded859983f821e8f167691b52a529bdc5e9a1269646ab1d1466f/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/5c853382c912e5cddd3dc6c726bce09fdbec753a89b60b6cf334214c34216cbc/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1d5f384f212ecf62aafa44e8cd1dbabf5bc0fa28aeb10759da9e372494d82661/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/7d8ab2baa7b00523538b0921b0fc6c9d6aee1d121db8752fdb38bdce877784b7/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/1669d88641a2eafd7dc3c841e9f71ef6014d61bd986bc527e899b2b972837279/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/7d0ded66f7e811fdb0fee58dba737494aca1e36510f8291d7ebca5f5a3ad9824/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/4ec65ad53b8ab3391a04cc2442fa3278b114fbe4791ca1b24cc7db213be8f896/shm
shm 64M 0 64M 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/2bb33f80ec8afc1639dd57094dbbd9c59e97640cb10df557f68d3bd8582a10f1/shm
tmpfs 3.0G 4.0K 3.0G 1% /run/user/1000
xxxxxxxxxxxx 9.8T 493G 9.3T 5% /software
```