# MicroShift Architecture

This document describes how MicroShift is built and how its components interact at runtime. For the design goals, principles, and reasoning behind architectural decisions, see [design.md](./design.md).

## Repository Layout

| Directory | Purpose |
|-----------|---------|
| `cmd/microshift/` | Main binary entry point (Cobra CLI) |
| `etcd/` | Etcd subprocess manager (separate binary) |
| `pkg/` | Core library packages (see [Package Structure](#package-structure)) |
| `assets/` | Kubernetes manifests for core components |
| `assets/optional/` | Manifests for optional components (OLM, Gateway API, cert-manager, SR-IOV, etc.) |
| `packaging/` | RPM spec, systemd units, CRI-O config, SELinux policy, Containerfiles |
| `test/` | E2E test framework (Robot Framework), scenarios, image blueprints |
| `scripts/` | Build tooling, verification, rebase automation |
| `docs/` | User and contributor documentation |
| `vendor/` | Standard Go vendor directory for third-party dependencies (managed by `make vendor`) |
| `deps/` | Vendored copies of `openshift/kubernetes` and `k8s.io/klog` with rebase patches applied, referenced via `go.mod` replace directives. Exists so that vendor checks in ART (the release build system) pass cleanly |

## Package Structure

### Control Plane Lifecycle

| Package | Purpose |
|---------|---------|
| `pkg/cmd/` | Cobra CLI commands: `run`, `version`, `show-config`, `backup`, `restore`, `healthcheck`, `c2cc-probe` |
| `pkg/controllers/` | Kubernetes control plane service wrappers — etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, OpenShift controllers |
| `pkg/servicemanager/` | Dependency-aware service lifecycle manager with startup recording |
| `pkg/admin/` | Backup/restore, data management, pre-run checks (version metadata, feature gates, health verification) |

### Post-Startup Components

| Package | Purpose |
|---------|---------|
| `pkg/components/` | Infrastructure services deployed after control plane is ready: DNS, ingress router, storage (CSI), service CA, CNI, Multus CNI |
| `pkg/kustomize/` | Kustomization engine for manifest deployment from configured paths |

### Configuration and Runtime

| Package | Purpose |
|---------|---------|
| `pkg/config/` | Configuration parsing and validation — maps `config.yaml` to Go structs covering apiserver, etcd, DNS, ingress, networking, storage, C2CC, and other settings |
| `pkg/release/` | Release version tracking and component image references (auto-generated by rebase) |
| `pkg/version/` | Version metadata (major/minor/patch, build timestamp, variant) |
| `pkg/node/` | Node initialization and kubelet configuration |
| `pkg/healthcheck/` | Cluster health verification, including optional workload awareness |

### Supporting Packages

| Package | Purpose |
|---------|---------|
| `pkg/mdns/` | mDNS service discovery for edge networking |
| `pkg/loadbalancerservice/` | LoadBalancer service type support |
| `pkg/gdp/` | Graceful Degraded Performance handling |
| `pkg/sysconfwatch/` | System configuration file watcher |
| `pkg/telemetry/` | OpenShift telemetry integration (enterprise variant only) |
| `pkg/apis/` | MicroShift-specific API types (CRDs) |
| `pkg/generated/` | Generated clientsets, informers, and listers for MicroShift APIs |
| `pkg/util/` | Utilities — crypto/certificates, signal handling, file operations |

## Startup Sequence

MicroShift runs as a single systemd unit (`microshift.service`). The main binary executes `RunMicroshift(cfg)` which follows this sequence:

### 1. Pre-Flight Checks

- Verify root privileges
- Load and validate configuration from `/etc/microshift/config.yaml`
- Set logging verbosity
- Manage data backups (pre-run)

### 2. Environment Setup

- Configure `NO_PROXY` environment
- Create data directories under `/var/lib/microshift/`
- Process version metadata and feature gates
- Generate or rotate cryptographic materials (certificates, keys)

### 3. Service Startup

The `ServiceManager` starts embedded services in dependency order. Each service registers:

- `Name()` — service identifier
- `Dependencies()` — list of services that must start first
- `Run()` — the goroutine that runs the service

Services start concurrently, respecting the dependency graph. The core services are:

1. **etcd** (separate process, managed by main binary)
2. **kube-apiserver**
3. **kube-controller-manager**
4. **kube-scheduler**
5. **kubelet**
6. **OpenShift controllers** (route, policy, etc.)

### 4. Component Deployment

After the control plane is ready, `StartComponents()` deploys infrastructure services in this order:

1. **Service CA**
2. **CSI snapshot controller** (if enabled in config)
3. **CSI plugin** (LVMS)
4. **Ingress router** (openshift-ingress)
5. **DNS controller** (openshift-dns)
6. **CNI plugin** (OVN-Kubernetes)
7. **Multus CNI** (if enabled in config)
8. **Certificate authority** (multinode mode only — unsupported, testing only)

### 5. Graceful Shutdown

Signal handling (SIGTERM, SIGINT) triggers ordered service termination with a 15-second timeout.

## Deployment Modes

MicroShift supports two deployment models:

- **ostree (RPM-based)**: MicroShift is installed as RPM packages on an rpm-ostree system (RHEL for Edge). Updates are delivered as new ostree commits that atomically replace the entire OS and application layer.
- **bootc (image mode)**: MicroShift is embedded in a bootable container image. The entire system — OS, MicroShift, and configuration — is delivered as a single OCI image. Updates replace the image atomically.

Both models support atomic rollback. The choice between them affects packaging, delivery, and the CI test infrastructure (see [CONTRIBUTING.md](../../CONTRIBUTING.md#test-harness)).

## Greenboot Integration

MicroShift integrates with [greenboot](https://github.com/fedora-iot/greenboot) for automatic health checking after updates. On boot, greenboot runs health check scripts. If MicroShift's health check fails (control plane not ready within timeout), greenboot triggers an automatic rollback to the previous ostree commit or bootc image.

The health check script verifies that MicroShift's core workloads and any installed optional workloads are running. See `pkg/healthcheck/` for the implementation.

For testing details, see [greenboot testing](./greenboot.md).

## Backup and Restore

MicroShift supports backup and restore of its data directory via `microshift backup` and `microshift restore` CLI commands. Backups capture etcd state, certificates, and cluster metadata. This is implemented in `pkg/admin/` and runs as a pre-flight step before updates to enable recovery.

For user-facing documentation, see [auto-recovery](../user/autorecovery.md).

## Data Directory

MicroShift stores all runtime state under `/var/lib/microshift/`:

- `certs/` — TLS certificates and keys
- `etcd/` — etcd database
- `kubelet-plugins/` — kubelet plugin data
- `resources/` — cluster resource state (kubeconfigs, API server secrets, scheduler/kubelet configs)
- `version` — data version metadata

Backups are stored separately at `/var/lib/microshift-backups/`.

This directory must persist across reboots. Its contents are version-dependent — MicroShift validates the data version on startup and refuses to start if there is a version mismatch (handled by `pkg/admin/prerun/`).

## Optional Packages

The base `microshift` RPM requires three subpackages: `microshift-selinux` (SELinux policy), `microshift-networking` (OVN-Kubernetes + Open vSwitch), and `microshift-greenboot` (health check scripts). These are not optional.

Additional functionality is delivered as optional RPM subpackages:

### Available Optional Components

| RPM Package | Delivery Mechanism | Functionality |
|------------|-------------------|---------------|
| `microshift-olm` | `manifests.d/001-microshift-olm` | Operator Lifecycle Manager |
| `microshift-gateway-api` | `manifests.d/000-microshift-gateway-api` | Gateway API (Service Mesh) |
| `microshift-cert-manager` | `manifests.d/060-microshift-cert-manager` | Certificate Manager |
| `microshift-sriov` | `manifests.d/070-microshift-sriov` | SR-IOV Network Operator |
| `microshift-observability` | `manifests.d/003-microshift-observability` + systemd service | OpenTelemetry Collector |
| `microshift-ai-model-serving` | `manifests.d/010-*-kserve`, `050-*-runtimes` | AI Model Serving (KServe) |
| `microshift-multus` | config file in `/etc/microshift/config.d/` + CRI-O config | Multus CNI (enables via config, not manifests) |
| `microshift-low-latency` | tuned profile | Baseline low-latency workload configuration |

Most optional components also have a `-release-info` RPM containing container image references used by image builder blueprints.

### Discovery Mechanism

Optional components are discovered via the manifest path system configured in `config.yaml`:

```yaml
manifests:
  kustomizePaths:
    - /usr/lib/microshift/manifests
    - /usr/lib/microshift/manifests.d/*
    - /etc/microshift/manifests
    - /etc/microshift/manifests.d/*
```

The glob pattern `/usr/lib/microshift/manifests.d/*` automatically picks up any installed optional RPM. Installing the RPM is sufficient to enable the component; no configuration change is needed.

### Config-Based Control

Some components have explicit enable/disable settings in `config.yaml`:

- **Multus CNI** (`pkg/config/network.go`): `.network.multus.status` — `Enabled` or `Disabled` (default: Disabled)
- **CSI Snapshot Controller** (`pkg/config/storage.go`): `.storage.optionalCsiComponents` — list of optional CSI components to deploy

### Health Check Integration

The health check system (`pkg/healthcheck/microshift_optional_workloads.go`) is optional-aware. It checks whether each optional component's directory exists on disk AND is included in the configured kustomization paths before expecting its workloads to be running.

## Asset Management

### Core Assets

Kubernetes manifests for built-in components live in `assets/`, organized by function:

- `assets/core/` — core cluster resources (namespaces, kubelet config, API service registrations, priority classes)
- `assets/components/` — infrastructure service manifests (DNS, router, CSI, service CA, OVN, Multus, C2CC, CSI snapshot controller)
- `assets/controllers/` — control plane controller manifests
- `assets/crd/` — custom resource definitions
- `assets/release/` — release metadata
- `assets/version/` — version tracking
- `assets/embed.go` — Go embed directives to bundle assets into the binary

These are embedded in the binary at build time and deployed via `pkg/components/` and `pkg/kustomize/`.

### Optional Assets

Manifests for optional components live in `assets/optional/`, organized by feature:

- `assets/optional/operator-lifecycle-manager/`
- `assets/optional/gateway-api/`
- `assets/optional/cert-manager/`
- `assets/optional/sriov/`
- `assets/optional/observability/`
- `assets/optional/ai-model-serving/`

These are packaged into the corresponding optional RPMs and installed to `/usr/lib/microshift/manifests.d/`.

### Rebase-Managed Files

The following are maintained by the rebase automation (`scripts/auto-rebase/`). See [rebase overview](./rebase/rebase.md) for details on the process.

| Files | Updated by |
|-------|-----------|
| `assets/` manifests | `rebase.sh manifests` |
| `pkg/release/release_*.go` (image refs) | `rebase.sh images` |
| `Makefile.version.*.var`, `Makefile.kube_git.var` | `rebase.sh buildfiles` |
| `vendor/`, `deps/` | `rebase.sh go.mod` + `make vendor` |

## Configuration Model

MicroShift is configured through `/etc/microshift/config.yaml`. The configuration is strictly declarative and uses minimal options with sensible defaults.

The config file maps to Go structs in `pkg/config/`:

| Config Section | Go Type | File |
|---------------|---------|------|
| `.apiServer` | `ApiServer` | `pkg/config/apiserver.go` |
| `.dns` | `DNS` | `pkg/config/dns.go` |
| `.etcd` | `EtcdConfig` | `pkg/config/etcd.go` |
| `.ingress` | `IngressConfig` | `pkg/config/ingress.go` |
| `.manifests` | `Manifests` | `pkg/config/manifests.go` |
| `.network` | `Network` | `pkg/config/network.go` |
| `.node` | `Node` | `pkg/config/node.go` |
| `.storage` | `Storage` | `pkg/config/storage.go` |
| `.debugging` | `Debugging` | `pkg/config/debugging.go` |
| `.telemetry` | `Telemetry` | `pkg/config/telemetry.go` |
| `.kubelet` | `map[string]any` | `pkg/config/config.go` |
| `.genericDevicePlugin` | `GenericDevicePlugin` | `pkg/config/gdp.go` |
| `.clusterToCluster` | `C2CC` | `pkg/config/c2cc.go` |

The default configuration is maintained in `packaging/microshift/config.yaml`. Changes to config structs must be reflected in the default config file, and vice versa.

## Build System

MicroShift uses Makefile-based builds with OpenShift build machinery (`openshift/build-machinery-go`).

Key binaries:

- `microshift` — main binary (all control plane + worker functionality)
- `microshift-etcd` — etcd subprocess
- `generate-config` — configuration generation utility

The build supports two variants controlled by build tags:

- `enterprise` (default) — includes OpenShift telemetry
- `community` — no telemetry, suitable for OKD builds

Version information is tracked in:

- `Makefile.version.x86_64.var` / `Makefile.version.aarch64.var` — OCP version per architecture
- `Makefile.kube_git.var` — Kubernetes git version metadata