# CPU templates Firecracker allows users to customise how the vCPUs are represented to the guest software by changing the following configuration: - CPUID (x86_64 only) - MSRs (Model Specific Registers, x86_64 only) - ARM registers (aarch64 only) - vCPU features (aarch64 only) - KVM capabilities (both x86_64 and aarch64) A combination of the changes to the above entities is called a CPU template. The functionality can be used when a user wants to mask a feature from the guest. A real world use case for this is representing a heterogeneous fleet (a fleet consisting of multiple CPU models) as a homogeneous fleet, so the guests will experience a consistent feature set supported by the host. > [!NOTE] > > Representing one CPU vendor as another CPU vendor is not supported. > [!NOTE] > > CPU templates shall not be used as a security protection against malicious > guests. Disabling a feature in a CPU template does not generally make it > completely unavailable to the guest. For example, disabling a feature related > to an instruction set will indicate to the guest that the feature is not > supported, but the guest may still be able to execute corresponding > instructions if it does not obey the feature bit. Firecracker supports two types of CPU templates: - Static CPU templates - a set of built-in CPU templates for users to choose from - Custom CPU templates - users can create their own CPU templates in json format and pass them to Firecracker > [!NOTE] > > Static CPU templates are deprecated starting from v1.5.0 and will be removed > in accordance with our deprecation policy. Even after the removal, custom CPU > templates are available as an improved iteration of static CPU templates. For > more information about the transition from static CPU templates to custom CPU > templates, please refer to > [this GitHub discussion](https://github.com/firecracker-microvm/firecracker/discussions/4135). > [!NOTE] > > CPU templates for ARM (both static and custom) require the following patch to > be available in the host kernel: > [Support writable CPU ID registers from userspace](https://lore.kernel.org/kvm/20230212215830.2975485-1-jingzhangos@google.com/#t). > Otherwise KVM will fail to write to the ARM registers. ## Static CPU templates At the moment the following set of static CPU templates are supported: | CPU template | CPU vendor | CPU model | | ------------ | ---------- | ------------------------------- | | C3 | Intel | Skylake, Cascade Lake, Ice Lake | | T2 | Intel | Skylake, Cascade Lake, Ice Lake | | T2A | AMD | Milan | | T2CL | Intel | Cascade Lake, Ice Lake | | T2S | Intel | Skylake, Cascade Lake | | V1N1 | ARM | Neoverse V1 | T2 and C3 templates are mapped as close as possible to AWS T2 and C3 instances in terms of CPU features. Note that on a microVM that is lauched with the C3 template and running on processors that do not enumerate FBSDP_NO, PSDP_NO and SBDR_SSDP_NO on IA32_ARCH_CAPABILITIES MSR, the kernel does not apply the mitigation against MMIO stale data vulnerability. The T2S template is designed to allow migrating [snapshots](../snapshotting/versioning.md) between hosts with Intel Skylake and Intel Cascade Lake securely by further restricting CPU features for the guest, however this comes with a performance penalty. Users are encouraged to carry out a performance assessment if they wish to use the T2S template. Note that Firecracker expects the host to always be running the latest version of the microcode. The T2CL template is mapped to be close to Intel Cascade Lake. It is only safe to use it on Intel Cascade Lake and Ice Lake. The only AMD template is T2A. It is considered safe to be used with AMD Milan. Intel T2CL and AMD T2A templates together aim to provide instruction set feature parity between CPUs running them, so they can form a heterogeneous fleet exposing the same instruction sets to the application. The V1N1 template is designed to represent ARM Neoverse V1 as ARM Neoverse N1. ### Configuring static CPU templates Configuration of a static CPU template is performed via the `/machine-config` API endpoint: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/machine-config' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "vcpu_count": 2, "mem_size_mib": 1024, "cpu_template": "T2CL" }' ``` ## Custom CPU templates Users can create their own CPU templates by creating a json file containing modifiers for CPUID, MSRs or ARM registers. > [!NOTE] > > Creating custom CPU templates requires expert knowledge of CPU architectures. > Custom CPU templates must be tested thoroughly before use in production. An > inappropriate configuration may lead to guest crashes or making guests > vulnerable to security attacks. For example, if a CPU template signals a > hardware vulnerability mitigation to the guest while the mitigation is in fact > not supported by the hardware, the guest may decide to disable corresponding > software mitigations which will make the guest vulnerable. > [!NOTE] > > Having MSRs or ARM registers in the custom CPU template does not affect access > permissions that guests will have to those registers. The access control is > handled by KVM and is not influenced by CPU templates. > [!NOTE] > > When setting guest configuration, KVM may reject setting some bits quietly. > This is user's responsibility to make sure that their custom CPU template is > applied as expected even if Firecracker does not report an error. In order to assist with creation and usage of CPU templates, there exists a CPU template helper tool. More details can be found [here](cpu-template-helper.md). ### Configuring custom CPU templates Configuration of a custom CPU template is performed via the `/cpu-config` API endpoint. An example of configuring a custom CPU template on x86_64: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/cpu-config' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kvm_capabilities": ["!56"], "cpuid_modifiers": [ { "leaf": "0x1", "subleaf": "0x0", "flags": 0, "modifiers": [ { "register": "eax", "bitmap": "0bxxxx000000000011xx00011011110010" } ] } ], "msr_modifiers": [ { "addr": "0x10a", "bitmap": "0b0000000000000000000000000000000000000000000000000000000000000000" } ] }' ``` This CPU template will do the following: - removes check for KVM capability: KVM_CAP_XCRS. This allows Firecracker to run on old cpus. See [this](https://github.com/firecracker-microvm/firecracker/discussions/3470) discussion. - in leaf `0x1`, subleaf `0x0`, register `eax`: - clear bits `0b00001111111111000011100100001101` - set bits `0b00000000000000110000011011110010` - leave bits `0b11110000000000001100000000000000` intact. - in MSR `0x10`, it will clear all bits. An example of configuring a custom CPU template on ARM: ```bash curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/cpu-config' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "kvm_capabilities": ["171", "172"], "vcpu_features": [{ "index": 0, "bitmap": "0b11xxxxx" }] "reg_modifiers": [ { "addr": "0x603000000013c020", "bitmap": "0bxxxxxxxxxxxx0000xxxxxxxxxxxx0000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" } ] }' ``` This CPU template will do the following: - add checks for KVM capabilities: KVM_CAP_ARM_PTRAUTH_ADDRESS and KVM_CAP_ARM_PTRAUTH_GENERIC. These checks are to ensure that the host have capabilities needed for the vCPU features. - enable additional vCPU features: KVM_ARM_VCPU_PTRAUTH_ADDRESS and KVM_ARM_VCPU_PTRAUTH_GENERIC - modify ARM register `0x603000000013c020`: - clear bits `0b0000000000001111000000000000111100000000000000000000000000000000` - leave bits `0b1111111111110000111111111111000011111111111111111111111111111111` intact. Information about KVM capabilities can be found in the [kernel source](https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/kvm.h). Information about vCPU features on aarch64 can be found in the [kernel source](https://elixir.bootlin.com/linux/latest/source/arch/arm64/include/uapi/asm/kvm.h). Information on how the ARM register addresses are constructed can be found in the [KVM API documentation](https://docs.kernel.org/virt/kvm/api.html#kvm-set-one-reg). ### Custom CPU templates language schema The full description of the custom CPU templates language can be found [here](schema.json). > [!NOTE] > > You can also use `_` to visually separate parts of a bitmap. So instead of > writing: `0b0000xxxx`, it can be `0b0000_xxxx`. #### Expansion of contracted bitmaps If a contracted version of a bitmap is given, for example, `0b101` where a 32-bit bitmap is expected, missing characters are implied to be `x` (`0bxxxxxxxxxxxxxxxxxxxxxxxxxxxxx101`). #### CPUID normalization and boot protocol register settings Some of the configuration set by a custom CPU template may be overwritten by Firecracker. More details can be found [here](cpuid-normalization.md) and [here](boot-protocol.md). #### Information about architecture-specific settings For detailed information when working with custom CPU templates, please refer to hardware specifications from CPU vendors, for example: - [Intel Software Developer Manual](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) - [AMD Architecture Programmer's Manual](https://www.amd.com/en/support/tech-docs?keyword=programmer%27s+manual) - [ARM Architecture Refernce Manual](https://developer.arm.com/documentation/ddi0487/latest) ## A note about configuration of both static and custom CPU templates If a user configured both a static CPU template (via `/machine-config`) and a custom CPU template (via `/cpu-config`) in the same Firecracker process, only the configuration that was performed the _last_ is applied. This means that if a static CPU template was configured first and a custom CPU template was configured later, only the custom CPU template configuration will be applied when starting a microVM.