# EVE Security Overview ## Introduction What makes EVE a secure-by-design system is that it has been developed from the ground up with security in mind. EVE doesn't rely on 3rd party or add-on components to provide trust in the system, but rather offers a set of well-defined principles on which a trust model can be built. You will notice that EVE doesn't call itself a trusted system -- but rather a trustworthy one. The distinction is subtle, but very important and was first articulated by [Joanna Rutkowska](https://www.darkreading.com/vulnerabilities---threats/rutkowska-trust-makes-us-vulnerable/d/d-id/1330587). Joanna, of course, is the architect of [Qubes OS](https://www.qubes-os.org) with which EVE shares a good deal of core security principles like [security by compartmentalization](https://www.qubes-os.org/doc/glossary/#qubes-os) and focusing on representing all user applications as VM-based abstractions. In EVE, the same VM-based approach to defining applications is built around [Edge Containers specification](ECOS.md). That VM-based default has one deliberate exception, described in [Isolation between applications and EVE](#isolation-between-applications-and-eve). We have made an effort to provide users of EVE with a system that is both practically secure and can be deployed in a zero-touch fashion. In order to achieve our goals we put forward a few guiding principles (some of them borrowed from Chromium's approach to securing mobile devices): * Provide strong, industry standard (such as [X.509](https://en.wikipedia.org/wiki/X.509)) cryptographic identity for Edge Nodes and software entities alike * Make Edge Nodes secure-by-default * Defense in depth (based on the hypervisor boundary protection guarantees) * Relying on state-of-the-art hardware elements (such as TPM and TEE) for providing robust root-of-trust and secure key management * Remote attestation and measured boot * Robust trust model between EVE and its controller * Workloads which are immutable In the rest of this document, we explain these principles and discuss some expected use cases for Edge Node devices running EVE. We then give a high-level overview of the threat model against which we will endeavor to protect our users, while still enabling them to make full use of their cloud-based controller orchestration service. ## EVE deployment model Reasoning about security and trustworthiness of any system is only meaningful if one understands how the system is going to be deployed and used. EVE main deployment targets are Edge Nodes. Typically, Edge Nodes are small form factor, ruggedized PC-like systems that are deployed in physically insecure environments with ad-hoc networking without any firewall-like guarantees. This type of environment is very similar to the one assumed by mobile computing systems (laptops, tablets and smart phones) and it is not a coincidence that EVE shares a lot of security principles with operating systems developed for mobile computing. For example, EVE's approach to thwarting physical attacks and protecting the integrity of the software stack running on the Edge Node may look similar to if you're familiar with Chromium OS. There is, however, one crucial difference between mobile computing environment and Edge and that is the notion of who controls the system. With mobile there's always a human being end user who has direct physical access to the hardware running mobile computing operating system, in EVE's case that entity is a remote, cloud-based controller. Any security related event that originates on EVE's side (e.g. measurements coming from boot sequence) has to be reliably passed and acknowledged by the controller and any action that is triggered on the Edge Node by EVE (e.g. storage key rotation) has to be scheduled by the controller. In short, EVE's deployment model forces us to protect against an opportunistic adversary with full physical access to and Edge Node and Edge Node's networks through a combination of system hardening, defense in depth, process isolation, secure autoupdate, measured boot, encryption, and seamless integration with EVE's controller security policies. ## Security capabilities vs. security policies At the end of the day, EVE only does what its controller instructs it to do. This puts an enormous premium on protecting the trust between EVE instance and its controller at all costs (in both direction) but it also allows us to have a clear division of responsibilities between the two: * EVE only implements security capabilities, never security policies * Controller orchestrates EVE's security capabilities in order to realize sophisticated policies On EVE's side, this separation allows us to focus on a small number of well designed capabilities (AKA security building blocks) that are guaranteed to have a well defined interactions between each other. For example, while EVE may provide a capability for storing encrypted binary blobs on the Edge Nodes it will be up to the controller's security policy to determine which pieces of data will be stored there. Controller and its management of security policies are out of scope for this document. ## EVE threat model Just as Chromium, we consider two different kinds of adversaries: * An opportunistic adversary * A dedicated adversary The opportunistic adversary is just trying to compromise an individual Edge Node and/or data. They are not targeting a specific user or enterprise, but they are very likely to have prolonged physical access to the Edge Node, and they may steal it or its storage drive and read or copy the drive offline. They are not going to disassemble the Edge Node or modify its hardware. This level of physical access also allows an opportunistic adversary a chance to replace trusted network and I/O connections with those of their own making. Unlike Chromium, we assume that this will facilitate things like DNS or other network-level attacks and will place them within an opportunistic adversary's reach. The dedicated adversary may target a user or an enterprise specifically for attack. They are willing and capable of modifying an Edge Node with extra hardware and software components. They may also do anything that the opportunistic adversary can do. Theft of an Edge Node or its drive is addressed in [Encrypted Application and User Storage](#encrypted-application-and-user-storage), which also states what a stolen drive still reveals. ## Isolation between applications and EVE The boundary between an edge application and EVE depends on the virtualization mode requested for that application in its configuration -- the `VmMode` field of the EVE API (see [vm.proto](https://github.com/lf-edge/eve-api/blob/main/proto/config/vm.proto)). By default that boundary is a virtual machine, and this holds for edge applications delivered as OCI containers just as much as for those delivered as VM images. For a container, containerd prepares the container filesystem as a snapshot and the hypervisor-specific loader boots a VM from it, as described in [CONTAINERS.md](CONTAINERS.md) and [TASKS.md](TASKS.md). On a node running the Kubernetes-based configuration ([EVE-K.md](EVE-K.md)) the same container is launched as a shim VMI. There are two deliberate exceptions, and in both the application runs as a native container on the host, with no VM around it. The first is an application configured with `VmMode = NOHYPER` -- "Do not use a hypervisor": * On the KVM configuration, such a task is handed to containerd rather than to the hypervisor, and runs as a container on the host. * On the Kubernetes-based configuration, it is created as a plain Kubernetes ReplicaSet rather than a KubeVirt `VirtualMachineInstanceReplicaSet`, so there is no launcher VM. The second is a workload deployed through native Kubernetes orchestration on the Kubernetes-based configuration ([EVE-K.md](EVE-K.md#modes)). Such workloads are deployed with Helm charts, manifests or other Kubernetes tooling rather than as EVE applications, and run as plain pods unless they request a KubeVirt VM. For these applications the isolation from EVE and from other applications is provided by the Linux kernel -- namespaces, cgroups and the container's capability set -- and not by the hypervisor. The guiding principle of defense in depth based on hypervisor boundary guarantees, stated above, does not apply to them, and the trusted computing base they can reach is considerably larger. A kernel vulnerability reachable from inside such a container is a path to the host. Two further properties follow from the absence of a VM: * A `NOHYPER` application cannot be assigned a PCI device by passthrough, since there is no shim VM to assign it to. When a physical network adapter is direct-attached to such an application, EVE moves the interface into the container's network namespace and adds `CAP_NET_ADMIN` and `CAP_NET_RAW` to its capability set. * The AppArmor profiles shipped with EVE confine EVE's own components, such as `swtpm`, `vtpm` and `vector`. They are not applied to edge applications. Deploying an application in `NOHYPER` mode is therefore a decision to trade isolation for direct access to the host's kernel and devices, and a controller's security policy should treat it as such. ## Establishing trust between EVE and EVE's controller ### EVE trusting its controller Recall that EVE's deployment model presupposes a controller that can exercise arbitrary control over Edge Nodes. EVE provides the following capabilities that can protect against an adversary trying to take control over an Edge Node by pretending to be a controller: * The controller's network address (hostname and port) cannot be changed by the controller itself, by an application, or over any network path. The one interface that can change it is the [local TUI](LOCAL-TUI.md), which renders only on a physically attached monitor and offers the operation only before the device is onboarded, and any such change is measured into the TPM (see below). The consequence is not only that the change is detected at the next boot: the measurement is part of the vault key access control policy, so the key no longer unseals and `/persist/vault` stays unavailable until the controller accepts the new measurements and releases the [encrypted backup key](#encrypted-application-and-user-storage) * The controller's identity is verified by a Root CA which is also considered immutable and sealed in TPM where possible. This is used in the TLS verification for API V1 and in the object signature verification in API V2 * The TLS identity of the controller is verified by a Root CA. This is a single Root CA in API V1 and a larger set of root CAs plus the ability to express trust in proxy certificates in API V2. The principle that EVE's node forever gets bound to a fixed controller may strike some as too restrictive, but it allows a much easier reasoning about security properties and, since controller gets identified by the DNS name still allows a certain flexibility in deployment. It does, however, stand in stark contrast to EVE's fundamental guarantee of never ever requiring a "truck roll" to manage software on Edge Nodes. Unfortunately, avoiding a "truck roll" in this particular case will require us to design a comprehensive asset transfer protocol that tackles a whole host of thorny problems like: * guaranteeing that at the completion of the asset transfer the Edge Node always ends up in a "pristine state" (so that the previous owner of the Edge Node doesn't accidentally leak something to the next one) * providing a way for the next owner to validate the hardware of the Edge Node (so that no additional components or firmware outside of the visibility of the TPM attestation can be implanted by the previous owner) Given the complexity of designing such a protocol for EVE (especially solving the hardware validation problem), we are currently punting on the problem and declaring software-based asset transfer out of scope for now. However, we are in no way suggesting that this should be a permanent state of affairs. If there's an attempted modification of either controller's address (stored in /config/server) and controller's Root CA (/config/root-certificate.pem) Edge Node should get disconnected from the controller and should be forced to do a hardware-assisted clear operation and start all over again. In the current implementation of EVE, if TPM is available, controller's address and controller's Root CA are measured into TPM as part of vault key access control policy meaning any changes in those data will prevent the system to get access to the vault key at the next boot up. Both files live in the CONFIG partition, which is measured into PCR 14 file by file; [MEASURED-CONFIG](MEASURED-CONFIG.md) describes which files are covered and how the resulting event log is used to identify what changed. ### EVE trusting side-channel configuration The use of [object signing](https://github.com/lf-edge/eve-api/tree/main/OBJECT-SIGNING.md) enables delivering device configuration using side channels such as USB sticks, as the [bootstrap configuration](CONFIG.md#bootstrap-configuration) does. Replay is prevented by the configuration timestamp: whatever its source, EVE rejects a configuration whose `config_timestamp` is older than that of the configuration it has already applied. A bootstrap configuration is in addition applied only once, since EVE records its SHA-256 hash. ### Identity of EVE's instance Each device running EVE has a unique device certificate. Further, if the device has a TPM, the device private key is generated and stored in the TPM to prevent cloning the device identity. If the device does not have a TPM/TEE for secure key generation and storage then the device certificate and private key are stored in a file in the /config partition, hence the confidentiality of the private key depends on the physical protection of the storage on the device. The device certificates are currently self-signed by the device with a long lifetime, since instead of certificate revocation for untrusted devices the controller can explicitly mark them as untrustworthy (the device certificate is only used to communicate to the controller). But in the future EVE can send Certificate Signing Requests over the EVE API to have the controller ask some backend CA to sign the device certificates in the cases where that facilitates managing the devices. The device generates additional key pairs and certificates, since different keys are required to have different usage. This includes an ECDH certificate used for object encryption (to minimize exposure of secrets in the configuration) which is generated using the TPM is available, and stored in a file in the /persist partition. There is also an attestation certificate (with the appropriate key usage settings to perform remote attestation from the TPM), and a certificate for the TPM endorsement key (which is needed by some vTPM use). All three additional certificates are signed using the device private key. Collectively we call these additional device-side certificates EdgeNode certificates. ### Controller trusting EVE As a result of on-boarding a device, the controller is told to trust the device with a particular device certificate (and also that it is "owned" by some particular user of the controller). Separately measured boot with remote attestation will ensure that the device is running the expected versions and hashes of firmware and of the EVE software. On successful attestation, Controller provides a particular one-time token (called integrity-token), which needs to be produced by EVE for a successful configuration request. Controller will reject incoming configuration requests, if the integrity-token does not match the expected value, and will reply with HTTP error code 403, to force EVE to re-attest and generate a new integrity-token. EVE stores this token in memory, and hence is valid only for the current boot session. This ensures that every boot session automatically triggers a fresh attestation cycle, thereby measuring and validating the software components all over again. #### Initial on-boarding There are several ways in which devices running EVE can be on-boarded to a controller. The details depend on the support available in the location, e.g., a factory, where the EVE software is installed on the device. One approach is that EVE is booted and the TPM is used to generate the device certificate, and that device certificate is securely delivered from the factory to the intended end user of the device. That end user can then upload the device certificate to the controller to claim the device. (A controller would presumably check for attempts for more than one party claiming ownership of the same device hence so that a disclosed device certificate can be detected.) Another approach is that an onboarding token (in the form of a certificate and private key) is added to the EVE image, but the factory does not need to boot EVE and upload information after installing the image. When the device boots EVE the first time at the installation site it will present the onboarding certificate and its hardware serial numbers using a register API call to the controller, and the user can scan and upload the serial number to the controller. This has weaker security especially if serial numbers can be guessed by an attacker having an account on the same controller, but different devices can be given different onboarding tokens to make such attacks more difficult. The onboarding token lives in conf/onboard.*.pem in the repo, and this is used to build the config disk image which ends up in /config/ when EVE is booted. Thus the onboarding key UUID can be extracted using ```shell openssl x509 -in conf/onboard.cert.pem -text | grep CN ``` The [generate-onboard.sh](../pkg/pillar/scripts/generate-onboard.sh) script can be used to generate such onboarding certificates. A variant of that approach is to use a random soft serial number. When EVE is installed it generates a /config/soft_serial which is a random 128-bit UUID. If installed from a USB stick this serial number (together with the less random hardware serial number) are written to a directory on the USB installer stick. A device that boots a live image directly, without ever running the installer, generates the same random 128-bit UUID on its first boot instead, before it creates its device certificate, and reports it on the console. An image written to more than one device leaves /config/soft_serial empty for that reason, since a value carried in the image would make every device written from it present the same one. When EVE is calling the register API it will present both the hardware serial and soft serial, hence if the controller has been told of the random soft serial for the device we avoid depending on guessable hardware serial numbers. In both cases of calling the register API the factory can choose the granularity of the onboarding token. A single onboarding token can be used for a large production run of devices, or for fewer devices, or even be generated for each individual device. The choice depends on the capabilities of the factory installation site and the logistics of conveying the information (device certificate, onboarding tokens and/or serial numbers) from the factory to the end user. ## Encrypted Application and User Storage EVE provides a security capability to enable storing sensitive information on the built-in storage of the Edge Node where EVE is running, while providing reasonable protections from this information leaking outside of the running EVE instance. Note that this is not an end-to-end encryption solution, but rather a capability that mitigates some of the attack vectors based on physical possession of the Edge Node. The data itself, while protected in-flight by the transport level security mechanism such as TLS, is expected to be un-encrypted before it lands on the Edge Node. One big driving factor for this is protecting Edge Containers and the Data Volumes they utilize. We expect a high number of Edge Containers deployed by EVE's users to receive and process business sensitive information from sensors and the Cloud. Data collected and processed by these Edge Containers is stored in their virtual storage, which is backed by the hardware storage on the EVE platform. It is important that even if the storage drive or entire device is stolen, the data remains secure. For this reason, data should be in encrypted form when it is stored. See [Encrypting Sensitive Information at Rest at the Edge](https://wiki.lfedge.org/display/EVE/Encrypting+Sensitive+Information+at+Rest+at+the+Edge) design specification. The storage encryption _location_ is in `/persist/vault/`, and encompasses application content and volumes. Only that subtree is encrypted, and the boundary is worth spelling out because it decides what physical possession of the disk yields. Device and application logs are collected under `/persist/newlog`, which is outside the vault: chunks already uploaded to the controller are retained there in a circular queue rather than deleted, so a running device holds a substantial window of log history readable from the raw disk. Device state published between EVE's own microservices lands in `/persist/status`, also outside. So does the CONFIG partition, whose contents are protected by measurement rather than encryption. An attacker in possession of the drive therefore recovers no application data or volumes, and recovers the device's log history and its configuration state only as far as that state is in the clear. The secrets the controller sends under [object level encryption](OBJECT-LEVEL-ENCRYPTION.md) — datastore credentials, WiFi credentials, cloud-init user data — are not: they sit in the checkpointed configuration in encrypted form and are decrypted in memory by the microservice that needs them, so reading them off the drive also requires the device's ECDH private key, which is held in the TPM where there is one. That carve-out is as wide as the controller's use of the mechanism and no wider, since it is per field: [wireless credentials](WIRELESS.md), for one, can be sent either encrypted or in cleartext. Keeping the logs outside the vault is also what makes them readable on a device whose vault will not unseal — the state in which they are most wanted. The storage encryption _mechanism_ uses standard filesystem encryption, fscrypt for the ext4 file system and native ZFS encryption for the ZFS file system. This storage encryption _key_ is symmetric and generated solely on the edge device using a TPM, and then sealed into TPM using PCR values. This means that the key can only be retrieved from TPM when the firmware and software booting sequence has not changed. The encryption key is not written to disk storage, and must be retrieved on each boot from the TPM, which is available only if the PCR values are identical, i.e. the boot chain is unchanged. When a system upgrades, by definition the boot process changes, leading to different values in the PCRs. and the TPM will not be able to unseal the encryption key. To handle this scenario, the "encrypted backup key" mechanism is used. This mechanism is similar to Network-Based Disk Encryption (NBDE) and Shamir Secret Sharing. It requires participation of both parties, controller and device, to use the backup key. 1. When the vault encryption key is generated, in addition to sealing the key to PCRs, it also is encrypted using regular TPM encryption, i.e. not sealed to PCR values. Because it is encrypted using the TPM, but not stored in the TPM, it is not available on the device by itself either. 1. The encrypted copy of the key is sent to the controller. Because it is encrypted using the TPM, the controller cannot read the key either. At this point, neither controller nor device by itself can read the backup key. 1. When the system upgrades, the controller validates via remote attestation that the new PCR values are acceptable, and then sends the encrypted key to the device. 1. The device decrypts the key using the TPM. It now has a valid vault decryption key, which it seals to the new PCR values. To decrypt the key, one has to be on the same device with access to the same TPM, and the firmware+software on that device has to pass the [remote attestation](https://wiki.lfedge.org/display/EVE/Measured+Boot+and+Remote+Attestation) check in the controller. Note what this means for a device pointed at a different controller, which an on-site operator can do from the [local TUI](LOCAL-TUI.md) before the device is onboarded. Accepting the attestation quote is not the step that unlocks the vault; returning the encrypted backup key is. That key was escrowed with the controller the device was talking to when the key was created, and no other controller has a copy, so a controller that a device has been redirected to cannot unlock the existing vault however readily it accepts the new PCR values. The one thing such a controller can cause is destruction rather than disclosure: if it returns an empty key, EVE removes and recreates the vault empty and reports the unlock method as recreated. That path is gated on the presence of `/persist/status/allow-vault-clean`, which is created when `/persist` is first initialized and removed as soon as the device has something to lose — the first volume or content it stores, or the first successful escrow of a key. A device in service therefore yields neither its data nor a wipe; a freshly installed one comes up with an empty vault. ### TPM Bus Protection via Parameter Encryption The vault encryption key must travel over the physical bus that connects the CPU to the TPM (typically LPC or SPI). This bus can be passively monitored with inexpensive hardware, so a plaintext key on it would be recoverable even by an opportunistic adversary. EVE protects seal and unseal operations using **salted HMAC sessions with AES-128-CFB parameter encryption**: 1. Before sealing or unsealing the vault key, EVE opens a TPM2 HMAC session bound to the Endorsement Key (EK). A random session salt is generated and RSA-OAEP encrypted with the EK public key, so only the specific TPM can recover it. 2. The shared session key derived from that salt is used to AES-128-CFB encrypt the key material in the TPM command (seal) and response (unseal). The plaintext never appears on the bus. 3. The session also provides an HMAC over each command, which prevents an attacker from replaying or modifying bus traffic. EVE probes AES-128-CFB support at runtime. On TPMs that do not support it, EVE falls back to the unencrypted legacy path so that compatibility with all hardware is preserved. ### Flexible PCR Policy EVE supports a flexible mechanism for defining which Platform Configuration Registers (PCRs) are used to seal the vault encryption key. While EVE ships with a sensible default set of PCRs (typically measuring firmware, bootloader, root filesystem and kernel configuration), this policy is not static. The controller can instruct EVE to adopt a specific set of PCRs for sealing operations. When a new policy is received, EVE validates it against the device's capabilities. If the policy is accepted, the vault key is automatically unsealed with the old policy and resealed with the new set of PCRs. This capability allows security administrators to tighten or loosen the binding of the encrypted data to the platform state as needed, without requiring physical access or a complete reinstall of the device. ## Disabling Remote Access EVE provides a mechanism to build an image with remote access disabled (edge-view and ssh), this can be done by configuring EVE when building an installer. Enabling remote access back requires access to the cloud controller to enable console keyboard access on the edge node, plus physical access to the edge node to issue `eve remote-access` command on the edge node. In addition changing remote access status from its initial value to anything else will result in change of PCR-14 value and subsequent failure in unsealing the vault key that needs to be handled using the cloud controller (see [MEASURED-CONFIG](MEASURED-CONFIG.md)). Check [config document](CONFIG.md#eve-configuration) for more information. ## Details on keys and certificates These details are specified in [KEYS-AND-CERTS](KEYS-AND-CERTS.md). ## Hardware Security These details are specified in [Hardware Security Recommendation](SECURITY-HARDWARE.md).