Introduction[edit]

This chapter is dedicated to give an introduction into the Whonix ™ source code. If you prefer to read and understand the source code just by reading scripts you may skip this optional chapter. It can be quite difficult to get started with hacking existing big complex projects.

build-steps

This is a high level overview. Whonix ™-Example-Implementation can currently be build in nine steps. (There is also a whonix_build script, which automates these eight steps for your convenience.)

1100_prepare-build-machine
1150_export-libvirt-xml
1200_create-debian-packages
1300_create-raw-image
1700_install-packages
2300_run-chroot-scripts-post-d
2400_convert-raw-to-qcow2
2500_convert-raw-to-vdi
2600_create-vbox-vm
2700_export-vbox-vm
2800_create-report

help-steps

Boring methods, which are required by the build-steps above.

delete-vbox-vm
mount-raw
chroot-raw
unchroot-raw
unmount-raw
pre gets sourced by all other scripts.
variables gets sourced by all other scripts.

whonix_build

Is a script, which simply runs all other build-steps. Actually it is "optional". It has very little functionality besides running all other steps. You are free to run all steps one by one. That is useful for learning and for debugging purposes. In case you want to fix a bug or in case you want to upgrade the distribution or in case you want to switch the operating system or whatever you are better off running the steps manually. Run it with --help to see available switches.

Overview:

prepare-build-machine: installs build dependencies and applies a few other required settings for building from source.
export-libvirt-xml: Copies files from libvirt folder to ~/whonix_binary folder and adds version numbers to them.
create-debian-packages: Recursively works through packages folder, builds them one by one and adds them to a local APT repository.
create-raw-image: grml-debootstrap ^[archive] creates virtual machine images in .raw image format. It keeps care of creating the image, the partition table, grub boot manager, minimal system debootstrap and apt installing all Whonix ™ packages. This step requires most time in the build process.
install-packages: Installs meta .deb packages whonix-shared-packages-dependencies/recommended, desktop, desktop-kde, kde-accessibility and whonix-*-packages-dependencies/recommended and whonix-workstation-default-applications. Those only include dependencies and recommended packages which are pulled from Debian's apt repository.
run-chroot-scripts-post-d: Post Chroot Scripts are applied.
convert-raw-to-qcow2: Only having effect when using --qocw2 switch. The .raw image gets converted to a .qcow2 image. Actually not converted, a new file will be created and the old .raw remains available until cleanup is run or manually deleted or grml-debootstrap runs again. The .raw format is more "generic". VirtualBox does not support raw (.raw) images, but .vdi and .vmdk (and others).
convert-raw-to-vdi: The .raw image gets converted to a .vdi image. (Actually not converted, as explained above.)
create-vbox-vm: A virtual machine (VirtualBox) will be created and the .vdi image will be attached.

Modularity:

All steps could be easily replaced to add support for additional virtualizers, operating systems and so on. The numbers before the script names (20_..., 25_..., 30..., ...) are honored by whonix_build (using run-parts ^[archive]), which runs these steps in lexical order. Therefore you can easily add steps to the beginning or the end or even wretch steps in between.

File sizes:

Do not get shocked by file sized. Initially the .raw is created with a size of 100 GB, but it will actually only take a fraction of that space on the harddrive. This is because .raw .vdi and .vmdk are all sparse files, which means they do not take their maximum size, but only as much data as really is inside these images.

Chroot Scripts

packages/anon-*-build-*/usr/lib/anon-dist/chroot-scripts-post.d/

Summary:

Thus, given the nature of the build step orientated scripts, you can easily work on the different aspects of Whonix ™. For example, once you have created a clean virtual machine with the operating system only, you can make a copy, run either the gateway or the workstation copy routine or Chroot Scripts as often as you need to test your changes and if something goes wrong, go back to backup. You don't have to build everything from scratch again. ^[1]

Another Introduction:

It is really not that difficult after all. If you like, you could read Manually Creating Whonix ™, which is a similar topic, which covers part of this in other words.

OpenPGP keys[edit]

OpenPGP public keys required for building Whonix ™ are stored inside various packages. These include.

tb-updater: /usr/share/torbrowser-updater-keys.d ^[archive]
anon-shared-build-apt-sources-tpo: /usr/share/anon-shared-build-apt-sources-tpo/tpoarchive-keys.d ^[archive]
- torprojectarchive.asc - Torproject's archive key https://www.torproject.org/docs/debian.html.en ^[archive]
whonix-repository:
- /usr/share/whonix-repository/ ^[archive]
  - whonix.asc
whonixcheck: /usr/share/whonix/whonix-news-keys.d ^[archive]
- derivative.asc - Whonix ™ maintainer key - used for systemcheck's Whonix News verification ^[archive]

To find out what the keys are good for, simply grep the source code.

cd /home/user/Whonix

grep -r torbrowser-updater-keys.d *
grep -r tpoarchive-keys.d *
grep -r /usr/share/whonix-repository *

If you are in luck, you never have to update the keys yourself and the Whonix ™ maintainer will keep them updated. Otherwise and also as a good precaution you can verify these keys manually. Follow the instructions from torproject.org to obtain the key. Then simply check if the keys match or update the old key with the new one.

Torproject's archive key expires on 2022-08-05.

gpg --fingerprint A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
pub   rsa2048 2009-09-04 [SC] [expires: 2022-08-05]
      A3C4 F0F9 79CA A22C DBA8  F512 EE8C BC9E 886D DD89
uid           [ unknown] deb.torproject.org archive signing key
sub   rsa2048 2009-09-04 [S] [expires: 2020-11-23]

https://github.com/QubesOS/qubes-template-configs/blob/master/R4.0/templates-community/whonix-15.conf ^[archive]

Chroot Scripts[edit]

What are Chroot Scripts?[edit]

Some operations for building Whonix ™ cannot run as part Debian maintainer scripts (preinst, postinst, prerm, postrm). Those are installed by whonix-initializer ^[archive] to /usr/lib/anon-dist/chroot-scripts-post.d. When those scripts are just installed, they do nothing. Those scripts are run in lexical order after package installation by a later build step of Whonix ™ build script (see #Introduction for overview).

How many chroot-scripts are there and what are they used for?[edit]

Introduction[edit]

To get an always up to date list of packages that do ship chroot-scripts, you could run find from within Whonix ™ source code folder.

find . -type f -ipath *chroot-script* | sort

Here is the result that has been written at time of Whonix ™ git tag 13.0.0.1.1.

./build-steps.d/2300_run-chroot-scripts-post-d
./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/20_sanity_checks
./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/75_sources
./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup

So let's go through them one by one.

run-chroot-scripts-post-d[edit]

./build-steps.d/2300_run-chroot-scripts-post-d

This is actually not a chroot-script, it is the build-steps.d/2300_run-chroot-scripts-post-d ^[archive] build step, which is responsible of running all these chroot-scripts at Whonix ™ build time.

anon-shared-build-sanity-checks[edit]

./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/20_sanity_checks

Details: Sanity tests that check the integrity of the build.
Used by default in Whonix ™ build script: Yes.
Negligible for Whonix ™ installations from repository: Up to you.
Convertible into postinst script: Probably not, because some tests such as checking if apt/dpkg are in sane states cannot run inside a postinst script because apt/dpkg would be still working at that time.
package: https://github.com/Whonix/whonix-initializer ^[archive]
script: https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/20_sanity_checks ^[archive]

anon-shared-build-remember-sources[edit]

./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/75_sources

Details: Logs GPL sources during build - for better obeying legal requirements.
Used by default in Whonix ™ build script: Yes.
Negligible for Whonix ™ installations from repository: Up to you.
Convertible into postinst script: Probably not, because for this script to have a chance of being effective, it must run after installation of all packages has finished.
package: https://github.com/Whonix/whonix-initializer ^[archive]
script: https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/75_sources ^[archive]

whonix-initializer[edit]

./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup

Details: Cleanup, and required for Verifiable Builds, therefore only useful for vm image builds.
Used by default in Whonix ™ build script: Yes.
Negligible for Whonix ™ installations from repository: Yes.
Convertible into postinst script: Probably not and probably not required. If the image is not supposed to be redistributed, there is no need to clean up. If the image is supposed to be redistributed, it should be cleaned up beforehand.
package: https://github.com/Whonix/whonix-initializer ^[archive]
script: https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup ^[archive]

Debugging[edit]

Stuff you may find helpful for debugging.

Build Configuration[edit]

See Build Configuration.

Building without whonix_build main script[edit]

most basic[edit]

Example using an apt cache (apt-cacher-ng) and building qcow2 images.

sudo apt install apt-cacher-ng

cd {{project_name}}

sudo ./build-steps.d/1100_prepare-build-machine --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/1150_export-libvirt-xml --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/1200_create-debian-packages --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/1300_create-raw-image --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/1700_install-packages --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/2300_run-chroot-scripts-post-d --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/2400_convert-raw-to-qcow2 --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/2500_convert-raw-to-vdi --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/2600_create-vbox-vm --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/2700_export-vbox-vm --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/2800_create-report --build --target qcow2 --flavor whonix-gateway

optional[edit]

For example for a qcow2 build, the following steps can be safely skipped, since these would be doing nothing anyway.

2500_convert-raw-to-vdi
2700_export-vbox-vm
2800_create-report (not in use currently)

When first creating a gateway build and then building a workstation build (or vice versa) the following steps do not need to be repeated:

1100_prepare-build-machine
1200_create-debian-packages

SSH[edit]

Mount and inspect images[edit]

Mount and inspect images

Interactive Chroot[edit]

https://github.com/Whonix/whonix-developer-meta-files/ ^[archive]

Interactively chroot Whonix-Gateway ™.

Open a bash shell inside the Whonix-Gateway ™ .raw image.

sudo ./debug-steps/interactive-chroot-raw -tg

Interactively chroot Whonix-Workstation ™.

Open a bash shell inside the Whonix-Workstation ™ .raw image.

sudo ./debug-steps/interactive-chroot-raw -tw

Check Whonix ™ version number.

Important before building builds for redistribution.

Build config variable.

anon_dist_build_version

cat /usr/share/whonix/build_version

Less Important Goodies[edit]

Unpacking .ova images[edit]

If you want for some reason to unpack an .ova, for example to get the .vdmk image, you can use tar.

tar -xvf Whonix-Gateway.ova

tar -xvf Whonix-Gateway.ova

Will show:

Whonix-Gateway.ovf
Whonix-Gateway-disk1.vmdk

Whonix-Gateway.ovf
Whonix-Gateway-disk1.vmdk

Converting .vmdk images to .raw images[edit]

#qemu-img info Whonix-Gateway-disk1.vmdk

#qemu-img info Whonix-Gateway-disk1.vmdk

qemu-img convert Whonix-Gateway-disk1.vmdk -O raw Whonix-Gateway.raw

qemu-img convert Whonix-Gateway-disk1.vmdk -O raw Whonix-Gateway.raw

#qemu-img info Whonix-Gateway.raw

#qemu-img info Whonix-Gateway.raw

Porting[edit]

These are some random notes about porting Whonix ™ update debs to rpm.

What would have to be done:

create rpm package
Find a replacement for config-package-dev, a package which allows third party packages (Whonix ™) to own files which are owned by other packages. Such as /etc/tor/torrc is owned by tor, but anon-gw-anonymizer-config includes its own config file.
add init scripts (currently done by debhelper)
add man pages (currently done by debhelper and ronn, see debian/rules)
minor: replacement for dh_apparmor

Build Script Features[edit]

Build Script Features - Implemented[edit]

Each item in the following lists (essential, non-essential, undecided) is currently supported by the Whonix ™ build script.

essential:

does not use non-deterministic binary base boxes (VM images) (which if compromised would compromise the resulting Whonix ™ image)
does not download anything that does not get gpg verified (apt, tb-updater do gpg verify everything)
prospective support to create deterministic images (once this is generally possible, Whonix ™ build script can also learn this)
supports using APT Cache to speed up builds
exit code checking everywhere
APT unreliable exit code checking workaround
--target virtualbox build VirtualBox ova images
- custom virtualbox VM settings (VBoxManage modifyvm "$VMNAME" --synthcpu on... etc.)
--target qcow2 build KVM images
- xz archive creation
- adding whonix-libvirt xml files to the xz
build Whonix-Gateway ™
build Whonix-Workstation ™
install Tor Browser by default inside Whonix-Workstation ™
default login user account creation, user "user" / password "changeme"

non-essential:

step based, build steps case be run manually to speed up development
injection of custom build steps
building Whonix ™ images that do not come with a desktop environment
building Whonix ™ images that do not come with Whonix ™ default applications

undecided:

easy to implement creating other image types (raw images etc.)
creation of hash sum verification and gpg signatures
automatically installs all required build dependencies on the host system
cleanup command removing temporary files and/or images
--arch parameter support (--arch amd64 or --arch i386)
--kernel and --headers parameter support (--kernel linux-image-amd64 --headers linux-headers-amd64)
build from local apt repository rather than from deterministic remote repository
install Whonix ™ packages from own custom remote repository
interactive error handler to repeat commands, open a shell or ignore them
optional build step skipping feature
--target root (for physical isolation)
--target raw build raw images
custom VM settings during build (these can of course be manually changed by the user anyhow), supported parameters with examples:
- --vmram 128
- --vram 12
- --vmsize 200G
build image that never had Whonix ™ repository enabled
build and install all Whonix ™ packages during Whonix ™ image build
protection from bad build surprises
- break or do not break when uncommitted changes are found
- break or do not break from non-tag
--confdir /path/to/config/dir
--tb none|closed|open
- none: Do not install Tor Browser.
- closed: Fail closed if Tor Browser cannot be installed.
- open: Fail open if Tor Browser cannot and installed.
use onion apt sources for building
install different packages for virtualbox (virtualbox-guest-x11) and kvm (spice...)

Separate VirtualBox / KVM builds ^[2] using --target virtualbox or --target qcow2.

Notes[edit]

sudo ~/Whonix/packages/whonix-developer-meta-files/debug-steps/interactive-chroot-raw --target raw --flavor whonix-host-xfce --build --freedom false

Why not Replace grml-debootstrap with 'X'?[edit]

Whonix ™ developers are not remotely close to exhausting grml-debootstrap's extensive feature-set yet. ^[3]

There are two sorts of VM image creation tools:

[A] Those that use virtualization, boot the image and perform actions.
[B] Those that use chroot (or maybe systemd-spawn).

[A] is incompatible with Whonix ™ design principles due to files being created during boot, such as entropy seeds. This is a less clean method and is not suitable for redistribution. Similar to Distro Morphing vs Builds.

In either case it would be necessary to compare: