Dev/Source Code Intro
< Dev(Redirected from Dev/SourceCodeIntro)
- 1 Introduction
- 2 OpenPGP keys
- 3 Chroot Scripts
- 4 Debugging
- 5 SSH
- 6 Mount and inspect images
- 7 Less Important Goodies
- 8 Porting
- 9 Build Script Features
- 10 Notes
- 11 Why not Replace grml-debootstrap with 'X'?
- 12 Footnotes
This chapter is dedicated to give an introduction into the Whonix ™ source code. If you prefer to read and understand the source code just by reading scripts you may skip this optional chapter. It can be quite difficult to get started with hacking existing big complex projects.
This is a high level overview. Whonix ™-Example-Implementation can currently be build in nine steps. (There is also a whonix_build script, which automates these eight steps for your convenience.)
Boring methods, which are required by the build-steps above.
- pre gets sourced by all other scripts.
- variables gets sourced by all other scripts.
Is a script, which simply runs all other build-steps. Actually it is "optional". It has very little functionality besides running all other steps. You are free to run all steps one by one. That is useful for learning and for debugging purposes. In case you want to fix a bug or in case you want to upgrade the distribution or in case you want to switch the operating system or whatever you are better off running the steps manually. Run it with --help to see available switches.
- prepare-build-machine: installs build dependencies and applies a few other required settings for building from source.
- export-libvirt-xml: Copies files from libvirt folder to ~/whonix_binary folder and adds version numbers to them.
- create-debian-packages: Recursively works through packages folder, builds them one by one and adds them to a local APT repository.
- create-raw-image: grml-debootstrap [archive] creates virtual machine images in .raw image format. It keeps care of creating the image, the partition table, grub boot manager, minimal system debootstrap and apt-get installing all Whonix ™ packages. This step requires most time in the build process.
- install-packages: Installs meta .deb packages whonix-shared-packages-dependencies/recommended, desktop, desktop-kde, kde-accessibility and whonix-*-packages-dependencies/recommended and whonix-workstation-default-applications. Those only include dependencies and recommended packages which are pulled from Debian's apt repository.
- run-chroot-scripts-post-d: Post Chroot Scripts are applied.
- convert-raw-to-qcow2: Only having effect when using --qocw2 switch. The .raw image gets converted to a .qcow2 image. Actually not converted, a new file will be created and the old .raw remains available until cleanup is run or manually deleted or grml-debootstrap runs again. The .raw format is more "generic". VirtualBox does not support raw (.raw) images, but .vdi and .vmdk (and others).
- convert-raw-to-vdi: The .raw image gets converted to a .vdi image. (Actually not converted, as explained above.)
- create-vbox-vm: A virtual machine (VirtualBox) will be created and the .vdi image will be attached.
All steps could be easily replaced to add support for additional virtualizers, operating systems and so on. The numbers before the script names (20_..., 25_..., 30..., ...) are honored by whonix_build (using run-parts [archive]), which runs these steps in lexical order. Therefore you can easily add steps to the beginning or the end or even wretch steps in between.
Do not get shocked by file sized. Initially the .raw is created with a size of 100 GB, but it will actually only take a fraction of that space on the harddrive. This is because .raw .vdi and .vmdk are all sparse files, which means they do not take their maximum size, but only as much data as really is inside these images.
Thus, given the nature of the build step orientated scripts, you can easily work on the different aspects of Whonix ™. For example, once you have created a clean virtual machine with the operating system only, you can make a copy, run either the gateway or the workstation copy routine or Chroot Scripts as often as you need to test your changes and if something goes wrong, go back to backup. You don't have to build everything from scratch again. 
It is really not that difficult after all. If you like, you could read Manually Creating Whonix ™, which is a similar topic, which covers part of this in other words.
OpenPGP public keys required for building Whonix ™ are stored inside various packages. These include.
- tb-updater: /usr/share/torbrowser-updater-keys.d [archive]
- anon-shared-build-apt-sources-tpo: /usr/share/anon-shared-build-apt-sources-tpo/tpoarchive-keys.d [archive]
- whonixcheck: /usr/share/whonix/whonix-news-keys.d [archive]
To find out what the keys are good for, simply grep the source code.
grep -r torbrowser-updater-keys.d * grep -r tpoarchive-keys.d * grep -r /usr/share/whonix-repository *
If you are in luck, you never have to update the keys yourself and the Whonix ™ maintainer will keep them updated. Otherwise and also as a good precaution you can verify these keys manually. Follow the instructions from torproject.org to obtain the key. Then simply check if the keys match or update the old key with the new one.
Torproject's archive key expires on 2022-08-05.
gpg --fingerprint A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 pub rsa2048 2009-09-04 [SC] [expires: 2022-08-05] A3C4 F0F9 79CA A22C DBA8 F512 EE8C BC9E 886D DD89 uid [ unknown] deb.torproject.org archive signing key sub rsa2048 2009-09-04 [S] [expires: 2020-11-23]
- https://github.com/QubesOS/qubes-template-configs/blob/master/R4.0/templates-community/whonix-15.conf [archive]
What are Chroot Scripts?
Some operations for building Whonix ™ cannot run as part Debian maintainer scripts (preinst, postinst, prerm, postrm). Those are installed by whonix-initializer [archive] to /usr/lib/anon-dist/chroot-scripts-post.d. When those scripts are just installed, they do nothing. Those scripts are run in lexical order after package installation by a later build step of Whonix ™ build script (see #Introduction for overview).
How many chroot-scripts are there and what are they used for?
To get an always up to date list of packages that do ship chroot-scripts, you could run find from within Whonix ™ source code folder.
find . -type f -ipath *chroot-script* | sort
Here is the result that has been written at time of Whonix ™ git tag
./build-steps.d/2300_run-chroot-scripts-post-d ./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/20_sanity_checks ./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/75_sources ./packages/whonix-initializer/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup
So let's go through them one by one.
- Details: Sanity tests that check the integrity of the build. See package description of the anon-shared-build-sanity-checks [archive] package for a more verbose explanation.
- Used by default in Whonix ™ build script: Yes.
- Negligible for Whonix ™ installations from repository: Up to you.
- Convertible into postinst script: Probably not, because some tests such as checking if apt-get/dpkg are in sane states cannot run inside a postinst script because apt-get/dpkg would be still working at that time.
- package: https://github.com/Whonix/anon-shared-build-sanity-checks [archive]
- Details: Logs GPL sources during build - for better obeying legal requirements.
- Used by default in Whonix ™ build script: Yes.
- Negligible for Whonix ™ installations from repository: Up to you.
- Convertible into postinst script: Probably not, because for this script to have a chance of being effective, it must run after installation of all packages has finished.
- package: https://github.com/Whonix/anon-shared-build-remember-sources [archive]
- script: https://github.com/Whonix/anon-shared-build-remember-sources/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/75_sources [archive]
- Details: Cleanup, and required for Verifiable Builds, therefore only useful for vm image builds.
- Used by default in Whonix ™ build script: Yes.
- Negligible for Whonix ™ installations from repository: Yes.
- Convertible into postinst script: Probably not and probably not required. If the image is not supposed to be redistributed, there is no need to clean up. If the image is supposed to be redistributed, it should be cleaned up beforehand.
- package: https://github.com/Whonix/whonix-initializer [archive]
- script: https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup [archive]
Stuff you may find helpful for debugging.
Build Configuration Folder
You can drop configuration file either in:
- or in
Files should have the extension .conf.
Method 2. is recommended for users.
- Contains examples. It is more difficult to use, since you would have to git commit your build config files. Rather use.
- When is your Whonix ™ source folder, you could use as your Whonix ™ build configuration folder. It is easier to use, since you don't have to git commit your build config files.
Below is an example how to use method 2.
sudo mkdir /etc/whonix_buildconfig.d
/etc/whonix_buildconfig.d/50_user.conf in an editor with root rights.
(Qubes-Whonix ™: In TemplateVM)
Add. (Replace it with whatever build configuration variable you wish to set.)
WHONIX_APT_REPOSITORY_OPTS='--enable --repository stable' export WHONIX_APT_REPOSITORY_OPTS
Skipping Build Steps
Rebuild the .ova images, while skipping the slow prepare-build-machine and create-debian-raw steps. (Of course, this assumes that these steps where run at least once previously.)
The  step takes far most of the build creation time. As long as no packages have been added or removed, you can repeat all other steps from a backup, which has been automatically created for you, by using.
sudo ./whonix_build --tor-gateway --build --fast1 ## or --tor-workstation
There is also, which skips all steps up to and including 1700_install-meta-packages.
sudo ./whonix_build --tor-gateway --build --fast2 ## or --tor-workstation
Skipping Chroot and Postinst Scripts
Some chroot scripts, especially the 70_update_command_not_found and 70_torbrowser take very long, while they are non-critical. You can either run update-command-not-found and the Tor Browser Updater after Whonix ™ has been build and booted, or if you don't wish to use them at all, just don't use it at all. Therefore you can conveniently disable them, to safe some time, which is useful for debug builds.
Create your own build configuration, seefolder. For see example , which contains
#export SKIP_SCRIPTS+=" 70_torbrowser "
You could create a folder fileand a file and add.
export SKIP_SCRIPTS+=" 70_torbrowser "
This works for the following folders.
Look at the file names and add them. export SKIP_SCRIPTS+=" another_file_name ". Don't forget empty spaces () before and after .
Building without whonix_build main script
Example using an apt cache (apt-cacher-ng) and building qcow2 images.
sudo apt-get install apt-cacher-ng
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/1100_prepare-build-machine --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/1150_export-libvirt-xml --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/1200_create-debian-packages --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/1300_create-raw-image --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/1700_install-packages --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/2300_run-chroot-scripts-post-d --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/2400_convert-raw-to-qcow2 --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/2500_convert-raw-to-vdi --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/2600_create-vbox-vm --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/2700_export-vbox-vm --build --target qcow2 --flavor whonix-gateway
sudo REPO_PROXY=http://127.0.0.1:3142 ./build-steps.d/2800_create-report --build --target qcow2 --flavor whonix-gateway
For example for a qcow2 build, the following steps can be safely skipped, since these would be doing nothing anyway.
- 2800_create-report (not in use currently)
When first creating a gateway build and then building a workstation build (or vice versa) the following steps do not need to be repeated:
Mount and inspect images
- Interactively chroot Whonix-Gateway ™.
Open a bash shell inside the Whonix-Gateway ™ .raw image.
sudo ./debug-steps/interactive-chroot-raw -tg
- Interactively chroot Whonix-Workstation ™.
Open a bash shell inside the Whonix-Workstation ™ .raw image.
sudo ./debug-steps/interactive-chroot-raw -tw
- Check Whonix ™ version number.
Important before building builds for redistribution.
Build config variable.
Less Important Goodies
Unpacking .ova images
If you want for some reason to unpack an .ova, for example to get the .vdmk image, you can use tar.
tar -xvf Whonix-Gateway.ova
Converting .vmdk images to .raw images
#qemu-img info Whonix-Gateway-disk1.vmdk
qemu-img convert Whonix-Gateway-disk1.vmdk -O raw Whonix-Gateway.raw
#qemu-img info Whonix-Gateway.raw
These are some random notes about porting Whonix ™ update debs to rpm.
What would have to be done:
- create rpm package
- Find a replacement for config-package-dev, a package which allows third party packages (Whonix ™) to own files which are owned by other packages. Such as /etc/tor/torrc is owned by tor, but anon-gw-anonymizer-config includes its own config file.
- add init scripts (currently done by debhelper)
- add man pages (currently done by debhelper and ronn, see debian/rules)
- minor: replacement for dh_apparmor
Build Script Features
Build Script Features - Implemented
Each item in the following lists (essential, non-essential, undecided) is currently supported by the Whonix ™ build script.
- does not use non-deterministic binary base boxes (VM images) (which if compromised would compromise the resulting Whonix ™ image)
- does not download anything that does not get gpg verified (apt-get, tb-updater do gpg verify everything)
- prospective support to create deterministic images (once this is generally possible, Whonix ™ build script can also learn this)
- supports using APT Cache to speed up builds
- exit code checking everywhere
- apt-get unreliable exit code checking workaround
- --target virtualbox build VirtualBox ova images
- custom virtualbox VM settings (VBoxManage modifyvm "$VMNAME" --synthcpu on... etc.)
- --target qcow2 build KVM images
- xz archive creation
- adding whonix-libvirt xml files to the xz
- build Whonix-Gateway ™
- build Whonix-Workstation ™
- install Tor Browser by default inside Whonix-Workstation ™
- default login user account creation, user "user" / password "changeme"
- step based, build steps case be run manually to speed up development
- injection of custom build steps
- building Whonix ™ images that do not come with a desktop environment
- building Whonix ™ images that do not come with Whonix ™ default applications
- easy to implement creating other image types (raw images etc.)
- creation of hash sum verification and gpg signatures
- automatically installs all required build dependencies on the host system
- cleanup command removing temporary files and/or images
- --arch parameter support (--arch amd64 or --arch i386)
- --kernel and --headers parameter support (--kernel linux-image-amd64 --headers linux-headers-amd64)
- build from local apt repository rather than from deterministic remote repository
- install Whonix ™ packages from own custom remote repository
- interactive error handler to repeat commands, open a shell or ignore them
- optional build step skipping feature
- --target root (for physical isolation)
- --target raw build raw images
- custom VM settings during build (these can of course be manually changed by the user anyhow), supported parameters with examples:
- --vmram 128
- --vram 12
- --vmsize 200G
- build image that never had Whonix ™ repository enabled
- build and install all Whonix ™ packages during Whonix ™ image build
- protection from bad build surprises
- break or do not break when uncommitted changes are found
- break or do not break from non-tag
- --confdir /path/to/config/dir
- --tb none|closed|open
- none: Do not install Tor Browser.
- closed: Fail closed if Tor Browser cannot be installed.
- open: Fail open if Tor Browser cannot and installed.
- use onion apt sources for building
- install different packages for virtualbox (virtualbox-guest-x11) and kvm (spice...)
Build Script Features - Missing
separate VirtualBox / KVM builds during official builds - https://forums.whonix.org/t/non-qubes-whonix-13-0-0-1-0-x-issues/2443/4 [archive](Fixed. Nowadays there is
sudo ~/Whonix/packages/whonix-developer-meta-files/debug-steps/interactive-chroot-raw --target raw --flavor whonix-host-xfce --build --freedom false
Why not Replace grml-debootstrap with 'X'?
Whonix ™ developers are not remotely close to exhausting grml-debootstrap's extensive feature-set yet. 
There are two sorts of VM image creation tools:
[A] Those that use virtualization, boot the image and perform actions. [B] Those that use chroot (or maybe systemd-spawn).
[A] is incompatible with Whonix ™ design principles due to files being created during boot, such as entropy seeds. This is a less clean method and is not suitable for redistribution.
In either case it would be necessary to diff:
The image created by grml-debootstrap versus the new tool. Create an image twice using the new tool and inspect the diff.
https://github.com/Whonix/Whonix/blob/master/help-steps/analyze_image [archive] https://www.whonix.org/wiki/Verifiable_Builds [archive]
- If something would go wrong, you would have to reinstall the whole operating system every time again. That's why we use separate steps.
- This is because means "one level below this folder".
- (replace X with g or w)
- We are expanding a bash array.
- https://forums.whonix.org/t/replacing-grml-debootstrap-debos-build-platform/5582/3 [archive]
This is a wiki. Want to improve this page? Help is welcome and volunteer contributions are happily considered! Read, understand and agree to Conditions for Contributions to Whonix ™, then Edit! Edits are held for moderation.
Copyright (C) 2012 - 2019 ENCRYPTED SUPPORT LP. Whonix ™ is a trademark. Whonix ™ is a licensee [archive] of the Open Invention Network [archive]. Unless otherwise noted, the content of this page is copyrighted and licensed under the same Freedom Software license as Whonix ™ itself. (Why?)