Jump to: navigation, search

Dev/KVM

< Dev


Contents

Links[edit]

Continuous Integration[edit]

Blockers[edit]

None.

Non-Blockers[edit]

Image Rebasing[edit]

https://www.suse.com/documentation/sles11/book_kvm/data/cha_qemu_guest_inst_qemu-img.html http://nairobi-embedded.org/manipulating_disk_images_with_qemu-img.html#working-with-base-and-derived-images

Play with image rebasing. Advantages: disk space savings, one time update for all derived images.

This feature can be used to create what Qubes calls "templates".

NAT Port Forwarding[edit]

Instructions for port forwarding to GW. Useful if configuring GW as Tor relay. Relevant if /when this is supported in Whonix

http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

Virgl3D[edit]

Blog gives complete minimum requirements and settings needed to enable. Check Stretch package versions after freeze and determine if its supported or not. Document how to enable 3D on wiki.

https://www.kraxel.org/blog/2016/09/using-virtio-gpu-with-libvirt-and-spice/

Audit Output of virsh domxml-to-native[edit]

virsh domxml-to-native qemu-argv ./usr/share/whonix-libvirt/xml/Whonix-Gateway_qemu.xml
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-system-x86_64 -name Whonix-Gateway -machine ubuntu,accel=tcg,usb=off -cpu qemu64,-kvmclock,+kvm_pv_eoi,+kvm_pv_unhalt -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d67e18a8-ea3c-4c6d-81eb-99a6324506a6 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6d:37:bc,bus=pci.0,addr=0x3 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ac:89:1e,bus=pci.0,addr=0x4 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5901,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on -device qxl-vga,id=video0,ram_size=262144000,vram_size=262144000,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x5 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 -msg timestamp=on

Same but rewritten for better readability.

LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name Whonix-Gateway \
-machine ubuntu,accel=tcg,usb=off \
-cpu qemu64,-kvmclock,+kvm_pv_eoi,+kvm_pv_unhalt \
-m 512 \
-realtime mlock=off  \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid d67e18a8-ea3c-4c6d-81eb-99a6324506a6 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,clock=vm,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 \
-drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6d:37:bc,bus=pci.0,addr=0x3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ac:89:1e,bus=pci.0,addr=0x4 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-device usb-tablet,id=input0 \
-spice port=5901,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=262144000,vram_size=262144000,bus=pci.0,addr=0x2 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x5 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-object rng-random,id=rng0,filename=/dev/random \
-device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 \
-msg timestamp=on

Does it look sense? Do we understand all of it? Is there anything we should change?

HulaHoop: Output makes sense. Needs to be updated because of recent config changes. Some upcoming.

See CI log for output of running against other xml files.

Kernel Direct Boot[edit]

Minor importance.

Document how to do a kernel direct boot. For passing special options for debugging options such as disabling apparmor in kernel. Interesting, because even if grub was broken for some reason, VM would still be bootable.

HulaHoop: There is an example here: https://libvirt.org/formatdomain.html#elementsOSKernel feel free to change it to suit Whonix.

Hardware side-channel research[edit]

Ongoing. Interesting developments worth watching out for at https://gruss.cc

Upcoming XML Changes - Stable Next[edit]

When Stretch hits stable disable:

vmport
smm
pmu

In Progress[edit]

None.

Done[edit]

Apply Clean-filter traffic?[edit]

Advantages are its easy to set a new IP for a new WS. Has many rules to prevent MAC, IP, ARP spoofing. Can also protect the GW from DoS by using connection tracking and rate-limiting.

Sounds good but I still think a completely separate internal network is the best option to ensure separartion between multi WSs. The GW should be able to withstand DoS on its own or there will be big problems. I think its safer if libvirt knows nothing about the traffic on the internal network and does not parse anything on there.

To have a multi-WS implementation with different internal networks likely requires a big effort on the Whonix development side. Not worth it when setting a new GW is easy and costs little resources - also provides more assurance against theoretical attacks on a shared GW.

RNG[edit]

Rate limit[edit]

This is necessary to prevent malicious entropy starvation of other VMs and the host.

Test code for /dev/random throughput[1]

dd if=/dev/random bs=1k count=1

Re-reading the article it seems the choice made is OK. The limit sets the max a guest can use to be 25% of that of the host.

/dev/random provides high quality numbers but stops when entropy is poor: the Linux kernel keeps a entropy pool of 4 KB (see /proc/sys/kernel/random/poolsize); when the pool is empty, the kernel blocks causing a delay until it can provide the requested random numbers.

https://www.certdepot.net/rhel7-get-started-random-number-generator/

Note: Just after <rng model=’virtio’>, you can add a line like <rate bytes=’1024′ period=’1000’/> to prevent a guest from consuming more than 1 KB per second of pseudo random numbers. Otherwise, one single guest could consume all the available pseudo random numbers at the expense of all the others.

Also applied for GW since some crypto requiring operations under the control of WS processes can abuse the entropy available to it if unchecked.

Haveged[edit]

Cross-posted

   haveged relies on the RDTSC instruction, that apparently is useless in some virtualized environments. Also, the quality of random numbers output by HAVEGE is unclear, and the topic of many discussions.

https://tails.boum.org/contribute/design/random/#rngd

haveged is not useful on KVM

rdtsc or any tsc counter provides very accurate clock tick information so better to block.

blkiotune[edit]

Its possible to limit IO throughput per VM however with disk speeds varying a lot it will be impossible to find an option that makes sense for everybody. Nonetheless still interesting to have. Documented below is a IO test and how to set limits.

https://fedoraproject.org/wiki/QA:Testcase_Virtualization_IO_Throttling

Secure Clipboard[edit]

https://bugzilla.redhat.com/show_bug.cgi?id=1320263


Biometric Fingerprinting[edit]

This problem is related to all hypervisors and noted here to keep track. New info added not in mail list message:

Upstream tickets:

https://lists.nongnu.org/archive/html/qemu-discuss/2016-03/msg00023.html

https://secure-os.org/pipermail/desktops/2016-March/000108.html

https://bugzilla.redhat.com/show_bug.cgi?id=1318813


https://marc.info/?l=linux-kernel&m=145877344732456&w=2

https://lists.freedesktop.org/archives/wayland-devel/2016-March/027607.html

https://lists.x.org/archives/xorg-devel/2016-March/049159.html

https://phoronix.com/scan.php?page=news_item&px=Anti-Keystroke-FP-Wayland-Kern

https://mailman.stanford.edu/pipermail/liberationtech/2016-March/015947.html

https://mailman.boum.org/pipermail/tails-dev/2016-March/010485.html

https://lists.torproject.org/pipermail/tor-talk/2015-July/038624.html

Testing sites listed here: https://forums.whonix.org/t/help-wanted-kvm-users-biometrics-fingerprinting-testing/2233

Related code:

https://cgit.freedesktop.org/spice/spice-gtk/tree/src/channel-inputs.h

https://cgit.freedesktop.org/spice/spice-gtk/tree/src/channel-inputs.c

http://git.qemu.org/?p=qemu.git;a=blob;f=hw/input/ps2.c


Virtualization is useful in presenting an identical environment and set of "hardware" for each user which goes a long way in creating an anonymity set of systems. That way a system attacker, advertisers and online trackers would not be able to fingeprint a user or their hardware.

The problem: Tracking techniques have become more sophisticated with time. They advanced from simple cookies to browser/device fingerprinting (which Tor Browser focuses on defeating) to user behavior fingerprinting. The latter is about profiling how a user types on a keyboard or uses a mouse [2].

Keystroke dynamics is a super creepy way to track users based on how long they press keys (dwell time) and the time between key presses (gap time). This is extremely accurate at identifying individuals because of how unique these measurements are. Advertising networks (Google, Facebook...) that fingeprprint users on both the clearnet and Tor can deanonymize users. This technique is already actively used in the wild [6][7].


Potential Solutions:

Since input devices are all emulated its a great opportunity to stop this profiling technique.

  • A security researcher designed a proof of concept plugin for Chrome browser that mitigates this. Implementing something like the PoC addon in [1] known as KeyBoardPrivacy. Some random delay in milliseconds in a 50 millisecond range for dwell and gap times for the emulated keyboards is enough to skew the values to render this attack useless while not affecting performance.


  • The changes made to Tor Borwser to make JS timers more coarse grained but constant (250ms for keyboard events) were not enough to stop keystroke dynamics fingerprinting because a malicious script can evict the cache and allow extrapolation of true timing events within 1-5ms accuracy .[3][5] Their goal is to instead add jitter to the timers [4]. A similar solution proposed in [4] can be implemented in all QEMU-KVM timers to mitigate both attacks.


EDIT: Second solution not the way to go. High precision timers in virtualizers have bad security implications. They enable sidechannel attacks on crypto operations running on other VMs as well as aid in biometric fingerprinting. A multi hypervisor study on timer accuracy shows most have high accuracy[8] by necessity. Increasing coarseness in the hypervisor's code can lead to guest slow downs or instability. This varies by hypervisor. CPU load does NOT affect timer accuracy [in KVM] however IO does. A stress/stress-ng daemon installed on GW that runs some varying workload may introduce a skew to timer data.

EDIT 2:

Even if the user's destination does not itself surreptitiously sniff their biometrics, any one observing the traffic of any interactive JS application can generate a model for your biometric statistics. [16]


[1] https://paul.reviews/behavioral-profiling-the-password-you-cant-change/

[2] http://jcarlosnorte.com/security/2016/03/06/advanced-tor-browser-fingerprinting.html

[3] https://www.lightbluetouchpaper.org/2015/07/30/double-bill-password-hashing-competition-keyboardprivacy/#comment-1288166

[4] https://trac.torproject.org/projects/tor/ticket/16110

[5] https://trac.torproject.org/projects/tor/ticket/1517

[6] http://scraping.pro/no-captcha-recaptcha-challenge/

[7] https://nakedsecurity.sophos.com/2013/11/01/facebook-to-silent-track-users-cursor-movements-to-see-which-ads-we-like-best/

[8] Achieving High Resolution Timer Events in Virtualized Environment https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4503740/pdf/pone.0130887.pdf

[9] http://dataprotectioncenter.com/featured/keystroke-profiling/

[10] http://arstechnica.com/security/2015/07/how-the-way-you-type-can-shatter-anonymity-even-on-tor/

[11] Spoofing key-press latencies with a generative keystroke dynamics model [DOI: 10.1109/BTAS.2015.7358795] http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=7358795

[12] "I Am Robot: (Deep) Learning to Break Semantic Image CAPTCHAs" http://www.cs.columbia.edu/~polakis/papers/sivakorn_eurosp16.pdf

[13] I can be You: Questioning the use of Keystroke Dynamics as Bio metrics http://flyer.sis.smu.edu.sg/ndss13-tey.pdf

[14] How I reverse-engineered Google Docs to play back any document's keystrokes https://news.ycombinator.com/item?id=8562483

[15] An Efficient User Verification System via Mouse Movements [DOI=10.1.1.220.9410] http://www.cs.wm.edu/~hnw/paper/ccs11.pdf

[16] Knowing the User’s Every Move – User Activity Tracking for Website Usability Evaluation and Implicit Interaction https://www.medien.ifi.lmu.de/pubdb/publications/pub/atterer2006www/atterer2006www.pdf

[17] Eliminating Fine Grained Timers in Xen http://cseweb.ucsd.edu/~bvattikonda/docs/xentimers-ccsw11.pdf

Mouse Tracking[edit]

User re-authentication via mouse movements https://doi.org/10.1145/1029208.1029210

Relatively successful tracking active readers however mouse movements are not enough to fingerprint users. That was back in 2004.

ON USING MOUSE MOVEMENTS AS A BIOMETRIC http://www.cs.sjsu.edu/faculty/pollett/papers/shivanipaper.pdf http://www.cs.sjsu.edu/faculty/pollett/masters/Semesters/Spring04/Shivani/CS298Presentation.pdf

Needs more work to achieve high accuracy. 2004.


An Efficient User Verification System via Mouse Movements http://www.cs.wm.edu/~hnw/paper/ccs11.pdf


high accuracy achieved in limited situations - actice authentication during log-on. Does not clear EU false positive requirements however so they recommend it for combining with keystroke dynamics as extra confirmation:

In the scenario of static verification, a user is required to perform a series of mouse movements and its mouse data is verified within a certain amount of time (e.g., login time). A good example of this scenario is a click-based graphical password for user login, where five clicks are estimated to be made in no more than 25 seconds.

By contrast, in the scenario of continuous verification, a user’s mouse data is continuously collected and verified throughout the entire session. This is non-intrusive to users and meets the goal of passive monitoring. However, the frequency of user mouse actions varies significantly in different sessions. In general, the average frequency of user mouse actions will be much lower than that of the static scenario.

Even for the same user at different times, the number of mouse events per unit time varies a lot. However, to the best of our knowledge, our work is the first to achieve high accuracy with a reasonably small number of mouse events.

search terms:

Mouse movement authentication

CPU Masking[edit]

KVM cpus support a baseline of features by default. You can mask out the problematic ones and don't have to worry about the extra ones it doesn't support because it will be masked out anyhow (because it was never supported in the first place).

The only bad instructions we should filter out are a subset of whatever instructions are listed under the virtual cpu from the output of the cpu_map.xml list

cat /usr/share/libvirt/cpu_map.xml

I figured out safe defaults and will do a pull request. NB clflush was abused to carry out the rowhammer attack so its blacklisted.aes will be passed through for crypto performance - it doesn't mess with random number generation.

<cpu mode='custom' match='exact'>
   <model fallback='forbid'>qemu64</model>
   <topology sockets='1' cores='2' threads='1'/>
   <feature policy='disable' name='tsc'/>
   <feature policy='disable' name='clflush'/>
   <feature policy='optional' name='aes'/>
</cpu>


https://libvirt.org/formatdomain.html#elementsCPU

Informative link about cpu flag functionality: https://unix.stackexchange.com/questions/43539/what-do-the-flags-in-proc-cpuinfo-mean

https://www.berrange.com/posts/2010/02/15/guest-cpu-model-configuration-in-libvirt-with-qemukvm/

"Every hypervisor has its own policies for what a guest will see for its CPUs by default, Xen just passes through the host CPU, with QEMU/KVM the guest sees a generic model called “qemu32” or “qemu64”. "


cat output:

     <model name='qemu64'>
      <!-- These are supported only by TCG.  KVM supports them only if the
           host does.  So we leave them out:

           <feature name='abm'/>
           <feature name='lahf_lm'/>
           <feature name='popcnt'/>
           <feature name='sse4a'/>
      -->
      <feature name='apic'/>
      <feature name='clflush'/>
      <feature name='cmov'/>
      <feature name='cx16'/>
      <feature name='cx8'/>
      <feature name='de'/>
      <feature name='fpu'/>
      <feature name='fxsr'/>
      <feature name='lm'/>
      <feature name='mca'/>
      <feature name='mce'/>
      <feature name='mmx'/>
      <feature name='msr'/>
      <feature name='mtrr'/>
      <feature name='nx'/>
      <feature name='pae'/>
      <feature name='pat'/>
      <feature name='pge'/>
      <feature name='pni'/>
      <feature name='pse'/>
      <feature name='pse36'/>
      <feature name='sep'/>
      <feature name='sse'/>
      <feature name='sse2'/>
      <feature name='svm'/>
      <feature name='syscall'/>
      <feature name='tsc'/>
    </model>

Passed to Command Line[edit]

Minor importance.

Libvirt "only" runs the qemu-kvm command line utility with the correct command line parameters?

Is there a way to make libvirt tell with what parameters libvirt is calling qemu-kvm?

HulaHoop: Yes its possible. Conversion of commands is supported in bot directions. See documentation here: http://libvirt.org/drvqemu.html#xmlexport

Nice and useful. Added to unit_test:
https://github.com/Whonix/whonix-libvirt/commit/4bcac51b264b9d78042899be005e4c06549224ab

Haveged test for KVM + virtio-rng Result[edit]

NB Patrick has conducted the tests without virtio-rng and they worked too.

Reults with virtio-rng:

0 failed individual tests with THRESHOLD 0.001000 on 514 individual tests
PASS: nist/test.sh
==================
All 2 tests passed
==================
make[2]: Leaving directory `/home/user/haveged-1.4'
make[1]: Leaving directory `/home/user/haveged-1.4'

Two Separate Whonix-Gateway's[edit]

https://www.whonix.org/old-forum/index.php/topic,397.msg2973.html#new Possible and documented.

Preserve Comments in libvirt / virsh XML Files[edit]

Asked on stackexchange:
http://unix.stackexchange.com/questions/148163/is-it-possible-to-use-comments-in-virsh-libvirt-xml-files

Asked on libvirt mailinglist and here is their response: https://www.redhat.com/archives/libvir-list/2014-August/msg00681.html

A description attribute can be used to carry human readable metadata.

Activate Watchdog?[edit]

libvirt can be told to ignore a guest's request for restart. The actual action would be a silent shutdown. Same for when a crash happens, libvirt would just leave the machine in its shutdown state instead of booting it again.

This all hinges on what you think about the attack i described earlier. Should we just advise people not to do things on their host that would leak timestamps? Like browsing for instance. From what I got from the wiki, this is the only thing done on a Linux host that could leak this information.

Later I learned that TCP Sequence numbers could still leak time information, but its more difficult to exploit than other methods.

Patrick wrote: "the attack i described earlier" in unclear for other readers and I am also not so sure about it anymore.

Patrick wrote: Not sure exactly what you want to defend here, but I guess this info might invalid its premises. The ACPI reset command is not required. An adversary could simply use kexec. It allows to reboot a machine without ACPI reset command. kexec's purpose is to save time by not going through BIOS initialization routines. Therefore KVM ignoring ACPI reset command probably won't be a big help.

HulaHoop: Agreed. I went ahead and re-added it.

Add pvspinlock when supported in more recent version of libvirt[edit]

Was not accepting it because its a fresh feature with not support in the version of libvirt I run. Its been added as of libvirt 1.2 as per the reply. [2]

HulaHoop: Done.

Preconfigured Shared Folders[edit]

Patrick asks: For better usability, should we add the necessary settings for shared folders by default to whonix_gateway.xml and whonix_workstation.xml?

Patrick comments: I could then, if kvm is detected (easy using virt-what), automatically mount the shared folder in Whonix VMs. Could be implemented into the shared-folder-help package.

Patrick asks: If we decide to do so, what would be good default paths for the host shared folder? /mnt/shared_gw and /mnt/shared_ws?

Patrick said: HulaHoop reported, that "virsh define" will fail if the shared folder in the xml does not exist.

See also: https://github.com/Whonix/Whonix/issues/223

Kernel Samepage Merging (KSM)[edit]

Any changes to the images required? Probably adding this init script:
https://github.com/dnaeon/ksm-init.d-debian (looks good and simple - trivial to package) Any other changes required?

Any changes to documentation or xml required to make use of it?

While not needed to enable this functionality for Whonix directly,its useful for Debian host users. Its inclusion in Whonix will definitely help when running Nested Virtualization though so please add this.

To make use of it, it needs to be enabled host-side following this guide[3]. It is used automatically by libvirt without further intervention, but certain parameters such as huge page sharing can be, optionally, used through the XML.[4]

Using it on the host is as simple as starting the two services ksm and ksmtuned following the guide above. No need to adjust the number of pages it uses anymore as newer versions automatically handle it with bigger RAM amounts as to give a bigger performance improvement compared to earlier versions that used just 2000 pages of RAM.

How to verify that KSM is working and looking at its stats.[5]

Notes: What appeared to be effects of KSM was actually just the Linux Guests not using all allocated RAM by default. Non Linux guests usually take over the entire assigned RAM allocated space while Linux does not. By default KSM kicks in when there is very little free free RAM on the host. Settings must be tweaked in ksmtuned.conf to change this behavior. Experimenting in progress...

KSM security implications?

3.
ATTACK ON MEMORY
DEDUPLICATION
Memory deduplication may be subject to a memory disclosure
attack from the attacker’s VM to the victim’s VM. The attack
guesses a process running on a victim’s VM or a file downloaded
by a browser.
Memory deduplication shares 4KB pages which have the
same contents. When data is written to a deduplicated page, the
page is re-created with a copy of its contents (Copy On Write).
This causes the write access time to be slower than normal,
because it includes the overhead to re-create the same page. An
attacker can measure access time to determine whether a page was
deduplicated by another VM. 
[6]

Enable Hugepages for guests? Yes. I am seeing benchmarks of 40% improvement of performance for all workloads in a guest according to the literature. However using it will prevent use of memory ballooning and swapping[7] and is subject to lack of adequate sVirt protections.[8] Just confirmed from Dan Walsh, SELinux policy developer that this is no longer the case. The sacrificing of one performance feature for another may not be a problem afterall. Since 2012 there has been an effort to teach KSM how to play nice with THP transparent Hugetables. [9]

More info[10]

Documentation: A kind soul posted how to interpret what ksmuned.conf's parameters mean in plain English in the first post. Its worth mirroring in our documentation: http://forum.proxmox.com/threads/5041-KSM-with-2-6-35

No need to play around with the defaults. ksmtuned should handle memory overcommitment adjustments dynamically when the host is under pressure.

More accurate instructions on starting ksm and ksmtuned services and keeping them that way at boot. They will be summarized in the documentation too:

http://www-01.ibm.com/support/knowledgecenter/api/content/linuxonibm/liaat/liaatbpsharing.htm http://www-01.ibm.com/support/knowledgecenter/api/content/linuxonibm/liaat/liaatbpksm.htm http://www-01.ibm.com/support/knowledgecenter/api/content/linuxonibm/liaat/liaatbprunksm6.htm http://www-01.ibm.com/support/knowledgecenter/api/content/linuxonibm/liaat/liaatbpksmmemsavings6.htm

TO DO: packaging https://github.com/Whonix/Whonix/issues/39#issuecomment-53991872

TO DO: Document on KVM wikipage how users can enable this permanently.

Documentation and risk assessment completed.

Done.

Pass qemu commands via libvirt?[edit]

Not a good idea, has potential security and stability risks. https://www.redhat.com/archives/libvir-list/2014-September/msg00288.html

Useful in some cases such as guest debugging via qemu's gdb while retainging libvirt's security framework. Relevant documentation added.

memory balloon[edit]

https://rwmj.wordpress.com/2010/07/17/virtio-balloon/

Any changes to documentation, xml or images required to make use of it?

HulaHoop: Just having the device added is enough to take advantage of it. At the moment this feature needs manual intervention to make use of it which limits its utility in the real world. However, towards the end of 2013 work is being done to have this feature automated and mainlined [11] This space will be watched for developments; as soon as its incorporated upstream and an option added in libvirt I will enable it in the configuration.

SPICE security implications[edit]

http://lists.freedesktop.org/archives/spice-devel/2013-December/015810.html

http://lists.freedesktop.org/archives/spice-devel/2014-January/015815.html

Protections applying to QEMU apply here too.

How to check if virtio storage access is in use or an eventually existing fallback driver?[edit]

Question[edit]

How to check if virtio storage access virtio_blk really is in use or an eventually existing fallback driver?

http://www.linux-kvm.org/page/Boot_from_virtio_block_device
http://wiki.libvirt.org/page/Virtio

Asked on unix.stackexchange[edit]

http://unix.stackexchange.com/questions/148055/kvm-how-to-check-if-virtio-storage-access-is-in-use

lsmod won't help[edit]

The following.

lsmod | grep virtio

Only shows, that the virtio kernel module is loaded. It isn't hard to load arbitrary kernel modules for hardware that you don't have installed ("sudo modprobe virtio-blk"). The question remains, is this kernel module actually in use or an eventually existing fallback driver?

modinfo won't help[edit]

modinfo virtio-blk

Won't help, because it only shows general information it would also show on VBox systems (after running "sudo modprobe virtio-blk").

ls Test[edit]

/dev/sda and /dev/sda1 should not exist.

ls -la /dev/sda
ls -la /dev/sda1

/dev/vda and /dev/vda1 should exist.

ls -la /dev/vda
ls -la /dev/vda1

hwinfo Test[edit]

See which driver is loaded for each particular device.

hwinfo --short

How to check if virtio network is really in use or an eventually existing fallback driver?[edit]

How to check, that virtio_pci is really being used and not some fallback driver?

/sys/devices/virtio-pci/0/net/eth0/statistics/rx_bytes seems not to exist on Debian Wheezy.

How to enable AppArmor / SELinux for KVM on Debian?[edit]

Details on how to install, configure and enable SELinux enforcing mode: https://wiki.debian.org/SELinux/Setup#Steps_to_setup_SELinux <-- This is a start, but incomplete. When following these instructions it gets difficult at the "audit2why -al" step, because it shows a ton of policy violations which are non-trivial to fix without learning a lot about SELinux.

Why Bother? SELinux is more robust than Apparmor because its label based vs file-path based. But his comes at the expense of being difficult to write policies. However the plans I have for it in mind, it won't matter since we won't be writing SELinux profiles, but leveraging the one already present in sVirt as a protection layer around application sandboxes.

HulaHoop: This is probably a result of not having done a relabel of your filesystem after enabling SELinux. This aspect can be handled by a script from the link below. I'll need to test this at some point. Any files created before SELinux was prsent creates conflicts with policy when it is enforced. A relabel must also run if you want to isolate software that has a policy but was introduced after SELinux was enabled. Example KVM if we decide to ship it for nested purposes in the future.

http://debian-handbook.info/browse/stable/sect.selinux.html Best guide has everything in one place: https://www.wzdftpd.net/docs/selinux/using_debian_selinux.html

Another guiede to test with: http://blog.danielcosta.pt/?p=440

Note: Both MAC systems cannot be run simultaneously run. This is not supported by LSM and may also be a source of conflicts.


For Debian hosts, AppArmor makes more sense.

http://libvirt.org/drvqemu.html#securitysvirtaa

AppArmor profiles for Ubuntu could be ported to Debian: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/utopic/libvirt/utopic/files/head:/debian/apparmor/

DONE. DOCUMENTED.

Set cluster_size 512 - 2MB for better performance?[edit]

HulaHoop1 wrote: The qcow2 defaults seem fine to me, however the one parameter of interest we should play with is image's : 'cluster_size' 512 - 2MB the larger the size the better performance but the size is bigger. I think we should try the largest and see if the image size stays reasonable.

Patrick wrote:

When using (Whonix 7.7.8.0):

     qemu-img \
        convert \
           -p \
           -O qcow2 \
           -o preallocation=metadata \
           "$WHONIX_BINARY/$VMNAME.img" \
           "$WHONIX_BINARY/$VMNAME-$whonix_build_whonix_version_new.qcow2"

du -h --apparent-size file.img
100 GB

du -h file.img
2,5 GB

When using (Whonix 7.7.8.0):

     qemu-img \
        convert \
           -p \
           -O qcow2 \
           -o cluster_size=2M \
           -o preallocation=metadata \
           "$WHONIX_BINARY/$VMNAME.img" \
           "$WHONIX_BINARY/$VMNAME-$whonix_build_whonix_version_new.qcow2"

du -h --apparent-size file.img
2,9 GB

du -h file.img
2,9 GB

File size grew a bit. Not too much. We could use this.

It remains to be tested, if the images work well using these settings. (The apparent size should be shown as 100 GB, but is only 2,9 GB. Needs testing if 100 GB can be used after the image is in use.)

"qemu-img info file.img" however, shows 2,9 GB actual size and 100 GB virtual size.

git tag 7.8.0.1 and above will be using the "-o cluster_size=2M" option.

HulaHoop said: I am yet to document this in the KVM page lets do this at the end. Is there any tangible speed gains?

Exporting/Cloning of virtual machines[edit]

Important links:

Archived thread SE

I'm a bit lost with virt-manager / libvirt / KVM.

I've got a working KVM VM (Windows XP) which works nicely.

The VM is backed by a 4GB file or so (a .img).

Now I want to do something very simple: I want to duplicate my VM.

I thought "OK, no problem, let's copy the 4GB file and copy the XML" file.

But then the libvirt FAQ states in all uppercase: "you SHOULD NOT CARE WHERE THE XML IS STORED"

http://wiki.libvirt.org/page/FAQ

OK fine, I shouldn't care. But then how do I duplicate my VM?

I want to create a new VM that is a copy of that VM.

The most convenient is simply:

virt-clone --connect=qemu://example.com/system -o this-vm -n that-vm --auto-clone

Which will make a copy of this-vm, named that-vm, and takes care of duplicating storage devices. Nothing new here except detail.

More to the point, What the FAQ is saying is that the XML domain descriptions are not directly editable, you need to go through libvirt. To complete the steps taken by the virt-clone command, you could:

You cannot "clone" a running vm, stop it. suspend and destroy are also valid options for less graceful cloning

virsh shutdown this.vm

copy the storage.

cp /var/lib/libvirt/images/{this-vm,that-vm}.img

dump the xml for the original

virsh dumpxml this-vm > /tmp/that-vm.xml

hardware addresses need to be removed, libvirt will assign new addresses automatically

sed -i /uuid/d /tmp/that-vm.xml
sed -i '/mac address/d' /tmp/that-vm.xml

and actually rename the vm: (this also updates the storage path)

sed -i s/this-vm/that-vm /tmp/that-vm.xml

finally, create the new vm

virsh define /tmp/that-vm.xml
virsh start this-vm
virsh start that-vm
# dump the xml for the virtual isolated network
virsh net-dumpxml whonix > /tmp/whonix.xml

# hardware addresses need to be removed, libvirt will assign
# new addresses automatically
sed -i /uuid/d /tmp/whonix.xml
sed -i '/mac address/d' /whonix.xml

management tools[edit]

There are a lot management tools. Are any of these useful for Whonix's use case?

Nested Virtualization[edit]

As a minor point, documenting how to use KVM with nested virtualization would be nice.

Forum thread:
https://forums.whonix.org/t/nested-virtulization-needs-64-bit-whonx-guest

This is only possible using a 64 bit kernel. Install a 64 bit kernel. Don't get confused by the package name. Works with AMD and Intel.

sudo apt-get update
sudo apt-get install linux-image-amd64

qemu i386 support[edit]

Why are our qemu xml files currently called ${VMNAME}_qemu-x86-x64.xml. Are they dependent on x86-x64? Should they work on i386 systems as well? Maybe just <pae/> must be removed? Can you test please?

Any QEMU architecture works on any host supported by QEMU.i386 tested and working but pointless when x86-64 works on i386 hosts too.


Disable Microphone Input[edit]

Figure out how to disable Microphone input to Guests. While a nice feature for VOIP, is a very dangerous setting to have by default as its unexpected.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/section-libvirt-dom-xml-sound-devices.html

http://libvirt.org/formatdomain.html

Somewhat Solved: After speaking with Cole Robinson, libvirt developer, its clear that this is a feature provided by SPICE rather than something having to do with the sound card. I found that all sound cards have this after testing yesterday. When I changed Spice to VNC this no longer happens. Before you say that we shouldn't use spice, its important to note that audio does not function without it at all.

I've asked on the mailinglists if there is some xml attribute to disable the feature. From what I've seen in the documentation this is not possible. However it can be accomplished through the Host's audio popup menu in the taskbar. The devices show up with the name "virt-manager: record" and can be set to mute as with any other device on the host.

http://libvirt.org/formatdomain.html#elementsGraphics

Other spice channel functionality provided:

Valid channel names include main, display, inputs, cursor, playback, record (all since 0.8.6); smartcard (since 0.8.8); and usbredir (since 0.9.12). 

Mailinglist reply: https://www.redhat.com/archives/virt-tools-list/2014-August/msg00002.html Currently not supported through libvirt but planned. Not a problem, we can just document how to disable its easy and gui based. For those running from commandlines, they should just revert to a non SPICE adapter and they'll be fine. They are not missing much because they are not using the workstation as a desktop.

Bugzilla Ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1126641

Document: how to leave KVM when no X is running?[edit]

Situation... User is in terminal in a VM... No X is running ("sudo service kdm stop"). User wants to switch back to the host... How to do this?

The emulated tablet device handles this by not allowing the mouse to be captured by the guest. Its still possible though:

serverfault.com/questions/457603/any-way-to-release-focus-on-a-kvm-guest-in-virt-manager-without-havin-g-to-click

press Ctrl_L & Alt_L


Audio Retasking[edit]

SPEAKE(a)R: Turn Speakers to Microphones for Fun and Profit https://arxiv.org/ftp/arxiv/papers/1611/1611.07350.pdf


conclusion:

While virtual sound devices might be retaskable with software too the malware still cannot access the soundcard settings of the host to retask it too given that the hypervisor is not broken.

Replies from the QEMU devs. They confirm my conclusion that no risk posed to host. Also virtual soundcards optionally have half-duplex modes that make audio input impossible.

https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04754.html2 Replies: https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04899.html2 https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04906.html2 https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04904.html2


libguestfs import/export feature request[edit]

Let's together write a feature request for an import/export feature for libguestfs, similar to VirtualBox's import/export feature. Then post in libguestfs's bug tracker.

Irrelevant see virt-manager feature instead.

virt-manager import/export feature request[edit]

Let's together write a feature request for an import/export feature for virt-manager, similar to VirtualBox's import/export feature. Then post in virt-manager's bug tracker.

Update on import/export Support:

Gnome Boxes, another simplified GUI for libvirt will have an OVF import feature added. Note that OVF is not the vmware vmdk specfic format OVA. In this sense making it a true standard.

Its in the works and the Gnome contributor behind it has posted that he plans to make it into a stand-alone library so it could be used by other projects. Meaning it could be easily included in virt-manager too.

His post: http://zee-nix.blogspot.com/2014/05/berlin-dx-hackfest-boxes-rain-and.html

Initial commit: https://gitorious.org/govf


Update 2016:

Past efforts have come to a stop. Opened new discussion: https://www.redhat.com/archives/libvir-list/2016-March/msg00165.html

virt-v2v[edit]

Importing KVM to KVM - Not what its for: https://rwmj.wordpress.com/2015/09/18/importing-kvm-guests-to-ovirt-or-rhev/#content

How to install virt-v2v on Debian?[edit]

virt-v2v is not yet in Debian. wishlist request

Can we copy and paste virt-v2v in meanwhile?[edit]

Since virt2v2 is not yet in Debian, maybe it is a single script that can replace the whonix_libvirt_import script?

HulaHoop: It is my opinion that we shelve the automated script idea completely and use virt-v2v when its possible/available. If virt-v2v does not work out, we should instead do the following: 1) ship multiple templates according to containment systems we want to support 2) A readme file with instructions on [a] how to copy the sparse file properly to any location a user wants if they wish [b] how to introduce the sparse file into priviledged dirs like /var/lib.. [c] how to import xml for machines [d] how to autostart the isolated network [e] how to edit xml files in the case that they want to use non-default paths

The build process would be scripted to update the version number of the name and disk img xml fields and thats it.

The absence of an easy way to import a machine is too big of an issue. We cannot compensate for a shortcoming as big as this coming from upstream. It would take tremendous effort as we have seen and some people might not be happy that it doesn't do X. So I say we leave it as described above.

How to export a VM using virt-v2v?[edit]

This is what probably must be done in the build script. We got qcow2 files and xml. What command must be used to export a VM?

How to export a VM using virt-v2v?[edit]

This is probably what users have to do to import. What command must be used to import a VM?

Closed[edit]

Looks like we've settled on not using virt-v2v for either importing Whonix VMs or for cloning VMs. Probably mainly because v2v is not in Debian and hence doesn't make things simpler.

Shared Folder[edit]

We should add instructions on how to use shared folders. http://www.linux-kvm.org/page/9p_virtio

Done: KVM#KVM_Shared_Folders

Script/automate creation and upload of raw images?[edit]

Because raw images have better performance than raw images? http://www.linux-kvm.org/page/Qcow2

True that performance is better, but I don't think it supports snapshots AFAIK. For something that supports snapshots and good performance lets stick with qcow2.

write generic import script[edit]

Maybe write generic import script as per:
https://www.whonix.org/old-forum/index.php/topic,159.msg2412.html#msg2412

A hellish solution and way too much work. Your time is spent better elsewhere. I'll use my time to get xml import validation working.

Script/automate creation of VM description files?[edit]

So not following manual instructions is required anymore, but copy/paste some commands end up with the correct VM configuration.

Once the necessary commands are documented, Patrick will be happy to automate the steps with a script.

HulaHoop created xml files, because kvm xml files can not be created using libvirt as VirtualBox VM setting files can be created using vboxmanage. This is not a problem.

How to add a random time offset?[edit]

Having a small random offset (similar to Advanced_Security_Guide#Network_Time_Synchronization) would be a good thing.

Patrick said: Thanks to bootclockrandomization this is no longer necessary.

xml does not validate[edit]

virsh --version
0.9.12.3
virt-xml-validate whonix_gateway.xml
whonix_gateway.xml:24: element pm: Relax-NG validity error : Element domain has extra content: pm
whonix_gateway.xml fails to validate
virt-xml-validate whonix_workstation.xml 
whonix_workstation.xml:24: element pm: Relax-NG validity error : Element domain has extra content: pm
whonix_workstation.xml fails to validate
virt-xml-validate whonix_network.xml 
whonix_network.xml:1: element network: Relax-NG validity error : Invalid attribute connections for element network                                                                  
whonix_network.xml fails to validate

kvmclock had to be under rtc for it to work. The Network xml validates on my machine without changes however. Please check if it works and post back.

The virt-xml-validate I tested with is a newer version. Try updating to a more recent one and try again, I think it may be buggy or not updated enough to interpret new schema they may have added since .9x was released.

Patrick said: HulaHoop said this is only due to different KVM versions and won't cause issues.

Patrick said: Leaving automatic validation out for now. Can still be done manually after editing the xml files.

Run virt-xml-validate during build process[edit]

We should use.

virt-xml-validate libvirt.xml

Patrick will be happy to add this command to the build process when using --kvm (or --qemu?) switch. Scripting this should be simple and the build process could fail early and closed (break).

Patrick said: Since validation is too version specific (see above), we're not doing this.

Weaker warning against KVM in whonixcheck[edit]

As soon as all blockers have been sorted out. We could recommend using KVM instructions.

Done.

Leak Tests?[edit]

HulaHoop1 sorted that out in Special:AWCforum/st/id306/.

Hardware serials?[edit]

HulaHoop1 sorted that out in Special:AWCforum/st/id306/.

How to Edit in nano with virsh?[edit]

su

  1. EDITOR=nano virsh edit GuestMachineName

whonixcheck KVMClock[edit]

Whonixcheck should check if /sys/devices/system/clocksource/clocksource0/current_clocksource is kvm-clock and warn if it does.

Implemented: https://github.com/Whonix/Whonix/commit/2584c3bdcd9a98181c7535f963a615d257e0fefd

Is it possible to add 64, 128 or even more MB to the virtual graphics card using the command line to boost graphics performance?[edit]

Yes. Can be realised by virsh command as per the steps in here: http://serverfault.com/questions/346301/how-to-increase-video-memory-in-libvirt-kvm-gui

KVM Instructions recommend to use a no cache policy?[edit]

As per blog post it is recommended to use a no cache policy. Do our KVM instructions need changes to incorporate this?

Yes if its recommended then lets do this, Its easily done through the GUI by going to: disk -> Cache policy: set to none

Script/automate creation and upload of qcow2 images?[edit]

Done.

Verifiable Builds for qcow2 images[edit]

The https://github.com/Whonix/Whonix/blob/master/help-steps/analyze_image requires supports raw, vdi, vmdk, qcow and qcow2 images as well as .ova appliances.

metadata preallocated images[edit]

For better performance. (As per blog post.) Done in 7.7.8.0.

qcow2 compression[edit]

When uncompressed, upload would take ~250 hours per image. Now compressing using tar. [Not using qcow2's compression feature.]

How to disable KVMClock?[edit]

In xml file add: <timer name='kvmclock' present='no'/>


What does KVMClock do? "kvmclock or KVM pvclock lets guests read the host’s wall clock time." This is bad. source


How to adjust KVM to have clock=vm on host.

See http://www.linux-kvm.org/page/KVMClock

http://libvirt.org/formatdomain.html#elementsTime

https://www.redhat.com/archives/libvir-list/2010-March/msg01293.html

https://doc.opensuse.org/products/draft/SLES/SLES-kvm_sd_draft/cha.qemu.running.html#cha.qemu.running.gen_opts.rtc

ls /sys/devices/system/clocksource/clocksource0/current_clocksource

To ensure that you have successfully isolated the vm clock form the host's you must run this in the vm:

sudo chmod -x /etc/init.d/sdwdate

You should verify that kvmclock is indeed disabled by running this command as root: cat /sys/devices/system/clocksource/clocksource0/available_clocksource

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

Make sure kvmclock is not listed as a response any longer.

Documented for reference but is irrelevant as of now since this problem is solved.


https://bugzilla.redhat.com/show_bug.cgi?id=783921#c6 < This link provides the solution to disabling kvmclock and verifying that it's so.

whonixcheck (since Whonix 8) would tell you if kvmclock is enabled. whonixcheck --verbose will tell you, that it isn't enabled.

Fix subnet range in KVM instructions[edit]

As found out by zweeble in https://www.whonix.org/old-forum/index.php/topic,328.0.html, subnet range must be adjusted to make KVM work.

HulaHoop: I think until we can be sure that this is a problem it shouldn't be considered a blocker

Readme Commands and Warnings[edit]

Until we compile and test a list of virsh commands needed for operating on vm templates and associated disk storage, we should shot release KVM version to the public.

# virsh net-define [path]/whonix_network.xml
Internal network 'Whonix' defined from network xml file residing in the following directory path.
# virsh net-autostart Whonix
Network 'Whonix' marked to be autostarted upon boot.
# virsh net-start Whonix
Network 'Whonix' has been started


Leak[edit]

Instructions include how to set clock to UTC?[edit]

As per http://libvirt.org/formatdomain.html#elementsTime

The clock setting.

Quote:

The offset attribute takes four possible values, allowing fine grained control over how the guest clock is synchronized to the host. NB, not all hypervisors support all modes.

How to check, that it is set to utc?

It is set to utc by default. A quick look at the xml file will show: <clock offset='utc'>

How to use -rtc clock=vm ?[edit]

KVM's -rtc option as per https://doc.opensuse.org/products/draft/SLES/SLES-kvm_sd_draft/cha.qemu.running.html#cha.qemu.running.gen_opts.rtc

This is the original qemu patchset that implements clock=vm in the timer back-ends: https://www.redhat.com/archives/libvir-list/2010-March/msg01293.html

If you need to isolate the VM Guest clock from the host one, specify clock=vm instead of the default clock=host.

How to use -rtc clock=vm ?

How to check, that KVM -rtc is set to clock=vm ?


Does not seem to be a valid or accepted parameter by libvirt as its always sanitized from the xml file upon saving. But it is accepted by QEMU directly. However I noticed that even with kvmclock enabled, I no longer see any change in a guest's time in response to a host skew of the time. In the recent past this was not true, as any change was immediately reflected in the Guest clock immediately. kvmclock on its own can be disabled and confirmed to be so as outlined below.

What is timer pit?[edit]

What is timer pit?

HulaHoop: Whatever it was, its now gone. I overlooked this by mistake.

We are using.

    <timer name='kvmclock' present='no'/>
    <timer name='hpet' present='no'/>

But why are we using?

    <timer name='pit' tickpolicy='no'/>

Why are we not using?

    <timer name='pit' present='no'/>

virt-image[edit]

Users Expected Scenario[edit]

source

sudo apt-get install virt-manager

Extract Whonix-Gateway-8.3.qcow2.xz.

Import.

virt-image Whonix-Gateway-8.3.xml

Start Whonix-Gateway in virt-manager.

xml creation[edit]

How can we create an xml file, that can be imported using virt-image?

Possible help:
https://fedoraproject.org/wiki/Features/ApplianceTools#Release_Engineering

Maybe we need to learn from their fedora-aos.xml to structure whonix xml.

Since virt-image has been deprecated, this is no option. Source:
http://blog.wikichoon.com/2014/04/deprecating-little-used-tool-virt-image1.html

How to get virtio running for storage?[edit]

Initial Instructions[edit]

Virtio-blk now works

Here are the steps, which I'll throw altogether here before the server downtime. Will move into the wiki later:

http://www.linux-kvm.org/page/Boot_from_virtio_block_device < Original steps that needed to be changed so it can apply to Debian.

Quote

   in guest os, change /boot/grub/device.map from "(hd0) /dev/sda" to "(hd0) /dev/vda"
   in guest os, change /boot/grub/grub.cfg from "root=/dev/sda1" to "root=/dev/vda1", if you are using UUID, then no need to do this step. 


I changed all mention of sda1 to vda1 in the grub file. I changed the disk controller type from virtmanager to virtio which takes care of this at the libvirt xml level. Now the vm boots normally with no hangups and I feel a little faster.

A bit of advice, that I'll add so you can decide:


Quote

   <iggy> bancfc: the better option would be using LABELs and UUIDs and such in the source image
   <iggy> unless this is a one time conversion kind of thing

Using LABELs[edit]

Patrick asks: How to use LABELs?

Patrick says: Nevermind. We're now using UUIDs.

Using UUIDs[edit]

Patrick writes: In Whonix 9, fstab will look like this.

cat /etc/fstab
/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6 /  auto    defaults,errors=remount-ro 0   1
proc           /proc        proc    defaults                      0   0
/dev/cdrom     /mnt/cdrom0  iso9660 ro,user,noauto                0   0
# some other examples:
# /dev/sda2       none         swap    sw,pri=0             0   0
# /dev/hda1       /Grml        ext3    dev,suid,user,noauto 0  2
# //1.2.3.4/pub   /smb/pub     cifs    user,noauto,uid=grml,gid=grml 0 0                                                                                                            
# linux:/pub      /beer        nfs     defaults             0  0                                                                                                                    
# tmpfs           /tmp         tmpfs   size=300M            0  0                                                                                                                    
# /dev/sda5       none         swap    sw                   0  0

The relevant part here is.

/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6 /  auto    defaults,errors=remount-ro 0   1

The UUID 26ada0c0-1165-4098-884d-aafd2220c2c6 is a fixed one. (Feature by grml-debootstrap.)

grep 26ada0c0-1165-4098-884d-aafd2220c2c6 /usr/sbin/grml-debootstrap                                                                                        
           tune2fs "$TARGET" -U 26ada0c0-1165-4098-884d-aafd2220c2c6

Maybe using UUID and even a fixed one will aid this feature.


HulaHoop: Yes it will. I found very encouraging answers to your inquiries on the forum that I'll post here:

1. activate virtio-blk in a booted image
2. check virtio-blk is really in use (https://www.whonix.org/wiki/Dev/KVM#How_to_check_if_virtio_storage_access_is_in_use_or_an_eventually_existing_fallback_driver.3F)
3. see how we can automate this using Whonix's build script / Whonix's packages

1. From my IRC chats with KVM devs, the virtio driverof concern has been included in all distros for a while now and is used by simply having libvirt select it so.

2. lsmod : lists all currently running modules

   modinfo <module name> : information about associated devices
   hwinfo --short : see which driver is loaded for each particular device

3. UUID is the answer from what I recall from different sources inluding KVM documentation. From a post towards the bottom of a thread, someone describes how to configure the kernel to use UUID to boot: forums.gentoo.org/viewtopic-p-6668919.html

/boot/grub/grub.cfg issue[edit]

Patrick comments: I think /boot/grub/grub.cfg is a unstable solution. That file says on top.

# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub

So when there is a kernel upgrade, /boot/grub/grub.cfg will be regenerated and the changes will be lost.

Patrick asks: How do we apply these settings in a way, so they do survive an "sudo update-grub" (which is automatically run on every kernel upgrade).

Patrick says: this is no longer required, since we're now using UUID's in grub.cfg.

System gets unbootable?[edit]

Patrick comments: After editing /boot/grub/grub.cfg, in worst case, maybe, the system will be no longer bootable then. You can test/simulate this case by running "sudo update-grub". Please try.

HulaHoop comments: I found a solution, one that survives kernel and grub upgrades: http://grumpymole.blogspot.nl/2007/05/ubuntu-how-to-edit-grub-boot-parameters.html

2.2 Making a change survive upgrades

Depending on how frequently you upgrade your system, you might find that when a new kernel is installed, or grub is upgraded (there may be other scenarios), the /boot/grub/menu.lst file may be replaced by a newer version, or a new line might be added for the new kernel. In this case, your "permanent" changes might be as permanent as you thought.

The best way to address this is to change the "#kopt..." line in /boot/grub/menu.lst. Note that the line is not commented out, even though it starts with a "#". See the excerpt below.

    ### BEGIN AUTOMAGIC KERNELS LIST
    ## lines between the AUTOMAGIC KERNELS LIST markers will be modified
    ## by the debian update-grub script except for the default options below

    ## DO NOT UNCOMMENT THEM, Just edit them to your needs

    ## ## Start Default Options ##
    ## default kernel options
    ## default kernel options for automagic boot options
    ## If you want special options for specific kernels use kopt_x_y_z
    ## where x.y.z is kernel version. Minor versions can be omitted.
    ## e.g. kopt=root=/dev/hda1 ro
    ## kopt_2_6_8=root=/dev/hdc1 ro
    ## kopt_2_6_8_2_686=root=/dev/hdc2 ro
    # kopt=root=UUID=fccafcc7-d7cc-4594-9459-a8f0db7b9f7f ro

Add the parameter changes to the end of the line "#kopt...". These parameters will then be automatically added to any new kernel entries and should also survive an upgrade of grub

Patrick comments: Have you tested this?

Patrick comments: I am not sure editing /boot/grub/grub.cfg is a robust solution. We cannot directly ship a /boot/grub/grub.cfg in a deb, because it is an auto generated file by grub-mkconfig. (When that deb was updated, /boot/grub/grub.cfg would be outdated, system unbootable, you know?) Maybe a package could be created, that appends /boot/grub/grub.cfg, but still, any solution not touching /boot/grub/grub.cfg should be preferred. If we could edit/extend /etc/default/grub, /etc/default/grub.d or /etc/grub.d, that would be much more robust. What is the official suggestion by kvm devs to solve this as a distro?

Patrick says: this is no longer required, since we're now using UUID's in grub.cfg.

Virtio Block Latency[edit]

Not really important but is interesting:
http://www.linux-kvm.org/page/Virtio/Block/Latency

HulaHoop: Yes it could have been the situation that there is latency as the data shows, but I'm sure this was improved by now.

Well what'ya know It's a bug![edit]

HulaHoop wrote:
The reason I struggled so much to get this working is because of an ancient bug that has a workaround in the build process by naming disks to vdX instead of sdX. Please generate a test image with these changes.

https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/517067 Please read this comment for the workarounds: https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/517067/comments/4

> b.) linux/udev : you could suggest that the vdX devices should be symlinked or other to sdX so that your fstab finds the devices independent of how they're connected. -> A suggestion for linux/udev. Not Whonix.

Patrick says: this is no longer required, since we're now using UUID's in grub.cfg.

Solved[edit]

https://www.whonix.org/old-forum/index.php/topic,401.msg3099.html#msg3099

Enable virtio by default?[edit]

Probably yes?

https://wiki.debian.org/DebianKVMGuests

Boot from virtio block device?[edit]

What is http://www.linux-kvm.org/page/Boot_from_virtio_block_device good for? Required?

Disable Microphone Feature Request[edit]

Let's together write a feature request for an option to disable sound. Maybe like this.

Adds only speaker, no microphone.

  <devices>
    <sound model='ich6'>
      <codec type='speaker'/>
    <sound/>
  </devices>

Then post it... Where?

Done:
https://bugzilla.redhat.com/show_bug.cgi?id=1126641

Debian Network NAT Issue[edit]

error: Requested operation is not valid: network 'default' is not active

See: https://www.whonix.org/old-forum/index.php/topic,690.msg6076.html#msg6076

Archived Discussions[edit]

Archived IRC Logs[edit]

* Now talking on #kvm
* Topic for #kvm is: KVM wiki website is http://www.linux-kvm.org - try status and faq pages, no really, read them! || you can also check the qemu manual at http://wiki.qemu.org/download/qemu-doc.html || don't paste on chan, use http://pastebin.com/ instead || libvirt/virt-manager support is #virt on irc.oftc.net
* Topic for #kvm set by aliguori at Wed Mar 30 13:38:24 2011
<bancfc> Hi How do I check if virtio storage is in use or an existing fallback driver for an emulated device?
<iggy> there's no fallback in kvm like in xen
<bancfc> so a simple grep should verify if a virtio device is used then - not merely attached I mean?
<iggy> well... if you really want to check, it's going to be /dev/vdX
<iggy> and lspci will show virtio block device
<bancfc> Thanks iggy I have a few more questions that I'd really appreciate your help with
<iggy> ask away
<bancfc> I am trying to isolate the vm clock so no changes on the host can affect it. This is currently possible with -rtc clock=vm afaik,
<bancfc> but it would be much more helpful for my project's purpose if this can be done through something like libvirt
<bancfc> is this possible with libvirt?
<iggy> that I don't know
<bancfc> the appealing part of using libvirt is the xml settings file it creates, but I think something similar can be done with qemu-kvm cfgfiles option
* iggy hates xml
<bancfc> But that means all settings would have to be imitated but directly for the qemu-kvm utility which is fine if I can get it to do the same things
<bancfc> namely the functionality of isoalted internal networking between two vms
<bancfc> is that possible with qemu-kvm directly?
<iggy> libvirt has an option to pass options directly to qemu-kvm
<bancfc> please tell me how this is done :-)
<bancfc> or where I can know this.
<bancfc> ok nevermind, I just want to know if this means that a setting in the xml file is passed directly to qemu-kvm then?
<iggy> ... You might try asking in the libvirt channel
<bancfc> I did but no one seems to reply...
<iggy> I know it's possible, I don't remember it off the top of my head
<bancfc> Thank you so far for your replies. I am actually working on a project that uses virtualization to route all of a vm's traffic through a TOR instance running on another network facing gateway vm.
<bancfc> Its known as Whonix
<bancfc> Just letting you know that I'd appreciate any help on getting the main issues with porting it to KVM solved
<bancfc> so we can prepare this for a greater good.
<bancfc> This is the bugtracker page I am working on resolving: https://www.whonix.org/wiki/Dev/KVM#How_to_get_virtio_running_for_storage.3F
<bancfc> iggy do you mind if I post the contents of this thread on our forum?
<iggy> no...
<ancfc> Is there any security drawbacks from using a raw image file?
<iggy> bancfc: security drawbacks? like what?
<bancfc> iggy: are its contents directly interpreted by the host or is that when lvm is used only?
<iggy> I'm not sure what you mean by "contents directly interpreted", but raw devices/files just means there's a 1:1 correlation between guest request and host request
<bancfc> like a virus or advanced persistent threat running in the vm
<iggy> the host doesn't "interpret" what it's reading/writing... it just becomes a read/write in the host
<bancfc> ok good just checking. iAnother question is raw snapshotting possible albeit not directly?
<bancfc> I saw something about a differencing image applied in qcow format, yes?
<iggy> only if there's something underlying the files that supports it (btrfs, lvm, zfs, etc)
<bancfc> so not at the "block" level I guess?
<bancfc> iggy: just a question or two about storage if thats ok, becuase we are trying to see which can give maximum performance
<iggy> lvm > raw files > qcow2 > everything else
<bancfc> Is qcow2 the standard? Is qed recommended instead? When is the qcow"3" changes included into the present format?
<bancfc> didn't mean to send this as you had already responded
<bancfc> but anyhow qed seems to have been designed as a speedier replacement for qcow2. But qcow2 were adding changes to become more competitive again
<bancfc> I wanted to know what you think about this and thats all. thanks
<iggy> bancfc: qed is deprecated (as all of it's performance improvements that actually ended up helping were backported to qcow2)
<iggy> the features are handled by feature flags in the qcow2 format, so there's technically no need for a "qcow3" format
<iggy> as long as you don't use the features that qed didn't support (i.e. snapshots, etc.), qcow2 should be as fast
<bancfc> iggy: thanks alot for your help
<iggy> np
* Now talking on #tor
* Topic for #tor is: Welcome to #tor, the discussion channel for Tor users and operators | https://www.torproject.org/ | https://www.torproject.org/docs/faq | https://tor.stackexchange.com/ | Developer discussion on #tor-dev | It's 'Tor', not 'TOR' | non-tor talk => #nottor
* Topic for #tor set by ChanServ!services@services.oftc.net at Sun Feb  2 13:56:09 2014
* Riastradh (~riastradh@82JAAD1W2.tor-irc.dnsbl.oftc.net) has joined #tor
<expedient> I am working on the Whonix KVM port and ran the leaktest for this bug reported here: https://lists.torproject.org/pipermail/tor-talk/2014-March/032503.html
<expedient> It turns out that this issue applies to us too and its an upstream bug
<arma> i believe you
<arma> upstream meaning what -- the kernel? iptables?
<expedient> I am thinking iptables
<expedient> You guys can probably get to them easier than we can
<arma> is there a patch?
* drtor (~drtor@9YYAAK2T4.tor-irc.dnsbl.oftc.net) has joined #tor
<expedient> We are looking at some workaround in iptables but are not sure how well this will work
<expedient> but one thing for sure is it needs to be fixed upstream because it affects any Linux platform that uses Tor as a transproxy
<expedient> AFAIK Patrick  -Whoix leader- doesn't write kernel level code
<expedient> https://www.whonix.org/old-forum/index.php/topic,243.0.html
* realitygaps has quit (Remote host closed the connection)
<expedient> According to my findings, the leak can be seen in tcpdump running on the host, as direct communication to Google without going through any Tor node. Its a catastrophe by all means
<velope> a kernel patch is not necessarily appropriate, and upstream might well not consider it a bug
<velope> the iptables workaround i gave is in a followup post from mike in that thread
<velope> it would appear that "it needs to be fixed upstream" is just false
<velope> i only skimmed that forum page quickly, and i really don't want to wade through every bit of it
<velope> it contains my workaround based on --state INVALID
<expedient> That's fine we will apply your advice, but are you sure that there is nothing fundamentally wrong with the kernel or iptables though?
<velope> but it doesn't clearly seem to have been used
<velope> apply it and test, a lot
<velope> there is not yet any solid case made that there is something "fundamentally wrong"
<velope> using iptables to redirect for transproxying relies on connection tracking
<arma> i think the conclusion is "it is hard to use these things safely"
<arma> which isn't really a bug in upstream so much as a challenge with the approach
<velope> you have to test
<velope> people don't test thoroughly
<velope> they just read a man page or get some cookbook approach from some post, and run with it
<velope> that is why something like this goes unnoticed for so long
<velope> (gee, sounds like some other notable problems ...)
<expedient> Sorry to throw another link your way, but so far this is what we do when testing:
<expedient> https://www.whonix.org/wiki/Dev/Leak_Tests#FIN_ACK_.2F_RST_ACK_-_Leak_Test
<expedient> Any advice if this is enough or anything we should add to ensure its safe after a workaround?
* quiliro has quit (Quit: Leaving.)
<velope> do violent tests with your networking connection
<expedient> You are very sharp btw. That bug was a great find
<velope> interrupt it and restore it at various moments, for various periods
* blumenkraft (~eristisk@a1.networktest.34sp.com) has joined #tor
<velope> well actually i never announced or reported the problem, so even though i fixed the problem for myself a couple years ago, mike appropriately gets credit for finding it. i just get credit for a better workaround.
* amiller has quit (Remote host closed the connection)
<velope> y'all need to understand that connection tracking was never intended to be more than a loose fit around categories of traffic, not some leakproof box
<expedient> do you know if this applies to nftables too?
<expedient> Its gradually becoming the successor to iptables and was added in 3.13
<velope> when the kernel's info on a connection is lost or dropped (timeout), it's gone forever, so subsequent traffic won't be redirected
<velope> i haven't used it and don't know. unless there's been some massively radical re-architecting of connection tracking, it's unlikely to be totally different, though there could be minor differences in behavior.
<arma> the tails approach is starting to look a bit smarter
* eristisk has quit (Ping timeout: 480 seconds)
<expedient> tails is robus but it has to secure an entire rich os stack to protect the user
<arma> approach to torifying the traffic i mean
<expedient> ok
<arma> i don't argue that wrapping stuff in vm's can make it harder for an attacker to break out
<velope> in that tails doesn't rely on transproxy redirection, yes it's better
<expedient> what specifics are you talking about for torifying maybe we can replicate that?
<arma> but torifying outgoing streams is harder to make safe than explicit proxy settings
* darkclown (~darkclown@p2011-ipbf7306marunouchi.tokyo.ocn.ne.jp) has joined #tor
* anong0 has quit (Ping timeout: 480 seconds)
<velope> right, we've always warned that transproxying is not as great as it seems
<arma> "set your proxy settings to use tor, only allow tor to use the network"
* Gentlecat has quit (Ping timeout: 480 seconds)
<arma> so, implicitly, "don't allow anything that isn't configured to use tor to use the network"
<velope> and most people just ignore the warnings about transproxying, because "just stuffing everything through tor" seems just what they want, and so simple. but not really.
* mdik is now known as Guest6988
* mdik (~mdik@brln-4d0c41d4.pool.mediaWays.net) has joined #tor
<expedient>  the problem is when a box is rooted the software that is configured to use Tor can do things it shouldn't do
<expedient> but I get your point
* Guest6988 has quit (Ping timeout: 480 seconds)
<expedient> its those people who think stuffing flash through a transproxy pipe think their safe
<velope> that problem only exists for you because you insist on running tor in isolation
* huseby has quit (Ping timeout: 480 seconds)
<arma> "be sure to send unexpected traffic over tor so it can't hurt us"
<arma> vs "be sure to drop unexpected traffic so it can't hurt us"
<arma> option b sure seems wiser
<arma> then a bad guy who breaks in can, at worst, generate traffic that looks expected so it goes through tor
<sd> Useability wise, just having unique circuit per PID might be reasonable compromise.
* TheCthulhu1 has quit (Remote host closed the connection)
<arma> i think that's a totally separate topic
<expedient> arma:  an important goal we are trying to achieve is for operators of hidden services to be able to have a fighting chance in case they are targeted.
<sd> Trying to educate users "don't throw * at tor, but selectively firewall everything" seems like a losing battle though.
<arma> it does? screw those users then.
<arma> better to write software that works well
<arma> and the ones who want to use the safe software will use it
<velope> right, it can only be won or lost if you consider it to be a battle. users screw themselves, every day
<arma> and we can't do much for the ones who choose to do it wrong
<velope> a better analogy might be leading a horse to water
* Rym has quit (Ping timeout: 480 seconds)
<sd> That sounds like somewhat elitist stance. "You dont deserve privacy because you're dumb" vs "Safe defaults built in."
<velope> obviously better defaults are better, if they actually exist and are actually safer
<sd> (I can see how it can go downhill from there if users are babysitted too much, though)
<velope> but defaults never cover everything, because software is flexible, and people want it to be
<expedient> The less decisions that need to be made in security, the better everyone will be
<expedient> and hence why transproxy is an idea worth working towards
<arma> so give them only the applications that you know behave well, with those applications configured to use tor correctly
<velope> that principle is correct but the conclusion about transproxying is wrong
<arma> letting them pick their own apps and hope they are safe is a recipe for surprises again and again
<expedient> we do, but we can't guarantee that nothing dangerous happens when these apps are compromised unless it *all* goes through Tor
<sd> Ultimately the problem with transproxy is just that it's difficult to identify the source app.
* sd uses assigned source port ranges, but it is far from ideal
<expedient> Kgpg, TorBrowser and xchat come by default
<velope> even if it all goes through tor, you can't guarantee that nothing dangerous happens, you only have considered the first layer of potential problems
<expedient> updated system and AppArmor are other layers we have
<velope> that's all very nice and good
<expedient> but against a well funded adversary you must throuw everything in the toolbox and the toolbox itself to protect users
<velope> actually it's just more and more desperate band-aiding
<velope> the effort would be much better directed towards thoroughly auditing apps
<expedient> Auditing firefox is out of our league
<expedient> but we have verifiable builds for Whonix
<sd> Velope, that goal is far from realistic when we're talking huge blobs of code such as web browser or libpurple.
<velope> it's the true longterm challenge that people get scared away from all the time
* mttp_ (~mttp@9YYAAK2WG.tor-irc.dnsbl.oftc.net) has joined #tor
<velope> and the "community" pays for it dearly with things like heartbleed

Footnotes[edit]

  1. https://unix.stackexchange.com/questions/13771/dd-if-dev-random-is-randomly-bottlenecked-with-big-time-lags-but-i-have-no-i
  2. https://www.redhat.com/archives/libvir-list/2014-August/msg01010.html
  3. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/chap-KSM.html
  4. http://libvirt.org/formatdomain.html#elementsMemoryBacking
  5. https://www.suse.com/documentation/sles11/book_kvm/data/kvm_qemu_ksm.html
  6. https://staff.aist.go.jp/k.suzaki/EuroSec2011-suzaki.pdf
  7. https://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory#Detailed_Description
  8. https://bugzilla.redhat.com/show_bug.cgi?id=515741
  9. http://www.linux-kvm.org/wiki/images/1/19/2012-forum-memory-mgmt.pdf
  10. https://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory#Benefit_to_Fedora
  11. http://www.linux-kvm.org/page/Projects/auto-ballooning

Random News:

We are looking for maintainers (developers).


Impressum | Datenschutz | Haftungsausschluss

https | (forcing) onion
Share: Twitter | Facebook | Google+
This is a wiki. Want to improve this page? Help welcome, volunteer contributions are happily considered! See Conditions for Contributions to Whonix, then Edit! IP addresses are scrubbed, but editing over Tor is recommended. Edits are held for moderation. Whonix (g+) is a licensee of the Open Invention Network. Unless otherwise noted above, content of this page is copyrighted and licensed under the same Free (as in speech) license as Whonix itself.