Author Topic: [Help Welcome] KVM Support - staying the course  (Read 11937 times)

zweeble

  • Jr. Member
  • **
  • Posts: 51
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #75 on: May 06, 2014, 07:42:20 pm »
I really don't want to be nasty, but I mentioned before that this +100 GB image doesn't make sense anyway^^
Where is the problem to decrease workstation to let's say 10 GB and gateway to 4 GB from the beginning? I'd prefer to download 2 huge files I can use instead of 2 small ones I can't use...
« Last Edit: May 06, 2014, 07:48:54 pm by zweeble »
Freedom is not worth having if it does not include the freedom to make mistakes.

Patrick

  • a maintainer of Whonix
  • Administrator
  • *****
  • Posts: 3108
  • (adrelanos)
    • View Profile
    • Patrick Schleizer – Profile Page
Re: [Help Welcome] KVM Support - staying the course
« Reply #76 on: May 06, 2014, 08:21:32 pm »
I really don't want to be nasty, but I mentioned before that this +100 GB image doesn't make sense anyway^^
Where is the problem to decrease workstation to let's say 10 GB and gateway to 4 GB from the beginning? I'd prefer to download 2 huge files I can use instead of 2 small ones I can't use...
I don't find this question nasty/cynic or anything. I value your constructive comments. You have a valid point here.

Shipping a workstation with 10 GB max space would make users wanting to exceed that limit complain. They could grow the virtual hdd size (https://www.whonix.org/wiki/Grow_Virtual_Harddisk) but that is rather cumbersome and complicated due to missing easy GUI access to the required functions.

Operating system's / rsync / browser issue is:
They don't support sparse files well.

gzip's issue is:
It doesn't support sparse files. (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=535987) And I should not have taken gzip for that reason in the first place. (Took gzip, because it can produce deterministic archives.)

KVM's issue here is:
KVM has no alternative to VirtualBox's .ova feature. If they had one, we would not have to have this discussion. That's why we have to compress them.

The new compression method will be:

Code: [Select]
tar \
   --create \
   --sparse \
   --xz \
   --mtime="2014-05-06 00:00:00" \
   --directory="$WHONIX_BINARY" \
   --file "$WHONIX_BINARY/Whonix-Gateway-$version.qcow2.xz" "Whonix-Gateway-$version.qcow2"

Which produces a deterministic sparse archive.

You can then unpack using:

Code: [Select]
tar xvf Whonix-Gateway-8.2.qcow2.xz

(`unxz` won't work!)

And will end up with a spare image. Apparent size 100 GB, can grow up to 100 GB space, but will initially take no more than ~2GB space after extraction. I will re-compress, sign and upload soon.

You could still say you preferred non-compressed smaller 10 GB images that must be grown to take more than 10 GB? You would still have a point. In an ideal Whonix distro world, we would offer both types of images. In an ideal world, other projects would solve their issues with sparse files. [Best would still be, if KVM added an alternative to VirtualBox's ova feature.]
« Last Edit: May 07, 2014, 02:33:09 am by Patrick »
HOT: How to Ask Smart Questions | How to Report Bugs Effectively
Impressum | Datenschutzerklärung | Haftungsausschluss
If Whonix (g+) is useful to you, please consider a reoccurring donation so I (e-mail) (gpg) (g+) can work full time on Whonix.
Need more attention? Get Professional Support!

zweeble

  • Jr. Member
  • **
  • Posts: 51
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #77 on: May 07, 2014, 10:03:22 am »
Well, maybe we should ask around who really uses up to 10 GB with whonix. I want to point at the fact that using KVM opens the way for using LVM, that would improve performance considerably and makes resizing the images a piece of cake. So these imo very few users that might complain about a small 10 GB image are no subject anymore, plus using sparse files is not needed any more (also sparse files present a performance handicap).
If I didn't make a mistake, I can only add advantages:
- no trouble with the compressing process
- much smaller but easy resizeable images
- two possibilities to improve performance
Freedom is not worth having if it does not include the freedom to make mistakes.

Patrick

  • a maintainer of Whonix
  • Administrator
  • *****
  • Posts: 3108
  • (adrelanos)
    • View Profile
    • Patrick Schleizer – Profile Page
Re: [Help Welcome] KVM Support - staying the course
« Reply #78 on: May 07, 2014, 01:35:05 pm »
The thing I dislike about KVM is, that VirtualBox has no such issues with max disk size (which is set to 100 GB but could be increased to anything) and really using only used disk space. (Well, VirtualBox has other issues, such as only supporting vmdk images for exported ova's, which make things harder later as well.)

Well, maybe we should ask around who really uses up to 10 GB with whonix.
Want to write a blog post? (In this style?)

Quote
I want to point at the fact that using KVM opens the way for using LVM,
Since I am not that knowledgeable about KVM, I don't know what difference KVM makes in relation to LVM compared to VirtualBox. Please explain.

Quote
that would improve performance
KVM or LVM would improve performance? KVM has worse graphics performance, even when using SPICE. I am not sure we discussed this here or in the lost old forum. LVM slightly worsens performance (but so slightly, that no one minds)?

Quote
considerably and makes resizing the images a piece of cake.
How so?

When not using LVM, I can grow my file system using gparted. When using LVM, this isn't possible. Are there finally any GUI tools supporting growing lvm file systems?

Quote
(also sparse files present a performance handicap).
After quick research, I haven't found any references for having a performance penalty for using sparse files.

Also we're using qcow2 images with metadata preallocation. According to this blog post (That HulaHoop shared.) and earlier discussions with HulaHoop, qcow2 images using metadata preallocation are a good choice, size wise, performance wise and feature wise (support snapshots).

Quote
If I didn't make a mistake, I can only add advantages:
- no trouble with the compressing process
Without compression, upload would (100 kb/s upload) would take me 2-3 days for 20 GB. Still bearable if this is the best solution, but I like uploading the ~1GB images more.
HOT: How to Ask Smart Questions | How to Report Bugs Effectively
Impressum | Datenschutzerklärung | Haftungsausschluss
If Whonix (g+) is useful to you, please consider a reoccurring donation so I (e-mail) (gpg) (g+) can work full time on Whonix.
Need more attention? Get Professional Support!

zweeble

  • Jr. Member
  • **
  • Posts: 51
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #79 on: May 07, 2014, 03:45:21 pm »
Maybe my misunderstanding is how you build whonix?? I thought that the KVM version is build in a qemu/KVM VM using qcow2 or raw LV...

No need to make performance tests here: image based VMs (no matter if OVA using sparse files or qcow2) are slower than VMs that use a LVM raw partition. Working on files in an image file for sure is at least more disk IO.
LVM itself is the GUI to manage LVs, also to resize them. But there is a also a simple command in the terminal.

And qcow2 images still can be compressed with gzip, so no worries about huge uploads. Watch out, a gzipped qcow2 image will be about 10% smaller than the native qcow2 zlib compression :)
Freedom is not worth having if it does not include the freedom to make mistakes.

Patrick

  • a maintainer of Whonix
  • Administrator
  • *****
  • Posts: 3108
  • (adrelanos)
    • View Profile
    • Patrick Schleizer – Profile Page
Re: [Help Welcome] KVM Support - staying the course
« Reply #80 on: May 07, 2014, 04:50:14 pm »
Maybe my misunderstanding is how you build whonix??
For VM images (both ova's and qcow's):
- Using Whonix's build script (https://github.com/Whonix/Whonix/tree/Whonix8)
- that uses grml-debootstrap (https://github.com/Whonix/Whonix/blob/Whonix8/build-steps.d/1300_create-debian-img)
- that creates a bootable (grub) raw image without lvm [which is a difficult task, especially making it bootable (https://github.com/grml/grml-debootstrap/blob/master/grml-debootstrap)]
- later converted to qcow2 (https://github.com/Whonix/Whonix/blob/Whonix8/build-steps.d/2400_convert-img-to-qcow2)
- later compressed (https://github.com/Whonix/whonix-developer-meta-files/blob/master/release/compress_qcow2)

Physical Isolation:
- https://www.whonix.org/wiki/Dev/Build_Documentation/Physical_Isolation

Quote
I thought that the KVM version is build in a qemu/KVM VM using qcow2 or raw LV...
- Redistributed Whonix VM images are build without starting any VMs.
- Using qemu-img, mount, chroot and so forth.
- Everything is scripted and automated.
- No manual interventions for image creation.

Quote
No need to make performance tests here: image based VMs (no matter if OVA using sparse files or qcow2) are slower than VMs that use a LVM raw partition.
Quote
Working on files in an image file for sure is at least more disk IO.

Ah, you meant raw LVM partitions in your previous post? I was talking about VM images using LVM. I could get convinced, that we should use LVM for VM images. That discussion however, should be unrelated to this KVM thread. But I guess you didn't raise that point.

KVM LVM partitions with Whonix are theoretically possible. Perhaps they would have even better performance. No one is working it it at the moment. The problem is, how to redistribute KVM LVM partitions with Whonix? It would make the setup more host operating system specific. More instructions would be required. A new option for Whonix's build script would be required. Instructions would look a bit similar to build instructions for physical isolation (https://www.whonix.org/wiki/Dev/Build_Documentation/Physical_Isolation), I think.

What we're working on here are VM images, we agreed to use qcow2 images for advantages stated in previous post. It is much simpler to redistribute them. The plan roughly is: 1. download qcow2.xz 2. extract 3. use our kvm instructions [or in future hopefully, run our kvm setup script] 4. done.

Creating a KVM partition however looks much more scary (data loss), difficult for the user (instructions) and technically difficult to implement to me. But if anyone wants to work on this, by all means, go ahead.

Quote
LVM itself is the GUI to manage LVs, also to resize them. But there is a also a simple command in the terminal.
What's the GUI package for resizing?

Quote
And qcow2 images still can be compressed with gzip, so no worries about huge uploads. Watch out, a gzipped qcow2 image will be about 10% smaller than the native qcow2 zlib compression :)
So if you don't complain about compression. Great. I guess you will like the new qcow2 images. They're already briefly tested by me (that they do boot up, 100 GB apparent size, ~2GB really used space) and uploaded. Just hoping all mirrors updates already. Please try.

http://mirror.whonix.de/8.2/Whonix-Gateway-8.2.qcow2.xz
http://mirror.whonix.de/8.2/Whonix-Gateway-8.2.qcow2.xz.asc

http://mirror.whonix.de/8.2/Whonix-Workstation-8.2.qcow2.xz
http://mirror.whonix.de/8.2/Whonix-Workstation-8.2.qcow2.xz.asc

Code: [Select]
tar xvf Whonix-Gateway-8.2.qcow2.xz

(`unxz` won't work!)

gpg (date/time must match for this release):

Code: [Select]
gpg --verify Whonix-Gateway-8.2.qcow2.xz.asc
gpg: Signature made Wed 07 May 2014 02:41:41 AM CEST using RSA key ID 77BB3C48
gpg: Good signature from "Patrick Schleizer <adrelanos@riseup.net>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 916B 8D99 C38E AF5E 8ADC  7A2A 8D66 066A 2EEA CCDA
     Subkey fingerprint: 6E97 9B28 A6F3 7C43 BE30  AFA1 CB8D 50BB 77BB 3C48

Code: [Select]
gpg --verify Whonix-Workstation-8.2.qcow2.xz.asc
gpg: Signature made Wed 07 May 2014 02:41:56 AM CEST using RSA key ID 77BB3C48
gpg: Good signature from "Patrick Schleizer <adrelanos@riseup.net>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 916B 8D99 C38E AF5E 8ADC  7A2A 8D66 066A 2EEA CCDA
     Subkey fingerprint: 6E97 9B28 A6F3 7C43 BE30  AFA1 CB8D 50BB 77BB 3C48

sha512:

Code: [Select]
5266eaacb3446f366ce644fe4788858aa4f3dbb073bec9b347f7adffeb969aa5bfe8e08fd073c32389c3345cb9a4e5c62ff5c206871d27740b20016b41012659  Whonix-Gateway-8.2.qcow2.xz
feec2956fe78a7ad7d47c48f39faac469bb634600d03f550d4f0ba76ff0edf483d9b96a54e845af271c07170d81bed022dc9788a6a10e602456106a71038d9d0  Whonix-Gateway-8.2.qcow2.xz.asc

Code: [Select]
2e84f51d1f905b28227e8b2df1114e0ea6f3f021a374866ab36d8ae8fab8d0f9bce0c84f7f804bbee33d030c5c555a87a1d3320d860030dc27489cf7be18022e  Whonix-Workstation-8.2.qcow2.xz
fe6cec1e5858aa61f3b013aee9650b0297858a7eb4141cfa6d257af47a508c08e1de840098729da119279f241da6e2ea365e7181c66faea05bded3bcf3f28bc6  Whonix-Workstation-8.2.qcow2.xz.asc
HOT: How to Ask Smart Questions | How to Report Bugs Effectively
Impressum | Datenschutzerklärung | Haftungsausschluss
If Whonix (g+) is useful to you, please consider a reoccurring donation so I (e-mail) (gpg) (g+) can work full time on Whonix.
Need more attention? Get Professional Support!

zweeble

  • Jr. Member
  • **
  • Posts: 51
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #81 on: May 07, 2014, 06:43:49 pm »
ok, now I see... ;) thanks for your answer-package^^
I will test the new images and also go through your build instructions.
Let's see if we find more answers and solutions :)
Freedom is not worth having if it does not include the freedom to make mistakes.

zweeble

  • Jr. Member
  • **
  • Posts: 51
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #82 on: May 09, 2014, 07:24:08 pm »
Well, this is embarrasing. I cannot get the new qcow images running, in the best case I get a frozen start screen with many funny colourful acsii codes... The systems used for testing were KVM on Ubuntu and openQRM/KVM on Debian Wheezy.
Maybe I pass by in a couple of months again, I simply cannot afford to spend more time on this any more.
Freedom is not worth having if it does not include the freedom to make mistakes.

ii

  • Newbie
  • *
  • Posts: 6
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #83 on: May 22, 2014, 06:39:01 pm »
hi, this is more of a curiosity question. I was not able to resolve the "resolution problem" that some seem to have with virtbox, that is achieving a higher resolution tried here and here and for now am just dealing. After seeing about the KVM work i wanted to ask if it looked like this ongoing common issue with resolutions would be an issue with KVM like it is in virtbox? I'd be curious to hear anyones experiences.

zweeble

  • Jr. Member
  • **
  • Posts: 51
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #84 on: May 28, 2014, 07:26:00 pm »
Just thinking about another approach:
How about following the "Install Whonix-Gateway/Workstation on hardware" guides to build the gateway and workstation on 2 KVM guests? Was this tried already or am I missing something?
Freedom is not worth having if it does not include the freedom to make mistakes.

Patrick

  • a maintainer of Whonix
  • Administrator
  • *****
  • Posts: 3108
  • (adrelanos)
    • View Profile
    • Patrick Schleizer – Profile Page
Re: [Help Welcome] KVM Support - staying the course
« Reply #85 on: May 28, 2014, 11:55:38 pm »
Just thinking about another approach:
How about following the "Install Whonix-Gateway/Workstation on hardware" guides to build the gateway and workstation on 2 KVM guests? Was this tried already or am I missing something?
Not required, since we're providing .qcow2 images. Missing would be the VM settings.

Either manual:
https://www.whonix.org/wiki/KVM#KVM_Setup_Instructions

Or hopefully some day automated:
https://www.whonix.org/wiki/Dev/KVM#Script.2Fautomate_creation_of_VM_description_files.3F

(I am still hoping, that HulaHoop contributes that part some day.)
   
What you are suggesting is certainly possible. We had a discussion some time ago about this:
Building Whonix with --bare-metal option in a VM
https://www.whonix.org/forum/index.php/topic,7.msg17.html#msg17

In summary, I for one, do not have time to maintain such a set of separate instructions. And I also find too many options too confusing. And I still don't see why "Building Whonix with --bare-metal option in a VM" has any advantage over current VM build instructions. The "Building Whonix with --bare-metal option in a VM" imho overcomplicates things, because you always have to argue "but don't forget the VM settings". Anyhow. If someone wants to maintain such instructions or such a feature, by all means, you're welcome.
HOT: How to Ask Smart Questions | How to Report Bugs Effectively
Impressum | Datenschutzerklärung | Haftungsausschluss
If Whonix (g+) is useful to you, please consider a reoccurring donation so I (e-mail) (gpg) (g+) can work full time on Whonix.
Need more attention? Get Professional Support!

HulaHoop

  • Hero Member
  • *****
  • Posts: 667
  • Maintainer of Whonix KVM
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #86 on: May 31, 2014, 03:13:02 am »
hi, this is more of a curiosity question. I was not able to resolve the "resolution problem" that some seem to have with virtbox, that is achieving a higher resolution tried here and here and for now am just dealing. After seeing about the KVM work i wanted to ask if it looked like this ongoing common issue with resolutions would be an issue with KVM like it is in virtbox? I'd be curious to hear anyones experiences.

The resolution can be adjusted from the KDE screen settings dialogue and whatever higher resolution will automatically take effect and fill in the screen when using KVM. But this isn't recommended because you ill stick out from the group running Whonix I think because resolution is a fingerprintable attribute. There  may be safeguards against this in TorBorwser but I'm not knowledgeable enough to confirm this.
Understand your freedom.  Assert it. | Resources: Whonix KVM Wiki & Whonix KVM Support Forum

Nothing can Five Eyes yield from the Onion field.

HulaHoop

  • Hero Member
  • *****
  • Posts: 667
  • Maintainer of Whonix KVM
    • View Profile
Re: [Help Welcome] KVM Support - staying the course
« Reply #87 on: May 31, 2014, 03:17:42 am »
Just thinking about another approach:
How about following the "Install Whonix-Gateway/Workstation on hardware" guides to build the gateway and workstation on 2 KVM guests? Was this tried already or am I missing something?

Like Patrick aid this is not necessary and complicates. things in light o f there being qcow2 images available. The only thing remaining is the automation of a specific set of settings once we there is a consensus on what should be included.
Understand your freedom.  Assert it. | Resources: Whonix KVM Wiki & Whonix KVM Support Forum

Nothing can Five Eyes yield from the Onion field.

Patrick

  • a maintainer of Whonix
  • Administrator
  • *****
  • Posts: 3108
  • (adrelanos)
    • View Profile
    • Patrick Schleizer – Profile Page
Re: [Help Welcome] KVM Support - staying the course
« Reply #88 on: May 31, 2014, 12:52:41 pm »
hi, this is more of a curiosity question. I was not able to resolve the "resolution problem" that some seem to have with virtbox, that is achieving a higher resolution tried here and here and for now am just dealing. After seeing about the KVM work i wanted to ask if it looked like this ongoing common issue with resolutions would be an issue with KVM like it is in virtbox? I'd be curious to hear anyones experiences.

The resolution can be adjusted from the KDE screen settings dialogue and whatever higher resolution will automatically take effect and fill in the screen when using KVM. But this isn't recommended because you ill stick out from the group running Whonix I think because resolution is a fingerprintable attribute. There  may be safeguards against this in TorBorwser but I'm not knowledgeable enough to confirm this.
Not at time of writing. We can document the latest status of that topic here:
https://www.whonix.org/wiki/Tor_Browser#Maximizing_Browser_Window
HOT: How to Ask Smart Questions | How to Report Bugs Effectively
Impressum | Datenschutzerklärung | Haftungsausschluss
If Whonix (g+) is useful to you, please consider a reoccurring donation so I (e-mail) (gpg) (g+) can work full time on Whonix.
Need more attention? Get Professional Support!

Patrick

  • a maintainer of Whonix
  • Administrator
  • *****
  • Posts: 3108
  • (adrelanos)
    • View Profile
    • Patrick Schleizer – Profile Page
Re: [Help Welcome] KVM Support - staying the course
« Reply #89 on: June 01, 2014, 07:57:50 pm »
So there is no intention to make whonix a solution for hosting hidden services? Everything I tested was a disaster and posts about it are regarded as not necessary...
This is sad as this is throwing away a huge potential.
Moved and answered here:
https://www.whonix.org/forum/index.php/topic,328.0.html
HOT: How to Ask Smart Questions | How to Report Bugs Effectively
Impressum | Datenschutzerklärung | Haftungsausschluss
If Whonix (g+) is useful to you, please consider a reoccurring donation so I (e-mail) (gpg) (g+) can work full time on Whonix.
Need more attention? Get Professional Support!

 

Legal

Impressum Datenschutz Haftungsausschluss Contact

Links

Homepage Blog Issues Github

Misc

Contribute Donate Investors Free Support Professional Support