Virtualization questions [SOLVED!!!!!!!!]

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

I've been running VMWare Server for awhile now, and unfortunately it's few small bugs have annoyed me to the point that I'm contemplating making a switch to KVM (and I'm open to other suggestions). The problem is I have a few specific requirements. If KVM can't meet them, I won't be able to swap over. So without further ado, here's what I need:

1) Direct access to disks. Basically, one of my VM's will be a Windows-based fileserver that needs to be able to mount drives and treat them as it's own. No filesystem within a filesystem type stuff. I need to maintain the ability to physically disconnect the drives from the host, plug them into a non-Virtualized window's box and have it recognized 100% without the need to install anything extra. This applies to USB drives as well.

2) No X. I need to be able to accomplish everything with a combination of VNC with SSH (for the Linux VM's) and RDP (for the Windows ones). This includes installing. If I understand it correctly, I can use VNC to do the install. I'm not sure how, but I'm hoping it's obvious and easy to figure out.

3) It needs to run on an Intel i5. So VT but not VT-d.

4) Bridged networking (wired).

5) It needs to run on 64-bit Gentoo host.

6) It needs to support Server 2008 as a guest.

From everything I've read so far, most of this is a no-brainer. Server 08 is supported in Xen and KVM, they both run on 64-bit Gentoo, the both have no issues with wired bridged networks. Also, I believe they both run without any issues without VT-d as long as VT is available. I believe I can use VNC to install Windows guests without having X on the host with KVM, I'm not sure how/if this can be done with Xen. Direct access to disks I couldn't find information on. And that would be a deal breaker. It is absolutely imperative that I be able to shut down the host, remove the drives, plug them into any other windows machine and have them work.

To clarify that further... These are separate drives in the Guest as well. For example, I have my host and all guests installed on 2 mirrored 500GB drives. Those drives would NOT be required to work if transplanted to a windows-only machine. I do have some 1TB internal SATA II drives, a couple of 250GB IDE drives, and a few external USB drives that were partitioned in a Windows-only system and are formatted with NTFS. Those "data" drives contain no operating system information, just data. But they need to work in the VM as if they were native windows drives, directly connected to that machine's hardware.

All this works in VMware Server for me. The reasons I would like to switch:

1) Performance is less than stellar.

2) The web interface is buggy. VERY buggy. To the point sometimes it just won't load (blank page titled "Loading...") which Google finds thousands of pages of people complaining about without any working solutions. Sometimes it works with Firefox, sometimes with IE, sometimes with either one, sometimes with neither, sometimes it won't work for a week then suddenly work as if nothing were wrong, etc. It is the biggest pain, usually because when I want to do something it doesn't work. I have no problem SSH'ing in to the host to configure stuff via command line, but VMWare Server 2 CLI support is less than ideal as well. At first I thought, "Wow, web interface! Awesome!" After a few hours of that, you realize it may look impressive to the crowd who love GUI's and amaze all your friend, but isn't really any more powerful or useable than a CLI...especially with it only works maybe 20% of the time.

So, I'm hoping someone can either tell me how to get KVM or Xen to do what I want. Paravirtualized would be best, obviously. I like fast. Some sites I've found says KVM supports paravirtualization, others say no. I'm hoping it's a question of it didn't used to but now it does so some sites are just out of date. I dunno. But I'm sure someone here does, and I'd appreciate their input.

And as I said, if there's another option, or if there's a reason I should look at Xen or KVM in your opinion, I'm all ears. Thanks for your time.

jormartr · Apprentice Joined: 02 Jan 2008 Posts: 174

I have not used 2008, but I am trying KVM on a hosted machine.

I can attach the output of the console machine to the terminal, that way I can operate directly into the vm console with ssh at the host. This is posible only if the output of the vm console is text, if it gets into graphics (example: windows, linux framebuffer or X) you see a beauty text saying that you are into VGA mode, and that's all.

I have a windows installed (not 2008), and network configuration is done with VNC, after that, I just use RDP (with the posibility to acces via VNC to the vm console, any problem can hit...).

There are para-virtualized drivers (i am trying net and disk now).

One think I like about KVM is the posibility to work directly with vm console through ssh. It also gives you the posibility to attach with the machine running disks and netword cards (at least I tried that on linux, dunno how that works on windows).

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

drescherjm · Posted: Mon Nov 23, 2009 12:02 am Post subject:

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

I'm waiting for a new power supply to be delivered, once that's done I'll let you know what my experience with KVM is like.

drescherjm · Posted: Mon Nov 23, 2009 1:13 am Post subject:

Thanks. I would be very interested in that.
BTW, have you tried the version update to vmware-server-2.0.2.203138. I have not tried that version yet.
_________________
John

My gentoo overlay
Instructons for overlay

born2chill · n00b Joined: 14 Aug 2006 Posts: 10

hi,

you might want to look into ESXi, its quite performant and allows direct access to disc iirc. Configurable via a windows-client or (with a little hack) or via ssh. It's free too. Downside: you can only run ESXi on the box, nothing else.

http://www.vmware.com/products/esxi
http://professionalvmware.com/2009/05/unsupported-console-and-ssh-on-esxi-4

br,
b2c

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

Dagger · Retired Dev Joined: 11 Jun 2003 Posts: 765 Location: UK

Personally I believe KVM is a way ahead. On my server I've got 23VMs running (from Fedora, Gentoo, BSD to Windows7 and Windows 2008R2). I moved my configuration from XEN to KVM/libvirt (3) and I really love it (1). Problem with XEN is - it's not in mainline, so if you need features from newer kernels, you're either screwed or use unofficial patches.

Configuration is easily handled by virt-manager (4) (or virsh from console) - you can use username/password or x509 certs for auth. libvirt allows you to use whatever you want as a storage backend (from file image - qcow2/iso/raw/lots lots others - to block device /dev/sdaxx, LVM2, iSCSI, NFS, Fiber - you name it - it's there). The only annoying thing for me was bridging - it was bridging all virtual interfaces (and not physical) and forwarded traffic to network via physical interface. I had to manually adjust it to bridge all interfaces - (physical and virtual - 2) to avoid extra routing. libvirt also supports other hypervisors backends - so if you want to play with XEN/VB/KVM - you just need to flip the switch and you're ready. VERY powerful tool.

Performance wise - everything is better then VMware server. It might be free, but when I reached 10VMs it was choking and CPU usage was over 40% when idle. From my personal experience, XEN and KVM work pretty much the same.

So from my personal experience (and I've been playing with virtualisation for the last three years), stick with libvirt.

1) 14:57:11 up 113 days, 23:01, 2 users, load average: 0.20, 0.31, 0.23
2)

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

Thanks for all the replies folks. Is there some way to start KVM VM's at boot in a specific order like VMWare had? I'm installing my VM's as we speak, headless through VNC, and it seems a little snappier so far, but we'll see how it works once they're up and running with RDP.

Also, is it possible to have more than 4 hard drives loaded with KVM? I noticed the qemu documentation says to use the flags "-hda -hdb -hdc -hdd", but says nothing about whether or not that can be carried on with -hde, hdf, etc as needed....

EDIT: Just tried it...no luck though. Also, I couldn't seem to mount a file as scsi. So...I'm not quite sure what else I can do. I have 5 drives that one of my VM's is supposed to access. I suppose I COULD, in theory, RAID0 3 of them that hold data and mount them as 1 drive, but that'd be less than ideal.

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

Ok, so...KVM is blazing fast. It's like lightning. It almost feels like I'm running on bare metal, and I'm not even using the paravirtualized drivers. I've got direct disk access, and I work entirely with VNC/RDP/SSH so I don't miss the lack of a GUI at all. If anyone is considering trying KVM, GO FOR IT! It is way better than vmware-server. Networking was pretty straightforward as well, which is what people seem to normally complain about.

I'm still having issues mounting my SATA drives though. I need to mount more than 3 on one machine, which it won't allow (it cave have 4, but the OS image counts as 1 leaving 3). For now I'm mounting the extra drives to a different VM and them sharing them out to the one that needs them, but that's less than ideal.

I did try using them as scsi devices (created a disk .img file and mounted it) but qemu ended up not showing them as drives and I kept getting IO error messages on the host. Is SCSI broken, does it only work with real SCSI disks, or do I need some strange SCSI support included in my hosts kernel to do it?

drescherjm · Posted: Sat Dec 05, 2009 4:39 pm Post subject:

Anon-E-moose · Posted: Sat Dec 05, 2009 6:10 pm Post subject:

drescherjm · Posted: Sat Dec 05, 2009 6:14 pm Post subject:

I should stated my reason for the question. SMP with windows guests in virtualbox has been problematic (slow at times with unusually high system load + random crashes) at best. I was wondering if it was the same under KVM.
_________________
John

My gentoo overlay
Instructons for overlay

Anon-E-moose · Posted: Sat Dec 05, 2009 6:24 pm Post subject:

drescherjm · Posted: Sun Dec 06, 2009 6:12 am Post subject:

I tried KVM and it does appear significantly faster than vmware and also virtualbox. I say appear because I have not been able to convert enough real data to kvm to fully test. Got to figure out how to convert .vdi and .vmdk files to kvm. Also I did give KVM an advantage in that I used a raw partition instead of a large fixed disk file on top of the ext4 fs.
_________________
John

My gentoo overlay
Instructons for overlay

Anon-E-moose · Posted: Sun Dec 06, 2009 11:58 am Post subject:

# These lines convert a VirtualBox VDI disk image (disk.vdi) to a KVM disk image (disk.qcow).
# You need VirtualBox and KVM/QEmu installed on your system.

VBoxManage clonehd -format RAW disk.vdi disk.img
qemu-img convert -f raw disk.img -O qcow2 disk.qcow

---

If I remember correctly the qemu-img program should be able to convert .vmdk files directly to qcow2
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

Ok, more problems.

Two USB hard drives I use for backups have the same vendor:device ID... so QEMU won't pass them to the guests if they're both connected.

So I can't connect all my hard drives, and I can't get at my backups.

As fast as KVM may be, it seems that it doesn't have the ability to give me access to all my storage requirements. :-(

drescherjm · Posted: Sun Dec 06, 2009 5:35 pm Post subject:

Thanks about the .vdi to .raw.

Anon-E-moose · Posted: Sun Dec 06, 2009 6:22 pm Post subject:

Instead of host:vendor_id:product_id you should be able to use host:bus.addr which should be unique for each device.

I am assuming that the command line takes multiple usbdevice paramaters.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

drescherjm · Posted: Mon Dec 07, 2009 2:28 pm Post subject:

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

dtjohnst · Apprentice Joined: 23 Apr 2006 Posts: 178

Ok, for the benefit of anyone who may come searching for answers, I'm going to explain how I made it work. For those who wanted a comparison, I set up a Xen system and KVM performed about the same once I had the paravirt drivers working. I didn't run specific benchmarks though, that was just from eyeballing it. Personally, I'm going to stick with KVM. The fact that it's in the kernel and allows me to run my beloved Gentoo underneath are enough reasons for me to side with it. Also, the network was trickier to set up if I was going to use virsh. I did get it working, but it was much trickier if I was using xml files for virsh instead of just passing parameters to qemu-system-x86_64.

I tried it using libvirt and virsh and I tried it with my own scripts. Personally, I like running it with my own scripts better. virsh was better at monitoring and it allows hot-swapping some hardware with certain guests. As the only guests I'd want to swap with are Server 2008R2 guests and hot-swapping doesn't work with them, that wasn't enough to keep me around. If you don't like scripting it would probably be easier to stick with virsh. The only thing I have left to do is to write a daemon. Once I do I'll see if there's enough interest and post my source and scripts.

As for hardware limitations: KVM will allow up to 4 IDE devices. Assuming you leave the drive image the OS is installed on as an IDE device, you're left with 3. If you ever want to mount a cd-rom (iso or direct access to /dev/cdrom) you're left with 2. That was a rather strict limitation. I had come up with 3 plans:

1) Find a way to mount all drives on my fileserver as they were with VMware.
2) Mount my 3 main storage drives on my fileserver, mount 2 less vital data drives to my Domain Controller and share them out then mount them on my fileserver, and lastly mount my USB backup drives on my e-mail server and share them out then mount them on my fileserver as well
3) Abandon KVM.

I don't quit easily so 3 was out. I determined 2 was my last resort. So I tried EVERYTHING I could think of. If I mounted my USB hard drives as direct-access usb devices using the DEVICEID:VENDORID trick, I could only use one at a time. I WAS able to mount them using the bus and device # method, but only using virsh. However, even getting them into the OS that way left me without any way to install "mass storage device" drivers. Apparently the virtualizing of the hardware bugs that up somewhere and it doesn't work on later 64-bit Windows OSes. So that option was out.

I also tried using if=scsi when I couldn't get the virtio block device drivers to work. The problem with that is that LSI never released a 64-bit driver, not any driver for that matter later then Windows 2000. In Linux that driver is accessible in the kernel, but in Windows you're apparently just hooped.

So I had to go back to virtio block device. The driver from linux-kvm.org didn't work. It caused my machines to crash and become unrepairable. I did download the exact same version of the drivers in an ISO from linux-kvm.com though, and those drivers I was able to load. I couldn't load them in test mode though, that still caused my machines to lock up, fail to boot and be unrepairable. If I disabled driver signing checking at boot (press F

, I was able to install them. Downside: I have to press F8 everytime I reboot the machine. However, now that the drivers are working, I'm going to use the Windows Logo Kit to sign my own drivers and that should be that. Also, it won't work if I use a disk img file. That also causes my Server 2008R2 to lock up. It only works if I pass the device's root node, so /dev/sdc for example.

So what I ended up doing was mounting OS image as IDE and then mounting my 3 main storage drives, my 2 supplementary storage drives and my 2 external USB hard drives as virtio SCSI block devices. And now everything works. Thanks to everyone who helped out.