| View previous topic :: View next topic |
| Author |
Message |
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
 Posted: Fri May 08, 2020 2:20 pm Post subject: QEMU/KVM unable to boot amd64 guests on amd64 host |
 |
|
Hey,
I'm struggling with my QEMU/KVM setup. The Gentoo system in question is more than 10 years old and was continuously updated, this could very well be an issue of me having messed up something at some point. Hardware-wise I'm running on an Intel Core2Quad CPU.
My main issue is that QEMU with enabled KVM works fine with 32 bit guests, but is unable to boot anything that is built for 64 bits. I simplified the problem to simply invoking
| Code: | | qemu-system-x86_64 --enable-kvm -m 384 -boot d -cdrom debian-10.3.0-amd64-netinst.iso |
This will start the VM, the boot menu will appear, but booting any 64bit linux kernel like that will simply reboot the virtual machine without any error message. I verified that by manually invoking the kernel from the GRUB2 shell:
| Code: | linux /boot/.../vmlinuz
boot
|
will just lead to an immediate restart of the VM.
Playing around with various QEMU settings such as "-cpu host" didn't change that, however as soon as I use that switch to simulate a 32 bit CPU, e.g. "-cpu pentium3", any attempt to boot a linux kernel will hang with a message that I need a 64 bit CPU. Which leads me to believe that my regular VM is a 64bit one and something else is wrong here. Creating the VM without KVM support (without --enable-kvm) "just works" with all my 64 bit ISOs. Moreover, with enabled KVM, 32 bit guests work correctly as well. I tried various ISOs for different operating systems, but none of those 64 bit images work: Windows bluescreens with "IRQL_NOT_LESS_OR_EQUAL", Solaris just reboots without any errors.
I'm currently running QEMU 4.2.0 and kernel 4.19.82 with KVM support:
| Code: | CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m |
PS: 64bit guests in Virtualbox run fine with VT-x enabled.
Any help is greatly appreciated! |
|
| Back to top |
|
 |
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Fri May 08, 2020 3:51 pm Post subject: |
|
|
Hi sylence,
welcome to the Gentoo forums!
Is virtualization enabled in the BIOS? It is/was disabled by default, at least on older machines.
Mike |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23087
|
| Posted: Sat May 09, 2020 12:22 am Post subject: |
|
|
| Are you sure that the 64-bit guest you are trying to run is actually compatible with your real CPU? When using software emulation without KVM, qemu can emulate instructions that you do not have. When using KVM acceleration, I am not sure if qemu will fall back to emulating instructions that are not natively supported by the CPU. |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 1:03 pm Post subject: |
|
|
Hey,
| mike155 wrote: | | Is virtualization enabled in the BIOS? It is/was disabled by default, at least on older machines. |
Yes, I just verified that - it's enabled. If I understood the documentation correctly, KVM would complain in case it's disabled.
| Hu wrote: | | Are you sure that the 64-bit guest you are trying to run is actually compatible with your real CPU? When using software emulation without KVM, qemu can emulate instructions that you do not have. When using KVM acceleration, I am not sure if qemu will fall back to emulating instructions that are not natively supported by the CPU. |
Well, the ISOs I'm trying to boot are mostly just regular x86_64 Linux distributions such as the Debian installer or the grml live CD. The 64bit version of Windows 7 doesn't work as well (bluescreen - see my initial post). All of those should run fine on my Core2Quad CPU. The crashes and reboots happen with '-cpu host' (passthrough) as well as '-cpu qemu64' or any other 64 bit option in the built-in CPU list. However, a 32bit ISO boots correctly with any 32 bit CPU (such as '-cpu qemu32'), even with KVM enabled.
I'm really out of ideas here.  |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54827
Location: 56N 3W
|
| Posted: Sun May 10, 2020 1:41 pm Post subject: |
|
|
sylence,
Do you have a 64 bit install on the host?
Most virtualisers will allow you to run 64 bit guests on 32 bit hosts, but not all of them.
What does the hosts show.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 1:52 pm Post subject: |
|
|
Yeah, the host's Gentoo installation is 64 bit:
| Code: | | Linux shade 4.19.82-gentoo.shade.v2 #2 SMP Thu Apr 9 12:49:07 CEST 2020 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux |
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54827
Location: 56N 3W
|
| Posted: Sun May 10, 2020 1:55 pm Post subject: |
|
|
sylence,
Do you have
| Code: | $ ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Dec 30 2017 /dev/kvm |
and is the user running qemu-system-x86_64 in the kvm group?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 2:17 pm Post subject: |
|
|
Looks fine (the user running qemu is 'sylence'):
| Code: | # ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Mai 10 15:19 /dev/kvm
# grep kvm /etc/group
kvm:x:78:qemu,sylence |
|
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54827
Location: 56N 3W
|
| Posted: Sun May 10, 2020 2:21 pm Post subject: |
|
|
sylence,
That's fine if sylence has logged out and back in again since being added to the kvm group.
If kvm is not listed in groups, when run as sylence, thats a problem.
| Code: | $ groups
tty wheel uucp audio cdrom video games kvm cdrw usb users vboxusers scanner wireshark plugdev roy |
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 2:54 pm Post subject: |
|
|
Looks correct as well.
| Code: | # groups
wheel cdrom video games kvm usb users libvirt docker sylence plugdev qemu vboxusers wireshark |
In the meantime, I removed the VirtualBox kernel modules I had installed as well, just to be sure: So far, same behaviour. |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Sun May 10, 2020 3:17 pm Post subject: |
|
|
Looks good so far...
- Please post the output of
Please post your kernel config using wgetpaste.
|
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Sun May 10, 2020 4:14 pm Post subject: |
|
|
Thanks for the data. I couldn't find anything wrong in the output of 'emerge --info' and your kernel config.
You seem to build KVM as modules:
| Code: | CONFIG_KVM=m
CONFIG_KVM_INTEL=m |
Are all KVM modules loaded when you start QEMU/KVM? Have you tried to compile KVM directly into the kernel? |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 4:17 pm Post subject: |
|
|
If by all you mean all two of them (kvm and kvm_intel), then yes, those are loaded when I'm starting QEMU.
I haven't tried yet to build them into the kernel and can try to do that, but would be surprised if that changes anything.
Thanks for having a look at my configuration, though.  |
|
| Back to top |
|
|
alamahant
Advocate
Joined: 23 Mar 2019
Posts: 3949
|
| Posted: Sun May 10, 2020 4:54 pm Post subject: |
|
|
Do you have libvirt also installed?
Maybe if you had a full qemu/libvirt/virt-manager setup and created a guest graphically?
Sometimes trying to be smartly minimal in many things ends up inviting more problems than not..........
I have seen this in USE flags for example, the kernel .config flags and in many other places...
We strive for smart seek minimal setups(which is a fantastic thing),but some times we break things up like this.................
|
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 5:15 pm Post subject: |
|
|
I do have libvirt and virt-manager installed. That's where I noticed that something is wrong - my original goal was to set up an Android VM, which didn't work. Only then I noticed that I can't boot anything that's x86_64. The minimal QEMU command above is just my attempt to reduce the problem to something tangible. Sadly, the 'normal' way of managing KVM VMs doesn't work either.  |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54827
Location: 56N 3W
|
| Posted: Sun May 10, 2020 5:24 pm Post subject: |
|
|
sylence,
What error do you get from virtmanager?
I'll guess, none at all, the guests just reboot.
That points the finger at the host. I have an AMD and Intel system that are KVM hosts. I can share the kernel .configs if you want.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 5:33 pm Post subject: |
|
|
| NeddySeagoon wrote: | What error do you get from virtmanager?
I'll guess, none at all, the guests just reboot. |
Correct, no error anywhere. The guest just reboots (or bluescreens in the case of a Windows guest). I was playing around with QEMUs logging options ("-d") for invalid opcodes (amongst others), but didn't get anything from there either. I'm not sure if another kernel config would help. Guess I'll try to two things now: Building a 5.x kernel (maybe it's really just a very specific kernel bug) and booting some other distribution to see whether KVM works from there. Could as well be a hardware issue with the VT-X extension. |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Sun May 10, 2020 6:26 pm Post subject: |
|
|
You could also try a BIOS update. I found some reports from users with a Core 2 CPU who reported that virtualization worked for them only after a BIOS update.
12 years ago, virtualization was quite new and mainboard and BIOS manufactures had to puzzle out how to use it... |
|
| Back to top |
|
|
ununu
n00b
Joined: 19 Apr 2020
Posts: 31
|
| Posted: Sun May 10, 2020 7:59 pm Post subject: too little ram |
|
|
greets,
this could be due to little RAM on your qemu options
I think 64 bit distros needs a little bit more due to KASLR or something like that.
Also make sure that qemu is built with:
| Code: | QEMU_SOFTMMU_TARGETS="i386 x86_64"
QEMU_USER_TARGETS="i386 x86_64" |
I forgot to add this:
or
| sylence wrote: | Hey,
...
system is more than 10 years old ... running on an Intel Core2Quad CPU. |
:cough:AMD:cough: <--- |
|
| Back to top |
|
|
sylence
n00b
Joined: 08 May 2020
Posts: 10
|
| Posted: Sun May 10, 2020 9:46 pm Post subject: |
|
|
Problem solved, even though I'm still not sure what exactly the problem was.
I tried various things, all of them without success:
* BIOS is up to date (I used VT-x a lot previously with VirtualBox, under Windows etc., so that should work)
* Not enough RAM: The problem persisted also with 2GB or 4GB of RAM (-m 2048, -m 4096)
* My QEMU_*_TARGETS are both set to just "x86_64", which should give me at the very minimum support for 64 bit
* Different "-cpu" settings didn't help
I verified that it's not a hardware problem by booting another USB live linux (also with kernel 4.19) and running QEMU/KVM from there (same command, same ISO) - this worked fine. As a last resort I finally upgraded my kernel to 5.4.38. I built that one with the exact same config as I used before for 4.19.x (make olddefconfig). Now it works fine, no idea what was wrong with the old one. I also tried to integrate KVM into the kernel or build it as a set of modules - no difference.
Well, thanks for all the suggestions! |
|
| Back to top |
|
|
|
Kernel & Hardware
