[solved] gentoo-sources-2.6.34-r2 and xen pci devices.

[Arrta]

I decided to use the gentoo-sources-2.6.34-r2 kernel on a domU.
I got the DomU working, kernel loads, but lspci shows none of the PCI devices that I passed through.
I rechecked my kernel config and verified I enabled all the XEN devices. That is when I discovered that there is no CONFIG_XEN_PCIDEV_FRONTEND option anywhere in the gentoo-sources kernel.
According to the Xen docs, "Latest Linux kernels (2.6.27 and newer) are good for domU use." So I am not sure why it is missing.
Is the PCI frontend not in the linux base? or did it get renamed and I'm just not finding the ducumentation?

Arrta

[idella4]

[Arrta]

I tried the xen-sources but they get stuck after

[idella4]

You just need too adjust your xen kernel config.
It's not too hard. Your kernel is mounting root, so it's very close. You have most important things right already.

This has been trodden well before you. Read closely this post and this other post and they should get you there. Note the other links in those posts, they have the info.

If not, I can supply you with a config that will probably be a prefect fit, tomorrow
_________________
idella4@aus

[Arrta]

If I can get the PCI devices to show to the gentoo-source-2.6.43-r2 that would work for me.
Otherwise I'm not sure what to do to fix the xen-source-2.6.34 unless it has to do with me not having pv-grub and a grub menu.list for the domU?

Edit: Seems not, pv-grub has something to do with booting off the local disk. I have a separate partition for the domU

Edit2: http://wiki.xensource.com/xenwiki/XenPCIpassthrough tells me that getting gentoo-sources to see the pci devices is not possible because "upstream kernel.org Linux kernel doesn't have Xen pcifront driver yet." So I have to use xen-sources at this time.
For the life of me I can't figure it out atm. My genkernel built gentoo-sources works, but the hand built xen-sources hangs at that step, genkernel won't work on xen.

Edit3: Not sure what changed, but genkernel is working for me with the xen-sources now. Previously it would bomb at the "Compiling 2.6.34-xen bzImage..." line. We will see how this turns out in a few mins once it finishes.

Edit4: Ok that failed miserably. Turns out I wasn't building off the config I loaded and genkernel picked its own without any options. I did however find out why my config is failing. There seems to be something in my config that turns off CONFIG_HAVE_KERNEL_BZIP2=y so I'll have to recreate the kernel config again and try to find it. That is unless you can get me that config you think will work.

Edit5: Last one, going to bed. Turns out enabling Xen-compatible disables the bzip option which causes genkernel to fail. I read in one of the posts to disable Xen-compatible for guests, but if I do that it also disables the PCI frontend. I thought setting Xen-compatible and "CONFIG_XEN_PRIVILEGED_GUEST is not set" would work, as when "CONFIG_XEN_PRIVILEGED_GUEST is not set" then "CONFIG_XEN_UNPRIVILEGED_GUEST=y", which in my opinion would be a domU.
There has to be something else I am missing, I'll take another look in the morning. Otherwise I think I will need a more obvious hint than those threads and the links inside of them. Maybe just getting me that config would work.

[idella4]

Arrta;

ok, let's get the environment clear.

[Arrta]

Gentoo is my host system, x86_64
My cpuinfo I got a core2quad:

[Arrta]

Here are the options for the working gentoo-sources just to see what I have.
They are spread out all over the place so here is a grep of the .config instead.

[idella4]

you are very close. well you can now move on and try what I've pointed out.
Uncheck Xen compatable.
Check paravirt guest.
Enable its content.
correct your bzip2
re-compile
repost.

What about grub-install /dev/md...

I can get your environment now. I think you have enough correct tips, you now piece them together and execute,
and if you're really stuck, I'll give you the effective config which you just need to modify to your hardware
_________________
idella4@aus

[Arrta]

[idella4]

ok, let's ensure

[Arrta]

I just compiled a xen-sources kernel enabling both frontend and backend. Previously I had compiled 2 separate kernels and only enabled the backend drivers on the dom0(host) kernel and the frontend drivers on the domU(guest) kernel. Am I misunderstanding the backend/frontend concept?

[idella4]

Arrta

you have the concept quite right. We're making headway.

[Arrta]

Ok, I missed the window at 11:00am EST. Had the contractor that is going to remodel our bathroom show up unexpectedly.
I have about hour before I have to switch the system back to a working state. Then at 6 EST I'm good for the rest of the night for testing.

At this point I'm not sure if you are telling me I need pygrub or pv-grub. I havent been able to find either of them having been installed with xen-tools even thou I reinstalled tools with the pygrub flag. I did read somewhere that pygrub is outdated and to use pv-grub, but all information I found says it is a part of xen-tools, which leads back to where is it, as xen-tools is not installing it.

Edit: Ok, well that bombed. Modified grub/menu.lst to boot from the new xen host, and they system is not comming back up. I'm working until 8 tonight so I won't be able to see what went wrong until at least 8:15ish when I get home. Depends on how many calls come in tonight.
The only things I changed were adding the frontend devices as modules and changed the PCI to Passthrough from Virtual PCI, I completely spaced on that setting.

[idella4]

Arrta

[idella4]

..
_________________
idella4@aus

[Arrta]

So that was dunb, I enabled RAID0 and not RAID1 on my rebuilt kernel so it hung not being able to mount the md devices.
Fixed that, but now we have taken a step backwards. These are on the guest.
Not sure why you thought this was solved..

[idella4]

Arrta,
That message was a slip, i made it for another thread and entered it into yours instead.

[Arrta]

I am able to boot the gentoo source built using genkernel in paravirt.

The genkernel randrive didnt help at all.

Building the frontend drivers into the kernel brought me back to the original issue.

Removing PCI Frontend had no effect on the "hang" other than making it happen slightly faster because the kernel did not detect and load the drivers for the 3 passed pci devices.

Going to test paravirt on the xen-sources with drivers built in and see what happens. I expect to be able to load up fine but without the passthrough items. Which will place me in the same situation as when I boot from the gentoo-sources paravirt kernel.

[idella4]

Just how are you passing the devices to the guest? It seems to me they need be added in your
root='/dev/xvda ro console=tty0 pass-device-arg'

from man xm

[Arrta]

ok the paravirt xen-source failed at the same point as the non paravirt versions. "init-early.sh used greatest stack depth: 4024 bytes left"

This is how you pass PCI devices to the guest.
First find the addresses of the devices you want to pass

[idella4]

Arrta

ok, I can follow that. You're having a tough time with it. That xen overlay appears to be like my xen-source, it looks the same. Anyway,
-your kernel building should be effective. I suppose I'll give you the kernel config.
-Your system has no reason not to compile the xen tools. I've got it working here, so probably fix that. There's only a couple of points to have in place. There is another approach to consider which is probably too long winded.

catch you tomorrow.
_________________
idella4@aus

[Arrta]

haha, haha, haha,... *slams head on desk*

As you can tell I didn't goto bed. I took a look at the boot consol on the dom0, and it too got to the "init-early.sh used greatest stack depth:" but right after that it loads OpenRC.
I remembered that my raid script complained about not being run on baselayout 2 so I emerged that and that I had to add OpenRC to my package.keywords on the dom0.
Since I copied all the configs over, it was i nthe guide I read, the same things got carried over. once I went removed openrc, and switched back to baselayout 1 for the domU and I got a different "hang" point,

[idella4]

Arrta,
well this really is becoming a protracted essay. I didn't go to bed either, was following your progress.
good on ya mate the well expected breakthrough.

once I copied your setting of

That one important point is highlighted in the debian docs that you were never exposed to tackling it gentoo style. I was missing that for a little while. So that's good news, about time hey. I helped overcome at least one important hurdle.
Your components of raid baselayout requirements are are different to mine, so I couldn't point that out, you needed to sort that yourself. Next is to tweak that kernel. To clarify, does the bootup at this point complete to a login? I've never cottoned on to using ssh which is heavily cited in using xen.
_________________
idella4@aus

[Arrta]

No I do not get a login. xenconsole lets me watch the boot until "starting local" and then it just sits there. At that point I can ssh in to the system.

And WOOT!!!