| View previous topic :: View next topic |
| Author |
Message |
kybber
Apprentice
Joined: 25 Jun 2002
Posts: 228
Location: Trondheim, Norway
|
 Posted: Wed Jan 25, 2006 1:37 am Post subject: [IGNORE] Why do I have to choose between NUMA and eth2/3? |
 |
|
Edit: Don't spend time reading this. I was just confusing myself by assuming that two blades were identical, when it later appeared that the BIOS settings
were different. 
Hi,
I have a HP BL 25 server with two Opteron 250s. This server has four network cards, which I can see if I boot the livecd:
| Code: | livecd root # lspci |grep Eth
0000:02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
0000:02:02.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
0000:05:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet (rev 10)
0000:05:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703 Gigabit Ethernet (rev 10)
|
However, when I booted my own kernel, I could only see the first two. Also missing from the lspci output, compared to that of the livecd, were a couple of
PCI bridges, namely these:
| Code: | livecd root # lspci |grep 04\:
0000:04:09.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:04:09.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:04:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:04:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
|
So basically everything with PCI address 04 and above was missing with my own kernel. After some googling, browsing of this forum and comparing the
livecd .config (/proc/config.gz) with my own, I decided to enable CONFIG_PCI_MMCONFIG.
In order to do this, I had to enable ACPI first. So with the only differences between the old kernel and the new being CONFIG_PCI_MMCONFIG=y
and CONFIG_ACPI=y, I rebooted. However, now there is suddenly no NUMA, according to dmesg:
| Code: | Scanning NUMA topology in Northbridge 24
Number of nodes 2
Node 0 using interleaving mode 1/0
No NUMA configuration found
Faking a node at 0000000000000000-00000000f57fa000
Bootmem setup node 0 0000000000000000-00000000f57fa000
On node 0 totalpages: 1005465
DMA zone: 3999 pages, LIFO batch:1
Normal zone: 1001466 pages, LIFO batch:31
HighMem zone: 0 pages, LIFO batch:1
|
However, the livecd doesn't have this problem:
| Code: | Scanning NUMA topology in Northbridge 24
Number of nodes 2
Node 0 MemBase 0000000000000000 Limit 000000007fffffff
Node 1 MemBase 0000000080000000 Limit 00000000f57f6000
Using node hash shift of 24
Bootmem setup node 0 0000000000000000-000000007fffffff
Bootmem setup node 1 0000000080000000-00000000f57f6000
On node 0 totalpages: 524287
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 520191 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
On node 1 totalpages: 481270
DMA zone: 0 pages, LIFO batch:1
Normal zone: 481270 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
|
The liveCD uses kernel 2.6.11-gentoo-r3-k8 #1 SMP Sun Mar 20 21:39:10 UTC 2005 x86_64 while my own kernel is 2.6.14-gentoo-r5 #3 SMP PREEMPT
Wed Jan 25 03:55:41 CET 2006 x86_64.
What is the reason why I all of a sudden am unable to use NUMA, and how can I combine having four Ethernet cards with having NUMA enabled just like
the livecd kernel does? There has to be a kernel setting somewhere, and I would much appreciate being pointed in the right direction.
Please let me know if there's any more info you need to consider this problem.
Last edited by kybber on Mon Jan 30, 2006 11:37 am; edited 1 time in total |
|
| Back to top |
|
 |
kybber
Apprentice
Joined: 25 Jun 2002
Posts: 228
Location: Trondheim, Norway
|
| Posted: Wed Jan 25, 2006 3:15 am Post subject: |
|
|
Hmm... seems like I have been confusing myself. I was actually comparing two blades that were supposed to be identical, but it now seems like maybe
they aren't after all. The one blade constantly gives me
| Code: | # dmesg | grep NUMA
Scanning NUMA topology in Northbridge 24
No NUMA configuration found
|
While the other one shows NUMA as being on. This is regardless of any ACPI settings. So I've switched on ACPI now and am able to use all four network
cards.
So my next problem is figuring out why NUMA is disabled on one blade which should have identical hardware as the other blades which do have NUMA.
Could this be a BIOS setting? I've noticed the following difference between the dmesg's from the non-NUMA blade and a NUMA-enabled blade:
non-NUMA Blade:
| Code: | # dmesg | grep aperture
Checking aperture...
CPU 0: aperture @ 8000000 size 32 MB |
NUMA-enabled Blade
| Code: | # dmesg|grep aper
Checking aperture...
CPU 0: aperture @ 4000000 size 32 MB |
Can anyone comment on this difference?
Also, is there any way in which I can find out more about the hardwre installed in the blades so that I can compare them without having to actually open
them? I am specifically interested in checking which memory banks are full, to see it that may be why there's no NUMA on one blade. Any other suggestions?
Edit: I have found out that the one blade that was having NUMA trouble had NUMA disabled in BIOS
Sorry for your time. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
