4gb RAM and 64bit Ethernet Problem

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

I'm having a slight problem with my new computer, I've also posted this issue on the forums for my motherboard because I'm not sure if its a bios issue of if its a gentoo issue.

I'm running 4gb of ram with a AMD 64bit X2 Processor on a DFI Motherboard. In order to use any and all memory over 4gb there is a setting in the DFI bios called "Memory Hole Remapping" I believe and with this turned on the hardware and bios do indeed see all 4+ gb of ram. I'm also using 64bit gentoo (of course) and I'm on kernel 2.6.17-gentoo-r7. Everything works fine without the Memory HOle remap Bios option activated, and as expected, then both Gentoo and the bios only see 3.25gb of ram or so.

The problem is that with that bios option activated, neither of the ethernet ports on my board work anymore. I believe it has something to do with hardware addresses but I'm not sure if the problem is with the bios setting the hardware addresses or with gentoo reading those addresses? And it seems like that is the only difference. It sees all the ram with that option activated just the ethernet ports don't work. In fact everything at boot comes up like it normally does and it looks like the ethernet settings are set right and they startup at boot correctly, they just don't work. Instead I get some kind of loop (or thats what it looks like based on the ethernet monitor in gkrellm2. The motherboard is a DFI LanpartyUT-SLI and has both an nvidia and a marvell lan port on it, of which I normally use the marvell.

I know that all the Ram is good as I've run some benchmarks and even tested booting the computer with each 1gb module of ram separately and all seem to be read accurately. The ram is Corsair TwinX dual channel, two pairs of 1gb modules for a total of 4 modules of 1 gb each. They're set up on opposite channels and come up as reading 128bit so I'm pretty sure its not a memory problem.

As I said, I've also posted this problem on the DFI forum as I'm not entirely sure where the problem lies. According to their other posts about 4gb on their forum, activating the "Memory Hole Remap" option clears up the problem of not seeing all 4gb of ram, which it does. I can't seem to find anything on their forums as to problems occuring after that point soI started wondering if it was a problem specifically with Gentoo.

This machine is a dual boot machine with windows XP, but my XP version is only 32bit so it encounters other problems with the Memory Holr Remap Option enabled. My XP side also isn't set up to go on the internet out of paranoia so I can't even test to see if the problem occurs there as well.

I appreciate any advice on this topic. My first impression was it had something to do with bios settings etc, but I thought I would see what some other people running gentoo might think. I'd be happy to at least know for sure if its a bios problem or a gentoo problem, at least then I'd know what forum to concentrate on and where the problem actually occurs.

Thanks,
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

rsa4046 · Posted: Wed Nov 01, 2006 3:54 am Post subject:

Do you have IOMMU support enabled in the kernel?

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Um, I'm not exactly sure, where is IOMMU in the kernel? And can you give a little explanation about what it does?

Thanks I appreciate the help rsa4046,
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Ok, wait, I did a quick look through my kernel and I have GART_IOMMU enabled. If there are other IOMMU options I haven't found them yet. I seem to remember turning that option on when I first built this machine thinking that I would need it, (at least according to the description).

Thanks again, sorry for the double post.

Looks like back to square one?
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

rsa4046 · Posted: Wed Nov 01, 2006 6:30 pm Post subject:

I can't check the other posts right now, but IIRC this has been discussed elsewhere, e.g.
Check https://forums.gentoo.org/viewtopic-t-441770-highlight-iommu.html

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

thanks rsa4046, that forum kind of does confirm some of what I'm thinking is going on (I have a vague understanding that could be entirely incorrect) and it prompted me to do a little test.

I booted my machine with Memory Hole enabled in my bios, this does what they talk about in the forum you sent me, it lets the bios remap the memory to above 4 gb with a 64bit processor and according to the bios it does it correctly because 4.15 gb or so is found. When I go into gentoo, like I said my ethernet doesn't work, but I checked in kinfocenter really quick and see that it find 3.85 gb of ram. I then rebooted and turned off the Memory hole option in the bios and the ethernet comes back up and kinfocenter only finds 3.15 gb of ram.

Now, according to my calculations this actually makes sense but doesn't solve my problem. I've read some explanations on other forums and they talk about hardware addresses being assigned to address above the Ram range of addesses. So if you have 2gb of ram, then hardward addresses take up addresses that would be used by additional ram beyond 2 gb. Because I have 4 gb, and the system can only handle 4 gb, those hardware addresses are located on addresses used by ram, so that part of the ram cannot be seen.

If we assume that 0.3gb of ram is used for hardware then the numbers work out right for this theory. When I boot without Memory Hole, the system finds 3.45 gb and kinfocenter finds 3.15 gb. When I boot with Memory Hole, the system finds 4.15 gb and kinfocenter finds 3.85 gb both are exactly 0.3gb less.

Now I don't know if thats how this actually works or not, I'm not a hardware tech at all so I cdon't know whats going on at that level, but I'm just trying to understand what other sites have said (I wish I could remember the exact site).

Either way this still doesn't solve my problem because I still cannot access the ethernet with Memory Hole enabled. I thought originally it was because the kernel cannot access memory above 4gb like they talk about in the forum you sent but I'm on the 64bit kernel and high memory options don't exist so shouldn't it do it automatically?

I think I'll post on that other forum you sent too me and see if they can help.

Thanks again
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

chalex20 · n00b Joined: 27 Sep 2006 Posts: 27

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Thanks chalex, how do I pass that ioomu=force to the kernel?

I'm going to check right now on whether or not I can enlarge the AGP apeture or the IOMMU in my bios. I don't seem to recall seeing that, but this is DFI overclocking bios so there are tons of options.
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Sorry for the typos in the last message.

I checked my bios and I don't see anything about AGP aperture or Iommu size. I'm going to read through /usr/src/linux/Documentation/x86_64/boot-options.txt. and see if maybe if that solves the problem.

Thanks for the help, I really appreciate it.
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

chalex20 · n00b Joined: 27 Sep 2006 Posts: 27

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Yeah, my boad is PCI and PCI-e only so that makes sense.

I'm actually using lilo is there a way to pass kernel options in lilo? apparently I can't just call out iommu options as then I get a lilo error.

I guess I can switch to grub if necessary it just always seemed like so much more of a hassle than lilo to set up.

Ok, I think we may have pinpointed at least part of the problem. When running "dmesg | grep -i iommu" the only results are
"PCI-DMA: Disabling IOMMU."

So now I'm really confused. I have IOMMU enabled in the kernel but PCI-DMA is disabling it? is this some sort of conflict in kernel options or something else entirely?

Thanks again. I believe we're making some progress.
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

chalex20 · n00b Joined: 27 Sep 2006 Posts: 27

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

As far as I can tell thats the only IOMMU entry. I'm going to try what you've said about lilo and see if that comes up with anything else.

Thanks again, hopefully that will work. If it doesn't I'll check whats loading related to IOMMU again and post that and maybe that will shed some light on this problem.
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Success!

I started with the additions to lilo that you laid out chalex and that seemed to clear up the issue. Now let me know if I'm understanding this correctly.

In this line:
[code]append="iommu=memaper=3,noagp,force"[/code]

the iommu=memaper=3 maps the iommu space to 256 ram. When I run dmesg and grep for iommu after adding that I get a line that says
PCI-DMA: Reserving 256MB of IOMMU area in the AGP aperture

Now, this board is PCI and PCI-E only I shouldn't need an AGP aperture right? So I tried it as:

[code]append="iommu=noagp,force"[/code]

and I get a line that says:
PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

So I'm assuming that the 64mb is the default size for the aperture. This setting works too so I'm curious. Is there anyway to set the AGP aperture to 0? And if I'm using the noagp option why is it setting an AGP aperture at all? Maybe I'm jost not understand exactly what is happening.

Thanks for the help, this seems to have fixed the problem with the ethernet but I still have those few questions.
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com

chalex20 · n00b Joined: 27 Sep 2006 Posts: 27

dank_waldmb · n00b Joined: 31 Oct 2006 Posts: 22

Thanks Chalex,

I kind of figured what I was thinking wasn't right and what you're describing makes sense. I have actually been running some benchmarks just out of and have notices some change with the smaller AGP aperture so I think I'll stil with the 256 value for the time being.

Thanks for all the help It probably would have taken me months to figure out what you've eplained in a few days, I appreciate it.
_________________
Take care,

D.A.N.

Owner/Editor - Sights & Sounds from the Fifth Column
The Fifth Column Online Magazine - www.fifthcolumnmagazine.com

A division of Fifth Column Media -- Freelance Graphic/Web/Multimedia Design
www.fifthcolumnmedia.com