2.6.19/20 performance problems + RAID issues [SOLVED]

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

I've been happy with 2.6.17 for ages, but a hardware change has forced me into upgrading to 2.6.19. The upgrade hasn't gone well.

I have two potentially related issues.

1) 2.6.19 (and 2.6.20) are really slow to boot. Boot time has gone from 75 seconds with 2.6.17 to over 12 minutes with 2.6.19. Once booted, everything seems to work but some things still take too long (e.g. running init scripts take 10-15 seconds each, minimum). Other timings for 2.6.19 are:

Boot to 'INIT: version 2.86 booting' - 15 seconds (that's about the same as 2.6.17).
Boot to 'INIT: entering runlevel: 3' - 5 minutes (and it then takes 2 minutes to get any further).

2) The killer - the kernel simply refuses to accept my Adaptec 2400A RAID controller. It'll find it OK, try to access the devices, fail, and then hang soon after. It all works flawlessly on 2.6.17, but neither 2.6.19 nor 2.6.20 will even boot with the drivers present. This hardware is an I2O device and I'm using the native I2O (CONFIG_I2O) support.

Hardware spec: Athlon XP1700+ (Palomino core), PC Chips motherboard circa 2001 (SiS chipset), 1GB RAM.

emerge --info (from 2.6.17): here
lspci (from 2.6.17): here
2.6.17 dmesg: here
2.6.19 dmesg (without I2O): here
2.6.19 boot messages (with I2O - from I2O startup): here

In the last case, I copied from the console by hand, so please excuse any typos. I left it at the last message for one hour to make sure it wasn't just being slow.

If you need any additional information, let me know. Despite my newbie status here I'm a Gentoo user of over three years standing (and this is the first problem I've hit that I've not been able to solve from a search on these forums or with the aid of Google). I am, however, much more confident in userspace than hacking around inside the kernel.

Dan · Veteran Joined: 25 Oct 2005 Posts: 1302

udev issues? Check udev version and is there an upgrade? did you etc-update or dispatch-conf after...
make sure you atleast have cat /usr/src/linux/.config|grep I2O
# CONFIG_SCSI_DPT_I2O is not set
# I2O device support
CONFIG_I2O=y
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_CONFIG=m
CONFIG_I2O_CONFIG_OLD_IOCTL=y
#CONFIG_I2O_BUS is not set
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=y
I bet its udev that slowing you down...
_________________
- Failure is not an option. It's bundled with your software.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

That's a possibility. I'd been doing a udev update about the same time, but I've done with that now (and it's working with 2.6.17). I'm in the process of an 'emerge -uD world', so I'll try again to make sure when that's done.

As for the kernel config, mine is similar (I don't use modules unless I have to):

# CONFIG_SCSI_DPT_I2O is not set
# I2O device support
CONFIG_I2O=y
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_CONFIG=y
CONFIG_I2O_CONFIG_OLD_IOCTL=y
# CONFIG_I2O_BUS is not set
CONFIG_I2O_BLOCK=y
# CONFIG_I2O_SCSI is not set
CONFIG_I2O_PROC=y

I'll post the full .config if needed, but I'll need to eliminate some things first to avoid confusion (they're there to aid transition to a new motherboard if I have to but don't affect the booting issues).

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

Well, I checked out /etc/udev and it looks OK and up-to-date, but everything I know about udev can probably be written on the back of a postage stamp, so I may have missed something.

Any other ideas please?

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

I decided to try going to 2.6.18-gentoo-r6.

I have my RAID controller back on this version, so at least that seems to have eliminated the hardware issues, and I have the most important drivers I need for a future hardware upgrade.

However, anything involving init (i.e. boot, shutdown, running anything in /etc/init.d) is still horribly slow.

Don't think it's anything to do with udev. I'm fully patched and etc-updated, and I've never had any need to change udev rules, so they're all at defaults.

Any more suggestions for either problem, or where I can take this elsewhere to get suggestions, as I'm not having much luck so far here?

Thanks.

msutton · n00b Joined: 20 Jan 2005 Posts: 56

I have the same issue with my raid controller adaptec 2015s. Kernel 2.6.18.x works went to 2.6.19-r5 and it hangs on boot with the i2o driver.
System is fast on boot though no complaints.

Hardware Specs:
# Tyan S2881 Motherboard
# 4 gigs of RAM
# 2 x AMD Opteron Processor 248
# Adaptec 2015s Scsi Raid Card
# 2 x 70 gig scsi drives

Dan · Veteran Joined: 25 Oct 2005 Posts: 1302

Tried changing versions of sys-apps/sysvinit && dispatch-conf ?
Available versions: 2.86-r5 2.86-r6 ~2.86-r7
_________________
- Failure is not an option. It's bundled with your software.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

OK, well some good news on the I2O front (if you want to call it that). I've started to see reports of I2O being broken in 2.6.19 popping up on other distros as well, so I'll take it as a sure sign that I2O support is fubar'd in 2.6.19.

That just leaves the slow system. I thought it was just init, but it isn't. I currently have a bzip2 process that's been running for nearly 3 hours and has produced 17MB of output (~6MB/hour). I know bzip2 is generally slow, but it's not supposed to be that slow (compared to 2.6.17, it's about 10-20x slower). It's using about 90% CPU too, so it's not as if something else is getting in the way.

It's not disk-related - getting normal speeds with various dd read/write tests on both the IDE and RAIDed disks, but other than that, I've not been able to pin it down and don't really know where to start.

Any help appreciated.

msutton · n00b Joined: 20 Jan 2005 Posts: 56

I have had problems intermittently with some kernels. If I set the Timer Frequency under Processor type and features down to 250 or 100 it fixes the problem. When my machine was slow I would get 500+ ms pings on the LAN and boot up took forever.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

OK - checked those. Timer frequency is set to 250 HZ, pings are currently averaging 0.18ms.

RazielFMX · l33t Joined: 23 Apr 2005 Posts: 835 Location: NY, USA

You might also want to check your io/scheduler settings. There are three of them, and if things are taking as long as you say due to kernel issues, then changing this might see you some performance increase. I can't remember which one I use as I'm booted into Windows right now, but it works great for me. I know it is not 'Deadline' and it is not "Anticipatory". It is the one that says 'recommented for desktop use'. I also have my timer frequency cranked pretty high, since if you are using a desktop environment, you need it. This is on both my systems, which are running 2.6.19-gentoo-r5 without any issues (well, once I found where they moved my SATA drivers to anyway).
_________________
I am not anti-systemd; I am pro-choice. If being the latter makes you feel that I am the former, then so be it.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

The other one is CFQ. I'll try that, and also setting the timer to 1000 HZ, when I get home.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

Well, I changed the following

Scheduler: Pre-emptive (low latency desktop) to Voluntary (desktop)
IO Scheduler: Anticipatory to CFQ
HZ: 250 to 1000

Seems to have helped. It's still slower than it should be, but at least a cronjob that compresses a file with bzip2 every 24 hours doesn't take 36 hours to run (with the obvious consequence that would have had).

Thanks for the help so far. I would of course welcome any further advice.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

Eventually nailed it down. With:

# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set

Performance is slow, but if I change it back to:

CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set

Then everything works as it should.

I'd turned on high memory support after getting this warning:

Warning only 896MB will be used.
Use a HIGHMEM enabled kernel.

I'm about to replace the motherboard on this system, and will be adding another 1GB RAM at the same time. Here's hoping I can actually use that new RAM.

This gives rise to a few questions:

1) Are there some other setting(s) I need to change to make a kernel work with high memory support.
2) Is there a boot option to disable high memory support on kernels with it enabled?

RazielFMX · l33t Joined: 23 Apr 2005 Posts: 835 Location: NY, USA

In answer to #1, I recently placed an additional GB of RAM into my PC, making my total 2GB, and could not understand why it only saw 1GB. I realized I needed to change my kernel, so I enabled the 4GB mode and took no performance it. It was the only change I made.

I was under the impression that enabling HIGHMEM had no impact on performance, only the kernel size. I guess I was wrong.
_________________
I am not anti-systemd; I am pro-choice. If being the latter makes you feel that I am the former, then so be it.

Mark Gray · n00b Joined: 14 Feb 2007 Posts: 14 Location: Berkshire, UK

It's supposed to only have a very small performance hit (reported as unnoticeable).

I also read something about problems with onboard graphics, which I have, and it's also a rather old motherboard (close to five years old), so I can only assume the combination finished it off.

Having done some further research, I can now safely mark this as [SOLVED]. Thanks for all the help.