View previous topic :: View next topic |
Author |
Message |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Wed Sep 26, 2012 10:04 pm Post subject: trouble with new kernel compilation. [SOLVED] |
|
|
I can't believe I'm having to post this, but it has been so long since I compiled a new kernel, I actually need some help! I have run the same zen kernel since may of 2011, and it has served me well, however, it's time to upgrade as things are starting to get quirky. I hate to even ask for help, but I thought I might try out a newer gentoo-sources. I manually build and compile using pappy's seeds, but I am so rusty, I am having trouble figuring out what settings I need, so hopefully someone can point out what I've missed. I really appreciate any help I can get! the .config is here: http://pastebin.com/raw.php?i=p8JY3xSP . my lspci -n: http://pastebin.com/raw.php?i=3kBVcX3m .
Again, let me say thanks, I truly do appreciate the help I get here!
Todd
Last edited by todd93 on Mon Oct 01, 2012 9:38 pm; edited 1 time in total |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Thu Sep 27, 2012 5:41 am Post subject: |
|
|
http://www.gentoo.org/doc/en/xorg-config.xml reference for vga card; note that this cannot be modules, requires builtin firmware, kms needs these, if you have kms and an improper config the screen will hang during boot
do not use this section: Quote: | < > ATA/ATAPI/MFM/RLL support (DEPRECATED) ---> | your choices in Quote: | <*> Serial ATA and Parallel ATA drivers ---> | are all that you need along with your scsi choices the deprecated drivers may interfere _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Thu Sep 27, 2012 10:53 pm Post subject: |
|
|
DONAHUE wrote: | http://www.gentoo.org/doc/en/xorg-config.xml reference for vga card; note that this cannot be modules, requires builtin firmware, kms needs these, if you have kms and an improper config the screen will hang during boot
do not use this section: Quote: | < > ATA/ATAPI/MFM/RLL support (DEPRECATED) ---> | your choices in Quote: | <*> Serial ATA and Parallel ATA drivers ---> | are all that you need along with your scsi choices the deprecated drivers may interfere |
Okay, I get that, and have fixed it, but the kernel still won't boot, I continue to get the following error: "vfs cannot open root device sda3 or unknown-block 0 0". I have searched this issue out, and still can't find a solution that works for me. I do understand that the drivers must be built in and not modules, and they are. I think there is obviously more that I have missed here, and hopefully I can get a little more insight on what I need to do.
Thanks!
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Thu Sep 27, 2012 11:51 pm Post subject: |
|
|
pls run lspci -k and post results and post your current kernel config _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Fri Sep 28, 2012 9:35 am Post subject: |
|
|
DONAHUE wrote: | pls run lspci -k and post results and post your current kernel config |
sure, no problem! here is lspci -k: http://pastebin.com/raw.php?i=kJNNC6Ay . and here is my current .config: http://pastebin.com/raw.php?i=SEuNHGRE .
Once again, I would like to thank everyone for their kind help. I know I probably shouldn't have waited this long before compiling a new kernel, but things were so stable that I didn't want to upset any apple carts, so to speak, until I had to!
Thanks again!
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
toralf Developer
![Developer Developer](/images/ranks/rank-dev.gif)
![](images/avatars/10034462014367c4dfe33c1.gif)
Joined: 01 Feb 2004 Posts: 3943 Location: Hamburg
|
Posted: Fri Sep 28, 2012 9:50 am Post subject: |
|
|
Code: | "vfs cannot open root device sda3 or unknown-block 0 0" | Hhm, sounds that either a needed feature is n
built as a module rather than compiled in - or there's a missing option, usually either block device related or a missing file system. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Fri Sep 28, 2012 5:31 pm Post subject: |
|
|
I have taken your config and compared it against pappy's and mine then opened it with make oldconfig and then opened it with make menuconfig to add missing items. the only item i saw that could remotely be linked to your panic was config_ahci _platform=y. very unlikely. Other possibilities:
If you are using /dev/hda nomenclature in /etc/fstab and /boot/grub/grub.conf you should be using /dev/sda type names.
Are you sure /boot partition was mounted when you compiled and copied the new kernel?
to get the new (hopefully improved) kernel:
boot cd, mount gentoo partitions, enter the chroot,
Code: | cd /usr/src/linux
mv .config .configold
make clean
make mrproper
wget -o /usr/src/linux/.config http://bpaste.net/raw/48156/
ls -al /usr/src/linux/.config | verify you have nice new .config Code: | make oldconfig
make menuconfig | look around; change something; change it back; exit saving the config.
Code: | make && make modules_install && make install | if you prefer manual copying of the kernel to make install, the choice is yours, in either case ensure the new kernel name is exactly the same in /etc/boot and /boot/grub/grub.conf
if the panic recurs a picture of the screen posted on a free photo site might be helpful
http://swift.siphos.be/linux_sea/ch07.html#idp35213232 good discussion of boot panics _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Fri Sep 28, 2012 11:15 pm Post subject: |
|
|
DONAHUE wrote: | I have taken your config and compared it against pappy's and mine then opened it with make oldconfig and then opened it with make menuconfig to add missing items. the only item i saw that could remotely be linked to your panic was config_ahci _platform=y. very unlikely. Other possibilities:
If you are using /dev/hda nomenclature in /etc/fstab and /boot/grub/grub.conf you should be using /dev/sda type names.
Are you sure /boot partition was mounted when you compiled and copied the new kernel?
to get the new (hopefully improved) kernel:
boot cd, mount gentoo partitions, enter the chroot,
Code: | cd /usr/src/linux
mv .config .configold
make clean
make mrproper
wget -o /usr/src/linux/.config http://bpaste.net/raw/48156/
ls -al /usr/src/linux/.config | verify you have nice new .config Code: | make oldconfig
make menuconfig | look around; change something; change it back; exit saving the config.
Code: | make && make modules_install && make install | if you prefer manual copying of the kernel to make install, the choice is yours, in either case ensure the new kernel name is exactly the same in /etc/boot and /boot/grub/grub.conf
if the panic recurs a picture of the screen posted on a free photo site might be helpful
http://swift.siphos.be/linux_sea/ch07.html#idp35213232 good discussion of boot panics |
Okay, followed the steps laid out, and it has changed things a bit, however, I'm still getting a kernel panic. but now it's the following:
Code: |
vfs unable to mount root fs on unknown-block
|
I have checked and double checked my configurations. I am using sda nomenclature, and have been for quite some time now. So that is eliminated as a possibility. I think the answer lies within the kernel settings for my hardware controller, it's just finding it that's the issue. Allow me to post a few lines in my grub.conf:
Code: |
default 0
timeout 10
splashimage=(hd1,0)/boot/grub/splash.xpm.gz
title Gentoo Linux Gentoo Sources 9/26/2012
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sda3
title Gentoo Linux Zen 5/5/2011
root (hd1,0)
kernel /boot/vmlinuz-2.6.38-zen20110501 root=/dev/sda3
|
let us note that the second option, the Zen, boots without the panic, but the first option, the Gentoo Sources, is having the panic. Have I entered anything wrong in my grub.conf? Or is that part correct. I am happy to post a picture with my panic if you think it would be helpful, but it will be tomorrow.
The link provided did offer some insight as to what is causing my problem, and it was a very good page and allowed me to discover that I had forgotten a couple of things, but that did not help. I really do appreciate the help I am getting here, I know you all must have much more important work to do, so thanks so much!!
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Fri Sep 28, 2012 11:20 pm Post subject: |
|
|
did you import and use the provided kernel? _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sat Sep 29, 2012 12:18 am Post subject: |
|
|
DONAHUE wrote: | did you import and use the provided kernel? |
Yes, I followed the steps you provided. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Sat Sep 29, 2012 1:35 am Post subject: |
|
|
boot cd
edit your grub.conf so that root=/dev/sda3 is replaced in a multitude of menu entries by each and every partition that is listed by fdisk -l; for example:
Quote: | default 0
timeout 10
splashimage=(hd1,0)/boot/grub/splash.xpm.gz
title sda1
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sda1
title sda2
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sda2
title Gentoo Linux Gentoo Sources 9/26/2012
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sda3
title sda4
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sda4
etc
title sdb1
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sdb1
title sdb2
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sdb2
title sdb3
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sdb3
etc
title sdc1
root (hd1,0)
kernel /boot/vmlinuz-3.4.9-gentoo root=/dev/sdc1
etc, etc |
reboot and try each menu entry in turn until one works, I'd probably try /dev/sdb3 first.
if luks or lvm or raid or such variation is involved in your install please let us know. _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sat Sep 29, 2012 10:28 am Post subject: |
|
|
actually, lvm is involved. I never thought about this, but revdep-rebuild tried to rebuild lvm, and at first it was failing because it claimed that it could not find a kernel .config file. Now it is failing because of something else: http://pastebin.com/raw.php?i=6ttjFDUn
That's as far back as I can go. I did read some things about a bug in lvm2.2.02.95-r1, but I am trying to use the stable version. This is actually the reason that I am attempting to upgrade my kernel. I do see warning messages when I boot into the zen kernel that lvm failed to start, but it boots anyway. I hope this is relevant.
Thanks
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Sat Sep 29, 2012 2:34 pm Post subject: |
|
|
have you run through the grub.conf drill I proposed earlier?
I am 99.99% confident that a kernel compiled with the current config would boot a new basic handbook install.
It is possible the zen kernel is naming the hard drives differently than the gentoo-sources kernel, the drill should eliminate or confirm.
or
lvm?
Could you post the result of
have you a backup of your personal files?
when did you last run Code: | emerge --sync
emerge -uND world | succsessfully?
what version is installed? _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 11:19 am Post subject: |
|
|
DONAHUE wrote: | have you run through the grub.conf drill I proposed earlier?
I am 99.99% confident that a kernel compiled with the current config would boot a new basic handbook install.
It is possible the zen kernel is naming the hard drives differently than the gentoo-sources kernel, the drill should eliminate or confirm.
or
lvm? |
I actually haven't run through the drill yet simply because of the fdisk -l output below. I am a bit confused about the drive names. Is it possible that the drives could be re-named by a kernel and still be set as "sda1, 2, 3 etc"? Or does the kernel just say it, and grub happily do it?
Quote: |
Could you post the result of |
Sure:
Code: |
# fdisk -l
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd80ff29d
Device Boot Start End Blocks Id System
/dev/sda1 * 63 224909 112423+ 83 Linux
/dev/sda2 224910 2281229 1028160 82 Linux swap / Solaris
/dev/sda3 2281230 1953520064 975619417+ 83 Linux
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa08b42f3
Device Boot Start End Blocks Id System
/dev/sdb1 * 2048 206847 102400 7 HPFS/NTFS/exFAT
/dev/sdb2 206848 1953521663 976657408 7 HPFS/NTFS/exFAT
|
Quote: |
have you a backup of your personal files?
when did you last run Code: | emerge --sync
emerge -uND world | succsessfully? |
Yes, I backed them up this morning, and I try and sync and emerge -uDNv world every day or so if I am able.
Quote: |
what version is installed? |
Code: | * sys-fs/udev
Latest version available: 192
Latest version installed: 192
Size of files: 1,374 kB
Homepage: http://www.freedesktop.org/wiki/Software/systemd
Description: Linux dynamic and persistent device naming support (aka userspace devfs)
License: LGPL-2.1 MIT GPL-2
|
I really appreciate all your help, I will run through the drill you posted earlier and see what happens!
Thanks
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 11:59 am Post subject: |
|
|
**Update**
I tried the grub drill that you showed me, I did sda, b, and c 1- 9 on each, and each fail with the same kernel panic as before. I was so hoping that it would work. I hope this is useful information, because I kind of feel like I am hitting a brick wall.
Thanks
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Sun Sep 30, 2012 2:49 pm Post subject: |
|
|
does the panic screen have anything like Driver: sd or Driver: sr
or
Available partitions:
if partitions, which are listed
boot to your Bios; what modes are your hard drive controllers in? what choices are available. _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 3:37 pm Post subject: |
|
|
DONAHUE wrote: | does the panic screen have anything like Driver: sd or Driver: sr
or
Available partitions:
if partitions, which are listed
boot to your Bios; what modes are your hard drive controllers in? what choices are available. |
Actually, no, here are the images of the panic screen. the two images are of the same screen, I just took two for the sake of clarity.
Shot 1
Shot 2
I hope these help!
I will now boot into my bios and look at the harddrive controllers!
Thanks
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 3:48 pm Post subject: |
|
|
okay, had a look at my bios, and the options for the SATA controller are: Native IDE RAID and AHCI. Native IDE is selected, should it be AHCI? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Sun Sep 30, 2012 3:50 pm Post subject: |
|
|
pls try ahci _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 5:05 pm Post subject: |
|
|
DONAHUE wrote: | pls try ahci |
Okay, now nothing works, even after switching it back. I can't even get windows to boot now, and can't get it to fix the mbr. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
DONAHUE Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/2468228754a8b1493ea37b.gif)
Joined: 09 Dec 2006 Posts: 7651 Location: Goose Creek SC
|
Posted: Sun Sep 30, 2012 5:12 pm Post subject: |
|
|
the native ide should have been there to allow booting the windows cd/dvd
perhaps set bios defaults
or clear cmos _________________ Defund the FCC. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 5:20 pm Post subject: |
|
|
well, I can boot into the windows live dvd, however, I am having difficulty fixing the mbr |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
NeddySeagoon Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/3946266373f47d606a2db3.jpg)
Joined: 05 Jul 2003 Posts: 54849 Location: 56N 3W
|
Posted: Sun Sep 30, 2012 5:30 pm Post subject: |
|
|
todd93,
unknown-block(0,0) in your panic message means the kernel cannot communicate with your HDD at all.
In your BIOS, Native IDE mode is provided only to allow windows XP users to install the AHCI drivers.
Native IDE normally has missing functionality is iintended to be used once only by Windows XP users and never by anyone else.
Choose AHCI mode. Raid mode is for fakeraid with your drives set up the way you have, you don't want that.
DONAHUE wrote: |
I have taken your config and compared it against pappy's and mine then opened it with make oldconfig and then opened it with make menuconfig to add missing items. the only item i saw that could remotely be linked to your panic was config_ahci _platform=y. very unlikely |
Strange that. My desktop has the same AMD chipset as my HP Microserver. The microserver needs that option but my desktop doesn't.
I have not looked at DONAHUEs .config file but he usually knows what hes doing.
Maybe something went wrong with the kernel install, so you didn't actually use DONAHUEs kernel. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 6:13 pm Post subject: |
|
|
NeddySeagoon wrote: | todd93,
unknown-block(0,0) in your panic message means the kernel cannot communicate with your HDD at all.
In your BIOS, Native IDE mode is provided only to allow windows XP users to install the AHCI drivers.
Native IDE normally has missing functionality is iintended to be used once only by Windows XP users and never by anyone else.
Choose AHCI mode. Raid mode is for fakeraid with your drives set up the way you have, you don't want that.
DONAHUE wrote: |
I have taken your config and compared it against pappy's and mine then opened it with make oldconfig and then opened it with make menuconfig to add missing items. the only item i saw that could remotely be linked to your panic was config_ahci _platform=y. very unlikely |
Strange that. My desktop has the same AMD chipset as my HP Microserver. The microserver needs that option but my desktop doesn't.
I have not looked at DONAHUEs .config file but he usually knows what hes doing.
Maybe something went wrong with the kernel install, so you didn't actually use DONAHUEs kernel. |
Okay, back up and running, it was a dim-witted oversight on my part, grub reinstalled, all is good. I do notice an increase in speed in AHCI mode. maybe I need to go back to that post and try again. I did notice when I used make oldconfig that there were a lot of options to choose and I may have messed that part up.
Thanks!
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
todd93 l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
Joined: 30 Oct 2006 Posts: 625 Location: Missouri
|
Posted: Sun Sep 30, 2012 6:26 pm Post subject: |
|
|
Okay, I have some promising news. First of all, we're still not quite there yet. Secondly, there is no more kernel panic!! that is done, the kernel itself actually boots. there are some problems, though, first of all, LVM is still not loading. Also, it times out while trying to start eth1 (not sure why). and after the kernel finishes booting, I get a small icon for a few seconds, then a black screen. Something tells me that my display driver may not be exactly right on that one. I would really rather not say what I did to get passed the panic, but I will. I re-compiled the kernel by using: make && make modules_install && make install, rebooted, and there we have that. I was excited for a bit, but now I realize I'm up against a new problem.
Thanks everyone for the help!!
Todd |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|