View previous topic :: View next topic |
Author |
Message |
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Sun Jan 01, 2006 11:16 pm Post subject: <0>Kernel Panic - not syncing: Fatal exception in inte |
|
|
Hello,
I have bootable RAID 1 setup (intel ich6). I'm new to gentoo.
i've installed dmraid, dm, etc and my setup boots fine from the RAID 1 array.
I've successfully installed samba and am planning to use the setup as mostly as a fileserver.
I get a kernel panic when I try to copy many large files (movies mostly) to the setup. the target share is a FAT32 partition on the raid array.
I've created a share and am using windows to copy files. there is no problem if i copy a file individually - even the one on which there was a kernel panic. so it is not specific to data.
i seem to get a panic after a few gigs of data transfer. i've got it twice now. i was hoping to enable some kind of monitoring, etc to do something about it. but my knowledge is very limited in this area.
I've got a photo like a screenshot of the panic message (sorry - could not think of another way - unless this data has been saved). if someone is interested i can send the photo (<200kb). if there is a way to upload it here, please let me know.
i also noticed pdflush taking a lot of cpu - even more than smbd.
i've typed some information from the screen photo. This is as follows.
Code: | submit_bh
__block_write_full_page
fat_get_block
block_write_full_page
fat_get_block
fat_writepage
fat_get_block
mpage_writepages
fat_writepage
do_writepages
__sync_single_inode
__writeback_single_inode
dm_any_congested
dm_any_congested
dm_table_any_congested
sync_sb_inodes
pdflush
writeback_inodes
wb_kupdate
__pdflush
wb_kupdate
kthread
kthread
kernel_thread_helper
Code: 53 8b 6c 24 14 8b 5c 24 18 8b 74 24 1c 8d 7d 10 89 f8 e8 46 e0 9c c7 56 53 55 e8 0e fa ff ff 89 c3 89 f8 e8 45 e3 9c c7 83 c4 0c <f0> ff 4b 20 0f 94 c0 84 c0 75 07 5b 5e 5f 5d c3 89 f6 8d 7d 28
<0>Kernel Panic - not syncing: Fatal exception in interrupt |
I have only typed the function names of the stack trace. there are hex numbers that i've omitted. Please let me know if you want more info.
thanks for reading this.
cheers
i keep updating this post as i get more information.
the next time i ran this (copying large files across), same panic message - except this time a lot of function calls were in [dm_mirror]. i also have pic of this. please let me know if you want more info.
if anyone may be able to help ... even with how i can get more useful information, please let me know.
thanks again. |
|
Back to top |
|
|
widan Veteran
Joined: 07 Jun 2005 Posts: 1512 Location: Paris, France
|
Posted: Mon Jan 02, 2006 3:08 am Post subject: Re: <0>Kernel Panic - not syncing: Fatal exception in |
|
|
stillnoname wrote: | I get a kernel panic when I try to copy many large files (movies mostly) to the setup. the target share is a FAT32 partition on the raid array. |
Do you really need to use FAT32 ?
stillnoname wrote: | if there is a way to upload it here, please let me know. |
You can put it on ImageShack and post the link.
stillnoname wrote: | i also noticed pdflush taking a lot of cpu - even more than smbd. |
pdflush is the kernel thread that writes modified pages back to disk, so it's normal to see it active when you have disk activity. |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Mon Jan 02, 2006 1:50 pm Post subject: images |
|
|
thanks for the response.
these are the urls for the images
http://img499.imageshack.us/img499/512/dsc000141be.jpg
http://img499.imageshack.us/img499/7462/dsc000173ep.jpg
I can make this happen quite regualrly. I've made it happen atleast 5 times. sometimes there is no panic information printed, but the system just crashes - no response either. surprisingly the ping to the system works (when not panic). keyboard does not work. no local login or remote ssh login.
most times it panics. other things is that it seems to work without problems when i transferred 3 gigs of digital photos and mp3 files. so it seems to hold well with small files. it seems to break down when i transfer a few gigs of movie mpeg, avi, etc. - many of the files are full cd length. i finally transferred in small batches and it worked mostly - without too many crashes. it seems to hold better if i'm reading from the disks - that slows down transfers a bit, but seems to add stability (i may be wrong here). the panic seems random - even when i deleted all and did same transfer, it paniced at different place - so it should not be specific to data. i've also read of someone else having issue with 2.6 series kernel with this (and other) version of samba - when transferring a number of large files.
fyi, i'm using linux 2.6.12-gentoo-r10 kernel (genkernel) with patch for my network driver (sk98lin). samba version is 3.0.10. if you need more version or other information
i'm using fat32 cause i want the data to be accessible from windows as well. i intend to make this a dual boot system. i'm still not so comfortable with linux to remove windows completely. may sometime down the line - if i can run it without any panics
looking forward to hearing more from people.
cheers. |
|
Back to top |
|
|
widan Veteran
Joined: 07 Jun 2005 Posts: 1512 Location: Paris, France
|
Posted: Mon Jan 02, 2006 5:34 pm Post subject: Re: images |
|
|
Actually those won't help much, because the most interesting part of this kind of message (that tells what kind of error happened, and in which part of the code) is the top, which unfortunately is off-screen here ...
The trace is quite deep, and there were issues with older kernels with layered I/O. Maybe you got a kernel stack overflow, especially if you have 4KB stacks enabled in the kernel configuration (but usually those overflows involved NFS, not Samba...). Try to disable 4K stacks and see if it helps:
Code: | Kernel hacking --->
[*] Kernel debugging
[ ] Use 4Kb for kernel stacks instead of 8Kb |
You can also try a newer kernel (2.6.14 instead of 2.6.12 for example). |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Tue Jan 03, 2006 12:50 am Post subject: not much impact (but something more) |
|
|
hello,
thanks for the previous reply.
i recompiled the kernel with 4k stacks disabled.
very similar result - failed. However, on the root terminal i left "vmstat 1" running. It continued running even after the server was lost on the network. previous times i had top running on the it froze. not sure this time.
so i thought maybe samba failed. ran rc-update and that showed samba started. i then issued "/etc/init.d/samba stop". then it stuck there - nothing printed. so i think the script didn;t run.
i went to second console and tried logging in as root. it printed out:
Quote: | login(pam_unix)[10077]: session opened for user root by (uid=0) |
then it stuck at the next line.
went to next terminal - logged in - same result.
then i tried logging in remotely (ssh). that accepted login and password but printed nothing else.
went to next terminal (alt+f4) - tried logging in as root. It printed out
Quote: | login(pam_unix)[10079]: session opened for user root by (uid=0)
Last login: Mon Jan 2 23:37:39 on vc/1
|
on next terminal, same thing.
Looks to me that any disc access locks it. And I may be completely wrong here.
let me know if any of this helps to get this further.
i tried getting 2.6.14 from world, but it looks like it has not been release.
as i'm new to gentoo (and linux) i'm not sure how to install this. possibly vanilla kernel. but that would be a real problem since i need dmraid - my bootable disk is on raid 1. but maybe there's an easy way (using initramfs of previous build??). let me know if you need any more information, etc.
finally, i can change virtaul terminals using keyboard shortcuts (alt+f1, etc), but there is no impact of ctrl+c or ctrl+alt+del. so i'm a bit lost on this.
any other things i could try before trying kernel 2.6.14. i really want to avoid upgrading to a kernel not in world. any other diagnostics, etc i could switch on? something else i could do to help understand it better ?
again, thanks. |
|
Back to top |
|
|
widan Veteran
Joined: 07 Jun 2005 Posts: 1512 Location: Paris, France
|
Posted: Tue Jan 03, 2006 2:16 am Post subject: Re: not much impact (but something more) |
|
|
stillnoname wrote: | went to next terminal (alt+f4) - tried logging in as root. It printed out
Quote: | login(pam_unix)[10079]: session opened for user root by (uid=0)
Last login: Mon Jan 2 23:37:39 on vc/1
|
on next terminal, same thing. |
Something probably crashed somewhere in the block layer, and in doing so it failed to release all the locks it had on disk control structures. If that's the case, disk accesses after the crash are likely to deadlock.
stillnoname wrote: | i tried getting 2.6.14 from world, but it looks like it has not been release. |
It has been released for some time now...
Code: | widan@stephanie ~ $ emerge -s gentoo-sources
Searching...
[ Results for search key : gentoo-sources ]
[ Applications found : 1 ]
* sys-kernel/gentoo-sources
Latest version available: 2.6.14-r5
Latest version installed: 2.6.14-r2
Size of downloaded files: 38,435 kB
Homepage: http://dev.gentoo.org/~dsd/genpatches
Description: Full sources including the gentoo patchset for the 2.6 kernel tree
License: GPL-2 |
stillnoname wrote: | as i'm new to gentoo (and linux) i'm not sure how to install this. possibly vanilla kernel. but that would be a real problem since i need dmraid - my bootable disk is on raid 1. but maybe there's an easy way (using initramfs of previous build??). |
First run "emerge sync" to update the portage tree (if you don't see gentoo-sources-2.6.14, then it's not up-to-date), then check the 2.6.14 kernel appears:
Code: | widan@stephanie ~ $ emerge -pv gentoo-sources
These are the packages that I would merge, in order:
Calculating dependencies ...done!
[ebuild NS ] sys-kernel/gentoo-sources-2.6.14-r5 -build -doc -symlink (-ultra1) 0 kB
Total size of downloads: 0 kB |
Then simply emerge it normally, and use genkernel to build it like you did for the first one. It should create the needed initramfs for the RAID (you probably need the "--dmraid" option to genkernel though).
stillnoname wrote: | finally, i can change virtaul terminals using keyboard shortcuts (alt+f1, etc), but there is no impact of ctrl+c or ctrl+alt+del. so i'm a bit lost on this. |
VT switch (Alt-F1, ...) is something that is provided by kernel code, so it's always available as long as the kernel isn't completely crashed. Ctrl-C sends a signal to the process, but if the process is executing in the kernel at the time (which it will be if it is blocking in a system call) it won't receive the signal until it returns to userspace. Ctrl-Alt-Del actually runs a program ("shutdown -r now" by default), so it won't do anything if it can't read the program from disk. |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Wed Jan 04, 2006 1:32 am Post subject: unsuccessful after kernel upgrade also |
|
|
hello,
after much trial and error, i managed to update the kernel as suggested (2.6.14-gentoo-r5).
it is giving me the same problem. this time "vmstat 2" stopped running. i'm guessing that sometimes its a kernel panic, other times something else.
anything that anyone wants to suggest? what can i do to help diagnose this further?
i'm currently emerging samba-3.0.14a-r2. I'll let the forum know if it is successul. i am not too sure.
i have noted the booting from such a setup is still experimental. How can i help progress this further? any help will be appreciated.
thanks in advance |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Wed Jan 04, 2006 1:54 am Post subject: failed with latest samba also |
|
|
hi,
just to let all know that the new samba version stuck exactly like the previous version.
If anyone can shed any light on what to do (logging, etc) that would be appreciated.
thanks again. |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Wed Jan 04, 2006 9:12 pm Post subject: bump |
|
|
can anybody help?
thanks in advance |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Mon Jan 09, 2006 9:40 pm Post subject: one more try |
|
|
hi all,
i'll give this one more try, hoping someone can help.
thanks in advance |
|
Back to top |
|
|
Doctor Corndog Tux's lil' helper
Joined: 05 Dec 2005 Posts: 107 Location: Maryland
|
Posted: Tue Jan 10, 2006 1:12 am Post subject: |
|
|
I'm not sure if this will help you, but I have had similar errors. When I put my computer together, I loaded up Gentoo first, so I had nothing else to compare it's performace to. I began receiving Kernel Panics whenever I would try to compile large amounts of data (happened twice about 7/10 through emerge gnome). On my system, it looked like a hardware fault right away. Happened here and there, and always under stress. I asked around a many people told me it could be a memory problem. All my memtests ran great, so I was a little confused. In the end, I bumped up the voltage to my memory to 2.7V, and I've never had a problem since. Don't know if that will help ya, but it might be worth a shot. Make sure you find out the max voltage on your memory first though. |
|
Back to top |
|
|
stillnoname n00b
Joined: 01 Jan 2006 Posts: 8
|
Posted: Tue Jan 10, 2006 11:03 am Post subject: similar issue seen on dm-devel |
|
|
hello,
thanks for that. i'll look into this as well. i've have been compiling and emerging big packages. Plus, I've not overclocked or changed any voltage anywhere. So, I'm not sure this should be the case. But probably worth trying.
I've also notices that there was some traffic in "dm-devel" mailing list about file system io locks when doing large transfers. And this was in Apr 2005. I've not been able to track this down further. I'm hoping to update the device-mapper to a later version to see if this is still a problem. I'm very new to gentoo (and linux in general), so i'm not exactly sure how to go about it. Will read up on that. I'll probably need to apply a patch to the kernel. Or wait for a new stable ebuild of gentoo sources. as of yesterday, the version I used is the latest. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|