bender86
Guru
Joined: 18 Mar 2005
Posts: 484
|
 Posted: Fri Apr 14, 2017 7:17 pm Post subject: Out-of-memory kernel panic at boot after glibc upgrade |
 |
|
Hello,
I have a problem with my SPARC machine (Netra X1).
I'm fairly sure the problem was caused by a glibc upgrade (2.22-r4 -> 2.23-r3). That is, I upgraded every other package, I rebooted and upgraded glibc. The last thing I could see (I was using a tmux session) was the message "safely unmerging existing instance" (or whatever text emerge prints in this case). At that point I could not issue any command in the tmux session, and neither open a shell over SSH. I suspect there was a memory leak which saturated the swapfile.
Then I hard rebooted.
At this point, after the kernel is loaded, I get a out-of-memory kernel panic. At first it was something related to "loadkeys", so I rebooted with "init=/bin/sh" and removed "keymaps" from the boot runlevel. Now I have a similar problem with mount. Attached is the complete log from when I power on the machine (lines like --> [NNN lines] mean that the previous line was repeated NNN times).
Any idea on what could have been the problem? I had a similar issue few years ago when after a glibc upgrade every binary dynamically linked to it failed and had to throw away my installation and start from scratch, but this time if I boot with "init=/bin/sh" I can run commands.
If I manually mount my filesystems and unmount /dev and mount /proc I can run emerge. However, I then get an out-of-memory kernel panic.
I have a cross compiler and emerge, so I could create a new installation, but I'm not sure if I'll be able to merge the packages.
| Code: | lom>poweron
lom>
LOM event: +0h42m7s host power on
ü
Sun Netra X1 (UltraSPARC-IIe 400MHz), No Keyboard
OpenBoot 4.0, 512 MB memory installed, Serial #50660900.
Ethernet address 0:3:ba:5:6:24, Host ID: 83050624.
Initializing Memory -
Boot device: disk File and args:
SILO Version 1.4.14_git20120819_p1
\
Select which kernel boot:
* linux : default (4.4.39)
maintenance : maintenance (4.4.39)
emergency : /bin/sh (4.4.39)
old : maintenance (4.1.15-r1)
boot:
Allocated 64 Megs of memory at 0x40000000 for kernel
|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\Loaded kernel version 4.4.39
PROMLIB: Sun IEEE Boot Prom 'OBP 4.0.14 2001/11/07 17:16'
PROMLIB: Root node compatible: sun4u
Linux version 4.4.39-gentoo (root@netra) (gcc version 4.9.3 (Gentoo 4.9.3 p1.5, pie-0.6.4) ) #1 Sun Jan 1 15:57:23 CET 2017
bootconsole [earlyprom0] enabled
ARCH: SUN4U
Ethernet address: 00:03:ba:05:06:24
MM: PAGE_OFFSET is 0xfffff80000000000 (max_phys_bits == 40)
MM: VMALLOC [0x0000000100000000 --> 0x0000060000000000]
MM: VMEMMAP [0x0000060000000000 --> 0x00000c0000000000]
Kernel: Using 2 locked TLB entries for main kernel image.
Remapping the kernel... done.
OF stdout device is: /pci@1f,0/isa@7/serial@0,3f8
PROM: Built device tree with 71105 bytes of memory.
Top of RAM: 0x67ec8000, Total RAM: 0x1fe36000
Memory hole size: 1152MB
Allocated 16384 bytes for kernel page tables.
Zone ranges:
Normal [mem 0x0000000000000000-0x0000000067ec7fff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x0000000007ffffff]
node 0: [mem 0x0000000020000000-0x0000000027ffffff]
node 0: [mem 0x0000000040000000-0x0000000047ffffff]
node 0: [mem 0x0000000060000000-0x0000000066ffdfff]
node 0: [mem 0x0000000067000000-0x0000000067dfffff]
node 0: [mem 0x0000000067e80000-0x0000000067eaffff]
node 0: [mem 0x0000000067ec0000-0x0000000067ec7fff]
Initmem setup node 0 [mem 0x0000000000000000-0x0000000067ec7fff]
Booting Linux...
CPU CAPS: [flush,stbar,swap,muldiv,v9,mul32,div32,v8plus]
CPU CAPS: [vis]
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 64796
Kernel command line: root=/dev/sda1 ro softlevel=default
PID hash table entries: 2048 (order: 1, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 6, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 5, 262144 bytes)
Sorting __ex_table...
Memory: 499680K/522456K available (3507K kernel code, 199K rwdata, 888K rodata, 144K init, 518K bss, 22776K reserved, 0K cma-reserved)
SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:2048 nr_irqs:2048 1
clocksource: hbtick: mask: 0xffffffffffffffff max_cycles: 0x148020aa9, max_idle_ns: 440795202069 ns
clocksource: mult[b400012e] shift[24]
clockevent: mult[16c16bf] shift[32]
Console: colour dummy device 80x25
console [tty0] enabled
bootconsole [earlyprom0] disabled
PROMLIB: Sun IEEE Boot Prom 'OBP 4.0.14 2001/11/07 17:16'
PROMLIB: Root node compatible: sun4u
Linux version 4.4.39-gentoo (root@netra) (gcc version 4.9.3 (Gentoo 4.9.3 p1.5, pie-0.6.4) ) #1 Sun Jan 1 15:57:23 CET 2017
bootconsole [earlyprom0] enabled
ARCH: SUN4U
Ethernet address: 00:03:ba:05:06:24
MM: PAGE_OFFSET is 0xfffff80000000000 (max_phys_bits == 40)
MM: VMALLOC [0x0000000100000000 --> 0x0000060000000000]
MM: VMEMMAP [0x0000060000000000 --> 0x00000c0000000000]
Kernel: Using 2 locked TLB entries for main kernel image.
Remapping the kernel... done.
OF stdout device is: /pci@1f,0/isa@7/serial@0,3f8
PROM: Built device tree with 71105 bytes of memory.
Top of RAM: 0x67ec8000, Total RAM: 0x1fe36000
Memory hole size: 1152MB
Allocated 16384 bytes for kernel page tables.
Zone ranges:
Normal [mem 0x0000000000000000-0x0000000067ec7fff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000000000000-0x0000000007ffffff]
node 0: [mem 0x0000000020000000-0x0000000027ffffff]
node 0: [mem 0x0000000040000000-0x0000000047ffffff]
node 0: [mem 0x0000000060000000-0x0000000066ffdfff]
node 0: [mem 0x0000000067000000-0x0000000067dfffff]
node 0: [mem 0x0000000067e80000-0x0000000067eaffff]
node 0: [mem 0x0000000067ec0000-0x0000000067ec7fff]
Initmem setup node 0 [mem 0x0000000000000000-0x0000000067ec7fff]
Booting Linux...
CPU CAPS: [flush,stbar,swap,muldiv,v9,mul32,div32,v8plus]
CPU CAPS: [vis]
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 64796
Kernel command line: root=/dev/sda1 ro softlevel=default
PID hash table entries: 2048 (order: 1, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 6, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 5, 262144 bytes)
Sorting __ex_table...
Memory: 499680K/522456K available (3507K kernel code, 199K rwdata, 888K rodata, 144K init, 518K bss, 22776K reserved, 0K cma-reserved)
SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:2048 nr_irqs:2048 1
clocksource: hbtick: mask: 0xffffffffffffffff max_cycles: 0x148020aa9, max_idle_ns: 440795202069 ns
clocksource: mult[b400012e] shift[24]
clockevent: mult[16c16bf] shift[32]
Console: colour dummy device 80x25
console [tty0] enabled
bootconsole [earlyprom0] disabled
Calibrating delay using timer specific routine.. 11.13 BogoMIPS (lpj=55655)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 8192 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 8192 bytes)
devtmpfs: initialized
Performance events: No support for PMU type 'ultra12'
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
NET: Registered protocol family 16
/pci@1f,0: PCI IO[1fe02000000] MEM[1ff00000000] MEM64[1ff00000000]
/pci@1f,0: SABRE PCI Bus Module ver[0:0]
PCI: Scanning PBM /pci@1f,0
sabre f0069d08: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io 0x1fe02000000-0x1fe02ffffff] (bus address [0x0000-0xffffff])
pci_bus 0000:00: root bus resource [mem 0x1ff00000000-0x1ffffffffff] (bus address [0x00000000-0xffffffff])
pci_bus 0000:00: root bus resource [mem 0x1ff00000000-0x1ffffffffff] (bus address [0x00000000-0xffffffff])
pci_bus 0000:00: root bus resource [bus 00]
pci 0000:00:03.0: quirk: [io 0x1fe02002000-0x1fe0200203f] claimed by ali7101 ACPI
pci 0000:00:03.0: quirk: [io 0x1fe02004000-0x1fe0200401f] claimed by ali7101 SMB
pci 0000:00:05.0: can't claim BAR 6 [mem 0x1ff00080000-0x1ff000bffff]: address conflict with Video RAM area [??? 0x1ff000a0000-0x1ff000bffff flags 0x80000000]
vgaarb: loaded
SCSI subsystem initialized
/pci@1f,0/isa@7/rtc@0,70: RTC regs at 0x1fe02000070
clocksource: Switched to clocksource hbtick
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 2, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 0, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 0, 8192 bytes)
NET: Registered protocol family 1
pci 0000:00:07.0: Activating ISA DMA hang workarounds
power: Control reg at 1fe02002000
futex hash table entries: 256 (order: -1, 6144 bytes)
Initialise system trusted keyring
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 1024 (order 0, 8192 bytes)
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler cfq registered (default)
f0076dcc: ttyS0 at MMIO 0x1fe020003f8 (irq = 7, base_baud = 115387) is a 16550A
Console: ttyS0 (SU)
console [ttyS0] enabled
f0078ab8: ttyS1 at MMIO 0x1fe020002e8 (irq = 7, base_baud = 115387) is a 16550A
scsi host0: pata_ali
scsi host1: pata_ali
ata1: PATA max UDMA/66 cmd 0x1fe02010200 ctl 0x1fe02010218 bmdma 0x1fe02010220 irq 12
ata2: PATA max UDMA/66 cmd 0x1fe02010210 ctl 0x1fe02010208 bmdma 0x1fe02010228 irq 12
mousedev: PS/2 mouse device common for all mice
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc_cmos rtc_cmos: no alarms, 114 bytes nvram
NET: Registered protocol family 17
Loading compiled-in X.509 certificates
Kernel unaligned access at TPC[634634] mpi_read_buffer+0xd4/0x120
Loaded X.509 cert 'Build time autogenerated kernel key: 42ebc4053e8333df398250e8135e9626ea63a026'
ata1.00: ATA-5: ST38410A, 3.03, max UDMA/66
ata1.00: 16841664 sectors, multi 0: LBA
rtc_cmos rtc_cmos: setting system clock to 2017-04-14 18:27:19 UTC (1492194439)
ata1.00: configured for UDMA/66
scsi 0:0:0:0: Direct-Access ATA ST38410A 3.03 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 16841664 512-byte logical blocks: (8.62 GB/8.03 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2 sda3 sda4 sda5
ata2.00: ATA-7: Maxtor 7Y250P0, YAR41BW0, max UDMA/133
ata2.00: 490234752 sectors, multi 0: LBA48
sd 0:0:0:0: [sda] Attached SCSI disk
ata2.00: configured for UDMA/66
scsi 1:0:0:0: Direct-Access ATA Maxtor 7Y250P0 1BW0 PQ: 0 ANSI: 5
sd 1:0:0:0: [sdb] 490234752 512-byte logical blocks: (251 GB/234 GiB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdb: sdb1
sd 1:0:0:0: [sdb] Attached SCSI disk
EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
EXT4-fs (sda1): INFO: recovery required on readonly filesystem
EXT4-fs (sda1): write access will be enabled during recovery
EXT4-fs (sda1): recovery complete
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
devtmpfs: mounted
INIT: version 2.88 booting
OpenRC 0.23.2 is starting up Gentoo Linux (sparc64)
Press I to enter interactive boot mode
random: nonblocking pool is initialized
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
--> [139 lines]
* Mounting /proc ...
* Mounting /proc ...
* Mounting /proc ...
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
--> [66 lines]
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
proc is already mounted or /proc busy
mount: proc is already mounted or /proc busy
proc is already mounted on /proc
proc is already mounted on /proc
proc is already mounted on /proc
proc is already mounted on /proc
--> [61 lines]
proc is already mounted on /proc
proc is already mounted on /proc
proc is already mounted on /proc
proc is already mounted on /proc
[ !! ]
[ !! ]
[ !! ]
[ !! ]
[ ok ]
[ !! ]
[ !! ]
[ !! ]
[ !! ]
[ !! ]
mount invoked oom-killer: gfp_mask=0x24200ca, order=0, oom_score_adj=0
CPU: 0 PID: 3718 Comm: mount Not tainted 4.4.39-gentoo #1
Call Trace:
[00000000004ba764] oom_kill_process+0x284/0x3a0
[00000000004babf0] out_of_memory+0x2d0/0x320
[00000000004bec20] __alloc_pages_nodemask+0x640/0x740
[00000000004d4468] wp_page_copy.isra.91+0x48/0x3a0
[00000000004d58cc] do_wp_page.isra.95+0x6c/0x580
[00000000004d79b4] handle_mm_fault+0x614/0xc40
[000000000044d8e8] do_sparc64_fault+0x268/0x720
[0000000000407918] sparc64_realfault_common+0x10/0x20
Mem-Info:
active_anon:20159 inactive_anon:0 isolated_anon:0
active_file:141 inactive_file:56 isolated_file:40
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:52 slab_unreclaimable:26727
mapped:181 shmem:0 pagetables:9085 bounce:0
free:346 free_pcp:33 free_cma:0
Normal free:2768kB min:2824kB low:3528kB high:4232kB active_anon:161272kB inactive_anon:0kB active_file:1128kB inactive_file:448kB unevictable:0kB isolated(anon):0kB isolated(file):320kB present:522456kB managed:499824kB mlocked:0kB dirty:0kB writeback:0kB mapped:1448kB shmem:0kB slab_reclaimable:416kB slab_unreclaimable:213816kB kernel_stack:45904kB pagetables:72680kB unstable:0kB bounce:0kB free_pcp:264kB local_pcp:264kB free_cma:0kB writeback_tmp:0kB pages_scanned:11944 all_unreclaimable? yes
lowmem_reserve[]: 0 0
Normal: 0*8kB 47*16kB (UM) 31*32kB (M) 12*64kB (M) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB = 2768kB
234 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
65307 pages RAM
0 pages HighMem/MovableOnly
2829 pages reserved
[ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 430] 0 430 251 32 3 1 0 0 init
[ 431] 0 431 346 71 3 1 0 0 openrc
[ 560] 0 560 480 82 3 1 0 0 init.sh
[ 561] 0 561 480 82 3 1 0 0 init.sh
[ 562] 0 562 480 82 3 1 0 0 init.sh
[ 563] 0 563 480 82 3 1 0 0 init.sh
[ 564] 0 564 480 82 3 1 0 0 init.sh
--> [121 lines]
[ 686] 0 686 480 82 3 1 0 0 init.sh
[ 687] 0 687 480 82 3 1 0 0 init.sh
[ 1226] 0 1226 380 69 3 1 0 0 mount
[ 1237] 0 1237 380 85 4 1 0 0 mount
[ 1240] 0 1240 380 69 4 1 0 0 mount
[ 1247] 0 1247 380 69 4 1 0 0 mount
--> [49 lines]
[ 1376] 0 1376 380 65 3 1 0 0 mount
[ 1380] 0 1380 380 68 4 1 0 0 mount
[ 1382] 0 1382 380 67 3 1 0 0 mount
[ 1383] 0 1383 324 44 4 1 0 0 fstabinfo
[ 1384] 0 1384 380 39 4 1 0 0 mount
[ 1385] 0 1385 380 24 4 1 0 0 mount
[ 1386] 0 1386 380 24 4 1 0 0 mount
[ 1387] 0 1387 380 69 3 1 0 0 mount
[ 1388] 0 1388 380 37 4 1 0 0 mount
[2227 lines, mostly with "mount" but a couple times "fstabinfo", a series of "openrc" and one "init.sh"]
[ 3946] 0 3946 380 170 3 1 0 0 mount
[ 3947] 0 3947 380 0 mount
[ 3948] 0 3948 380 59 3 0
Out mKcess )2 or sacrifillecess 3944 (mounB, anon-rss:184^@
Type 'go' to resume
ok power-off
lom>
LOM event: +0h50m1s host power off
lom>
|
|
|
Gentoo on Sparc
