How to parse a 22GB messages file to find panic / reboot

RayDude · Last edited by RayDude on Mon Feb 17, 2025 2:53 am; edited 1 time in total

My server crashed last night.

I was attempting to make a backup with borg backup and I thought the external drive may have run out of space... But that's not the case.

I'm attempting to diagnose the issue. I need to parse a 22 GB messages file. I grepped -i for the keyword panic, but found nothing.

How can I figure out where the relevant messages are in the log. Vim is efficient enough to load it, but searches are slow.

Is there an easier way to diagnose the problem?

Thanks.

Update: I have a bigger problem. My messages kernel log is so big because it's spammed with, "kwin_core: parsing edid failed"

I know why, because I'm using an external DisplayPort to HDMI adapter to get 4K 120 Hz to run. I need that warning to go away.

Update: As for this, I changed my log level to 1. And it doesn't suppress the messages.

Update: I continued the backup today and after an hour or so of compressing (that's what it spends most of its time doing) the computer reboot while I was watching SNL 50th.

It may be a USB issue. Since I turned the log down to 1 though it looks like there's nothing in /var/log/messages...
_________________
Some day there will only be free software.

spica · Guru Joined: 04 Jun 2021 Posts: 345

Hu · Administrator Joined: 06 Mar 2007 Posts: 23124

A panic will not be written to the log, because the panic stops all processing, so the syslog daemon is never given the chance to write the panic text to the log. You might find some pre-panic messages of interest, though.

gentoo_ram

RayDude · Posted: Mon Feb 17, 2025 6:41 am Post subject:

Thanks everyone.

I'm running syslog-ng, I'll figure out how to filter k_win and ACPI unhandled event.

Since the issue is reproducable, I should be able to look at the console when I make it crash the next time.

Since I didn't have this problem before, it's either a hardware failure, which seems unlikely or an issue with the latest kernel.

Right now I don't have the time to pursue it, so I'll put it off until I have some time.

If I can't trap the issue (there's no visible panic screen, the system just rebooted), I'll hook up a serial port and another computer to try to capture the console via the serial port.

Well, that's what I want to do, but it's time consuming and I have a lot on my plate these days so we'll see.

I might just move back to a stable kernel and see if borg works with it.

I'm running kernel: 6.13.2-gentoo.

I have an older AMD Zen 3 CPU and I wonder if that's having new kernel issues. I've run into that with an older intel core processor where later kernels were unstable on the older processor. Or at least seemed to be.... You can never really know unless you can catch it in the act...

Again, thanks.
_________________
Some day there will only be free software.

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2207

If you have the persistent storage options set in your kernel config, you might find some useful info there. (That said, I've never seen anything useful myself.)
_________________
Greybeard

Ralphred · l33t Joined: 31 Dec 2013 Posts: 729

wjb · Posted: Tue Feb 18, 2025 12:43 am Post subject:

Borg keeps a lot in memory and in some circumstances this can be a problem. It could be that compression taking ages is just a symptom of running on swap.

There's some info here: https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#indexes-caches-memory-usage

gentoo_ram · Posted: Tue Feb 18, 2025 7:05 pm Post subject:

Here's an example of message filtering with syslog-ng. My actual configuration lists a lot more exclusions.