[SOLVED?] lshw - free(): invalid pointer Aborted

freke · Last edited by freke on Sat Jul 06, 2019 3:16 pm; edited 1 time in total

Hi,

Trying to figure out why /usr/sbin/lshw won't run (I think it used to at some point, and know it worked from the minimal-iso when installing)...
(not that I really need it anymore - but worried if there's other stuff not running properly...)

mike155 · Posted: Mon Jul 01, 2019 6:42 pm Post subject:

It could be...

a bug in lshw, most probably in src/core/pci.cc
a problem in the way you compiled the program (either 'lto' as you suggest, or 'hardened')
a configuration detail on your machine (one of the tmpfs mounts is too small, etc.)

It would be nice to find the line in the source code where the error happens. You could

emerge lshw with debug symbols, run lshw in gdb and create a backtrace after the error happens
add debug statements to the source code and try to find exactly the line in the source code. It guess you will find the error in function "load_pcidb()" in src/core/pci.cc or in one of the functions it calls.
use some other tracing technique

freke · Posted: Wed Jul 03, 2019 2:45 pm Post subject:

Thanks, I'll try to dig deeper and read up on debug symbols/gdb.

I tried with

mike155 · Posted: Wed Jul 03, 2019 3:28 pm Post subject:

Please take a look at: https://wiki.gentoo.org/wiki/Debugging

create /etc/portage/env/debugsyms and /etc/portage/env/installsources
add 'sys-apps/lshw debugsyms installsources' to /etc/portage/package.env
re-emerge lshw
run lshw in gdb
after the SIGABRT: enter "bt" to create a backtrace
post the backtrace

That's how I would start.

Hu · Administrator Joined: 06 Mar 2007 Posts: 21861

Neither stack-protector nor pie need to be disabled for debugging. If disabling them fixes the program, then there is something else wrong that should be found and fixed.

mike155's advice is good, but I want to clarify one point. In step 1, don't create them as empty files. Create them as described in the link he provides.

freke · Posted: Thu Jul 04, 2019 1:43 pm Post subject:

gentoo_ram · Posted: Thu Jul 04, 2019 2:56 pm Post subject:

The stuff in libc just says that some code crashed while calling into the C library. In my experience, that doesn't mean there's a problem with the C library, it means there's a problem with the program's code when calling into the C library.

The last function in lshw is load_pcidb(). Just looked at the source. That function is trying to load a pci.ids file so it can eventually translate PCI ID values to strings. The places it is looking for the file is:

{PREFIX}/share/lshw/pci.ids
/usr/local/share/pci.ids
/usr/share/pci.ids
/etc/pci.ids
/usr/share/hwdata/pci.ids
/usr/share/misc/pci.ids

Do any of those files exist or maybe are corrupted in some way?

Of course, there could have been an issue earlier in the program that messed up memory and the program didn't crash until it got to that point. Maybe it had problems scanning hardware before it got to that point?

I also see something in the thread about a stack protector setting. The variables in that function are stack variables and could become large. Is the program being compiled with something that would limit the stack size? That might cause it problems.

freke · Posted: Thu Jul 04, 2019 3:12 pm Post subject:

I already suspected/had a little look at pci.ids as my strace-output stopped in the middle of that file.

I *had* a /usr/share/misc/pci.ids.gz which I gunzip'ed - so now it's *just* /usr/share/misc/pci.ids

Re-emerged sys-apps/hwids-20180917 - no change
Deleted distfile + re-emerged sys-apps/hwids-20180917 - no change

mike155 · Posted: Thu Jul 04, 2019 3:54 pm Post subject:

After your first post, I analyzed your strace file and I guessed the error happens in load_pcidb(), more specifically in a constructor or destructor of a vector object. See my first post in this thread.

Your backtrace confirms that. :-)

I think that the error happens in lshw, not in glibc (see line #5 of the backtrace). The glibc calls you see (#4 -#0) happen after the error - they are used to output of the error message and to abort. If you like, you can add "sys-libs/glibc debugsyms installsources' to /etc/portage/package.env, re-emerge glibc and re-run the gdb test. The backtrace will tell you which glibc functions were called. That's a general recommendation: whenever you see "??" in a backtrace, add the corresponding package to /etc/portage/package.env, re-emerge the package and re-run the test: the question marks will be replaced by meaningful debug symbols.

The error is not related to configuration files or to the PCI or hwid databases.

I'm surprised that the backtrace doesn't show the exact line in which the error happened. Maybe that's because it happened in a constructor/destructor.

Please give me a couple of hours to think about the next step. I will write more later in the evening.

freke · Posted: Thu Jul 04, 2019 4:58 pm Post subject:

Thanks a lot

mike155 · Posted: Thu Jul 04, 2019 5:26 pm Post subject:

Let's see...

Below is a snippet from the original source code (file: src/core/pci.cc)

freke · Posted: Thu Jul 04, 2019 5:39 pm Post subject:

I get this after adding glibc to my 'debugging environment':

mike155 · Posted: Thu Jul 04, 2019 6:10 pm Post subject:

Your backtrace looks better now. :-)

Let's try to find the bug:

Open lshw with gdb:

Code:

gdb /usr/sbin/lshw
Set a breakpoint at the start of function load_pcidb()

Code:

break load_pcidb
Run the program until it reaches the function:

Code:

run
Advance to line "parse_pcidb(lines)". Enter
Code:

next
until you are at the line "parse_pcidb(lines)"
Run another "step". The program will probably crash, right?
Quit gdb and repeat steps 1-4
When you are at the line "parse_pcidb(lines)", use
Code:

step
instead of "next". "step" will enter the function.
Run "step" until you see something strange. I can't predict what you will see, but I'm sure that you will recognize the error when it occurs
Please post the output

Below is the output on my machine:

freke · Posted: Thu Jul 04, 2019 6:40 pm Post subject:

Crashed after first 'next'

freke · Posted: Thu Jul 04, 2019 7:05 pm Post subject:

Added my no-lto.conf and getting output more similar to yours:

freke · Posted: Thu Jul 04, 2019 7:32 pm Post subject:

Currently emerging gcc-8.3.0-r1 - but that's an all night project on this hardware.

Thanks for the help/guiding so far

mike155 · Posted: Fri Jul 05, 2019 1:05 am Post subject:

Your latest backtrace shows that lshw calls free() in glibc. And free() decides to throw an error:

freke · Posted: Fri Jul 05, 2019 4:48 am Post subject:

ltrace output - http://dpaste.com/2MN1241

Currently running ltrace without a filter, taking quite some time it seems - and /tmp/log currently at +100MB

[EDIT]
and ltrace output without filter - http://dpaste.com/1A50SE9
[/EDIT]

mike155 · Posted: Fri Jul 05, 2019 11:29 am Post subject:

Thanks for your ltrace data. I analyzed it and made some tests on my machine. The error happens in function "string hw::strip(const string & s)". The backtrace should look like

freke · Posted: Fri Jul 05, 2019 3:28 pm Post subject:

freke · Posted: Fri Jul 05, 2019 5:58 pm Post subject:

Just found out that lshw is actually working on two (or rather - now one) of my servers (identical hardware/configuration).

I tried re-emerging lshw on one - and now same error.....

I think it might be time to crawl through emerge-logs and compare/see what packages have been updated after lshw was last emerged on the remaining working server....

It seems like I was at gcc-8.1.0-r1 and glibc-2.27-r2 at the time of last updating lshw on that server.

The binary from my working server works fine when copied to the four others.

[EDIT]
Thinking about it - that's actually a bit worrying?

That means that the environment in some way is the problem?

Hu · Administrator Joined: 06 Mar 2007 Posts: 21861

Some program bugs are environment specific. For example, if only the systems where it fails have a PCI network card, and the bug is triggered by trying to describe PCI network cards, then the other systems would seem fine.

Have you checked whether valgrind can show you the origin of the error? Although it runs slowly, it often provides very good guidance.

freke · Posted: Sat Jul 06, 2019 12:03 pm Post subject:

Valgrind-output here - https://pastebin.com/45BxNPe8

freke · Posted: Sat Jul 06, 2019 2:17 pm Post subject:

A simple make of the source-code actually produces a working binary.

This means that I should take a closer look at flags in /etc/portage/make.conf?

freke · Posted: Sat Jul 06, 2019 3:14 pm Post subject:

Pinpointed!