[Solved] md5sums don't match-potentially compromised system

SpackmanChris · n00b Joined: 24 Mar 2019 Posts: 7

I had to send my computer to be repaired. I got it back yesterday, and, being slightly paranoid, I checked my system's important files' checksums with those of a backup made just before sending the computer to be repaired. Most of the md5sums match, but worryingly many files in /lib/security do NOT match.

I booted from an old gentoo livecd and mounted the internal drive and the usb backup drive separately under /mnt/. I ran md5sum on the files on the internal drive, saving the output to a file. I copied the file to the usb drive and checked the sums against the files there. Around 80 failed - not missing or a directory, but md5sum reported "Failed".

I did this for bin/ usr/ lib/ lib64/ and almost all of the other static directories (I didn't check proc or sys, for example). Only these files in /lib/security failed. Only using ext4, no unusual mount tricks or anything like that. Backups were created with rsync -avx to the usb drive. I usually check md5sums in this way a couple times a year. This is the first time any have failed.

The dates displayed by ls -l for the files in each /lib/security are identical, but the sizes are noticeably different.

I assume the computer is compromised, but how can I rule out the possibility that the backup is compromised? I suppose I could mount everything, chroot, and run equery --check, but I don't like the idea of even giving that much power to a potentially compromised system. (I've not booted into the gentoo system since I received the computer back.) Where does portage keep the data to compare against when equery --check is run? Would looking at that data (just in terminal or text editor) allow me to compare without having to chroot into a potentially compromised system?

Is there a good rescue tool that can scan for viruses / rootkits / etc?

Thanks for any help, pointers, or advice.

EDIT: turned out to be user error - my backup of /lib was waaaaay out of date due to the switch to 17.1 profile. Files in sbin, bin, etc, and other system dirs were 100% okay, so confident it was not a system compromise.
_________________
--

Chris Spackman
ESL Coordinator
Linux user since 1998
Linux user #137532

NeddySeagoon · Posted: Tue Oct 27, 2020 9:19 pm Post subject:

SpackmanChris,

There is no rescue for a compromised system.

You either rebuild it from scratch or reinstall from trusted backups.
Since you can't determine what an attacker has done, you can't clean it up.
There may be something obvious, that you are supposed to find to put your mind at rest and something you don't find.

Anyone with physical access to your system who knows a little gentoo could have updated it. Booting with a liveCD and chrooting will give them root access.

Boot a liveCD and look at the end of /var/log/emerge.log What and when is there?
Is there timestamps when the box was not in your possession?

What of /usr/portage/metadata/timestamp.chk. That's the date and timestamp of your ::gentoo repo,

Lastly, heres my /lib/security and md5sums.

All I have in /lib/security is

SpackmanChris · n00b Joined: 24 Mar 2019 Posts: 7

Thanks you for the info. Our sums don't match, but my Gentoo hasn't been updated in several weeks, so probably missed any recent updates. Also, would ours match, even if we had the same versions, if we compile with different settings?

The usr/portage/metadata and var/log/emerge.log are fine. they are both at the day that I shipped the computer off - I did an update and backup right before sending it. So, nothing suspicious there. Although, as you say, who knows?

I am confused now, though, because:

* on the computer, lib is its own dir, but it is full of 32-bit files (probably put there by the repair people because ???)
* on the computer, there is no lib32 dir
* on the backup, lib is a symlink to lib64
* on the backup, there is a lib32 dir

I installed Gentoo on this computer about a year and some months ago (and been using Gentoo for many years), but I think I've kept up with any layout changes that mattered. Are we supposed to have lib -> lib64 and a lib32 dir? If yes, I'm just going to restore from backup and stop worrying about it.

And, yeah, I was aware that md5sum is obsolete. I just started with that because I was in a hurry (even though it probably didn't actually save that much time). I didn't realize it was trivial to spoof it now, though.

Thanks for your help!
_________________
--

Chris Spackman
ESL Coordinator
Linux user since 1998
Linux user #137532

SpackmanChris · n00b Joined: 24 Mar 2019 Posts: 7

Okay, with a little more research, it seems that maybe the lib and lib64 dirs on the internal drive are correct and the backup is a bit mixed up?

If that is correct, it may be because rsync never got rid of the lib -> lib64 symlink? And in all the output of rsync, I could have missed it complaining about lib32 not existing?

I'm going to grep on the backup for the sha512 sums of the files in lib on the internal drive.

All other sha512sums for files from boot, bin, sbin, opt, etc, and most of usr matched (too many files!), so I'm thinking it is likely not compromised, but perhaps my error. If I find all the sha512sums from lib files in other places in lib / lib32 / lib 64 in the backup, it will be pretty well confirmed. If so, another example of needing to be careful and always check your backups.

Of course, this is assuming that the current Gentoo standard is lib as a real dir and lib64 as a real dir, and no lib32 anymore.
_________________
--

Chris Spackman
ESL Coordinator
Linux user since 1998
Linux user #137532

NeddySeagoon · Posted: Wed Oct 28, 2020 12:10 am Post subject:

SpackmanChris,

Different USE settings, different CFLAGS, different versions of gcc and other things will affect the md5sums.

Do not just change the /lib /lib64 whatever settings lightly.

The /17.0/ to /17.1/ profile change changes these settings. It also needs all your 32 bit code rebuilt.
Did you do the profile update after the backup?

Compare the output of

SpackmanChris · n00b Joined: 24 Mar 2019 Posts: 7

I've not done anything with the /lib and /lib64 on either the internal drive or the backup.

I did do the switch to the 17.1 profile and new lib layout - I don't remember exactly, but soon-ish after it came out.

I reread the news item - if I'm understanding correctly, my system should have only /lib (NOT a symlink) and /lib64. This makes me think those parts of the backup were not correctly updated after the 17.1 switch.

When I tested the backup, I usually just checked bin, boot, etc, sbin, maybe usr/sbin, so I could have missed the lib confusion.

So, I think the most likely explanation is that the lib part of the backup was messed up and there is no problem with the internal drive. If I can find the files from internal lib on the backup (but in the old / wrong location), that would be strong confirmation that nothing had been tampered with on the internal drive.

Thanks for your help.
_________________
--

Chris Spackman
ESL Coordinator
Linux user since 1998
Linux user #137532