Gentoo 1.4 Webserver got rooted tonight.

chmod

I am running a Gentoo 1.4 box on a dual Xeon 2.4Ghz Intel board. I installed 1.4RC2 on it and have kept it updated by emerging "-u world" every few days.

Tonight I went to a site of mine, to find the index page had been moved with a new one in it's place. Also the intruder kindly removed the /var/log directory, and my last year or so of logs.

Where do I start even looking to find out who did this? Is there a likely hole in gentoo because of this? I'm no security buff, but I have kept openssh updated, and have no other users of this machine.

Can someone help me start hunting for what caused this? I'm not as much interested in catching the cracker as I am in preventing this from happening again.

SouthOfHeaven · Tux's lil' helper Joined: 05 Apr 2003 Posts: 128

sorry, out of curiosity why would you "emerge -u world" every few days ??

iwasbiggs · Apprentice Joined: 17 Jan 2003 Posts: 203

Horrible, too bad you coudlnt' read the logs.
_________________
www.ruinedsoft.com
Freeware development.

chmod · Posted: Thu May 22, 2003 8:02 am Post subject:

antik · Apprentice Joined: 01 Oct 2002 Posts: 212

iwasbiggs · Apprentice Joined: 17 Jan 2003 Posts: 203

chmod

Tuna · Guru Joined: 19 Jul 2002 Posts: 485 Location: Berlin

cat .bash_history ? ok if hes smart enough to delete the logs he proppably deleted that one also.. just to remember.. better not using that maschine in that state any longer on the net.. you never know what he changed/installed to get easy access anytime again.

chmod · Posted: Thu May 22, 2003 8:21 am Post subject:

Koon · Retired Dev Joined: 10 Dec 2002 Posts: 518

Odd.

What services are visible (exposed through your firewall) ?
Is everything installed from emerge packages or did you install binaries manually ?

-K

uxbod · Guru Joined: 30 Dec 2002 Posts: 474 Location: UK

Were you using Apache? If so what version?

gigel · Guru Joined: 14 Jan 2003 Posts: 370 Location: .se/.ro

you can call/contact your isp to tell who was logged/accesing into/your machine the day it was *rooted*

backup logs is not the issue here...unless you put them on a read-only media
putting a cron job to cp /var/log/* into a different place can fool a stupid cracker but i think it's not a good security option either...

on a server the most important things are the kernel and firewall configuration and the services that are allowed..

it's not a gentoo fault cause youve been cracked...

now post what services were u running in that day??
and also the version numbers..

to investigate more,you might annalize what has he modified into the kernel(cause i might think he has modified it) comparing it with a new one taken from kernel.org ...that way you might find something usefull...
try looking what processes are running right now,try looking what open ports have you now...try scanning your machine (nessus and nmap)

:cry:

...i just feel sad..that all...
//edit
AFAIK when you delete a file you are actually deleting the inode,so the basic file is still there as long as it's not overwritten by some other files(usualy this happens when you move files,or full the harddisk)
so i know there must be a program that restores inodes...try looking for one..and maybe(with luck) you can restore your logs and see who was messing with your machine.....
_________________
$emerge sux

elykyllek · Posted: Thu May 22, 2003 10:35 am Post subject: root kit?

could you run chkrootkit and see if you were rooted by a known rootkit?

chmod · Posted: Thu May 22, 2003 11:56 am Post subject: Re: root kit?

Roc · n00b Joined: 19 Jul 2002 Posts: 35

What kernel are you using? Is it possibile that someome used the ptrace bug to break into your system? This maybe can also be done by uploading a PHP script containing shell commands.

ARC2300 · Apprentice Joined: 30 Mar 2003 Posts: 267

And I'm sure you know, but your root password is secure, long, and very hard to crack??

I'm always afraid of someone doing this crap to me, but, then again, if they can get my password(s) and figure it out, good for them.

Sorry to hear about your luck.

And just out of curiousity (I don't really know if it's possible), did you chroot your FTP daemon?? I've heard of people breaking into systems via FTP because it's so easily crashable.

antik · Apprentice Joined: 01 Oct 2002 Posts: 212

Koon · Retired Dev Joined: 10 Dec 2002 Posts: 518

rajl · Apprentice Joined: 25 Sep 2002 Posts: 287

I followed that link and looked at the script, but I couldn't tell what it was exploiting (besides what seemed to be a mailserver). What service/package does it target, and how would I make sure that my own system is secure against it?
_________________
-Rajl

-----------------------------------------------------------
It's easy to be brave once you consider the alternatives.

paul138 · Guru Joined: 09 Aug 2002 Posts: 370 Location: Ottawa, ON

All you can really do now is format the thing. It's not a good idea to use a comprimised box even if you think it's clean. It's a learning experience.

If they didnt install a rootkit then they must have just came in to install a DDoS client of some sort (hence the port 44999 listening). The idea being that they can use your box in a distributed denial of service attack.

Some things to do next time around:

Run only stable. Don't use the ACCEPT="~x86" (if you were). Sometimes the experimental code is full of holes.

Run only the necessary software. Gentoo is good about not setting a lot of default services on startup like the other distros. Only install/start the minimum number of programs.

Daemons like ftp are notorious for being full of holes. Before you use an ftp daemon on the machine, read all of the docs especially the ones pertaining to security.

Do not install X windows on a server. This has proved fatal many times (especially where web servers are concerned, it is possible to start an xterm on a remote system with the right peice of buggy code and netcat).

If possible, remove gcc alltogether. Keep a mirror of the system at your office and build the updates as packages at the office then scp them (or take them on a CDR) to the remote machine. Use emerge -k [package] to install it, no gcc needed.

Eliminate all non-vital users from passwd and group files.

Lock down ssh as to only use RSA keys for root (if you must use remote root accounts), or deny root login all together. There are MANY ssh options to set up.

Install Tripwire or one of the variants (www.tripwire.org - why is there no ebuild for this yet?)

Use a nice firewalling system. I recommend Shorewall, it's worked for me for over 2 years now. It is possible to firewall a system with only 1 NIC.

Double-up on IP access restrictions to services using tcp_wrappers along with iptables.

There are so many other things, just keep it simple and locked down.

It's sad to see a box get comprimised. But 2nd time's always a charm ;-)

GOOD LUCK!
-P
_________________
Talk is cheap because supply always exceeds demand.

paul138 · Guru Joined: 09 Aug 2002 Posts: 370 Location: Ottawa, ON

SouthOfHeaven · Tux's lil' helper Joined: 05 Apr 2003 Posts: 128

i agree with paul138, they probably just borke in to install a DDoS client, to me it doesnt make much sense, what exactly do you have on the server worth stealing ?? credit card numbers ? email adresses ? and if the hacker/cracker/***scriptkiddie*** was after something then why delete the logs ??

Anyway i dont have much experience with this im just throwing in my 2 cents. I would suggest something like tripwire and snort.

Roc · n00b Joined: 19 Jul 2002 Posts: 35

Are you also sure that noone has physical access to the machine? Can you trust your ISP where it is located?

I don't know if it's a good idea, but maybe an experienced user will be able to find out the vulnerability if you reconnect the machine for a limited period of time, post its IP and keep a strong eye on it. It's a risk, but I would like to know what has happened to keep it out next time.

paul138 · Guru Joined: 09 Aug 2002 Posts: 370 Location: Ottawa, ON

That wouldnt make much of a difference. When the daemon is installed for distributed exploits, the person who installed it usually distributes your IP as being available for use to any number of other persons.

Simply waiting for someone to connect to the daemon prob wont get your cracker, maybe only one of his pal's pal's pal.

When I came to work where I am now, there was an Oracle server on the network running Red Hat 6.0. I logged in, poked around the logs etc and happened to notice a ddos daemon running. Comprimised? Yes, syslog was not working, ps showed false processes (masked the daemon) and, a mass of other problems. The box was sitting connected to the Internet without anything to stop the nasties from getting in. When I asked how long it had been comprimised they responded "There is something wrong with it?"

Anyway...a little humor.

Wipe out the machine, start over new. :twisted:

<- I love that emoticon
_________________
Talk is cheap because supply always exceeds demand.

Koon · Retired Dev Joined: 10 Dec 2002 Posts: 518

I think chmod is not looking for advice on what to do now he is rooted, but rather is concerned by the fact he was running a pretty simple Gentoo setup, with regular security updates, and thought he would be protected.

There are three possible causes :

1- Unknown vuln in the wild !
That's the worst case scenario, that would mean any of us can be targeted too. But given the low hacker profile (why on earth would you deface a website *and* install a DDOS client ? DDOS client must keep a low profile to stay for a long time) I think case 2 or 3 are most likely

2- Known vuln in kernel/software used
Surely chmod didn't post his entire config, and maybe he installed from source/binary a package on which a vuln exist, that was not updated by portage, and a root-script exploits it.

3- Hole in config or PHP programs or weak password...
Like maybe something has been done in the FTP config (or others, like running ExtreMail under root priviledges, I don't know) which allowed for the intruder to gain access. If it's not a really common error, it may mean a medium/high-profile hacker, which contradicts the point I made in (1)

As a conclusion, we sure hope there is no unknown vuln in the wild in a common package, but we cannot help you without an intimate knowledge of everything that was done and installed on your machine, the physical context around it, etc...

Good luck !

-K