Wow, today freaked me right out. Towards the end of the day I did a random run of UFies with chkrootkit, a program to check for rootkits, the Unix version of trojans or viruses (kinda, sorta, not really but a good enough explanation). To my horror I got back “5 processes hidden, possible LKM trojan”. I did a bit of searching on LKM trojans, and it’s a Loadable Kernel Module, which means that in theory, nothing in or out can be trusted. You can’t tell if your listing of files is actually showing you files, or the process list, or anything. So I dug around a bit more, trying to determine if there was a rootkit actually installed or not.
Eventually I came to the conclusion that there either was, and it was very well hidden except for the output of chkrootkit, which still gave me warnings, or there wasn’t, but I wasn’t convinced there wasn’t. Fred was nice enough to wander by the co-lo after work and booted up the system with a KNOPPIX CD and checked some stuff out. It didn’t have drivers for the extra PCI IDE card we had in, so we couldn’t check the data on the RAID arrays, but he did check the root partition and it reported that ifconfig() was infected. Shit.
Ok, for me that meant that there was enough doubt in my mind that to be sure the system was ok it had to be rebuilt from scratch. I’ve been wanting to do this for a while, with Debian unstable kicking me in the gnards a few too many times, and I wanted to move back to their Stable branch (or maybe Gentoo, which has been running fine on my server here at home.
So I did my duty as a sysadmin and posted a notice on the front page, on the mailing list, and sent a copy to the people I knew used the system as a shell to alert them that their data wasn’t safe, and not to use their passwords on the system, or use the system to jump to other computers, because there was the possibility that there were network sniffers doing all sorts of nasty, nasty things.
Not a long time after my now best friend Christophe pointed me to this mailing list message and this bug report, which basically say that this is a known issue with Debian Unstable + 2.4.2x kernels and chkrootkit (see links for info). So after checking and confirming to myself (again) that this indeed wasn’t a problem (not sure about the ifconfig file, I think it might be the knoppix version being silly, as the binary is identical to the original (checked offsite), I undid things. Re-emailed the list, re-changed the frontpage, re-emailed the list of people I know use the system, called Fred and asked him to cancel the request for entering the colo tomorrow night to do a re-install….
It kinda sucks to not get an excuse to do maintenance on the server, but on the other hand, I’m glad I’m not infected 🙂