Published 2004-06-08 21:09:11
One of my clients's servers managed to get rootkit'ed, not a happy situation. The first symptom was the fact the server stopped working, as the rootkit was far from perfect.
When I came in, it was unable to boot, as mount was segfaulting. It took quite a while to realize that it was not just a fsck error or something. eventually after looking at things like bash_history, it became clear that someone had got in and installed rk.tgz (or tried various rootkits).
The first evening was spent attempting to recover the existing system. This proved futile, as it appears that the segfaulting mount was a symtom of the rootkit, It appears to modify all the major commands, cp, ls etc. in such a way that any executable becomes infected. If you try and fix them, you usually end up running another infected file, which then infects all the ones you have fixed.. so at the end of the evening it became pretty clear that a clean install was required.
Day2 consisted of installing a new debian on a new hard disk (with the assumption that we would copy the old data from the infected drive at a later point.) This went pretty well, give or take the fudging around to find drivers for dell's rather odd mix of hardware.
However, the machine in question runs an POS application supplied by a third party, so after getting most of the new system going, i copied across the old application, and tried it out. BANG!!! - the whole system got re-infected.. (this is when i relalized that the rootkit seems to infect all running applications as well as the core utilities.)
Oh well.. back to the drawing board - remove bin,sbin and usr and re-install over again..
