STFN - Suspicious /tmp File Notifier

Detect. Eradicate. Patch.

In June 2005 I became responsible for administering Linux web hosting servers. It didn't take long to find that /tmp was a hotbed of activity for hackers, spammers, and bots. The malicious things found in /tmp typically fell into one of these categories:

The malicious files and directories placed there often had common properties regarding their names or their contents. What follows is a list of the ones I found in /tmp from Nov 2005 to Nov 2008, sorted by frequency. As you go through the list, see how many similarities you can find just based on the names. Note that the double quotes weren't part of the names; quotes were used to see when spaces were used at the end of the name, as in "/tmp/. "

"/tmp/. "
"/tmp/. /"
"/tmp/. /.mec/stealth"
"/tmp/. /.mec/pop3-login"
"/tmp/. /.mec/pico"
"/tmp/. /.mec/flood/xdestroy"
"/tmp/. /.mec/flood/udp"
"/tmp/. /.mec/flood/trash"
"/tmp/. /.mec/flood/synk7"
"/tmp/. /.mec/flood/synk"
"/tmp/. /.mec/flood/synhose"
"/tmp/. /.mec/flood/stream2"
"/tmp/. /.mec/flood/stealth"
"/tmp/. /.mec/flood/smurf6-linux+LPG.c"
"/tmp/. /.mec/flood/smack"
"/tmp/. /.mec/flood/sl"
"/tmp/. /.mec/flood/s"
"/tmp/. /.mec/flood/rc8"
"/tmp/. /.mec/flood/juno"
"/tmp/. /.mec/flood/"
"/tmp/. /.mec/flood/bloop"
"/tmp/. /.mec/flood/alpha"

STFN caught each of those in 30 seconds or less, and sent me an email immediately upon being detected. This fast notification time had several important advantages:

  1. The faster a threat is contained, the shorter the window of time for long lasting damage to occur. Damages including getting blacklisted due to spam, being DoS'd because of IRC, and being rooted and rm'd.
  2. A malicious file found in /tmp almost always meant that a customer's account had recently been compromised. As a result of the compromise, there were usually recent logs that explained how the account was compromised. The faster you are at investigating hacked accounts, the less logs you'll have to dig through. Wait too long and the log with the evidence could be rotated, removing crucial evidence (of course, one can log everything to a remote syslog server with a long log retention time).

Here's a list of what the script checked for:

Also check the first 30 characters of all files for the following:

Due to the often large number of session files, they were scanned less frequently. If I'd known about inotify(7) back then, I would've certainly used it. The advantages would've been real time scanning, and less of it.

Let's talk about why some of those file and directory properties are often used by malicious parties. First, files or directories whose names start with a dot. When listing the contents of a directory, many Linux users just run `ls` without any arguments:

[user@host /tmp]$ ls
[user@host /tmp]$

As shown above, `ls` doesn't show any output. However, `ls` intentionally doesn't display entries that begin with a dot. This information can be found in the documentation:

       -a, --all
              do not ignore entries starting with .

Let's run `ls` again, this time with the '-a' argument:

[user@host /tmp]$ ls -a
. ..

Now we know that by not using the '-a' argument, we can miss directory entries. I also recommend using the '-l' argument to provide a long listing, which also shows the user and group ownership, and the access permissions:

[user@host /tmp]$ ls -al
total 8
drwxrwxrwt 18 root root   4096 Oct 28 10:55 .
drwxr-xr-x 24 root root   4096 Oct 10 17:39 ..

Here are some of the emails I received from this script. Reverse shell:

  File: "/tmp/dc"
  Size: 880
Permissions: 0644
UID: ( 32343 )   GID: ( 847 )
Access time: Fri Nov 25 22:26:34 2005
Modify time: Fri Nov 25 22:26:23 2005
Change time: Fri Nov 25 22:26:23 2005

Signature: filestrings
Summary: this file appears to be compiled C code or that of a scripting language.

The first 30 characters are: #!/usr/bin/perl
# Data Cha0s

Another reverse shell:

Local server time: Thu Apr 27 22:17:34 2006

  File: "/tmp/a"
  Size: 416
Permissions: 0644
UID: ( 32864 )   GID: ( 1367 )
Access: Thu Apr 27 22:17:34 2006
Modify: Thu Apr 27 22:17:14 2006
Change: Thu Apr 27 22:17:14 2006

The first 30 characters of this file are:

use Socket;

If I remember correctly, when sending email via Horde webmail that contained an attachment, Horde wrote files in /tmp whose names started with "php" followed by a few random characters. Those were common, but not when they were ELF executables (kernel exploits and the like):

Local server time: Sat Feb 10 17:56:44 2007

  File: "/tmp/phpB1bDTd"
  Size: 5986
Permissions: 0600
UID: ( 32822 )   GID: ( 32822 )
Access: Sat Feb 10 17:56:44 2007
Modify: Sat Feb 10 17:56:44 2007
Change: Sat Feb 10 17:56:44 2007

The first 30 characters of this file are:

 ^?ELF^A^A^A         ^B ^C ^A   <A4><84>^D^H4

Attempt at creating a hidden directory:

Local server time: Fri Feb 16 14:38:38 2007

  File: "/tmp/. "
  Size: 4096
Permissions: 0755
UID: ( 2754 )    GID: ( 2753 )
Access: Fri Feb 16 14:38:35 2007
Modify: Fri Feb 16 14:38:37 2007
Change: Fri Feb 16 14:38:37 2007

This directory starts with a dot:

Path: "/tmp/. "

Kernel exploit:

Local server time: Wed Dec  5 14:38:05 2007

  File: "/tmp/local/1"
  Size: 6898
Permissions: 0755
UID: (32438)     GID: (32439)
Access: Tue Dec  4 23:52:46 2007
Modify: Fri Jul 14 08:37:34 2006
Change: Tue Dec  4 23:52:46 2007

This is a directory with a very short name:

Path: "/tmp/local/1"