Uncovering Unknown Malware On a cPanel Server

19 Oct 2017

About 4-5 years ago, I discovered a new Apache-based malware on a customer's server while investigating a customer's website issues. During my investigation, I checked /proc and found that the currently running httpd was removed from the disk:

[root@host ~]# ps ax | grep apache[2]
(...)
 5751 ?        Sl     0:01 /usr/sbin/apache2 -k start
 5752 ?        Sl     0:01 /usr/sbin/apache2 -k start
[root@host ~]# ls -al /proc/5751/
(...)
lrwxrwxrwx   1 root     root     0 Oct 20 00:28 exe -> /usr/sbin/apache2 (deleted)

Well, that's not normal. I copied the executable to disk right away to preserve it:

[root@host ~]# cp /proc/5751/exe apache2.deleted

Next, I ran strings against that binary and the binary that was currently at /usr/sbin/apache2:

[root@host ~]# strings apache2.deleted /usr/sbin/apache2 > strings-apache2

Finally, I looked for any strings that existed in one of the binaries but not the other:

[root@host ~]# sort strings-apache2 | uniq -c | sort -n

Any output starting with "1" indicated a string that existed in one of the binaries, but not the other. A handful of the unique strings made it immediately clear that the customer's server was rooted:

open_tty
hangout
ptsname
Qkkbal

That was sufficient to determine that something similar to the following occurred:

  1. The original Apache binary was renamed or copied.
  2. A trojaned Apache binary was put in place of the original binary.
  3. Apache was restarted, executing the trojaned binary.
  4. The trojaned binary was removed from the disk.
  5. The original binary was put back in its original place.

I added a check for this in the cPanel tech support's automated server troubleshooting utility SSP. See the check_for_cdorked_A() subroutine.

For more information about Linux/Cdorked.A, see the writeup from WeLiveSecurity: Linux/Cdorked.A: New Apache backdoor being used in the wild to serve Blackhole