Lessons Learned - Defense

10 Nov 2017

Know your environment.

It is much easier to know when something is out of the ordinary when you understand what "ordinary" is. Is your server suddenly listening on TCP port 4444? Did the permissions on /etc/passwd just change? Did the packets per second outbound just increase above an average amount? Is the Exim queue suddenly growing in size? Is there network traffic taking place between two hosts where both ports are >1024 (and not related to a passive FTP session)?

You will always be a step behind. Know how to respond when something bad happens.

It is difficult to protect against unknown threats. At some point a customer's account will get compromised. What's your plan of action when this happens? You'll need to check the process list (ps, lsof, netstat, /proc), check for files owned by the user in world writable locations such as /tmp, check which items in their $HOME directory have been modified in the last 24 hours, check their $HOME/.lastlogin file (if cPanel), check the control panel login, error, and other logs for their username, check their domain logs, check the server's syslogs, notify the customer (or the account owner, if a reseller), etc etc.

Stay on top of vendor and security mailing lists, exploit databases, etc.

You can potentially mitigate threats quickly by staying on top of security resources. Get on mailing lists, RSS feeds, and so on. The faster you know about a vulnerability, the faster you can take actions to patch, notify vulnerable customers, create mod_sec rules, and so on.

Understand your security tools.

Know how to block hosts with iptables. Learn to use mod_sec.

Give vendors access for a limited time, and no direct root login.

If you have to give a third party access to your server, such as via ssh, make an account for them with a unique password. Be sure to remove the account when they are done. This sensible practice alone would have saved numerous business owners from being hacked during the cPanel ebury mess.

Monitor your environment. Monitor high value tagets.

Set up a Nagios instance to monitor all services on all servers. It's free. Also, monitor /root/.accesshash and /var/cpanel/resellers (if using cPanel). Would you know if root's access hash was accessed inappropriately, or unexpectedly created? If a reseller was given root access?

Learn to code.

I used bash scripting and perl to automate countless tasks ranging from pushing past Yahoo's greylisting so that our customers' email was delivered in a timely manner, to alerting me of anomalous server behavior, and much, much more. A little bit of perl goes a very long way. I made mistakes in the beginning, including ones with serious security implications. However, I preferred this over publicly available third party solutions that contained features we didn't need, leading to an increased attack surface.