Some days it’s a bad idea to ignore (or rather, just not check) email. And ignore weird stuff when you see it. Turns out for two days straight my server has been spewing out spam, after a user on the server had their password compromised. The server is a Ubuntu server with Postfix as the underlying mail transport.
The spam was in the form of a from address of xxx@domain.com, where xxx was a random string. The server doesn’t allow relaying, so to send mail from domain.com (hosted on my server) to random other domains (yahoo, gmail, etc) they’d have to be sending it as a user on my system.
To temporarily fix things, all the email that was in the queue got put on hold.
# postsuper -h ALL
This puts the mail on the back burner until you figure out what to do. The server won’t try to deliver it at all until you “un-hold” it. This had to be done a couple of times before I figured out where the spam was coming from. Thing is, what to do with 600,000+ emails sitting in the hold queue?
First thing was to figure out where the mail was coming from. I looked through the logs and it seemed like all the spam mail seemed to be sent through the same user.
Oct 27 10:01:14 amarok postfix/smtpd[25071]: C3E4FB1CEA3: client=unknown[46.253.82.98], sasl_method=LOGIN, sasl_username=bob
Ok, so looks like ‘bob’ got his email login compromised. Ok, now at least there’s a starting point. A bit of digging through the logs I found about where the spam started, and confirmed that by checking where the user was logging in from. Unless he could jump from Vancouver to Bulgaria in a minute, and then decided to send mail every second, it was pretty easy to figure it out.
Ok, so what now?
Next step, get a list of the IPs that the user was logging in from sorted and uniqued so I had each of the IPs used to spam.
# grep sasl_username=bob /var/log/mail.log | sort -u > iplist.txt
Now I tried a few different things to figure out if the IPs were real or not. I figure if they came from the Vancouver area they were probably legit, but if they were from Asia, Russia, or a host of other countries they were probably not valid. I used a few different methods to try to do an automated lookup of where the IPs were from, but the reverse lookup tools seem to be inconsistent at best for automated lookups.
In the end I basically used a network tool to do a lookup like this: http://networktools.nl/whois/5.57.75.82 where the IP I looked up sometimes was just the first number, i.e.: http://networktools.nl/whois/175.0.0.0. Honestly after a while I just deleted the IPs I knew were commonly used by Rogers and Shaw, and then deleted everything else:
root@server:/var/log# grep -wFf /root/iplist.txt m.log | cut -f 6 -d ' ' | cut -f 1 -d ':' | postsuper -d -
This greps for sources in iplist.txt, in m.log (which was a combination of mail.log mail.log.1 and mail.log.2 (the three days of log files I knew had relevant data). Any resulting log file messages are cut up until just the mail queue ID was left, and then that is piped into postsuper which deletes it.
After a few of those I was down to 5,000 messages in the queue, down from 600,000. Not bad. Still a few to deal with though.
# mailq | grep domain.com | awk '{ print $7 }' | sort -u
Now that’s a list of all the email addresses in the queue that are “bad” and need to be deleted. So using a great little tool for deleting postfix messages by to or from address I called pfdel, I did this to now run each of those bad emails through it and delete those messages from the queue:
# for i in `mailq | grep domain.com | awk '{ print $7 }' | sort -u` ; do pfdel.pl $i ; done
This took me down to 38 messages in the hold queue, which were easily looked at to see if they were legit (hint: if it was coming from a .ru or .br address, or a spammy looking domain, it got nuked).
So that’s it, short, sweet, and not the way I wanted to spend my Sunday night. Now dealing with removing my server from all the blacklists, that’s another issue… Ugh 🙁