This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature: Enterprise Applications

Reducing spam with OpenBSD and spamd

By Terrell Prudé, Jr. on April 11, 2007 (8:00:00 AM)

Share    Print    Comments   

We all know about the rampant spam email problem. Nearly all of the potential solutions offered for it are based on the idea of the mail server receiving messages, classifying them as either spam or legitimate, and then processing further (deleting or forwarding messages) as appropriate. The problem with this strategy is that you end up using extra resources on the mail server. Here's a way to get the same result while minimizing resource usage by preventing the spam from reaching the mail server.
For this task you can use the OpenBSD platform, for two chief reasons:
  1. OpenBSD is secure and solid.
  2. OpenBSD comes with a tool to stop the spam before it even gets sent: spamd, a "fake" Simple Mail Transfer Protocol (SMTP) server that accepts SMTP connections and decides whether a sender is a spammer or not.
Spamd is not an email content analyzer; it does not actually examine a given email's payload. What spamd does is determine -- before the email is ever allowed to be sent in the first place -- whether the sender itself is a spammer or not. Spamd sits in front of your real mail server and listens for SMTP conversations. Since it operates at the SMTP level, it will work with any and all RFC 2821-compliant SMTP MTAs, including sendmail, Postfix, exim, qmail, and even Microsoft Exchange Server. It also is a good complement to email content analyzers like SpamAssassin.

One key method that spamd uses is a relatively new one called greylisting, which initially rejects all email with an SMTP error of 451, which means "temporary failure, please try again later." Properly behaving SMTP gateways will indeed try again later, in accordance with the rules of RFC 2821. However, virtually all spammer operations do not try again in order to maximize the amount of spam they spew out.

The greylisting technique returns a 451 error code to any mail sender that spamd doesn't yet know about, and will continue to return that error code to that same sender for a configurable time (25 minutes by default) based on three fields:
  1. the sender's IP address,
  2. the From: field in the email header, and
  3. the To: field in the email header.
This trio, called a tuple, gets put in quarantine, so if the sender does retry before the 25-minute quarantine is over, it will continue to get the 451 error. At the expiration of the quarantine, that sender gets put on a temporary "whitelist" lasting for several hours (four hours by default, including the original 25 minutes). Well-behaved MTAs that try again will be successful, and spamd will pass the email through to your real MTA.

Spamd can also handle known spammers by "tarpitting," or slowing down their connection in order to waste the spammer's time. When a known spammer tries to send you email, spamd will decreases the TCP window length to one to slow the connection down to one byte per second and will not let the connection go. If it gets caught in the tarpit, a spammer's MTA can take as long as 10 minutes to send one message. I regularly see spammers get tarpitted for between 400 and 550 seconds (6 1/2 minutes and just over 9 minutes).

What's worse (for spammers) is that spamd, after wasting all that time, never does allow the spam email through to the real SMTP server. Instead, it sends back a 450 "mailbox busy" message. The spammer retries, and retries, and retries, getting stuck in the tarpit every time. I had one spammer that kept retrying -- and repeatedly getting stuck in my trap -- for a day and a half, and never once was that spammer able to actually transmit the spam message to me.

Tarpitting can be implemented for senders on the same SPEWS/Spamhaus blacklist that you're likely using with a different antispam tool. Spamd's default configuration automatically tarpits the following IP addresses:
  • any IP netblocks in either the SPEWS Level 1 or SPEWS Level 2 lists
  • any IP netblocks in China
  • any IP netblocks in Korea
The reason for the SPEWS lists is obvious. China and Korea are blocked because so many spam email servers are located in those two countries, and their ISPs and governments don't seem to have any interest in getting rid of them. I also have added all of Russia's IP netblocks to my configuration, for the same reasons.

If you need to direct non-spammers who are in a blacklisted network past spamd, you can add their mail servers to a permanent whitelist that gets processed before any greylisting or blacklisting occurs. This allows their mail servers to bypass the greylisting and blacklisting functions and go straight to your real mail server.

There's one other handy thing that spamd can do for us. Spamd can optionally monitor your mail logs and automatically whitelist the destination email servers of anyone to whom you send email.

Another additional optional feature of greylisting with OpenBSD is something called greytrapping. Spammers "harvest" anything that looks like an email address from Web pages throughout the Internet, looking for potential victims. If you post a fake email address on your site that does not actually exist on your real email server, you'll know that if someone tries to send email to that fake email address, it's a spammer. Spamd checks the recipient in the SMTP "RCPT TO:" information against a list of fake recipient email addresses that you've previously told it to watch for. If it sees fakeaddress@mydomain.com, it immediately tarpits the mail server's IP address.

There's not much not to like in spamd. How do you get it to work? We'll tackle that tomorrow.

Share    Print    Comments   

Comments

on Reducing spam with OpenBSD and spamd

Note: Comments are owned by the poster. We are not responsible for their content.

But, Why?

Posted by: Anonymous Coward on April 11, 2007 10:30 PM
I'm here scratching my head wondering why I would want to use spamd when Postfix alone already offers me all of these capabilities. Postfix allows me to identify spammers using RBLs such Spamhaus and reject their connections at the SMTP handshake. Postfix allows me to fend off a fair bit of spam using greylisting. (A side note, spammers have adapted and greylisting is failing rapidly.) Postfix allows me to do tarpitting. Postfix allows me to do header checks and content inspection. Postfix allows me to do all this and then pass whatever else gets through on to Amavis-New and Spamassassin for final classification. This works with all mail systems including Microsoft Exchange and Novell GroupWise.

So, I'm scratching my head wondering why I would want to add another layer of complexity and use spamd when I already have all of its functionality in Postfix?

#

Re:But, Why?

Posted by: Anonymous Coward on April 12, 2007 12:49 AM
Gentooo all the way!!!!

im sorry this only remind me of my loving System..
GENTOO GENTOOO GENTOO.

For the record i think openbsd, and netbsd are the best bsd base system out there.

Gentoo Rocksss!!!

Also good job u both right, the owner of the article + the post before this one.

Remember there is always more than one way to do things.

#

Re:But, Why?

Posted by: Anonymous Coward on April 12, 2007 01:05 AM
spamd operates on protocol level and doesnt eats all your mailservers resources, thats why!

#

Re:But, Why?

Posted by: Anonymous Coward on April 12, 2007 08:35 AM
Postfix draws no more resources than spamd when performing those same functions. That was the point of the question. Postfix, the already running MTA, can perform all of the functions of spamd described in the article, and then some. It does it all without an more resource drain than spamd itself. Therefore, why run yet another process to eat memory, crash, get rooted etc.

When you are already using Postfix I seen no reason to use spamd and your response has done nothing to convince me otherwise.

#

If you don't like it, don't use it

Posted by: Anonymous Coward on April 12, 2007 01:15 PM
Simple. If you've got something that works for you, by all means, continue using it.

#

Re:But, Why?

Posted by: Anonymous Coward on April 13, 2007 02:29 PM
Common... spamd has not even the half of postfix's stupid buttons, many dimensions easier configuration, very less code and thus way more security.

#

Resource-usage reduction and security

Posted by: Anonymous Coward on April 12, 2007 01:52 AM
OpenBSD's spamd is better at it while using less resources. Additionally, to my knowledge, Postfix doesn't do greytrapping. OpenBSD spamd does. The article writer is correct, it reduces spam a lot, matter of fact, lower than Postfix does. Not knocking Postfix, I use it myself, way better than sendmail. But experience has taught me that an MTA should be an MTA, and a spam stopper should be a spam stopper.

#

Re:Resource-usage reduction and security

Posted by: Anonymous Coward on April 12, 2007 08:47 AM
Could you provide any information, preferably benchmarks, that show that spamd does the job better or does it using less resources? I'm not able to find any facts to support this assertion if Postfix is already running as your MTA.

Postfix does do greylisting. See postfix/examples/smtpd-policy/greylist.pl However, I would recommend the <a href="http://postgrey.schweikert.ch/" title="schweikert.ch">Postgrey</a schweikert.ch> policy server module instead. It is more refined and has some more functionality than the example that comes with postfix.

My lack of understanding with spamd is that if I am already running Postfix and it does its job as well as all of the functions of spamd(at least the ones described in the article), why do I need spamd at all?

#

Re:Resource-usage reduction and security

Posted by: Anonymous Coward on April 12, 2007 01:17 PM
Actually, he said "greytrapping", not "greylisting". The two are different. Does Postfix do greytrapping as well?

#

Re:Resource-usage reduction and security

Posted by: Anonymous Coward on April 13, 2007 02:12 AM
Nobody is forcing you to use spamd; its entirely your choice.

#

Re:Resource-usage reduction and security

Posted by: Anonymous Coward on April 13, 2007 12:42 PM
As far as I understand, since Greylist (spamd, postgrey and others) works during RCPT TO by presenting temporary and brief SMTP time out, the spammers won't be able to send their DATA payloads (the messages) and thus this is non-CPU and bandwidth punishing process since there are no messages and images needed scanned and deeply analyzed.

Greylist is useful for fighting ZOMBIES than fully relying with contents checks that eats up too many processes and putting heavy loads on our bandwidth. Since zombies were merely infected computers accross the internet, it is the BOTS that sends spam and for sure, these were to send only once due to the nature of these bots and hence won't (or seldom) retry after a failed connection attempt which is exactly opposite to a real SMTP server that would keep on retrying until it found out that the failure is permanent.

There is a contributed patch for postgrey to enable it to tarpit and further making it harder for the bots circumventing it in case, some of these bots now were rewritten to retry after a greylist, by adding a "—retry-cont=2 and "—auto-whitelist-delay=3600" startup options.

The creator of the patch as well combined S25R (Selective SMTP Rejection) that uses only simple regular expressions that matches with dynamic ADSL/cable subscribers (the ones being infected with bots or becoming zombies) hostname strings.

<a href="http://k2net.hakuba.jp/targrey/index.en.html" title="hakuba.jp">http://k2net.hakuba.jp/targrey/index.en.html</a hakuba.jp>

<a href="http://www.gabacho-net.jp/en/anti-spam/anti-spam-system.html" title="gabacho-net.jp">http://www.gabacho-net.jp/en/anti-spam/anti-spam-<nobr>s<wbr></nobr> ystem.html</a gabacho-net.jp>

#

Re:But, Why?

Posted by: Anonymous Coward on April 12, 2007 02:47 AM
Isn't it better to have more layers of spam protection / security than not?

#

Re:But, Why?

Posted by: Anonymous Coward on April 12, 2007 03:56 AM
Yup. It's called Defense in Depth.

#

Re:But, Why?

Posted by: Anonymous Coward on April 12, 2007 07:17 AM
I call it "lard." Adding more layers also adds complexity, sucks up system resources, and introduces more possible errors.

If only we could take the direct, effective route and just shoot spammers. Then we wouldn't have to continually re-invent bigger and better waders to deal with floods of spam.

#

Re:But, Why?

Posted by: Anonymous Coward on April 13, 2007 08:38 AM
Heh, if you want lard, you use MS Exchange.<nobr> <wbr></nobr>:-D

But on that note, though, if someone's using that (Exchange) on their network, then at least they'll get *some* kind of protection. Plus, they'll get a decent firewall out of the deal (see the part II of the article for how to do that).

I like your idea of dealing with the spammers.<nobr> <wbr></nobr>:-) But until that becomes legal (please, Mr. Bush, please?), we have a way to reject their crap.

#

logs

Posted by: Anonymous Coward on April 12, 2007 07:22 AM
"Spamd can optionally monitor your mail logs..."

spamd monitors the pflog, not the mail log

#

What is in the tuple?

Posted by: Anonymous Coward on April 12, 2007 01:25 PM
>...based on three fields:

> 1. the sender's IP address,

> 2. the From: field in the email header, and

> 3. the To: field in the email header.



According to

<a href="http://www.openbsd.org/cgi-bin/man.cgi?query=spamd&sektion=8" title="openbsd.org">http://www.openbsd.org/cgi-bin/man.cgi?query=spam<nobr>d<wbr></nobr> &sektion=8</a openbsd.org>

the tuple is based on "...connecting IP address, HELO/EHLO, envelope-from, and envelope-to, or tuple..."



So apparently the tuple isn't merely a triplet but includes the HELO/EHLO. But more importantly, it uses SMTP envelope From and To (MAIL FROM and RCPT TO) which is different than the cosmetic MIME From: and To: noted in the header.

#

Re:What is in the tuple?

Posted by: Anonymous Coward on April 12, 2007 07:55 PM
You are right if you are looking at a man page for 4.1 which has changed quite a few things in spamd.

The author was probably looking at 4.1.

The version shipping with 4.1 is much improved over the one that keeps me very happy on 4.0.

Maybe after I have had it in use for a couple of months I should write up my experiences compared with other "solutions".

#

Re:What is in the tuple?

Posted by: Anonymous Coward on April 19, 2007 07:32 AM
Hi, I'm Terrell, the author. I was using 4.0, actually. My language could've been a little more precise here, I agree.

The greytrapping is a phenomenal tool. I make quite effective use of greytrapping, using a few addresses. My logs show all sorts of spammers trying to hit those fake addresses...and they get tarpitted and rejected every time. Even though they have my real email address, it's likely to be in the same list as my fake "greytrap", a.k.a. "spamtrap" ones. So, if they send to any of my spamtrap addresses within the greylisting quarantine period, they will immediately get BLACKLISTED.<nobr> <wbr></nobr>:-) Yes, this means that even their spam to my real email address gets rejected in this case. My logs are showing results consistent with this functionality. In other words, it's workin' like a charm.

Thank you, OpenBSD team!

--TP

#

Some noteable points

Posted by: Anonymous Coward on April 13, 2007 04:30 AM
The name "spamd" is somewhat misleading, since the SpamAssassin daemon is called "spamd" since years.

The article says that greylisting works for addresses on SPEWS. 1) SPEWS hasn't been updated since at least August 2006; 2) Greylisting is most effective when applied to a wider range of sources.

Further, the article states that greylisting is something "rather new". Postfix eg has it since at least somewhere in 2003 (<a href="http://groups.google.ch/group/mailing.postfix.users/browse_thread/thread/275ff7c4261529c0/652da5fda7e249da?lnk=st&q=greylisting&rnum=897&hl=en#652da5fda7e249da" title="google.ch">http://groups.google.ch/group/mailing.postfix.us<nobr>e<wbr></nobr> rs/browse_thread/thread/275ff7c4261529c0/652da5fd<nobr>a<wbr></nobr> 7e249da?lnk=st&q=greylisting&rnum=897&hl=en#652da<nobr>5<wbr></nobr> fda7e249da</a google.ch>)

#

Re:Some noteable points

Posted by: Anonymous Coward on April 13, 2007 08:33 AM
Greylisting's newer than several other other methods, for example, Bayesian filtering, keyword filtering, and other types of content-based filtering. The grey-TRAPping is apparently quite new, I hadn't heard of that before.

Yeah, SpamAssassin's been called "spamd" for a while, true. But that's what the OpenBSD team chose to call their spamd, so I guess that's why he called his article that.

#

Re:Some noteable points

Posted by: Anonymous Coward on April 20, 2007 04:30 AM
The first paper I found which discussed greylisting in any significant detail is Evan Harris's 2003 paper, available at <a href="http://www.greylisting.org/" title="greylisting.org">http://www.greylisting.org/</a greylisting.org>. Perhaps that's why Postfix had it in that same year. OpenBSD added greylisting support to its own spamd in early 2004.

The article didn't say that greylisting works for addresses on SPEWS. The article said that addresses in SPEWS get BLACKlisted, not greylisted. I think the article's pretty clear that greylisting is to be applied to everybody who isn't "known good" just yet.

It's great if Postfix supports this, too. The more widespread use that greylisting gets, the better.

#

Link to "Installing and configuring spamd"

Posted by: Anonymous Coward on April 13, 2007 05:03 AM
<a href="http://applications.linux.com/article.pl?sid=07/03/28/1631206&tid=115" title="linux.com">http://applications.linux.com/article.pl?sid=07/0<nobr>3<wbr></nobr> /28/1631206&tid=115</a linux.com>

#

shit from your mouth

Posted by: Anonymous Coward on April 14, 2007 08:01 AM
China and Korea are blocked because so many spam email servers are located in those two countries, and their ISPs and governments don't seem to have any interest in getting rid of them.

#

Nope, our logs back it up

Posted by: Anonymous Coward on April 19, 2007 12:08 AM
Wow, what language!

You're obviously either Chinese or Korean. Perhaps you should read this, from OpenBSD's Web page:

<a href="http://www.openbsd.org/spamd/" title="openbsd.org">http://www.openbsd.org/spamd/</a openbsd.org>

The OpenBSD default is to blacklist China and Korea. My company use spamd too, and our logs show that the majority of the spam does come from those two countries. Maybe one day that won't be true, but today, it is. Maybe you should talk to the OpenBSD team about it, since they're the ones who made it the default.

#

Re: Nope, our logs back it up

Posted by: Anonymous [ip: 209.161.34.4] on November 11, 2007 12:12 AM
Hell, I was blocking China and N. Korea long before I started using spamd. Pretty standard operating procedure amongst MANY professional sysadmins I am acquainted with.

One more comment for all the Postfix fanboys, I have been using Postfix since before it was called Postfix and imho spamd rocks.

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya