O'Reilly    
 Published on O'Reilly (http://oreilly.com/)
 See this if you're having trouble printing code examples


Using Qpsmtpd

by Matt Sergeant
09/15/2005

The Dark Ages

When it comes to web programming, Perl developers are spoiled. They have an incredible array of choices for extending web-based content delivery, whether plain old CGI scripts, FastCGI and SpeedyCGI, or mod_perl and PerlIS. Beyond that, they have a large selection of frameworks with which to build their dynamic content, from Maypole and Catalyst to AxKit and OpenInteract.

The same is not true of email. Those who administer an email server more than likely have put up with the pain of adding dnsbl lookups to something like Sendmail or Qmail, or adding recipient validation for their custom user database in Postfix. Extending email servers is painful, and for the most part you can't do it easily in Perl.

Wouldn't it be nice if you could do something like mod_perl in a mail server?

Qpsmtpd, the mod_perl of Email

In 2001, Ask Bjørn Hansen needed to improve the spam detection on perl.org's mail server. He decided the option of patching the existing Qmail installation was too painful. Instead he turned to the code in Jim Winstead's Colobus NNTP server for inspiration to re-create qmail-smtpd in Perl. The first version he hacked together comprised just 300 lines of code.

Others soon started using qpsmtpd due to its malleability. In version 0.10, they refactored the code, factoring out some of the core subsystems and creating a flexible plugin system.

Related Reading

sendmail 8.13 Companion
By Bryan Costales, George Jansen, Claus Assmann, Gregory Shapiro

Ask is very coy about his project, but he proudly states that when apache.org switched from qmail to qpsmtpd, the server load actually dropped, simply because it was rejecting so much spam. As an example of the scalability of qpsmtpd, the apache.org mail servers currently process more than 2 million emails a day on their primary MX, rejecting more than 80 percent of it as junk--all thanks to the power and flexibility of qpsmtpd.

The goals of the Qpsmtpd project are:

It is important to note here that qpsmtpd is just the SMTP component of an email setup. You still need something to perform onward delivery--either to your end users, or to remote sites when you are relaying mail. Most setups use either qmail or postfix to do this, but any mail server will do.

The initial design of qpsmtpd was to talk to qmail for onward processing (hence the Q in the name); however, qpsmtpd speaks natively to qmail, postfix, or exim, and for other servers you can use SMTP to deliver mail. While this sounds like you have to maintain two systems, the additional capabilities of qpsmtpd make it worth it. The next section shows how to make qpsmtpd talk to your other email server.

Setting up qpsmtpd

You can run qpsmtpd in several ways, depending on how you want it to manage the connections:

Each possible configuration has its pros and cons, and you should evaluate your needs depending on your current setup. The example in this article uses qpsmtpd-forkserver, because this approach does not require any extra software.

Required Perl modules

The following Perl modules are required:

If you use a version of Perl older than 5.8.0, you will also need:

The easiest way to install modules from CPAN is with the CPAN shell. Run it with

$ perl -MCPAN -e shell
Getting it all running

After installing the prerequisite modules, getting qpsmtpd running is very simple. Download the tarball from the qpsmtpd web site and extract it in the location of your choosing. Then cd into the directory and type:

$ mv config.sample config
$ ./qpsmtpd-forkserver -u $USER

Congratulations! You now have qpsmtpd running on port 2525. Telnet in to see it running:

$ telnet localhost 2525
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 jester.sergeant.org ESMTP qpsmtpd 0.31-dev ready; send us
your mail, but not your spam.

Just type QUIT to end the session.

Running as a daemon

There are multiple ways to set up qpsmtpd as a daemon. The most common is to use Dan Bernstein's daemontools kit (a run file is included in the distribution); however, it is also quite easy to start by using rc scripts. I will leave running it as a daemon as an exercise for the user--people get very protective of their own particular way of doing this. See the read-me file in the distribution for details of how to do this for daemontools.

Replacing Your Current SMTP Server

The simplest (and least scary, for those who worry about those kinds of things) strategy for installing qpsmtpd is to change your current SMTP server to run on a different port, and simply have qpsmtpd forward to that SMTP server after its spam filtering. That way you get the best of all worlds--you keep your current stable email environment, and you get all the spam-filtering goodness that qpsmtpd brings.

First, create a qpsmtpd configuration file for the plugins that you want to load. In your qpsmtpd directory, edit the file config/plugins to contain the following lines:

count_unrecognized_commands 4
check_badmailfrom
check_badrcptto
check_spamhelo

# this plugin needs to run after all other "rcpt" plugins
rcpt_ok

# we'll turn this on later
#spamassassin

# deliver to localhost:2525
queue/smtp-forward localhost 2525

If you are not currently running Qmail, you will also need to create the file config/rcpthosts, containing a list of domains for which you currently receive email. Qmail users can relax, as qpsmtpd will read /var/qmail/control/rcpthosts by default.

# rcpthosts - who I receive email for
# put your domain name in here
example.com

Make sure you do a simple test of this new configuration as shown above.

Next, convince your current SMTP server (qmail, postfix, exim, or whatever you currently use) to run on a port 2525. I'll leave that up to you to work out, as every SMTP server configures this differently. If you get stuck, ask on the qpsmtpd mailing list for help.

To run qpsmtpd on port 25, pass that option on the command line:

$ ./qpsmtpd-forkserver -u $USER -p 25

That's all there is to it, so test it. The best testing tool for SMTP servers is swaks (though it needs the swaks TLS debugging patch if you intend to test TLS). To use it, try the following command:

$ swaks -t matt@sergeant.org -f matt@sergeant.org -h \
    foo -s localhost

If everything is successful, you should see:

<-  250 Queued! 
 -> QUIT
<-  221 myserver.example.com closing connection. Have
a wonderful day.
=== Connection closed by foreign host.

Now, if you have a working installation of SpamAssassin and a running spamd daemon, feel free to enable the SpamAssassin plugin in config/plugins and restart the server. By default this will just add headers to the email, but see perldoc plugins/spamassassin for details on how you can get it to reject mail over a certain threshold.

Useful Plugins

Qpsmtpd comes with a wealth of useful plugins. Rather than go into great detail on each, here's a synopsis of some of the more useful ones.

Writing Your Own Plugin

Making use of all the good stuff in qpsmtpd is all well and good, but sometimes you need to go the extra mile and write your own plugin. Rather than use a dummy example, I prefer to show how to create a plugin that watches for repeatedly denied IP addresses and locally blacklists them.

For a qpsmtpd plugin, you need two things:

The line in config/plugins is very simple; it just needs to contain the name (or relative path) of your plugin. Qpsmtpd will split anything after the name of the plugin on white space and pass it to the plugin's init() method.

The plugin itself is a file containing ordinary Perl. Save the following file as plugins/deny_repeat_offenders.

The plugin starts with some setup code. Plugins are Perl classes, but they don't need all the usual framework that you have to provide with a Perl class; qpsmtpd builds that for you. To initialize the objects of this class, you can optionally provide an init() method. I've used that here to set up some parameters to the hooks:

use NDBM_File;
use Fcntl;

sub init {
    my ($self, $qp, $filename, $threshold) = @_;
    
    tie my %h, 'NDBM_File', $filename, O_RDWR|O_CREAT, 0666
        or die "Unable to tie $filename: $!";
    $self->{dbm} = \%h;
    $self->{deny_threshold} = $threshold;
}

Here, the filename and threshold are the parameters from the line in config/plugins (shown toward the end of this section). I've used NDBM_File to store the data, but you can use any DBM system if you prefer something different. This example doesn't do any locking of the DBM file, but you should do so in a production environment, or use something that doesn't suffer from concurrent write problems such as an RDBMS.

The plugin needs to hook into the DENY phase, which gets called whenever a mail transaction is denied for any reason. There, it will increment a value in a DBM file corresponding to the IP address.

#!perl -w

sub hook_deny {
    my ($self, $transaction, $plugin, $level) = @_;
    
    # We're only interested in DENY or DENY_DISCONNECT
    unless ($level == DENY or $level == DENY_DISCONNECT) {
        return DECLINED;
    }
    
    return DECLINED if $plugin eq $self->plugin_name;
    
    # continued...

Several things are going on here: first, the name of the sub is hook_deny so that qpsmtpd knows to call it during the DENY phase. There's no other setup necessary for this sub to get called. Second, it collects the arguments, which include the plugin object itself, the transaction, a Qpsmtpd::Transaction object, the plugin that caused the DENY, and the actual code used.

For our purposes, the only interesting codes are DENY or DENY_DISCONNECT. The other reasons this might be called would be for DENYSOFT and DENYSOFT_DISCONNECT. Returning DECLINED gives other deny hooks the chance to be called.

Finally, it returns DECLINED if the DENY came from this plugin to avoid ending up in a feedback loop.

    my $ip = $self->connection->remote_ip;
    my $now = time;

    my $record = $self->{dbm}->{$ip};
    # Is this IP in the DB?
    if (!$record) {
        $self->{dbm}->{$ip} = pack("NN", 1, $now);
        return DECLINED;
    }
    
    my ($count, $tlast) = unpack("NN", $record);

    # Denied within the last 8 hours?
    if ($tlast < ($now - 28800)) {
        # Not denied in last 8 hours so just reset count.
        $self->{dbm}->{$ip} = pack("NN", 1, $now);
        return DECLINED;
    }
    
    # Now just update the count
    $self->{dbm}->{$ip} = pack("NN", $count+1, $now);
    
    return DECLINED;
}

The rest of this hook is quite simple--it stores some details in a DBM file about when it last saw a DENY from the IP address. If the IP hasn't been denied in the last 8 hours, it resets the counter.

With all that information stored, it's time to do something with it:

sub hook_connect {
    my ($self, $transaction) = @_;
    
    my $ip = $self->connection->remote_ip;
    my $record = $self->{dbm}->{$ip} || return DECLINED;
    my ($count, $tlast) = unpack("NN", $record);
    
    # Ignore and delete entry if not denied in last 12 hours
    if ($tlast < (time - 43200)) {
        delete $self->{dbm}->{$ip};
        return DECLINED;
    }
    
    if ($count >= $self->{deny_threshold}) {
        return DENYSOFT, "You are a repeat offender. Go away";
    }
    
    return DECLINED;
}

First, the code extracts the details from the DBM regarding that IP address. Then it makes sure that the IP address has been denied within the last 12 hours. (Otherwise, it's a stale entry in the DB and needs deleting.) Finally, if this IP has been denied too regularly recently, it issues a DENYSOFT. At connect time a DENYSOFT will disconnect the client with a 450 error, meaning that even if this is a false accusation, it will come back later to try to deliver the email. If this is legitimate email on a legitimate SMTP server, then the email will eventually get delivered--but spammers probably will not try again.

That's it. To add the plugin, edit your config/plugins file and put a line like this somewhere near the top. (The order of plugins determines when they run, and you want this one to run early.)

# deny repeat offenders for 12 hours after 5 bad attempts
deny_repeat_offenders ip_deny_db.dbm 5

Remember to restart qpsmtpd for this change to take effect.

Conclusions

Qpsmtpd is a wonderful, flexible mail server that is fantastic for trying out experiments in filtering mail. As classic examples of this experimentation philosophy, Qpsmtpd was the first SMTPD to have SPF support, the first to have URIBL support, and the first to have early talker checks shipped in core. All of these plugins were very simple to implement, taking up only a few lines of Perl, just like the example you see above.

Extending your mail server with Qpsmtpd is both easy and fun. To learn more, look through the plugins in the distribution, and ask questions on the mailing list. With Qpsmtpd, anything is possible, and your incoming mail will become usable once again.

Matt Sergeant is a leading figure in the mod_perl community, having contributed many open source modules and much documentation work.


Return to the Sysadmin DevCenter

Copyright © 2009 O'Reilly Media, Inc.