This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature: Mail & Messaging

How to build a local IMAP server

By Keith Fieldhouse on September 09, 2008 (9:00:00 AM)

Share    Print    Comments   

The usual practice of configuring your email client to retrieve email from your ISP's servers works well, but not for all situations. Suppose you add a laptop as a compliment to your desktop machine, or you'd occasionally like to use your spouse's computer to read your email -- you can run into problems trying to keep all of your email clients in sync. You can use IMAP (Internet Mail Access Protocol) instead of POP3 (Post Office Protocol), but then you need to store all of your email on your ISP's servers indefinitely, which has its own drawbacks. Here's a way you can set up a single machine on your own network to fetch and store your email and serve it to any number of email clients.

We're not talking about setting up a full-blown Internet email server. A "real" Internet mail server is public server identified as the mail host for a particular domain. When other servers have email for that domain they contact the mail server and hand off the email. The mail server then distributes that email to the appropriate users. We'll leave the responsibility of securely managing a public email server to our ISP and configure our local email server to periodically retrieve our email from our ISP's servers. This way, our server does not need to be directly exposed to the Internet at all.

For our example, we'll use as a base a system running the latest version (Hardy Heron) of Ubuntu Server, but the principles and software described here are applicable to any Linux distribution.

The install of Ubuntu Server can, for the most part, follow the defaults. When you get to the Software Selection screen, select Mail Server to install some of the email software you'll require automatically -- in particular, Postfix, a program that handles the distribution of mail on the server. Because you selected Mail Server you'll be asked additional configuration questions as the installation proceeds. In particular, you'll be asked to enter a fully qualified domain name (FQDN) to be used in the event an email address doesn't have a domain name associated with it. This won't used in this configuration; you can enter the domain name of your most often used email address.

You'll also be asked to enter an SMTP (Simple Mail Transport Protocol) relay host. For now, enter the SMTP host that you use in your regular email client -- typically a server on your ISP's network.

If you're using another Linux distribution to build this server, just make sure that mail server software is installed. Postfix, qmail, or even sendmail would all work equally well in the configuration described here.

Fetching the mail

Since we're not connecting this server directly to the Internet and not being contacted directly by other mail servers, we'll need some way to obtain our email. Fetchmail is designed for just this purpose.

Fetchmail functions in two parts. One acts like an email client, retrieving email for specific accounts using the POP3 or IMAP protocol. Rather than storing these messages and displaying them, as an email client like Thunderbird would, the second part of Fetchmail takes over. It connects to the MTA (Mail Transport Agent) on the server and provides it with email messages, just as if the server had been contacted by an external mail server with messages. Then the local mail handling machinery on the server takes over.

Fetchmail is not installed by default on Ubuntu Server, but you can install it easily using sudo apt-get install fetchmail. To configure the software, you must enter the email addresses that you plan to "fetch" and the users that mail is for. If you're going to serve email to more than the one user you created, you should create the other accounts on the server now with the adduser command. Users' email addresses and their account names don't have to match (although that may help keep them organized). The usernames and passwords that you create at this time will be the ones that you use when you configure email clients later.

Fetchmail has a built-in "daemon" mode where it runs in the background and periodically checks for messages as specified in a configuration file. On Ubuntu Server you can arrange for Fetchmail to be started at boot time by editing the file /etc/default/fetchmail and changing the line that reads START_DAEMON=no to read START_DAEMON=yes.

To test the configuration before installing it, create a file named fetchmailrc in a convenient working directory and populate it with lines for each user:

poll pop.example.com proto pop3 user saul with password competent is saul here poll pop.example.com proto pop3 user fred with password stolid is fred here poll pop.example.com proto pop3 user orrie with password slick is orrie here

Fetchmail's configuration language is chatty but fairly easy to read. The lines above poll pop.example.com using the POP3 protocol to fetch the mail for the users saul, fred, and orrie on the remote server and place that mail in the inboxes for the corresponding users on our local mail server. It would have been just as valid to fetch mail from saul, fred, and orrie on the remote server and place all of that email into a single user's inbox on the local machine. The poll commands shown are fairly generic. The fetchmail documentation details a large number of additional options. One of the most common would be to add options ssl in order to use SSL for those servers (such as Gmail's POP server) that require or provide it.

To test your fetchmailrc configuration, run the command fetchmail -v -f fetchmailrc. This will run Fetchmail against your configuration file and print out a lot of information about how things are proceeding, including whether there are any problems. When you are satisfied with the way things are going, install the fetchmailrc file to /etc/fetchmailrc and start the Fetchmail daemon by issuing the command sudo /etc/inid.d/fetcmail start.

Serving the mail

Once you've retrieved the mail onto the server, you need to provide access to it. We'll do that using the Dovecot email server and the IMAP protocol. Dovecot is easy to configure, and is pre-installed when when you choose Mail Server during the Ubuntu installation. If you don't already have it on your server box, you can easily install it from your distributions package repository.

We only need to make one configuration change to the default Dovecot installation. Edit the file /etc/dovecot/dovecot.conf and find the section that describes the mail_location variable. Add a line to the file that reads:

mail_location=mbox:~/Mail:INBOX=/var/mail/%u

This tells Dovecot several things. First, mail will be stored in mbox format (a venerable and straightforward mail storage format). Second, the directory Mail in the account's home directory should be used to hold any IMAP folders the user creates with an email client. Finally, incoming mail is kept in an mbox file named after the user's account (%u) name in the /var/mail directory.

Once you've made this change, restart the Dovecot server with the command /etc/init.d/dovecot restart.

At this point you should have functioning mail server. To test it, start your favorite email client, add an account, and configure it to point to your server. By default, Dovecot is configured to listen to external IP addresses only on port 993, which is assigned to IMAP + SSL (also known as IMAPS), so specify that protocol when configuring your client. Use one of the accounts and passwords that you created on the server.

Processing the mail

Once you have connected your client to the Dovecot server, you may decide that you'd like to do some processing of the incoming mail on the server rather than in the client. This insures that such processing is done regardless of which client you happen to be connecting with.

You can do a lot of server-side processing with a program called procmail. Procmail has a pattern-based configuration language that allows you to process incoming email in just about any way you can imagine. As an example, we'll use procmail to filter out spam using another program called SpamAssassin.

The default server installation doesn't install SpamAssassin, but you can install it with apt-get. To activate it, copy the file /usr/share/doc/spamassassin/procmail.example to ~/.procmailrc for each of the users for which you want to run the filter. After you've copied the file, add the following line to the top of it:

MAILDIR=$HOME/Mail

With this setting, procmail changes to the directory specified so that further folder and file references are relative to that directory.

The supplied .procmailrc will move spam into two different folders, almost-certainly-spam and probably-spam, depending on how likely SpamAssassin considers it to be spam. You can add additional rules to the .procmailrc file to do other processing. For example, the following rule will move any mail that was sent to an address with "mail-list" in the "To:" field to the folder Mail-List in the Mail directory.

:0: * O_mail-list Mail-List

Procmail's configuration language is powerful and complex. The procmail homepage contains links to a variety of Web sites with examples of useful procmail "recipes."

That's all it takes to build your own local mail infrastructure. The server is powerful and flexible and capable of dealing with the email needs of small and medium-sized groups. Once set up, such a server requires little maintenance beyond making sure that /var/mail and the individual users' ~/Mail directories are appropriately backed up. Each of the components we used is time-tested and configurable enough to be adaptable to an organization's specific requirements. With a little effort up front, you'll have an email system that will serve your needs for a long time.

Share    Print    Comments   

Comments

on How to build a local IMAP server

Note: Comments are owned by the poster. We are not responsible for their content.

Saul, Fred and Orrie?

Posted by: Anonymous [ip: 217.67.37.208] on September 09, 2008 09:17 AM
I got your reference to Rex Stout's books and have to admit that I am a fan of them, too! :-)

#

Re: Saul, Fred and Orrie?

Posted by: kfieldho on September 09, 2008 01:16 PM
I wondered if there's any overlap between linux.com readers and Rex Stout readers. Apparently there are at least two :-)

#

How to build a local IMAP server

Posted by: PerlCoder on September 09, 2008 11:32 AM
An interesting idea. I can see how that might work for some people.

It would never work for me, though. I've got to have access to my e-mail /anywhere/ that I happen to be, so I just go with IMAP from the start. I don't really see any drawbacks to storing my e-mail on my domain server (I have e-mail accounts that came with my web hosting package).

If I really did want my e-mail to be stored on my own computer, I'd go ahead and get a static ip and set up a "full-fledged" mail server, and connect to it through IMAP. Is it really that hard to secure a mail server?

#

Re: How to build a local IMAP server

Posted by: kfieldho on September 09, 2008 01:27 PM
No, it's not that hard to secure a mail server (though the proliferation of spam that takes advantage of mis-configured servers indicates that people do get it wrong often enough).

In my case, I try to minimize exposure of services to the Internet on general principles. Since I already need to expose SSH so that I can access my network, I concentrate on securing that, and use the fact that ssh works quite nicely as a SOCKS proxy (see ssh -D) for Thunderbird and access my internal IMAP server that way. Alternatively, you could expose the IMAPS (IMAP + SSL) port and access the IMAP server directly.

#

Re(1): How to build a local IMAP server

Posted by: Anonymous [ip: 209.221.240.193] on September 09, 2008 02:24 PM
The part I've been stuggling with is sending/replying- I can pull from several POP3 accounts (different domains) into their respective folders, but then I want to be able to send from those respective accounts as well. Sending everything from one account doesn't really work, since spam filters tend to not like it when the from field and sending server don't match.

#

Re(2): How to build a local IMAP server

Posted by: Anonymous [ip: 68.172.123.100] on September 09, 2008 05:18 PM
I deal with this in two ways. First, I set up a set of identities in Thunderbird (look at "Manage Identities" in "Account Settings), then I can select whichever "From" is correct/convenient. The problem with this that I haven't found a way to automatically select an identity based on the the message I'm replying to (though I haven't looked too hard recently).

So, I also set up multiple accounts that I use on my internal IMAP server. Then I configure Thunderbird with multiple accounts pointing to the internal server (with the correct, associated e-mail address).

#

How to build a local IMAP server

Posted by: Anonymous [ip: 100.4.3.1] on September 09, 2008 04:28 PM
Hello! A quick thing, why do you use mbox instead of Maildir? There are huge advantages using Maildir instead of mbox. Because Maildir has a file per message (instead of all messages in a single large file like in mbox), you have a big advantage when it comes to large mail-bases. I use a local IMAP setup on my laptop as well, but since I cart around more than 12,000 messages (yup, I know I'm crazy) in their various folders, things are extremely slow with mbox. Maildir does not have the speed issues at all. Furthermore, Dovecot actually expects the Maildir format as it's default configuration, although it has been written well enough to auto-sense what's going on. Great piece of software! I migrated over to Dovecot from the Courier IMAP server, and I am very happy with it's performance.

Just as an aside, here is what I use:

(1) getmail instead of fetchmail (I don't like what fetchmail does to mail headers)
(2) Procmail to filter everything through into their respective mail folders (set up to do Maildir as opposed to mbox)
(3) Dovecot as IMAP
(4) Various IMAP clients (mostly Thunderbird, sometimes Kmail, Evolution, Claws or whatever I'm testing at a particular point in time).

Furthermore I use Google Desktop to index the lot through Thunderbird. The whole shebang makes a comfortable way to deal with all my multitudes of sins (mainly log messages from the various systems I maintain).

#

Re: How to build a local IMAP server

Posted by: Anonymous [ip: 68.172.123.100] on September 09, 2008 05:21 PM
Re: Maildir vs. Mbox. I'm afraid the answer is simple habit. I've been fidgeting with e-mail setups for a long time now and I've always happened to use Mbox format. You're comments regarding the advantages of Maildir are well taken and worth investigating for anyone considering a setup like this.

#

How to build a local IMAP server

Posted by: Anonymous [ip: 70.91.34.61] on September 09, 2008 05:55 PM
This is a good article, but it's a little confusing for me. I get what fetchmail does, and I get (and really like) the concept of getting the mail from the ISP and then serving it locally via IMAP.

What I don't understand is how Postfix, Procmail, and Dovecot interact. Could you expand on that a little more?

Thanks.

#

Re: How to build a local IMAP server

Posted by: kfieldho on September 09, 2008 11:53 PM
Postfix is a Mail Transport Agent (MTA) (qmail and sendmail are two other popular ones). It's primary responsibility is to accept mail (typically from other other servers) and make sure it goes where it belongs. Most commonly this means that mail is distributed to different user accounts on the system Postfix is running on. On Ubuntu (and most other distributions) Postfix (or whatever MTA the distribution uses) is configured to run Procmail whenever a piece of email arrives. Procmail is essentially a mail filtering program that processes an incoming email according to a set of rules a user can define in his or her .procmailrc. Usually this means that the mail is removed from the primary Inbox and placed in some other location but just about anything is possible. So, basically Postfix and Procmail work together to take the email that they receive from Fetchmail and put it in appropriate locations on the server.

Dovecot is primarily responsible to making mail thats on the server available to email clients like Thunderbird. As it happens, Dovecot doesn't particularly care (or know) how that mail comes to be where it is, it only knows how to make it available to email client.

To sumamrize: Postfix and Procmail manage the arrival of mail. Dovecot manages access to mail. As long as Postfix/Procmail puts the mail where Dovecot expects to see it, everything works.

#

Re(1): How to build a local IMAP server

Posted by: Anonymous [ip: 70.91.34.61] on September 10, 2008 01:33 PM
Excellent! Thanks for the clarification.

#

Other options

Posted by: Anonymous [ip: 216.150.131.207] on September 09, 2008 07:28 PM
All well and good, but it is worth mentioning that the alternative to all of this work is to install one of several "turn-key" email systems now available in the open source universe. One example would be Citadel [http://www.citadel.org] which installs with a single command and sets up SMTP/POP/IMAP/Webmail all pre-integrated with no editing of config files.

This article's approach is a valid one, too, and it's a decent writeup.

#

How to build a local IMAP server

Posted by: Anonymous [ip: 192.168.1.51] on September 09, 2008 09:29 PM
I am finding it a lot more these days, where people are looking at an "Appliance" solution, and are willing to pay for it. Things like Citadel pretty much help with that. I guess no one has successfully replicated MS's stronghold on their Exchange server/MS Office integration. But that's out of the scope of this article.

Where I can see this being useful is already having an existing server and your clients being sick of POP. Without too much mucking about, this can be up and running in no time, with the longest part being changing email client settings on all PC's.

SteveC

#

How to build a local IMAP server

Posted by: kfieldho on September 10, 2008 12:07 AM
Regarding "turn-key" solutions -- I'm aware that they exist but haven't used one myself. One of the things I hoped to accomplish with the article was to show how the various pieces of email oriented software fit together. With that information, evaluating solutions like Citadel becomes easier.

Thanks for mentioning Citadel. For myself, one of the best things about articles like this is the comments section where other approaches to a given problem are pointed out.

#

How to build a local IMAP server

Posted by: Anonymous [ip: 69.37.32.208] on September 10, 2008 01:58 AM
The solution for me is to use IMAP with evolution but also store email on my computer. This works for me and I think most clients support it. I can still use Gmail when I'm away from my computer but I also have local storage on my laptop.

#

How to build a local IMAP server

Posted by: Anonymous [ip: 72.137.5.104] on September 10, 2008 04:19 AM
Very nice article... thanks.

The issue for me is that I have years of business email in MBOX format and would like to convert it for Maildir. I am sure there must be programs that can process directories of MBOX files - but like one of the earlier posters my preference would be to use IMAP through my ISP for easy connectivity around the world without needing to set up my own home internet facing server box. I would then need to get it all uploaded to the ISP structure. A lot of work.

Again, thanks for the article.

#

How to build a local IMAP server

Posted by: Anonymous [ip: 193.134.254.115] on September 10, 2008 10:28 AM
You didn't cover configuring Postfix.
You didn't cover split configuration (DMZ, LAN and mailrelay vs. mailhub) (you even explicitly state to go to your ISP)

And most importantly, you didn't cover high availability (HA): you never even mentioned clusters! ERROR.

#

How to build a local IMAP server

Posted by: Anonymous [ip: 193.190.145.91] on September 10, 2008 03:21 PM
Can't fetchmail be configured to deliver directly to procmail? This way you would not to bother about an MTA.

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya