How to use fetchmail to collect mail from different mail servers

So everyone who administers mail services knows this one problem - people having configured mail forwarding in times of DKIM and SPF that protect mail address forgery without having the ability of setting up SRS (this has to be configured on both mail servers that are involved in mail forwarding). Usually people don’t configure SRS since they try to forward to/from some big public mail service - most of them are even a problem if everything is done right since they tend to mark small mail services as spam which of course hurts the landscape of an open federated E-Mail system and is done on purpose to centralize even more users since the usually reaction of users is to tell other to move to the large mail services since they seem to be working and “they never have any problems there”. Also usually people tend not to configure the target servers they forward mail to correctly to trust their own forwarding agents without doing DKIM or SPF validations.

So to make the long story short there is a solution to the forwarding problem and that’s turning the transfer direction around. The target mail servers should fetch mail from other sources and collect mail on a periodic basis - the source mail server doesn’t have to be configured any more, only the target one. For this job a tool called fetchmail has been developed - and has been around since 1996. Don’t be fooled, it’s still maintained and still the tool for such a job.

In this blog post I’m describing a setup with a local MTA (in my case postifx) that fetches mail from a remote mailbox. This is done by:

• Configuring the local mail server to trust localhost deliveries without doing SPF or DKIM checks. How this is done depends on your local server configuration.
• Setting up fetchmail to do a periodic fetchall on a bunch of external mailboxes

Note that - in my opinion - there is a really huge drawback: Fetchmail of course is required to know your clear text credentials.

The scenario in which I used the configuration described in this blog post has been a little bit different - we used this while migrating a mail service from a larger hoster to a custom mail server. This of course meant setting up the same mailboxes on our own machines, configuring the MTAs (Postfix in our case) and then at the last step redirecting the MX records in our DNS zone. Since DNS is a distributed database and caches records for a longer period of time - we started to decrement the TTL from multiple days to multiple hours so the whole process should only take 1-2 days to propagate through the whole DNS - mail will still arrive on the hosted machine during the transition while mail will also start to flow into the newly configured machine. So what happened was - during night - to redirect all clients via a configuration change to the new machine and start a first run of fetchmail to synchronize the mailboxes at the same time as re configuring the DNS records. Thus clients saw the same state as before the migration and after initial sync only newly inbound mails will have to be transferred from the hosted service to our own one. Mail was periodically (every 5-10 minutes which is instantaneous for E-mail that’s also allowed to take multiple days for delivery, never forget that) pulled from the old machine into the new mailboxes while one could see the inbound mail slowly transitioning to the new machine. After about a week there was no more mail incoming at the hosted service and after about a month of running this configuration the hosted service has been disabled.

• Installing fetchmail on FreeBSD
• Configuration
• Postfix / MTA configuration required

Installing fetchmail on FreeBSD

First one has to install fetchmail. This can be done either by packages or ports:

pkg install mail/fetchmail

or

cd /usr/ports/mail/fetchmail
make install && make clean

Configuration

Configuration of fetchmail is mainly done through /usr/local/etc/fetchmailrc

First one sets a few basic options for the daemon:

• The polling interval is set using set daemon supplying the number of seconds to poll. A typical interval is 300 seconds (5 minutes) or higher - the more mailboxes one syncs the larger the interval should get usually.
• One can set logging using syslog using set syslog which is a pretty good idea. Then the logs go to /var/log/maillog by default
• One configures the postmaster (typically either the full postmaster address or the local root user dependent on ones system configuration) using set postmaster
• In the end for my application it’s a good idea to disable bounce mail generation and redirect them to the local postmaster instead of the sender. This is done by setting set no bouncemail option. This is done in this case since all bounce handling is already done by the downstream server and the upstream server had all mailboxes configured correctly for sure.
• Then one can set some defaults:
  • A timeout. 300 seconds (5 minutes) is again a good option.
  • Setting antispam SMTP error codes or disabling this behavior. For the described application anti spam error code detection is not needed.
  • Configuring one’s batch limit. This limits how many messages are transferred per SMTP session before this session is torn down and a new connection is built up. One can disable this by setting batchlimit 0

This leads to the following header lines in fetchmailrc:

set daemon 300
set syslog
set postmaster root
set no bouncemail

defaults:
  timeout 300
  antispam -1
  batchlimit 0

Now one has to configure the actual actual remote mailboxes to poll and the (in my case local) mailboxes one wants to deliver the mail to. This is done line by line in the form:

poll mail.example.com protocol IMAP user "whoever@example.com" there with password "secret" is "localusername@example.com" here fetchall

In addition one might add the keep attribute to keep messages on the source mailbox. When one doesn’t specify them they’re deleted there automatically (which is what I wanted to do since this has been used in the migration process from one mail server to another one while the DNS records propagates through the network so old mail still arrived on the old mail servers).

Now one just has to enable the service in rc.conf and start up fetchmail:

echo "fetchmail_enable=\"YES\"" >> /etc/rc.conf
/usr/local/etc/rc.d/fetchmail start

Postfix / MTA configuration required

Note that the local MTA has to accept deliveries from the fetchmail client without checking for login (one can let fetchmail log into a mail server of course - it’s entirely possible to use the daemon to transfer messages from any mailbox on a remote host to another mailbox on another remote host anyways). This is most of the time done using permit_mynetworks in Postfix’s main configuration file /usr/local/etc/postfix/main.cf. In case one uses some kind of mail filters like OpenDKIM they also require configuration to pass mail with invalid recipient addresses, etc. to the given mailboxes - this is of course only possible when one has administrative control over the target mail server.

This article is tagged:

• Tutorial
• System administration
• FreeBSD
• Administration
• Internet
• Network
• Messaging