How to use fetchmail to collect mail from different mail servers

27 Sep 2022 - tsp
Last update 27 Sep 2022
Reading time 6 mins

So everyone who administers mail services knows this one problem - people having configured mail forwarding in times of DKIM and SPF that protect mail address forgery without having the ability of setting up SRS (this has to be configured on both mail servers that are involved in mail forwarding). Usually people don’t configure SRS since they try to forward to/from some big public mail service - most of them are even a problem if everything is done right since they tend to mark small mail services as spam which of course hurts the landscape of an open federated E-Mail system and is done on purpose to centralize even more users since the usually reaction of users is to tell other to move to the large mail services since they seem to be working and “they never have any problems there”. Also usually people tend not to configure the target servers they forward mail to correctly to trust their own forwarding agents without doing DKIM or SPF validations.

So to make the long story short there is a solution to the forwarding problem and that’s turning the transfer direction around. The target mail servers should fetch mail from other sources and collect mail on a periodic basis - the source mail server doesn’t have to be configured any more, only the target one. For this job a tool called fetchmail has been developed - and has been around since 1996. Don’t be fooled, it’s still maintained and still the tool for such a job.

In this blog post I’m describing a setup with a local MTA (in my case postifx) that fetches mail from a remote mailbox. This is done by:

Note that - in my opinion - there is a really huge drawback: Fetchmail of course is required to know your clear text credentials.

The scenario in which I used the configuration described in this blog post has been a little bit different - we used this while migrating a mail service from a larger hoster to a custom mail server. This of course meant setting up the same mailboxes on our own machines, configuring the MTAs (Postfix in our case) and then at the last step redirecting the MX records in our DNS zone. Since DNS is a distributed database and caches records for a longer period of time - we started to decrement the TTL from multiple days to multiple hours so the whole process should only take 1-2 days to propagate through the whole DNS - mail will still arrive on the hosted machine during the transition while mail will also start to flow into the newly configured machine. So what happened was - during night - to redirect all clients via a configuration change to the new machine and start a first run of fetchmail to synchronize the mailboxes at the same time as re configuring the DNS records. Thus clients saw the same state as before the migration and after initial sync only newly inbound mails will have to be transferred from the hosted service to our own one. Mail was periodically (every 5-10 minutes which is instantaneous for E-mail that’s also allowed to take multiple days for delivery, never forget that) pulled from the old machine into the new mailboxes while one could see the inbound mail slowly transitioning to the new machine. After about a week there was no more mail incoming at the hosted service and after about a month of running this configuration the hosted service has been disabled.

Installing fetchmail on FreeBSD

First one has to install fetchmail. This can be done either by packages or ports:

pkg install mail/fetchmail

or

cd /usr/ports/mail/fetchmail
make install && make clean

Configuration

Configuration of fetchmail is mainly done through /usr/local/etc/fetchmailrc

First one sets a few basic options for the daemon:

This leads to the following header lines in fetchmailrc:

set daemon 300
set syslog
set postmaster root
set no bouncemail

defaults:
  timeout 300
  antispam -1
  batchlimit 0

Now one has to configure the actual actual remote mailboxes to poll and the (in my case local) mailboxes one wants to deliver the mail to. This is done line by line in the form:

poll mail.example.com protocol IMAP user "whoever@example.com" there with password "secret" is "localusername@example.com" here fetchall

In addition one might add the keep attribute to keep messages on the source mailbox. When one doesn’t specify them they’re deleted there automatically (which is what I wanted to do since this has been used in the migration process from one mail server to another one while the DNS records propagates through the network so old mail still arrived on the old mail servers).

Now one just has to enable the service in rc.conf and start up fetchmail:

echo "fetchmail_enable=\"YES\"" >> /etc/rc.conf
/usr/local/etc/rc.d/fetchmail start

Postfix / MTA configuration required

Note that the local MTA has to accept deliveries from the fetchmail client without checking for login (one can let fetchmail log into a mail server of course - it’s entirely possible to use the daemon to transfer messages from any mailbox on a remote host to another mailbox on another remote host anyways). This is most of the time done using permit_mynetworks in Postfix’s main configuration file /usr/local/etc/postfix/main.cf. In case one uses some kind of mail filters like OpenDKIM they also require configuration to pass mail with invalid recipient addresses, etc. to the given mailboxes - this is of course only possible when one has administrative control over the target mail server.

This article is tagged:


Data protection policy

Dipl.-Ing. Thomas Spielauer, Wien (webcomplains389t48957@tspi.at)

This webpage is also available via TOR at http://jugujbrirx3irwyx.onion/

Valid HTML 4.01 Strict Powered by FreeBSD IPv6 support