27 Sep 2022 - tsp
Last update 27 Sep 2022
So everyone who administers mail services knows this one problem - people having configured mail forwarding in times of DKIM and SPF that protect mail address forgery without having the ability of setting up SRS (this has to be configured on both mail servers that are involved in mail forwarding). Usually people don’t configure SRS since they try to forward to/from some big public mail service - most of them are even a problem if everything is done right since they tend to mark small mail services as spam which of course hurts the landscape of an open federated E-Mail system and is done on purpose to centralize even more users since the usually reaction of users is to tell other to move to the large mail services since they seem to be working and “they never have any problems there”. Also usually people tend not to configure the target servers they forward mail to correctly to trust their own forwarding agents without doing DKIM or SPF validations.
So to make the long story short there is a solution to the forwarding problem and that’s
turning the transfer direction around. The target mail servers should fetch mail from
other sources and collect mail on a periodic basis - the source mail server doesn’t have to
be configured any more, only the target one. For this job a tool called
has been developed - and has been around since 1996. Don’t be fooled, it’s still maintained
and still the tool for such a job.
In this blog post I’m describing a setup with a local MTA (in my case postifx) that fetches mail from a remote mailbox. This is done by:
Note that - in my opinion - there is a really huge drawback: Fetchmail of course is required to know your clear text credentials.
The scenario in which I used the configuration described in this blog post has been a little
bit different - we used this while migrating a mail service from a larger hoster to a custom
mail server. This of course meant setting up the same mailboxes on our own machines, configuring
the MTAs (Postfix in our case) and then at the last step redirecting the
in our DNS zone. Since DNS is a distributed database and caches records for a longer period
of time - we started to decrement the TTL from multiple days to multiple hours so the whole
process should only take 1-2 days to propagate through the whole DNS - mail will still arrive
on the hosted machine during the transition while mail will also start to flow into the
newly configured machine. So what happened was - during night - to redirect all clients
via a configuration change to the new machine and start a first run of fetchmail to synchronize
the mailboxes at the same time as re configuring the DNS records. Thus clients saw the same
state as before the migration and after initial sync only newly inbound mails will have
to be transferred from the hosted service to our own one. Mail was periodically (every 5-10
minutes which is instantaneous for E-mail that’s also allowed to take multiple days for delivery,
never forget that) pulled from the old machine into the new mailboxes while one could see
the inbound mail slowly transitioning to the new machine. After about a week there was no more
mail incoming at the hosted service and after about a month of running this configuration
the hosted service has been disabled.
First one has to install
fetchmail. This can be done either by packages or ports:
pkg install mail/fetchmail
cd /usr/ports/mail/fetchmail make install && make clean
fetchmail is mainly done through
First one sets a few basic options for the daemon:
set daemonsupplying the number of seconds to poll. A typical interval is 300 seconds (5 minutes) or higher - the more mailboxes one syncs the larger the interval should get usually.
set syslogwhich is a pretty good idea. Then the logs go to
rootuser dependent on ones system configuration) using
set no bouncemailoption. This is done in this case since all bounce handling is already done by the downstream server and the upstream server had all mailboxes configured correctly for sure.
antispamSMTP error codes or disabling this behavior. For the described application anti spam error code detection is not needed.
This leads to the following header lines in
set daemon 300 set syslog set postmaster root set no bouncemail defaults: timeout 300 antispam -1 batchlimit 0
Now one has to configure the actual actual remote mailboxes to poll and the (in my case local) mailboxes one wants to deliver the mail to. This is done line by line in the form:
poll mail.example.com protocol IMAP user "email@example.com" there with password "secret" is "firstname.lastname@example.org" here fetchall
In addition one might add the
keep attribute to keep messages on the source mailbox. When
one doesn’t specify them they’re deleted there automatically (which is what I wanted to do since
this has been used in the migration process from one mail server to another one while the DNS
records propagates through the network so old mail still arrived on the old mail servers).
Now one just has to enable the service in
rc.conf and start up fetchmail:
echo "fetchmail_enable=\"YES\"" >> /etc/rc.conf /usr/local/etc/rc.d/fetchmail start
Note that the local MTA has to accept deliveries from the
fetchmail client without checking
for login (one can let fetchmail log into a mail server of course - it’s entirely possible to use
the daemon to transfer messages from any mailbox on a remote host to another mailbox on another
remote host anyways). This is most of the time done using
permit_mynetworks in Postfix’s
main configuration file
/usr/local/etc/postfix/main.cf. In case one uses some kind of
mail filters like OpenDKIM they also require configuration to pass mail with invalid recipient
addresses, etc. to the given mailboxes - this is of course only possible when one has administrative
control over the target mail server.
This article is tagged: