menuimage

Description

Usage

Amavis_compatibility

Welcomelist

Sender_d_actions

Ban_actions

DMARC_forward_actions

Authentication_checks_failures

Log_rotation

DSN_code_rewriting

Options

Philosophy

Dependencies

License

Changelog

Download

Donation

My_Other_Sites

My_Programs

Comments

relay-enforcer v3.5.3 [21 Sep 2020] by Dominic

Description

relay-enforcer enables a postfix-based mail server, which is relaying incoming emails on to Gmail ('gsmtp'), to act on reports which it has received back from Gmail about blocked emails. Depending on the reason given by Gmail in the text of the report relay-enforcer will either block the offending incoming client's domain_name/ip_address, or forward the email encapsulated as an attachment (to reach its intended recipient).

The intended use case is where Gmail is acting - without G Suite - as the backend for a domain's emails (including sending of emails from Gmail so that they appear to come from that domain). Setting up such a scenario, requiring your own mail server, is not covered here. But once set up, as your server relays emails into Gmail some of these will get blocked by Gmail as being in some way 'bad'; and if there are too many of these then Gmail may block your mail server entirely, so that emails for that domain will be unable to reach recipients' mailboxes. relay-enforcer provides workarounds/mitigations for these blockages so that emails blocked mistakenly by Gmail can still be delivered, while emails blocked correctly by Gmail result in the sender domain_name or ip_address being blocked by your own server, preventing a flood of such emails from being relayed to Gmail and damaging your server's reputation.

Usage

relay-enforcer is intended to be run at startup e.g. by adding it to /etc/rc.local thus:

nohup /bin/sh /opt/relay-enforcer -v </dev/null >>/var/log/relay-enforcer.log 2>&1 &

or by adding it as a root cron job thus:

@reboot nohup /bin/sh /opt/relay-enforcer -v </dev/null >>/var/log/relay-enforcer.log 2>&1 &

If required, if can be (re)started from command line (with superuser permissions) thus:

/bin/sh /opt/relay-enforcer -r -v </dev/null >>/var/log/relay-enforcer.log 2>&1 &


By default relay-enforcer is silent until a matching message is found; use verbose, test or debug options to see more. Monitor relay-enforcer's output to see what it is doing. If run with '-v' it outputs a dot for each line of the mail log that it has inspected, and starts a new line with a datetime hourly; this allows you to be confident that it is running and processing the mail log. Actions generate output unless it is run with '-q'.

relay-enforcer uses inotify (via inotifywait) to inspect lines as they are added to the mail log. If it takes action this is reported on standard output (unless quiet option is chosen) and, if -m option is used, by email as well.

If a block (rather than a forward) action is required, relay-enforcer normally (but see 'Sender_d actions' below) bans the IP of the offending sender. It does this by adding an entry in the system log which ends with the ip of the incoming client whose message gave rise to the bad report; this line is to be read by the matching fail2ban jail which then effects the ip ban from incoming mail ports for some period (see 'Ban actions' below).

relay-enforcer is currently shell-agnostic, and is tested under dash (Ubuntu 20.04, 18.04, 16.04); it may later be made bash-specific.

Amavis compatibility

relay-enforcer is compatible with (but does not require) re-injection by amavis acting as a content filter (where the queue-id of the email rejected by the onward server is not the same as the queue-id of the mail from the incoming client). relay-enforcer assesses whether to run with this compatibility automatically, however command line options can be used to force the desired behaviour - for instance if amavis is run as a milter. Warning: usage of relay-enforcer without amavis re-injection is untested.

Note that relay-enforcer is not currently compatible with re-injection (content filtering or post-queue filtering) by utilities other than amavis.

Welcomelist

A file named relay-enforcer.welcomelist in the same location as relay-enforcer is used to ensure that certain ips cannot generate a report by relay-enforcer and thus a potential ban; instead of a blocking or banning action, emails from a welcomelisted IP will be encapsulated and sent on (see DMARC forward actions below). If missing, relay-enforcer.welcomelist is created at startup containing local ips and all possible local-network ips. relay-enforcer.welcomelist uses plain, range or CIDR formats as understood by grepcidr; each pattern should start on a newline. Comments (starting with #) and any space immediately preceding them are ignored, as are blank lines. relay-enforcer.welcomelist can be updated while relay-enforcer is running.

Sender_d actions

Where the rejection message from Gmail specifies that the problem was with the sending domain, then instead of adding a log entry which might trigger a source IP ban by fail2ban, relay-enforcer will add this domain to $SENDER_D_BANFILE (normally: '/etc/postfix/sender_access') and hash it; provided this file is specified with a postfix restriction list for check_sender_access then this ensures that no more emails from this domain will (ever) be accepted.

Ban actions

When an IP is identified for banning, the actual banning action is carried out by fail2ban jail 'relay-enforcer' acting upon log entries added by relay-enforcer. This jail needs to be created, for instance:
(a) create /etc/fail2ban/filter.d/relay-enforcer.local:

[INCLUDES]
before = common.conf
[Definition]
_daemon = relay-enforcer
failregex = ^%%(__prefix_line)sbannable.* <HOST>$
ignoreregex =

(b) add entries in, say, /etc/fail2ban/jail.local like this:

[relay-enforcer]
enabled = true
maxretry = 1
bantime = 14700
findtime = 14700
port = smtp,465,submission
logpath = %%(syslog_ftp)s

DMARC forward actions

If the local mailserver is enforcing DMARC (e.g. with opendmarc), a rejection from an onward server should only occur if the offending email's 'From:' sender domain publishes a DMARC p=reject policy but the email lacks valid DKIM and relies for successful delivery on SPF (which is inevitably broken by forwarding). In such a case (which is rare) relay-enforcer will extract the failed email from a local backup (e.g. /var/mail/backup - postfix settings 'always_bcc = backup@localhost' and 'mail_spool_directory = /var/mail' can be used to save all mails here), and send it on as an attachment to the original recipient, with a covering message. To enable this DMARC forwarding action, first ensure that you have an effective DMARC enforcement policy on your local mailserver, and have all sent emails saved (via always_bcc as shown above) in a single mbox-type file, then use relay-enforcer's -f option to define the local backup file when starting relay-enforcer.

Authentication checks failures

Gmail rejects some emails on the grounds of 'authentication checks', these are usually round-robin emails from a Microsoft server. Experience has shown that they are always or nearly always good emails, so relay-enforcer will resend them encapsulated (same as for DMARC forward actions, and with the same requirement that recent emails are saved in a local mbox-type file).

Log rotation

To prevent relay-enforcer from hanging upon log rotation of the mail log file, add 'copytruncate' to the relevant section of the relevant logrotate file (e.g. /etc/logrotate.d/rsyslog).

DSN code rewriting

relay-enforcer does not rewrite the code responses that are received from Gmail and then relayed by the local mailserver to the original sender; this must be done separately as part of the postfix configuration, but only for messages to Gmail. 550 responses upon failure because of DMARC or authentication checks need to be re-written to 250 so that the original sender considers the message was delivered ok (as, after forwarding by relay-enforcer, it should be), and temporary failure codes should be rewritten as permanent (i.e. 4xx to 5xx).

Example file extracts for postfix configuration are shown here -
/etc/postfix/master.cf -

...
smtp_dsn_rw unix - - n - - smtp -o smtp_reply_filter=pcre:/etc/postfix/smtp_dsn_rw_filter.pcre
...

/etc/postfix/main.cf -

...
transport_maps = hash:/etc/postfix/transport
...

/etc/postfix/transport -

...
gmail.com smtp_dsn_rw
...

/etc/postfix/smtp_dsn_rw_filter.pcre -

# no change if there is a temporary problems with Gmail servers
/^(421 4\.[0-9]\.[0-9] Temporary )/ ${1}
# DMARC rejection by Gmail - change code to 250 OK
/^550 5\.7\.1( DMARC.+gsmtp)/ 250 2.0.0${1}-fmod_dmarc
# Authentication checks rejection by Gmail - change code to 250 OK
/^421 4\.7\.0( information.+gsmtp)/ 250 2.0.0${1}-fmod_lackauth
# Other temporary rejection by Gmail - make rejection permanent
/^421 4\.7\.28( .+gmstp)/ 554 5.7.28${1}-fmod_unsol
/^421 4\.([0-9]*\.[0-9]* review our Bulk.+gsmtp)/ 554 5.${1}-fmod_bulk
/^421 4\.([0-9]*\.[0-9]* .+gsmtp)/ 554 5.${1}-fmod_generic

Options

-a - force compatibility with amavis as content-filter (see -n)
-d - debug mode - lots of extra text output, implies verbose
-f /var/local/backup - specify a local backup mbox-style file which will contain copies of recently sent mails, so can be used to retrieve blocked mails for encapsulating and forwarding
-h - show this help
-l - show changelog
-m [mail@address] - send report on any action to mail@address (default: root)
-n - force non-compatibility with amavis as content-filter (see -a)
-q - be quiet even if reportable actions are found
-r - kill another instance of relay-enforcer (if any), then continue running
-s - stop another instance of relay-enforcer (if any)
-t - test mode (don't submit reports to system log, don't update count of lines in mail log)
-v - be verbose
-w columns - set terminal width e.g. 80 (for help/changelog word-wrapping only, normally determined automagically)

Options must be specified individually not combined i.e. -v -r not -vr

Philosophy

RFC5321 at https://tools.ietf.org/html/rfc5321#section-4.2, referring to the reply code+text sent by an onward smtp receiver (such as Gmail) says: 'an SMTP client MUST determine its actions only by the reply code, not by the text'. relay-enforcer does not follow that rule, so if you have a philosophical objection to this behaviour, don't use relay-enforcer.

Dependencies

grepcidr
fail2ban with relay-enforcer jail - (relay-enforcer cannot ban without this)
inotifywait (in inotify-tools)
Mail User Agent (MUA) - one of: s-nail mailx sendemail mutt
opendmarc (or similar DMARC enforcement service) - only required if using -f option
postfix
python3

License

Copyright © 2020 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

Note - little changes and bugfixes may occur between releases listed here, these may be indicated by a sub-minor version change (3.4.0->3.4.1).
3.5 [10 Jul 2020] - all references to whitelist (except in this changelog) changed to welcomelist (compatibility break)
3.4 [26 Jun 2020] - only one fail2ban ban type (relay-enforcer, not relay-enforcer-long and relay-enforcer-short; compatibility break)
3.3 [09 Jun 2020] - alter welcomelist ip format (compatibility break)
3.2 [17 Apr 2020] - add preceding comment when adding entry to /etc/postfix/sender_access
3.1 [01 Apr 2020] - work with logrotate copytruncate
3.0 [13 Dec 2019] - bugfix paths for programs run by relay-enforcer, tidy up some old code
2.9 [07 Dec 2019] - bugfix for postfix QueueIDs of >10 characters
2.8 [02 Dec 2019] - don't copy encapsulated emails to administrator
2.7 [22 Jul 2019] - add an extra check to each candidate line from log to ensure that we didn't already process it (otherwise duplication occasionally happens)
2.6 [27 Jun 2019] - forward (by encapsulating) on any 'authentication checks' Gmail response (previous behaviour was to do this only when the From header was matched in $LACK_AUTH_WHITELIST)
2.5 [02 Jan 2019] - tweak output in non-verbose non-quiet mode
2.4 [18 Oct 2018] - first published version
2.3 [05 Apr 2018] - bugfix ' sending .*domain' blocking
2.2 [29 Mar 2018] - add a 5s delay before triggering a fail2ban ban, to allow time for any outgoing email to original sender (also helps ensure that any subsequent recorded blocks by fail2ban represent *new* connection attempts by the now-banned host at that ip), also bugfix code for unexpected gsmtp responses
2.1 [12 Feb 2018] - introduce ' sending .*domain' blocking via postfix if cause of Gmail unhappiness is stated to be sender domain - permanent block by sender domain name
2.0 [04 Nov 2017] - introduce 'authentication checks' forwarding if email 'From:' is in $LACK_AUTH_WHITELIST
1.9 [16 Jul 2017] - help info updated/bugfixed, log entry upon DMARC forwarding no longer says 'bannable'
1.8 [13 Jul 2017] - email any unrecognised rejection messages to mail address (if defined by option -m)
1.7 [08 Jul 2017] - output any unrecognised rejection messages
1.6 [01 Jul 2017] - wait up to 20m (not 60s) for mail log to appear when starting
1.5 [01 Jun 2017] - fix when restarting after mail log rotation
1.4 [12 May 2017] - bugfix (grep -a option added throughout)
1.3 [01 May 2017] - added instructions to avoid hang upon rotation of monitored log file
1.2 [18 Mar 2017] - set Reply-To for forwarded emails
1.1 [11 Mar 2017] - many bugfixes
1.0 [22 Feb 2017] - add special DMARC treatment, option -f
0.9 [18 Jan 2017] - various bugfixes
0.8 [09 Jan 2017] - don't delete existing queued message
0.7 [01 Jan 2017] - delete existing queued message, bugfixes
0.6 [30 Dec 2016] - bugfixes
0.5 [22 Dec 2016] - skip attempt to ban if ip is already banned
0.4 [14 Dec 2016] - use inotifywait, run continuously
0.3 [14 Dec 2016] - work without amavis (untested)
0.2 [09 Dec 2016] - allow 'short' and 'long' ban specifications
0.1 [06 Dec 2016] - first version

Download relay-enforcer

Donation

I have provided this software free gratis and for nothing. If you would like to thank me with a contribution, please let me know and I will send you a link. Thank you!

My Other Sites

My Programs

Here is a selection of some (other) programs I have written, most of which run from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written for a conventional Linux server, they should run on a Raspberry Pi, and many can run under Windows using Windows Subsystem for Linux (WSL) or Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.

Backup Utilities

Debian/Ubuntu kernel and LVM Utilities

Dellmont / Three - VoIP and Mobile Phone Account Utilities

Miscellaneous Programs

Comments

If you have a comment or question, please email me, thank you.

No comments yet