relay-enforcer v3.5.0 [10 Jul 2020] by Dominic
Description
relay-enforcer enables a postfix-based mail server, which is relaying incoming emails on to Gmail ('gsmtp'), to act on reports which it has received back from Gmail about blocked emails. Depending on the reason given by Gmail in the text of the report relay-enforcer will either block the offending incoming client's domain_name/ip_address, or forward the email encapsulated as an attachment (to reach its intended recipient).
The intended use case is where Gmail is acting - without G Suite - as the backend for a domain's emails (including sending of emails from Gmail so that they appear to come from that domain). Setting up such a scenario, requiring your own mail server, is not covered here. But once set up, as your server relays emails into Gmail some of these will get blocked by Gmail as being in some way 'bad'; and if there are too many of these then Gmail may block your mail server entirely, so that emails for that domain will be unable to reach recipients' mailboxes. relay-enforcer provides workarounds/mitigations for these blockages so that emails blocked mistakenly by Gmail can still be delivered, while emails blocked correctly by Gmail result in the sender domain_name or ip_address being blocked by your own server, preventing a flood of such emails from being relayed to Gmail and damaging your server's reputation.
Usage
relay-enforcer is intended to be run at startup e.g. by adding it to /etc/rc.local thus:
or by adding it as a root cron job thus:
If required, if can be (re)started from command line (with superuser permissions) thus:
By default relay-enforcer is silent until a matching message is found; use verbose, test or debug options to see more. Monitor relay-enforcer's output to see what it is doing. If run with '-v' it outputs a dot for each line of the mail log that it has inspected, and starts a new line with a datetime hourly; this allows you to be confident that it is running and processing the mail log. Actions generate output unless it is run with '-q'.
relay-enforcer uses inotify (via inotifywait) to inspect lines as they are added to the mail log. If it takes action this is reported on standard output (unless quiet option is chosen) and, if -m option is used, by email as well.
If a block (rather than a forward) action is required, relay-enforcer normally (but see 'Sender_d actions' below) bans the IP of the offending sender. It does this by adding an entry in the system log which ends with the ip of the incoming client whose message gave rise to the bad report; this line is to be read by the matching fail2ban jail which then effects the ip ban from incoming mail ports for some period (see 'Ban actions' below).
relay-enforcer is currently shell-agnostic, and is tested under dash (Ubuntu 20.04, 18.04, 16.04); it may later be made bash-specific.
Amavis compatibility
relay-enforcer is compatible with (but does not require) re-injection by amavis acting as a content filter (where the queue-id of the email rejected by the onward server is not the same as the queue-id of the mail from the incoming client). relay-enforcer assesses whether to run with this compatibility automatically, however command line options can be used to force the desired behaviour - for instance if amavis is run as a milter. Warning: usage of relay-enforcer without amavis re-injection is untested.
Note that relay-enforcer is not currently compatible with re-injection (content filtering or post-queue filtering) by utilities other than amavis.
Welcomelist
A file named relay-enforcer.welcomelist in the same location as relay-enforcer is used to ensure that certain ips cannot generate a report by relay-enforcer and thus a potential ban; instead of a blocking or banning action, emails from a welcomelisted IP will be encapsulated and sent on (see DMARC forward actions below). If missing, relay-enforcer.welcomelist is created at startup containing local ips and all possible local-network ips. relay-enforcer.welcomelist uses plain, range or CIDR formats as understood by grepcidr; each pattern should start on a newline. Comments (starting with #) and any space immediately preceding them are ignored, as are blank lines. relay-enforcer.welcomelist can be updated while relay-enforcer is running.
Sender_d actions
Where the rejection message from Gmail specifies that the problem was with the sending domain, then instead of adding a log entry which might trigger a source IP ban by fail2ban, relay-enforcer will add this domain to $SENDER_D_BANFILE (normally: '/etc/postfix/sender_access') and hash it; provided this file is specified with a postfix restriction list for check_sender_access then this ensures that no more emails from this domain will (ever) be accepted.
Ban actions
When an IP is identified for banning, the actual banning action is carried out by fail2ban jail 'relay-enforcer' acting upon log entries added by relay-enforcer. This jail needs to be created, for instance:
(a) create /etc/fail2ban/filter.d/relay-enforcer.local:
before = common.conf
[Definition]
_daemon = relay-enforcer
failregex = ^%%(__prefix_line)sbannable.* <HOST>$
ignoreregex =
(b) add entries in, say, /etc/fail2ban/jail.local like this:
enabled = true
maxretry = 1
bantime = 14700
findtime = 14700
port = smtp,465,submission
logpath = %%(syslog_ftp)s
DMARC forward actions
If the local mailserver is enforcing DMARC (e.g. with opendmarc), a rejection from an onward server should only occur if the offending email's 'From:' sender domain publishes a DMARC p=reject policy but the email lacks valid DKIM and relies for successful delivery on SPF (which is inevitably broken by forwarding). In such a case (which is rare) relay-enforcer will extract the failed email from a local backup (e.g. /var/mail/backup - postfix settings 'always_bcc = backup@localhost' and 'mail_spool_directory = /var/mail' can be used to save all mails here), and send it on as an attachment to the original recipient, with a covering message. To enable this DMARC forwarding action, first ensure that you have an effective DMARC enforcement policy on your local mailserver, and have all sent emails saved (via always_bcc as shown above) in a single mbox-type file, then use relay-enforcer's -f option to define the local backup file when starting relay-enforcer.
Authentication checks failures
Gmail rejects some emails on the grounds of 'authentication checks', these are usually round-robin emails from a Microsoft server. Experience has shown that they are always or nearly always good emails, so relay-enforcer will resend them encapsulated (same as for DMARC forward actions, and with the same requirement that recent emails are saved in a local mbox-type file).
Log rotation
To prevent relay-enforcer from hanging upon log rotation of the mail log file, add 'copytruncate' to the relevant section of the relevant logrotate file (e.g. /etc/logrotate.d/rsyslog).
DSN code rewriting
relay-enforcer does not rewrite the code responses that are received from Gmail and then relayed by the local mailserver to the original sender; this must be done separately as part of the postfix configuration, but only for messages to Gmail. 550 responses upon failure because of DMARC or authentication checks need to be re-written to 250 so that the original sender considers the message was delivered ok (as, after forwarding by relay-enforcer, it should be), and temporary failure codes should be rewritten as permanent (i.e. 4xx to 5xx).
Example file extracts for postfix configuration are shown here -
/etc/postfix/master.cf -
smtp_dsn_rw unix - - n - - smtp -o smtp_reply_filter=pcre:/etc/postfix/smtp_dsn_rw_filter.pcre
...
/etc/postfix/main.cf -
transport_maps = hash:/etc/postfix/transport
...
/etc/postfix/transport -
gmail.com smtp_dsn_rw
...
/etc/postfix/smtp_dsn_rw_filter.pcre -
/^(421 4\.[0-9]\.[0-9] Temporary )/ ${1}
# DMARC rejection by Gmail - change code to 250 OK
/^550 5\.7\.1( DMARC.+gsmtp)/ 250 2.0.0${1}-fmod_dmarc
# Authentication checks rejection by Gmail - change code to 250 OK
/^421 4\.7\.0( information.+gsmtp)/ 250 2.0.0${1}-fmod_lackauth
# Other temporary rejection by Gmail - make rejection permanent
/^421 4\.7\.28( .+gmstp)/ 554 5.7.28${1}-fmod_unsol
/^421 4\.([0-9]*\.[0-9]* review our Bulk.+gsmtp)/ 554 5.${1}-fmod_bulk
/^421 4\.([0-9]*\.[0-9]* .+gsmtp)/ 554 5.${1}-fmod_generic
Options
-a - force compatibility with amavis as content-filter (see -n)
-d - debug mode - lots of extra text output, implies verbose
-f /var/local/backup - specify a local backup mbox-style file which will contain copies of recently sent mails, so can be used to retrieve blocked mails for encapsulating and forwarding
-h - show this help
-l - show changelog
-m [mail@address] - send report on any action to mail@address (default: root)
-n - force non-compatibility with amavis as content-filter (see -a)
-q - be quiet even if reportable actions are found
-r - kill another instance of relay-enforcer (if any), then continue running
-s - stop another instance of relay-enforcer (if any)
-t - test mode (don't submit reports to system log, don't update count of lines in mail log)
-v - be verbose
-w columns - set terminal width e.g. 80 (for help/changelog word-wrapping only, normally determined automagically)
Options must be specified individually not combined i.e. -v -r not -vr
Philosophy
RFC5321 at https://tools.ietf.org/html/rfc5321#section-4.2, referring to the reply code+text sent by an onward smtp receiver (such as Gmail) says: 'an SMTP client MUST determine its actions only by the reply code, not by the text'. relay-enforcer does not follow that rule, so if you have a philosophical objection to this behaviour, don't use relay-enforcer.
Dependencies
grepcidr
fail2ban with relay-enforcer jail - (relay-enforcer cannot ban without this)
inotifywait (in inotify-tools)
Mail User Agent (MUA) - one of: s-nail mailx sendemail mutt
opendmarc (or similar DMARC enforcement service) - only required if using -f option
postfix
python3
License
Copyright © 2020 Dominic Raferd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Changelog
Note - little changes and bugfixes may occur between releases listed here, these may be indicated by a sub-minor version change (3.4.0->3.4.1).
3.5 [10 Jul 2020] - all references to whitelist (except in this changelog) changed to welcomelist (compatibility break)
3.4 [26 Jun 2020] - only one fail2ban ban type (relay-enforcer, not relay-enforcer-long and relay-enforcer-short; compatibility break)
3.3 [09 Jun 2020] - alter welcomelist ip format (compatibility break)
3.2 [17 Apr 2020] - add preceding comment when adding entry to /etc/postfix/sender_access
3.1 [01 Apr 2020] - work with logrotate copytruncate
3.0 [13 Dec 2019] - bugfix paths for programs run by relay-enforcer, tidy up some old code
2.9 [07 Dec 2019] - bugfix for postfix QueueIDs of >10 characters
2.8 [02 Dec 2019] - don't copy encapsulated emails to administrator
2.7 [22 Jul 2019] - add an extra check to each candidate line from log to ensure that we didn't already process it (otherwise duplication occasionally happens)
2.6 [27 Jun 2019] - forward (by encapsulating) on any 'authentication checks' Gmail response (previous behaviour was to do this only when the From header was matched in $LACK_AUTH_WHITELIST)
2.5 [02 Jan 2019] - tweak output in non-verbose non-quiet mode
2.4 [18 Oct 2018] - first published version
2.3 [05 Apr 2018] - bugfix ' sending .*domain' blocking
2.2 [29 Mar 2018] - add a 5s delay before triggering a fail2ban ban, to allow time for any outgoing email to original sender (also helps ensure that any subsequent recorded blocks by fail2ban represent *new* connection attempts by the now-banned host at that ip), also bugfix code for unexpected gsmtp responses
2.1 [12 Feb 2018] - introduce ' sending .*domain' blocking via postfix if cause of Gmail unhappiness is stated to be sender domain - permanent block by sender domain name
2.0 [04 Nov 2017] - introduce 'authentication checks' forwarding if email 'From:' is in $LACK_AUTH_WHITELIST
1.9 [16 Jul 2017] - help info updated/bugfixed, log entry upon DMARC forwarding no longer says 'bannable'
1.8 [13 Jul 2017] - email any unrecognised rejection messages to mail address (if defined by option -m)
1.7 [08 Jul 2017] - output any unrecognised rejection messages
1.6 [01 Jul 2017] - wait up to 20m (not 60s) for mail log to appear when starting
1.5 [01 Jun 2017] - fix when restarting after mail log rotation
1.4 [12 May 2017] - bugfix (grep -a option added throughout)
1.3 [01 May 2017] - added instructions to avoid hang upon rotation of monitored log file
1.2 [18 Mar 2017] - set Reply-To for forwarded emails
1.1 [11 Mar 2017] - many bugfixes
1.0 [22 Feb 2017] - add special DMARC treatment, option -f
0.9 [18 Jan 2017] - various bugfixes
0.8 [09 Jan 2017] - don't delete existing queued message
0.7 [01 Jan 2017] - delete existing queued message, bugfixes
0.6 [30 Dec 2016] - bugfixes
0.5 [22 Dec 2016] - skip attempt to ban if ip is already banned
0.4 [14 Dec 2016] - use inotifywait, run continuously
0.3 [14 Dec 2016] - work without amavis (untested)
0.2 [09 Dec 2016] - allow 'short' and 'long' ban specifications
0.1 [06 Dec 2016] - first version
Download relay-enforcer
Donation
I have provided this software free gratis and for nothing. If you would like to thank me with a contribution, please let me know and I will send you a link. Thank you!
My Other Sites
- TimeDicer - Onsite/offsite data backup for Windows (uses rdiff-backup)
- Finding a 4D Backup Solution
- Web Scraping How To - extracting data from web sites
My Programs
Here is a selection of some (other) programs I have written, most of which run from the command line (CLI), are freely available and can be obtained by clicking on the links. Dependencies are shown and while in most cases written for a conventional Linux server, they should run even on a Raspberry Pi, and many can run under Windows using Cygwin. Email me if you have problems or questions, or if you think I could help with a programming requirement.
Backup Utilities
- TimeDicer - Onsite/offsite data backup for Windows (uses rdiff-backup) [ GNU/Linux & MS Windows©: 2008-20 ]
- rdiff-backup-regress - GNU/Linux script to regress an rdiff-backup archive. [ GNU/Linux: 2012-16 ]
Debian/Ubuntu kernel and LVM Utilities
- kernel-remove - GNU/Linux script to list the installed GNU/Linux kernels in a Debian-based distro (e.g. Ubuntu), and can be used to remove an unwanted kernel and related packages, updating grub appropriately. [ GNU/Linux-Debian/Ubuntu: 2010-15 ]
- lvm-usage - GNU/Linux script to show available disk space and how it is used; run as cron job to warn if usage is above a set percentage. Provides additional information if LVM is in use. [ GNU/Linux-Debian/Ubuntu: 2012-20 ]
- lvm-delete-snapshot - GNU/Linux script to remove LVM snapshot that has been left over by another process. [ GNU/Linux-Debian/Ubuntu: 2012-19 ]
- netnames - GNU/Linux script shows current name, biosdevname and 'predictable name' of network device - helps with name scheme migration. [ GNU/Linux-Debian/Ubuntu: 2020-20 ]
Dellmont / Three - VoIP and Mobile Phone Account Utilities
- dellmont-credit-checker - GNU/Linux script to check credit balance on many Dellmont / Finarea / Betamax portals such as voicetrading.com and voipdiscount.com. [ GNU/Linux: 2008-19 ]
- sms-sender - GNU/Linux script to send text messages using Dellmont’s voicetrading.com. [ GNU/Linux: 2012-18 ]
- get-vt-cdrs - GNU/Linux script to download CDRs (call detail records) from Dellmont’s voicetrading.com or voippro.com. [ GNU/Linux: 2010-19 ]
- saynoto0870 - For people in UK, a GNU/Linux script which performs automated lookup of the www.saynoto0870.com database, finding cheap or free geographic number replacements for expensive non-geographic (087* or 084*) numbers. [ GNU/Linux: 2012-12 ]
- three-credit-checker - GNU/Linux script which checks credit/calls/text/data remaining on a mobile phone account with three.co.uk. [ GNU/Linux: 2014-20 ]
Miscellaneous Programs
- sleepwalker - Windows© program which can be run from a remote machine to 'wake up' a Windows© machine behind a router, wait for it to start and then initiate Remote Desktop session. [MS Windows©: 2008-16]
- bind9-resolved-switch - GNU/Linux program for switching permanently between using bind9 or systemd-resolved as the system DNS resolver. [ GNU/Linux: 2016-20 ]
- unlock - GNU/Linux remote program for easy entering of decrypt passphrase on a remote machine which has root dm-crypt+LUKS. [ GNU/Linux: 2017-18 ]
- wifi-updown - GNU/Linux program to take down wifi interface if there is a working wired interface (or restore wifi if not). [ GNU/Linux: 2018-20 ]
- routefix - GNU/Linux program to restore a default ip traffic route if there is none such (e.g. after running wifi-updown). [ GNU/Linux: 2018-19 ]
- pdf-compress - GNU/Linux program to create smaller b/w pdf file from an original large pdf file, especially when original resulted from scanning. [ GNU/Linux: 2016-17 ]
- form-extractor - GNU/Linux program to extract form tags from a web page or downloaded file. [ GNU/Linux: 2012-20 ]
- mythic-dns-sync - GNU/Linux program to update DNS record at mythic-beasts.com to match local external ip. [ GNU/Linux: 2016-20 ]
- recover-space - GNU/Linux program to enable a virtual disk volume to be compacted. [ GNU/Linux: 2014-15 ]
- tiny-device-monitor - GNU/Linux program to test webpages (including password-protected) or machines to check they are live; use as a cron job for your own websites, for hardware presenting a webpage, or for any machines with a presence on your local LAN or on the internet. [ GNU/Linux: 2009-19 ]
- dutree - GNU/Linux program to show a tree-style list of files and directories at the specified location and greater than the specified size (default 1GB). [ GNU/Linux: 2012-20 ]
- Accounts - Multi-business multi-currency accounting software, uses Access [MS Windows©: 1996-2020]
- Rents Program - Residential lettings/landlord front office program, with many special features for UK market [MS Windows©: 1991-2020]