Rant: Google Apps’ Gmail Can’t Protect Me from Misconfigured Mailers 2

This entry was posted in Linux Rants and Raves Technology and tagged blocked senders Google Apps SPF split domain routing spoof on by (updated 3800 days ago)

Google AppsIt all started about a week ago. I got an alert email from a server, complaining about a RAID array. That’s nothing terribly out of the ordinary, as I have all of my systems set to alert me if something seems out of place. However, the problem in this case is that the alert didn’t actually come from one of my servers… even though the message claimed it did.

The message read (with domain names changed for security):

From: [email protected]

To: [email protected]

This is a RAID status update from mpt-statusd.  The mpt-status
program reports that one of the RAIDs changed state:

Report from /etc/init.d/mpt-statusd on MyPersonalDomain.org

Since I’m using Gmail via Google Apps, I clicked the small drop-down arrow next to the recipient’s name (which was “root”) to display more details about the message. The details showed mailed-by: MyPersonalDomain.org as one of the items, meaning the message had actually passed an SPF check! This concerned me, since my personal domain’s SPF record explicity disavows all mail sent from servers not specifically mentioned in the record. Here’s my current SPF record for MyPersonalDomain.org:

v=spf1 ip4:123.45.678.90 a mx include:_spf.google.com -all

The above SPF record translates to mean that the server with IP (v4) address 123.45.678.90 is allowed to send mail on behalf of MyPersonalDomain.org, as can any server that has a valid A or MX record in the domain’s zone file, and to also “include” the outbound Google Mail servers (since I’m using Google Apps). The -all directive at the end means that no other servers are allowed to claim they’re mine. And yet, one was… and it was successfully delivering mail to me as such.

Meaning something wasn’t right.

To investigate further, I selected the “Show Original” option in the message (which is one of my favorite features of Gmail). Here’s a copy of the full message shown, with all the headers:

Delivered-To: [email protected]
Received: by 10.224.78.135 with SMTP id l7csp81480qak;
        Fri, 29 Nov 2013 18:58:57 -0800 (PST)
X-Received: by 10.67.14.231 with SMTP id fj7mr36109613pad.115.1385780336744;
        Fri, 29 Nov 2013 18:58:56 -0800 (PST)
Return-Path: <[email protected]>
Received: from Backup.MyPersonalDomain.org (MachineName.MyPersonalDomain.org. [123.45.678.90])
        by mx.google.com with ESMTPS id ty3si36615612pbc.197.2013.11.29.18.58.56
        for <[email protected]>
        (version=TLSv1 cipher=RC4-SHA bits=128/128);
        Fri, 29 Nov 2013 18:58:56 -0800 (PST)
Received-SPF: pass (google.com: domain of [email protected] designates 123.45.678.90 as permitted sender) client-ip=123.45.678.90;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of [email protected] designates 123.45.678.90 as permitted sender) [email protected]
Received: by Backup.MyPersonalDomain.org (Postfix)
	id 1E5B5104242E; Fri, 29 Nov 2013 18:58:56 -0800 (PST)
Delivered-To: [email protected]
Received: from localhost (localhost [127.0.0.1])
	by Backup.MyPersonalDomain.org (Postfix) with ESMTP id 117F0104242D
	for ; Fri, 29 Nov 2013 18:58:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at MyPersonalDomain.org
X-Spam-Flag: NO
X-Spam-Score: -0.701
X-Spam-Level: 
X-Spam-Status: No, score=-0.701 tagged_above=-9999 required=5
	tests=[RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=unavailable
Received: from Backup.MyPersonalDomain.org ([127.0.0.1])
	by localhost (Backup.MyPersonalDomain.org [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id C89ZYg6dvXMZ for ;
	Fri, 29 Nov 2013 18:58:49 -0800 (PST)
Received: from mail-lb0-f171.google.com (mail-lb0-f171.google.com [209.85.217.171])
	(using TLSv1 with cipher RC4-SHA (128/128 bits))
	(No client certificate requested)
	by Backup.MyPersonalDomain.org (Postfix) with ESMTPS id 584C61042407
	for <[email protected]>; Fri, 29 Nov 2013 18:58:48 -0800 (PST)
DKIM-Filter: OpenDKIM Filter v2.7.1 Backup.MyPersonalDomain.org 584C61042407
Received: by mail-lb0-f171.google.com with SMTP id q8so7555170lbi.30
        for <[email protected]>; Fri, 29 Nov 2013 18:58:46 -0800 (PST)
X-Gm-Message-State: ALoCoQmBQrYrWIUfei0ARRr5LDO2uu+ma50UDDlrZilVv32AwC5YLg3wssFBYYNWX4IvUQ+fMju9kQS9JqHj7mrfx+z9kuibDBu7hJIkWCrYxhijzU0VTqvItdbRLHOCgw9/y+MAgpTXGoorX7evFuwo2NknHsoKOA==
X-Received: by 10.112.72.70 with SMTP id b6mr10451305lbv.0.1385780326594;
        Fri, 29 Nov 2013 18:58:46 -0800 (PST)
X-Received: by 10.112.72.70 with SMTP id b6mr10451283lbv.0.1385780326249;
        Fri, 29 Nov 2013 18:58:46 -0800 (PST)
Received: from MyPersonalDomain.org ([91.242.161.26])
        by mx.google.com with ESMTP id p10si16668919lag.46.2013.11.29.18.58.41
        for <[email protected]>;
        Fri, 29 Nov 2013 18:58:45 -0800 (PST)
Received-SPF: fail (google.com: domain of [email protected] does not designate 91.242.161.26 as permitted sender) client-ip=91.242.161.26;
Received: by MyPersonalDomain.org (Postfix, from userid 0)
	id 27C594D1; Sat, 30 Nov 2013 06:57:20 +0400 (MSK)
To: [email protected]
Subject: info: mpt raid status change on MyPersonalDomain.org
Message-Id: <[email protected]>
Date: Sat, 30 Nov 2013 06:57:20 +0400 (MSK)
From: [email protected] (root)

This is a RAID status update from mpt-statusd.  The mpt-status
program reports that one of the RAIDs changed state:

Report from /etc/init.d/mpt-statusd on MyPersonalDomain.org

For fellow mail geeks, you should be able to read the above and realize right away that a misconfigured mail server in Russia was causing all the problems by pretending to be MyPersonalDomain.org.

But for the slightly less geeky, here’s a walk-through of what was happening at every step of the message’s journey from the naughty server in Russia server all the way to my Google Apps inbox:

  1. A server claiming to be MyPersonalDomain.org (but that actually lives at 91.242.161.26 — which is an IP in Russia and not my server) attempted to deliver a message to the authoritative inbound MX server for MyPersonalDomain.org, which is Google.
  2. The Google MX server (10.112.72.70) receives the mail, Google checks the sender’s SPF record, and finds that the message “fails” the check, since 91.242.161.26 is not a “permitted sender.” However, instead of rejecting it here (like I believe it should have), Google decides to keep processing the message and allow it to continue on its journey. That’s strike 1 against Google, IMHO.
  3. Because I have MyPersonalDomain.org configured as a domain alias in Google Apps with split domain routing, and because the [email protected] address doesn’t match any local Google Apps mail accounts, a Google outbound mail server (mail-lb0-f171.google.com [209.85.217.171]) sends the message on to the Backup.MyPersonalDomain.org server, which is the Postfix server I have set up to receive mail for all recipients on the MyPersonalDomain.org domain that don’t have Google Apps accounts (and bounce messages for any invalid addresses).
  4. The Backup.MyPersonalDomain.org server scans for viruses and spam, and finds the message is clean, and then delivers the message to the “steve” account on Backup.MyPersonalDomain.org server.
  5. The Postfix server on MyPersonalDomain.org is configured to forward any messages for [email protected] on to [email protected], so that’s what it does. Note, however, that because the message was forwarded from the IP address that’s allowed in the MyPersonalDomain.org SPF record, the accepting Google mail servers declare that it passes the SPF check for the original sender! So even though it failed an SPF check earlier in the journey, because it eventually gets forwarded to Google by a permitted server, it’s allowed through! 
  6. The message gets delivered to my Google Apps/Gmail inbox at [email protected].

I was disappointed that the first failed SPF check hadn’t stopped the message dead in its tracks. This message was clearly a hacked or misconfigured mail server that was spoofing my domain and sending out mail on my domain’s behalf… and doing so every two hours! This is precisely what SPF records are supposed to protect me against… and yet Google let it slip through.

So I dug deep into the Advanced Settings sections of Google Apps, and found what I thought to be a promising featured called Blocked Senders. My optimism faded, however, when I realized it was impossible to block senders by IP address. Google only allows users to block individual sender email addresses or domains. But since I receive valid server messages from [email protected] (that really are sent from my servers) every day, I couldn’t block the spoofed sender or the domain! That’s strike 2 against Google Apps. You can whitelist IPs, but you can only blacklist domains or email addresses (and an hour on the phone with Google Apps support confirmed this). That makes no sense!

My next idea was to forego the split domain routing for the [email protected] address, so I created an alias in Google Apps for [email protected] so that any messages received for that account would be delivered directly to my Google Apps inbox. I hoped that by eliminating the server from the delivery route with IP address allowed by the SPF record, Google Apps would block or send the next message received from the misconfigured server to my Spam folder.

However, two hours after the previous message, a new one arrived, with these headers:

Delivered-To: [email protected]
Received: by 10.224.78.135 with SMTP id l7csp21489qak;
        Sat, 30 Nov 2013 14:58:48 -0800 (PST)
X-Received: by 10.152.244.130 with SMTP id xg2mr40712496lac.4.1385852327864;
        Sat, 30 Nov 2013 14:58:47 -0800 (PST)
Return-Path: <[email protected]>
Received: from MyPersonalDomain.org ([91.242.161.26])
        by mx.google.com with ESMTP id h3si2355769lbd.96.2013.11.30.14.58.46
        for <[email protected]>;
        Sat, 30 Nov 2013 14:58:47 -0800 (PST)
Received-SPF: fail (google.com: domain of [email protected] does not designate 91.242.161.26 as permitted sender) client-ip=91.242.161.26;
Authentication-Results: mx.google.com;
       spf=hardfail (google.com: domain of [email protected] does not designate 91.242.161.26 as permitted sender) [email protected]
Received: by MyPersonalDomain.org (Postfix, from userid 0)
	id 9CF1488D; Sun,  1 Dec 2013 02:57:22 +0400 (MSK)
To: [email protected]
Subject: info: mpt raid status change on MyPersonalDomain.org
Message-Id: <[email protected]>
Date: Sun,  1 Dec 2013 02:57:22 +0400 (MSK)
From: [email protected] (root)

This is a RAID status update from mpt-statusd.  The mpt-status
program reports that one of the RAIDs changed state:

Report from /etc/init.d/mpt-statusd on MyPersonalDomain.org

Google still “hardfailed” the message when checking the SPF record, but still delivered it directly to my newly aliased Google Apps inbox. Sigh. That’s really bad. Why bother setting strict SPF records if large mail handlers like Google won’t honor them?!

I tried contacting the admins of the IP address in question, but with no luck. Probably some Russian hackers, anyway. I ran a port scan on the remote IP, and found:

# nmap 91.242.161.26

Starting Nmap 5.21 ( http://nmap.org ) at 2013-11-30 21:39 PST
Nmap scan report for 91.242.161.26
Host is up (0.19s latency).
Not shown: 988 closed ports
PORT     STATE    SERVICE
22/tcp   filtered ssh
53/tcp   open     domain
80/tcp   filtered http
443/tcp  open     https
749/tcp  open     kerberos-adm
1111/tcp open     unknown
8081/tcp open     blackice-icecap
8082/tcp open     blackice-alerts
8083/tcp open     unknown
9080/tcp open     unknown
9081/tcp open     unknown
9100/tcp open     jetdirect

Nmap done: 1 IP address (1 host up) scanned in 12.93 seconds

So that tells me they’ve got the system locked down pretty tight, including some BlackICE anti-intrusion software. I tried poking around the open ports for anything that could help me identify and get in contact with the owners, but found nothing. I’m guessing it’s using those 8083, 9080, and 9081 ports for its outbound mail — which could be attempts at spam, or could just be misconfiguration.

Finally, I took the least elegant approach of all. I set up a content filter in Gmail to send any message with “Report from /etc/init/d/mpt-statusd on MyPersonalDomain.org” in the body directly to the Trash. I call that the least elegant, because that doesn’t bounce anything back to the misconfigured server, which tells the remote admin he/she has a problem.

So I’ve stopped the bogus messages every two hours, but I’ve also exposed what I believe to be a couple of weaknesses in the way Google handles hosted Google Apps mail domains. First, if they won’t want to strictly enforce SPF records, they should at least give the mail customer (me) the ability to do so. And second, they should allow blacklisting by IP address, so that customers can quickly silence misconfigured or spammy mail servers that are sending useless crap.

More from this site: