This issue is old and familiar to us, and crops up about once every six months or so. I thought it might help to document the situation more publicly.
On Cisco firewalls (PIX or the newer ASA), various protocol inspection engines are available. Generally, they assist in tracking connections of IP traffic through the firewall. That is, for a protocol such as FTP various additional TCP connections are made alongside the original connection, and the firewall needs to know to allow these through. The inspection engines perform simple analysis of traffic to watch for and set up these so-called pinhole ports, on demand.
Trouble is, some of the inspection engines have suffered feature creep and now try to work out, I guess, the semantics of the exchange taking place. If the engine thinks the conversation contains “illegal” requests, it’s blocked.
In particular the SMTP inspection engine (also known as a fixup in the Cisco docs) is fairly notorious for messing about with email transfer and preventing successful delivery. At best you might experience mysteriously missing attachments, at worst the remote server simply sees a TCP connection reset and has no idea why delivery failed.
Here’s how to tell if your Cisco firewall is interfering with your mail server’s operation. Telnet to the mail server (we assume the firewall sits in front of it) on the standard port of 25, and look at the “banner” response. On a regular mail server the banner looks something like this:
host:~$ telnet oxmail.ox.ac.uk 25 Trying 129.67.1.161... Connected to oxmail.ox.ac.uk. Escape character is '^]'. 220 relay0.mail.ox.ac.uk ESMTP Exim 4.69 Thu, 26 Nov 2009 19:28:51 +0000
However on an affected server, the banner is noticeably different:
host:~$ telnet suspectserver.example.com 25 Trying 192.0.2.1... Connected to suspectserver.example.com. Escape character is '^]'. 220 *****************************************************************************
Disabling the SMTP fixup (which is on by default, I believe) enables mail to flow as it should. I recommend you do this on any PIX or ASA devices in your network.
Note that the fixup seems to interfere with email going through the firewall in both directions, and problems occur regardless of the mail server software being used in the communication (after all, all servers are speaking the same SMTP protocol language).
This is still a Issue today! with the ASA Cisco simply renamed it to ESMTP Inspection. but the telnet tells the story with the Banner replaced by asterisks…
again as it shows above; a good telnet should look like something like this:
-bash-4.2$ telnet BSF01.BARRACUDA.COM 25
Trying 64.235.145.81…
Connected to BSF01.BARRACUDA.COM.
Escape character is ‘^]’.
220 bsf01.barracuda.com ESMTP (ae8b7cc02fd084c5f6b12f5a779b2717)
EHLO MyMail.com
250-bsf01.barracuda.com Hello barracuda.com [64.235.147.73], pleased to meet you
250-SIZE 134217728
250-STARTTLS
250-PIPELINING
250-8BITMIME
250 HELP
Also some of the other important info is masked! most important for TLS IS the “STARTTLS”
if you continue your telnet session you will see something like this: (Below) (Anywhere you see rows of XXXX’s is the ASA masking these as well. with STARTLS masked your device cannot communicate over TLS. And, even if it is not masked (can start TLS) the ASA will interfere with the email packet to the point that most communications will be ended before the transmission can be completed…
220 **********************************************
EHLO MYMAIL.COM
250-My Domain-Special-02.me.com Hello [10.0.2.27]
250-SIZE
250-PIPELINING
250-XXXXXXXXXXXXXA
250-XXXXXXXXXXXXXXXXXB
250-8BITMIME
250-XXXXXXXC
250-XXXXXXD
250 XXXXE
This was causing issues with our ZixGateway defaulting down to a port delivery method instead of delivering the e-mail via TLS like it should have since both servers supported TLS. Thanks for this info!
Still causing problems to this day!
Yes, indeed. This caught out one of our departments a couple of weeks ago 🙁
Still a valid issue with 9.1 on an ASA
Thanks so much for this article that has saved the remaining hair that I haven’t yet torn out 🙂
Extremely helpful in fixing a massive thorn in my side from a Network Upgrade
–> April 2014!
Still helping people as of March 2014. Thanks!! And seriously Cisco, get it together.
Still helping people, three years later… I had this problem for months at two small biz clients, Googled the daylights out of it but never turned up anything pointing to the firewall. Then more recently I stumbled on a reference to the smtp fixup, realized that both of these clients have Cisco firewalls (one ASA5505 and one UC520), found your clear explanation, disabled the thing, and Bob’s your uncle. Or mine, now. Thanks so much for taking the time to put up the documentation.
~A fan in Boulder, CO
I was wondering what was causing the asterisks (*). Thanks for that, ill check the pix now!