It's nearly 2015, we are in a post-Snowden, and yet we are still considering the design pattern of "start with untrusted/unencrypted connection, and then just use an optional 'upgrade' command to make it encrypted/secure" acceptable.
One Word: SSLStrip
We have had to completely overhaul HTTP/HTTPS because this "start insecure, go to secure" just does not work. And even then, its been a series of hacks like Strict Transport Security, preloading lists for HSTS into browsers, OSCP Stapling, hard coding revoke lists, etc.
And now we need to go through all this pain again for mail?!?!
This is incredibly stupid and short sighted. Any network protocol that is sending personal/private information, or sends authentication credentials, needs to do so over an encrypted channel. New protocols get a single port, encryption is required. Legacy protocols which do not have an encrypted channel, like SMTP, POP, IMAP, FTP, HTTP, need a separate, dedicated port number for the encrypted version.
Its support is routinely required and enforced by both clients and servers. Servers won't let clients do AUTH without STARTTLS first. Clients won't send emails if they are set to use it and it's not supported. One can certainly misconfigure the hell out of mail server, but then no "secure from start" will stand a chance against these leet skills either.
Besides, STARTTLS dates back to 2002, so not sure what's up with your "?!?!" and "now we need to ..." comments. Looks like a basic rant to me :-/
I think you're underestimating the number of poorly configured/written SMTP clients and servers out there in the wild. As an example, here's smtp.comcast.net on tcp/587 (SMTP submission):
Connection to smtp.comcast.net port 587 [tcp/submission] succeeded!
220 resomta-ch2-12v.sys.comcast.net comcast ESMTP server ready
EHLO [2601:b:x:x:x:x:x:x]
250-resomta-ch2-12v.sys.comcast.net hello [2601:b:x:x:x:x:x:x], pleased to meet you
250-HELP
250-AUTH LOGIN PLAIN
250-SIZE 36700160
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-STARTTLS
250 OK
AUTH LOGIN
334 VXNlcm5hbWU6
That's right, the customer-facing SMTP server for the largest ISP in North America will happily let clients authenticate via base64-encoded plaintext without doing STARTTLS first.
At least with tcp/465 (SMTPS), the SMTP session (and the potential to send plaintext credentials) won't even start until the TLS session is up. That limits the attack surface to, "Does the client do something stupid with TLS, like not verify the peer's certificate or accept an aNULL/eNULL ciphersuite?"
If the MUA has a "Use SSL?" checkbox in its SMTP settings and it's not checked, then the MUA will send its credentials in the clear on tcp/587 if the server advertises/accepts AUTH LOGIN/PLAIN/CRAM-MD5/etc. before/without SMARTTLS, like Comcast's SMTP servers do. This scenario can't happen with SMTPS.
With respect to the RFCs, it's not required to issue STARTTLS before AUTH so that's probably why billyhoffman said "optional". One is certainly free to use authentication without requiring encryption although, in practice, I'm not sure why you would want to do that (with an exception being when all MUA -> MTA traffic remains entirely within a private network, perhaps -- but even then, why not add encryption just for good measure?).
From our perspective, most people start (or at least should start) with the premise that email is not secure. There are things that we can do to make email more secure, but as an email service provider, we have to balance delivery with security. Most customers would rather have their email delivered to an inbox than us only deliver to an inbox where we can validate the TLS certificate presented by the server. In-fact, delivery is the entire reason companies like Mailgun exist.
That doesn't mean we don't care about security. We've rolled out opportunistic TLS about a year ago, we're working on rolling out mandatory TLS soon (with strict certificate verification), and have a slew of security feature coming down the pipeline that I think (some) customers will be really excited about. However, when it comes to delivery, the truth is that features like mandatory TLS are not really requested or demanded by customers. In the email world, delivery trumps security which is not surprising when you work from the premise that it is not secure to begin with.
Please don't take my comments as a knock on Mailgun. You guys rock.
It was more the frustration with the idea that 1) encryption is optional for certain network protocols, and that 2) you upgrade a plaintext connection to a secure connection. This simply fails in practice, and we have seen this time and again with HTTP/HTTPS.
1) is changing, which is good.
2) is something people keep pushing for because it should work. Afterall you can configure your server to require a STARTTLS, and have all your client's configured to always use STARTTLS and abort out if it doesn't become secure.
Stop that. You are already making this to complicated. Crypto is hard enough as is. (See lower comment re: Comcast and STARTTLS). Ignoring the fact that you are mixing Layer 5/6 concepts into your Layer 7 application prototol, you are creating an insecure-by-default system. Its plaintext, unless you do these things. Stop that. You are doing it backwards. Make it a separate port, and SSL/TLS happens before the application protocol is even accessed. "Is this connection secure" and tracking state and transition into/out of that is not the job of an application protocol. Make this simple and boolean.
So it's sort of a chicken-and-egg situation then, huh? "It's not secure" => "Can't make security mandatory" => "It's not secure".
Off topic: Another thing I don't get is how do the relays validate each others' certs? You can get a postfix box up and running with a single command, no CA's involved. How does Mailgun know that it's talking to my mail server at mail.example.com? Or is that what DKIM doing? If so, then aren't we just relying on DNS to distribute public key info, which is not secure? Also, what if I don't publish DKIM?
In reality, it doesn't really matter much if TLS is used. Most MTAs will use TLS by default when it is available (i.e. if it sees the "STARTTLS" verb in the server's response) but (also by default) most won't attempt to validate the SSL certificate provided by the server -- it can be self-signed, expired, have a messed up certificate chain, or have a hostname mismatch and it will still be accepted.
Proper validation can be enabled, of course, but it is typically not by default. Of course, connecting over a "plain text" connection and then "upgrading" to TLS has its own problems, as you pointed out. The most significant is that a man-in-the-middle can simply filter out the "250-STARTTLS" response from the server and the client will not attempt to use encryption.
Unfortunately, in order to be RFC-compliant, an MTA cannot refuse to transfer mail just because encryption (TLS) is not available.
I wholeheartedly agree that the e-mail protocols need updated -- or perhaps even completely overhauled -- but I think it will be a long time before we see that happen. Besides, even if new protocols were introduced tomorrow that fixed all of this, it would be years before there was 100% adoption meaning that we'd still have to support the current protocols until then.
I don't think you can say, fairly, that "this is incredibly stupid and short sighted", though. I've been working professionally in I.T. for nearly 20 years and SMTP was still around way way before that. The problems that we are currently trying to solve simply did not exist then and almost certainly could not have even been imagined that long ago (recall that RFC733 was authored in November 1977 and RFC822 in August 1982).
Overall, SMTP has proven to be a fairly robust protocol but, as with anything, it needs to adapt.
I think you are missing something here / a clarification.
Almost all mail clients (MUAs) I've seen will validate the TLS certificate.
Almost no mail servers (MTAs) will validate the TLS certificate. This is because a) TLS isn't required, so a MITM could just strip the STARTTLS and everything would need to continue to work, and b) It doesn't provide any security.
TLS on the client is really securing the client credentials - not the email. TLS on the server is per hop - not to the final destination, so it doesn't protect a MITM attack.
If you want encrypted email, encrypt the email - don't use per hop encryption.
I had to configure Thunderbird for a coworker last week, connecting to the dreamhost mail servers, and it does validate the server certificates.
You have to either add an exception (works, but not ideal), or find the exact hostname which will pass Thunderbird's certificate validation (dreamhost uses a wildcard certificate for * .mail.dreamhost.com, so after digging on its wiki you find out that the correct alternative is in the form of subn.mail.dreamhost.com, where n is the same number on the mail server's real hostname, which you cannot use because it has two components where the * can match only one).
(Edit: can't make the asterisks show up correctly.)
(Edit: misread MTA as MUA. Yeah, most MTA's can't validate the certificates.)
"Unfortunately, in order to be RFC-compliant, an MTA cannot refuse to transfer mail just because encryption (TLS) is not available."
If you are running a mail system like Postfix, with DNSSEC and DANE turned on, then your server will refuse to deliver mail to mine unless it can negotiate a TLS encrypted connection, and the certificate used matches the cryptographically signed fingerprint I publish in the DNS.
Yes, sorry, I should have been more clear -- I was specifically referring to it being a violation of RFC2487 to require TLS on a publicly referenced SMTP server (on 25/TCP anyways) that is accepting mail.
One can obviously configure his/her own MTA to not send out mail unless encryption is being used (although one would certainly end up with undeliverable mail), but RFC2487 says that you may not refuse to accept mail (again, on 25/TCP) that is not encrypted.
As with DNSSEC and DANE, like you mentioned, one can also require his/her MTA to validate the server's certificate although this isn't typically enabled by default.
There is a strong argument that encrypted links between cleartext relay stations is of very limited use, and what should be encrypted is the message body.
We could do with another attempt to redefine email, even if it's unlikely to get deployed. It might be worth considering how to unify email with IM in the process.
I went through the whole stack of SMTP RFCs just this week and the takeaway was this.
TCP/25 if you are talking directly to the recipient's mail server,
though it's discouraged if you are not a mail server yourself.
TCP/587 - the same as above, but you are a mail client. In particular,
it's the port for talking to "your" server, which you are tasking with
relaying your email to the destination.
* If you have username/password and want to "login", you want TCP/587.
On the other hand
TCP/465 is a deprecated way of *securing* an SMTP connection.
It's just SMTP over TLS. Deprecated, discouraged, but still widely supported. Functionally it can be either TCP/25 or TCP/587, depending on the setup at the server end.
To secure 25/587, you first speak a bit of SMTP and then initiate the TLS handshake with STARTTLS command. It's all actually really simple. Much simpler than it seems when you are looking at the SMTP configuration UI in a typical mail client :)
And for those that still aren't familiar with TLS yet, think of it as simply SSL but newer and better. So, even though SMTP starts unencrypted the STARTTLS command initiates an encrypted session.
Most mail servers these days use TLS. So, there is a good chance that the email you send with your confidential pricing lists is being sent encrypted over the network. Unfortunately, there really isn't a good way to tell if that happened (at best you can view the mail headers which some servers add headers that show a TLS session was used between mail servers).
Yeah, it's usually possible to tell just from the Received: headers.
For example, here's a (slightly censored) Received: header from a recent e-mail received by my personal mail server:
Received: from mail.foo.com (mail.foo.com [192.0.2.8])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by mail.bar.com (Postfix) with ESMTPS id MX8675309
for <jlgaddis@bar.com>; Mon, 20 Oct 2014 18:31:26 -0400 (EDT)
N.B.: "with ESMTPS".
I'm sure that others will correct me if I'm wrong (please do!) and this almost certainly isn't foolproof (and I'm likely missing some and there are probably exceptions as well), but just from my own observations: "ESMTP" refers to a standard, unencrypted SMTP session (25/TCP), "ESMTPS" to an SMTP session where STARTTLS was used (25/TCP), "ASMTP" to an authenticated SMTP session where "full" SSL/TLS was used (e.g. 465/TCP), and "ESMTPSA" to an authenticated SMTP session where STARTTLS was used.
Of course, you can't see these on any outgoing messages you send, only incoming mail.
To add just a little more to your great summary or perhaps clarify a bit ...
MTA to MTA (mail server to mail server) traffic will use 25/TCP (with a very few "special case" exceptions).
MUAs (i.e. desktop clients) -- Outlook, Thunderbird, etc. -- should almost always use 587/TCP (the "submission" ports) and use authentication. This is almost certainly the case if your ISP is not also your e-mail provider.
ISPs will often allow their own users (or, more correctly, hosts on their own IP networks) to send mail through their own mail servers on 25/TCP without authentication, but not always. If not, use 587/TCP and authentication.
Many ISPs will block traffic destined for 25/TCP to hosts outside of their network, except from their own mail servers. In these cases, you'll need to either use your ISP's mail servers as relay hosts or send mail via port 587/TCP on your e-mail provider's mail servers (and you'll have to authenticate, of course).
Also, it is entirely possible that your ISP's "dynamic" IP addresses have been blacklisted by popular third party blacklists, e.g. the Spamhaus PBL [0] or Zen [1] lists (Zen incorporates the PBL and others into a single list). If the remote mail server you're attempting to connect to uses these RBLs, you'll be unable to send mail via that server, even if your ISP isn't blocking 25/TCP outbound (which is becoming more and more popular nowadays).
As latitude mentioned, the primary difference between 465/TCP (which, again, is deprecated but still fairly widely used) and 587/TCP is in the use of TLS. 587/TCP starts off as "plain text" and switches to "encrypted mode" upon the client's issuance of the "STARTTLS" command while with 465/TCP the entire conversation is encrypted (such as with HTTPS). Both will require authentication (unless, as mentioned above, you are using your ISP's own servers and they allow unauthenticated relaying from their own IP space) but encryption (TLS) is (usually) optional with 587/TCP.
For sake of completeness, on the "mail receiving" side, POP3 and IMAP operate similarly. The "regular" POP3 (110/TCP) and IMAP (143/TCP) ports often (but not always) support "STARTTLS" (converting the "plain text" connection to an "encrypted" one) while the POP3S (995/TCP) and IMAPS (993/TCP) ports are fully encrypted from start to finish (again, such as with HTTPS).
I run my own personal mail servers as well as those at my employer (an ISP) and the biggest (e-mail related) issue we seem to have is with "off-net" customers who are trying to send mail -- typically this is a business customer (or a business customer's employee) who sets up their mail client on a laptop while on our network (where authentication is optional) and then attempts to send mail via our server while outside of our network but is denied due to either lack of authentication or port 25/TCP being blocked. This is easily solved by reconfiguring the client to use SMTP authentication and/or 587/TCP.
If you're responsible for managing end users' mail clients, just set them up to use 587/TCP, STARTTLS, and authentication from the start and save yourself some trouble down the road.
"The RFC proposed a split of the traditional message submission and message relay concept."
What this is misses is that the former specifications never accounted for message submission in the first place. In the beginning, an MTA was installed on every system, and "sendmail" was used to get email submitted into the MTA. So there was no need for any other protocol to do this. But this only worked on Unix.
When less-well Internet-connected hosts came along (eg. Windows), there was a need to be able to submit mail to an MTA that was on a different host, since one was no longer present locally. Lacking a specification for this, MUA implementers just re-used (and overloaded) SMTP, by making the MUA pretend to be an MTA.
This caused a mess because MTAs were liberal in what they accepted (Postel's law), so they needed to tidy up badly formed submitted email such as fixing up headers. But this broke MTA to MTA transmission because email flowing that way ended up getting modified in unexpected ways, breaking old assumptions that MTAs generally only added headers.
And finally, more recently, the spam problem meant that MTAs needed to authenticate clients requesting relaying, since they didn't want to be labelled as spam originators.
Proposing "a split of the traditional message submission and message relay concept" was really just about sorting out the mess that we de-facto ended up in.
Since we are making progress to use secure mail transports, why are we not seeing more use of S/MIME to secure the contents of the message? I set up signing/encrypting on my mail clients, but have yet to see anyone, or any company, use it.
Zero demand? Wouldn't it be nice to send confidential email to business partners and know that it's secure?
I am surprised that large businesses have tolerated this for so long. Maybe SMIME isn't the answer, but it's surprising that we haven't seen some method to guarantee secure delivery of mail between companies.
It's unlikely that anything anyone has to say using "Mailgun" needs much security. It's basically for spamming, after all.
Yeah, I read the "transactional email" bullshit. That usually means spamming to somebody who at some time in the past had some vague connection with you. The fact that Mailgun touts their own API as a means to send more email fast means they're marketing to spammers. Most businesses don't do enough transactions per second to have this as a problem. Amazon and eBay do, and they don't use Mailgun.
I send registration welcomes / email confirmations via Mailgun. Likewise, my company does the same thing with a different provider. It's not spam, just handy.
Similarly, there are attacks on the transition between unprotected
and TLS-protected traffic. A number of IETF application protocols
have used an application-level command, usually STARTTLS, to upgrade
a clear-text connection to use TLS. Multiple implementations of
STARTTLS had a flaw where an application-layer input buffer retained
commands that were pipelined with the STARTTLS command, such that
commands received prior to TLS negotiation are executed after TLS
negotiation. This problem is resolved by requiring the application-
level command input buffer to be empty before negotiating TLS. Note
that this flaw lives in the application layer code and does not
impact the TLS protocol directly.
SMTP and IMAP clients are affected by this, so best practice is to use their encrypted ports. The use of STARTTLS-type techniques in future protocols is deprecated due to the downgrade risk.
This of course has a bit of a problem as 465 was a private assignment, later unfortunately reassigned to a Cisco protocol ("URD, a transitional technology for IGMPv2/IGMPv3 & SSM"), and some routers could mess it up: "routers running URD intercept all packets using port 465, regardless of destination". (You do NOT, however, want to run URD, and if you ever see it on the internet… well, think the whole thing through… if you have a Cisco, and find you have trouble with 465 through it sometimes, and haven't turned on URD yourself, maybe get your router forensically examined! <g>) That was on the whole not a good day for the IETF/IANA.
I have limited time this morning: so here's the thread on UTA, see for yourself. https://www.ietf.org/mail-archive/web/uta/current/msg00176.h... Long story short: using 465 is probably OK, lots of people are doing it (I am). Maybe use another port, too, in case you find a crazy misconfigured Cisco in-path, which complicates a dual-assignment.
The advice in the article is wrong: you are not in breach of an RFC if you use port 465 - and you probably should, but there's no clear consensus or resolution yet, so it may just get another port assignment (have I overlooked that happening?).
One Word: SSLStrip
We have had to completely overhaul HTTP/HTTPS because this "start insecure, go to secure" just does not work. And even then, its been a series of hacks like Strict Transport Security, preloading lists for HSTS into browsers, OSCP Stapling, hard coding revoke lists, etc.
And now we need to go through all this pain again for mail?!?!
This is incredibly stupid and short sighted. Any network protocol that is sending personal/private information, or sends authentication credentials, needs to do so over an encrypted channel. New protocols get a single port, encryption is required. Legacy protocols which do not have an encrypted channel, like SMTP, POP, IMAP, FTP, HTTP, need a separate, dedicated port number for the encrypted version.