Hmm, sorry but I don't buy that the "correct way to validate" is not to validate the input.
Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Sure, the complete regex is complex, but it is defined and is hardly unconquerable. Look at Django's `EmailValidator` implementation for example [0] that is mature and well tested [1].
The author has not convinced me that ignoring validation is the right choice when options with a scope so thorough exist.
> Sure, the complete regex is complex, but it is defined and is hardly unconquerable.
If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses.
(it may, of course, be a context-free expression masquerading as a regular expression)
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)
I suspect that it will, but of course I could be wrong.
I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?
When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM…
If you accept that assumption, what about `addr-spec` isn't regular?
Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.)
> I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?
> When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`.
Well, sure. Let's look at what RFC 5322 defines as an addr-spec[1]:
Let's ignore quoted-string and obs-local-part for the moment. What is a dot-atom?
dot-atom = [CFWS] dot-atom-text [CFWS]
And what is CFWS?
CFWS = (1*([FWS] comment) [FWS]) / FWS
What's a comment?
comment = "(" *([FWS] ccontent) [FWS] ")"
So far, all of this has been matchable with a regular expression. But what's a ccontent?
ccontent = ctext / quoted-pair / comment
See that there? A comment is composed of a balanced pair of parentheses around, perhaps, another comment! Thus (this (is (a (heavily (commented (email \(address))))))foo@bar.example(some more (to prove (the point))) is a perfectly viable RFC5322 address!
Pair-balancing, of course, is impossible with regular expressions, since matching pairs requires push-down automata (which match CFGs) and cannot be done with finite-state machines, (which match regular expressions).
QED.
> Also, using that assumption, your "perfectly valud[sic] email addresses such as …" would appear to not be valid, as it has unbalanced quotes.
Nope, there are no unbalanced quotes in (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest): the first quote balances with the third, while the second quote is one of a quoted pair \" (which is allowed within a quoted-string, which is allowed within a local-part). It's all allowed per the spec.
I'll admit that it's a bit surprising, but it's true. One simply cannot match a valid RFC5322 addr-spec with a regular expression. One can, of course, match it with something which pretends to be regular but isn't really (as I noted).
Oh, shoot, I missed that. I saw "FWS" (meaning "folding white space"), and assumed that didn't include comments since they have nothing to do with folding whitespace.
I'm not sure if they are context-free, but talks about Parseable Expression Grammars, specifically Lua's LPeg, implies they might be. Conceptually, PEGs are to Context Free Grammars as RegEx is to Regular Expressions.
You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs.
Are those parens really permitted in the host name? I'm looking at RFC 3696.
> 3. Restrictions on email addresses
> .... The syntax of the domain part corresponds to that in the previous section.
and
> 2. Restrictions on domain (DNS) names
> Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space.
> Are those parens really permitted in the host name? I'm looking at RFC 3696.
RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1
It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \.
Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point))
Whether one should ever actually use such an address is, of course, another matter entirely.
The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
> The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
It is! Conceivably, it could be used to implement a new kind of domain-less mail system, e.g. foo@$megamail or something.
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)
If it doesn't, remind me to patch it to not accept that.
I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
> If it doesn't, remind me to patch it to not accept that.
You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference.
Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse?
> I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor.
In other words: it may be your site, but it's my address.
There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems. So something being in an RFC doesn't create any kind of binding requirement, and very often the things that get dropped are things that, on contact with the real world, turned out to be bad ideas.
Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.
> There are already things that are technically allowed by RFCs for various types of input but should be disallowed because they cause problems.
That may be true, but it's something else entirely for someone to drop them because the problems they cause … were enabled by him. An email address using characters you don't like is perfectly valid, and perfectly deliverable, right up until you refuse to deliver it. It doesn't cause any problems until you decide to make it cause a problem.
> Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
> Also, no halting-problem issues at all; my preferred approach would be to disallow certain class
I really don't see why you think it's a bad idea to disallow more than alphanumeric (and plus? and ampersand?) characters in the local-part. Cui malo?
I would prefer to allow plus, dot, hyphen, underscore, and alphanumeric characters in the local part and probably not much of anything else.
I prefer this because I work in a world where email addresses are provided to me, and possibly stored and then retrieved and used and even displayed, as text, and the RFCs allow some nightmarish things when you consider the interaction of the syntax the RFCs permit and the set of characters which are sensitive to one or more of the non-MTA components of that chain.
Your original example, for instance, contains characters that require escaping or at least careful handling for multiple situations, and even goes so far as to contain things that will be interpreted as escape sequences in some contexts. I'm sure that Robert(';DROP TABLE users;)(<script type="text/javascript">alert("Bobby Tables")</script>)@not.a.hacker(honest) will be terribly disappointed to know that he needs to use a different address to register on my site. I'll send him over to you instead.
It is true that regular expressions in the CS sense can't parse context-free grammars. However, PCRE, which is what most programmers are talking about when they say "regex", can do so. So you're both kinda right, I guess. But you're being a bit pedantic.
Interesting; I've never had to think about how regex interact with Unicode.
I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.
By all means, validate. But err on the permissive side. The odds are pretty great that you're wrong, and you're preventing real email addresses (and almost certainly real users) by doing so.
I'd never looked closely at Django's email validation, but after looking at it I'm inclined to stop using it. Example reason:
> # max length for domain name labels is 63 characters per RFC 1034
There is a high likelihood that most of your users have shorter domain names, and also a high likelihood that most if not all registrars currently enforce this. But it also used to be highly likely that domains didn't begin with a numeral, and even though it was disallowed, 3m.com existed.
Another: neither the tests nor the patterns account for unicode, which is valid for domain names irrespective of the local part, and if not allowed for the local part by existing providers almost certainly will be eventually.
Also, a thing I noticed in the Django tests that is just upsetting:
- name: `TEST_DATA`
- values: `(validator, value, expected)` (a validator of the data is not the data)
- unstructured groups of such that should almost certainly be structured
The 63 character limit on domain name labels in RFC 1034/1035 isn't arbitrary, it's a limitation in the way DNS packets are encoded. The upper two bits of the label length are used for compression of labels in DNS packets.
Without protocol changes, DNS cannot support labels longer than 63 bytes.
However, I agree that validating it is unnecessary.
You can use a regex as a simple pre-check but you absolutely have to do more than that if you expect high-quality results.
Back in the 90s, we ran the customer rewards program mailing list for a mainstream business you've heard of. A [gnarly] regex took care of the gross failures but we still had double-digit percentage of invalid addresses and many spam reports because people mistyped their username, used their old ISP's email address which had been disconnected, etc.
The only approach which produced satisfactory results back then was to have a bit of (horrible) PHP code which did a full SMTP connection to deliver the welcome email. Even that wasn't enough to ensure you'd delivered it to the right person, however, so we had to track clicks on an activation link.
I would be surprised if the average internet user has gotten significantly more reliable over the last couple of decades and that's born out by the wide variety of misdirected legitimate email my shortname@gmail.com account receives.
Definitely, I think email validation links are important too. However, it's pretty senseless to let an obviously invalid email address pass all the way through to that layer (and potentially get billed for sending messages to invalid email addresses).
Obviously invalid can be a tough measure, though - many people who used domains other than .com/.net/.org have reported sites incorrectly denying their address. If I was still doing this, I'd start by doing some loose validation on the address and checking whether the domain resolved in DNS before returning a form validation error.
The other key step is having some contact form which doesn't require the same test so people can report bugs if you get the logic wrong.
The marketing or product team might prefer the "cost"(?) of sending a bad email here and there to the cost of losing a customer because the email is incorrectly rejected.
I am not sure what you are trying to say. Yes, of course, it costs money. Just like serving websites costs money. And just as checking the syntax of an email address costs money (CPUs cost money and need power to run!). Now, how is that an argument for checking the syntax of an email address to avoid sending a single email and seeing whether it bounces?
Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation.
Email validation is actually really important if you send even a medium amount of email. Testing by bouncing is not a very good idea, especially since there are a lot of ways to test the deliverability of the email before you try and send it.
That's not to say use a regex instead, you can do some simple checks on the domain etc or you can use one of the many services to check for you (for of a fee of course).
> Sending a lot of bounced emails is going to get your mailservers reputation tanked, which is not an acceptable tradeoff for removing email validation.
?!?
OK, let's break this down: What could be reasons for an email to be bounced due to an invalid address?
1. Because the address is syntactically invalid: Your own email server will notice and reject/bounce the email. Noone outside your own server will notice.
2. Because the domain doesn't exist or has no inbound mailserver: Your own email server will do the DNS lookup(s) and bounce the email. Noone outside your own server will make any note of that (and you can't find out without doing DNS lookups anyway).
3. Because the localpart doesn't exist: You can't find that out without trying to actually send an email (and consequently risking a bounce).
So, what exactly was the point of email validation "without bounces" again?
The only thing you can actually do to avoid unnecessary bounces is to make sure your emails have a valid return path and to make sure that you somehow process all the bounces that arrive there to make sure that you don't continue sending emails to addresses that start sending bounces (i.e., that don't exist (any more)). If you send lots of mails, it's probably best to employ VERP for that.
> Validating against the RFC is somewhat complex though.
Hence, not obvious :-)
If there is one thing we can learn from all the misguided attempts at validating email addresses that we have seen over the years, it is that it is obviously not obvious which email addresses are syntactically invalid.
With respect to (1), the cases I had in mind as "obviously invalid" is something that doesn't contain all of the required components of an email address. For example, if the user submits their email as "foo" or "foo@bar" or "foo@bar." I should have provided examples as the way I wrote it was definitely ambiguous.
Both "foo@bar" and "foot@bar." are valid email addresses. "bar" can be a host name or a top-level domain name. "bar." is a top-level domain name.
That said, "foot@bar." may be valid, but ICANN says it should not be functional. They prohibit dotless domain names (that is, top-level domain names may not have A, AAA, or MX records).
"foo", ok, I agree with the blogpost, if there isn't even an @ in there somewhere, maybe it's sensible to catch that and reject it outright. But that's probably about it.
Right. Wholeheartedly agree. Sanitize all input. If you _KNOW_ input you're receiving is invalid, throw it away.
Also whether you use RegExp on email or not totally depends on what you're doing with the address. Are you throwing that email into Salesforce? Great! Salesforce does RegExp validation on all email addresses and the field is required. You _HAVE_ to do it. Solution? Use the same validation that Salesforce does. That's what I do.
Also the OPs math about how common invalid email addresses are is wrong. Do you know what the most common way of screwing up an email address is? Hitting tilde when you're tabbing away from the field. And that error is in the tenths or hundreds of a percent range (which is still significant). I have a dataset of 5MM email addresses typed in by jaded call center agents to prove it.
> Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Yes, they aren't special, and you shouldn't validate any of the other stuff either, unless you actually need to understand the semantics. So much breakage happens because people think that they need to validate all kinds of stuff, it's not funny. People have their legal real names rejected because they're "not a valid name", people have their correct postal address rejected because it's "not a valid postal address" ... just forget it, if someone tells you their name, just believe them it's their name, if someone tells you their postal address, just believe them it's their postal address, if someone tells you their email address ...
Oh, and don't ever think about "sanitizing" stuff. Just don't. If you think you need it, you are doing something wrong. The solution to SQL injection is not to disallow people using "'" characters in messages, the solution to cross site scripting is not to prevent people from using "<" characters in comments, ... that's what encoding/escaping is for.
Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Sure, the complete regex is complex, but it is defined and is hardly unconquerable. Look at Django's `EmailValidator` implementation for example [0] that is mature and well tested [1].
The author has not convinced me that ignoring validation is the right choice when options with a scope so thorough exist.
[0]: https://github.com/django/django/blob/master/django/core/val...
[1]: https://github.com/django/django/blob/a9215b7c36bff232bcc941...