The reason people disobey the
7-bit standard is that the 7-bit standard sucks.
True. Then again, the obvious alternative also sucks - at least in
the short term. Just because something sucks doesn't mean it's not
the best way to do something.
I do believe that migrating to utf-8 in message headers is the way
to go in the long term, and that a transition strategy similar to what
Dan is suggesting (support utf-8 on receipt soon, generation of utf-8
later) is ultimately the way to get there from here. (I suspect we
have very different ideas about what would be a reasonable interval
between those two events)
But there are several things I don't believe.
One, that this will return us to a world where messages are ordinary
text and can be treated as such. For instance, even if we allow utf-8
in message headers, there will still be a need for canonicalization of
certain fields before sending and/or before comparison of values
embedded in those fields,
Two, that this allows use of utf-8 in addresses or domain names.
Those are separate problems,
Three, that the existing ability of some user agents to display utf-8
in message headers is sufficient for proper processing of headers
containing utf-8,
Four, that user agents that support utf-8 receipt in the near term will
receive sufficient testing through ordinary usage to be reliable at
supporting it once generation of utf-8 in headers is endorsed.
(so some other testing will be needed - not the way IETF normally works)
Five, that supporting utf-8 rids user agents of the burden
of supporting 2047, IDNA, and similar encodings, at least for
reading/presentation purposes, probably within our lifetimes.
In other words, utf-8 is the right way to go for the long term -
but the devil is in the details and it's not nearly as simple
as it looks. And I do think that a two-step transition is necessary -
we can't just start cramming utf-8 into either email or usenet and
expect things to work reasonably.
I also think that usenet can serve as a good test case for email.
(translation: I'd much rather usenet suffer the initial disruption than
email, so email could perhaps learn from usenet's experience.)
--
Keith Moore http://www.cs.utk.edu/~moore/
27 February 1933 11 September 2001