A great deal of complexity comes from the fact that standards are
rarely created in a vacuum.
In this case, RFC 3261 SIP had to be upward-compatible from RFC 2543
SIP. And the early design of RFC 2543 SIP was influenced (I am told)
by the idea that SIP messages should be able to go through HTTP
proxies. (That didn't work out, but the idea is less silly than it
seems, especially if you are contemplating organizations whose only
permitted outside net traffic goes through HTTP proxies...)
So, much of the SIP header formatting was taken from HTTP, which got
it from RFC 822 e-mail message formatting. And of course, once a
variation is permitted, you *can't take it out*, because some product
with significant market share depends on it.
There is also the necessity to allow for extensibility, which favors
the use of various sorts of separators rather than a fixed sequence of
fixed-length fields.
As for CR-LF, that is conformity to the ASCII standard by RFC 822,
where "CR" stands for "carriage return" and "LF" stands for "line
feed". Exactly why those functions were given separate control
characters I don't know, but I suspect it was to allow overstriking.
The CF-LF standard was largely followed back in 1982, especially in
Digital Equipment Corporation systems, which were the backbone of the
Arpanet. So CR-LF remains the conventional line ending in Internet
protocols. The use of LF alone seems to come largely from the Unix
lineage of systems.
There is also a strong tendency to favor protocols that are readable
and writable by humans as text strings (among other things, that makes
debugging easier), so formatting alternatives are added to make those
tasks easier.
Dale