there is no perfect foresight. there is no perfect hindsight either.
nor is the market either efficient or reliable at choosing good solutions.
It appears the word "market" is overloaded with lots of social/political
ideology so rather than go down that road, let me redirect the point by
simply observing that running code allows the designer(s) to get
feedback about a design and its implementation implications, which is
the point about "letting the market decide".
Within IETF, "running code" has usually meant interoperability tests of
multiple implementations of a single protocol. These days we find we need
to be more concerned about the interaction between protocols than we used to
be. The notion that protocols can be examined in isolation no longer holds.
So for instance: we expect non-TCP-based protocols to share bandwidth fairly
with TCP-based protocols; and we expect all protocols to be reasonably secure
because when even a single protocol is compromised it can disrupt operation of
the network, other protocols, and hosts that weren't directly compromised.
The "market" is not in a good position to evaluate these characteristics
because many of these effects do not become visible to the "market" until
the "market" has already substantially invested in a particular path.
Indeed, the very discipline of "engineering" exists to minimize such
catastrophes. If you make design decisions based on guesswork, that's
guesswork; if you make design decisions based on analysis of how well a
solution meets well-defined criteria, that's engineering.
These days, a lot of people seem to be arguing that IETF shouldn't do
engineering.