At 11:09 PM -0800 1/26/02, Kyle Lussier wrote:
I seem to be getting two conflicting viewpoints:
#1 Vendors can only be trusted to be interoperable on their own,
and can not be forced to conform.
#2 Vendors absolutely can't be trusted to be interoperable,
without conformance testing.
Missing is "#3 Vendors can't be trusted to be interoperable without
interoperability testing." VPNC performs conformance testing for
IPsec, and there are plenty of examples of our members who conform
but do not interoperate without a lot of knob twiddling and an
occasional bug fix.
The long experience with IPsec interoperability events (which VPNC
does not hold) has shown that A, B, and C might conform, and A and B
can interoperate fine, but A and C cannot interoperate. This is
usually due to administrative interfaces either not having the right
knobs, the defaults for C being valid for conformance but not for
interoperability, or weird magic.
Is interoperability testing needed for end users? Possibly, but it
won't happen until someone comes up with a good business model for
the testing agency. When anyone comes up with one, I'd love to hear
it. VPNC was originally formed to do good interop testing for the
IPsec industry, but when we figured out what that would cost all of
the members, there was no longer any interest. The basic problem:
either each of the 35 members is responsible for running and
debugging the test with the other 34 members, or they are willing to
pay someone to run and debug the 1225 (35^2) tests for them. In the
former case, the best statement of why that was not attractive was
"if I have a staff person who has that much skill with our product
and the at least 100 hours it will take, I have much more important
work for them". In the latter case, there was immediate history of
another interop testing agency who both charged a large amount of
money to do the tests and a fair amount of vendor staff time to do
debugging in order to do about one fifth the number of tests. Thus,
VPNC is left doing conformance testing with verifiable results, which
is admittedly not nearly as valuable to end users. (See
<http://www.vpnc.org/conformance.html>, particularly near the end,
for more details.) VPNC also does some small-to-medium sized interop
demos, but these are not formal interop tests with formal results.
Of course, this is not to say that formal interoperability testing is
impossible. There are examples of where it happens today in the
Internet industry. But there are probably one or two orders of
magnitude of examples of where it does not happen. Informal
interoperabilty events, where there are no results published but lots
of good interaction between vendors, have helped the industry a great
deal but are largely invisible to end users (and still don't produce
the level of interoperability that people in this discussion say is
"required").
Does the IETF or ISOC want to get into either conformance or
interoperability testing? It is fairly safe to say this not going to
happen without a business model to pay for the short-term and
long-term costs. So far, no business model has appeared in the
discussion.
--Paul Hoffman, Director
--VPN Consortium