In our last episode ("Re: DRAFT status and Compatibility testing", shown on
3/28/00), William H. Geiger III said:
IMHO what we need to do is put together a suite of test data that conforms
to RFC 2440.
[...]
Some of the non-RFC2440 packets should be included (X.509, PhotoID,
...ect) so vendors can test that their software can properly handle
non-RFC2440 packets without crashing (this has been a problem with the pks
software).
I think this is an excellent idea. 2440 is a data format standard, and
consequently, the test suite mostly should consist of data. The only real
comment I have on your list is that PGP/MIME isn't part of 2440, it's a
separate RFC. (And in my opinion, this is a feature, not a bug. It's called
layering.)
There's a related issue that I want to bring up, though.
Many messages, when generated require random numbers to be used. In a
non-security environment, we'd have no objections to saying, "Here are the
inputs, here is the final data, make sure you generate this." Do we have an
objection to saying, "With a session key of K, encrypt message M, and your
result should look like R"? I don't -- I consider this to be the equivalent
of a crypto test vector. On the other hand, it still gives me a small
shudder, and I think this is one of those things on which gentlepersons can
disagree. An alternative is to come up with a test suite where an
implementation generates a message with known keys, and then writes out the
session keys and signature randoms, so that they can be verified. But on
this one, I don't see it's any better.
The problem, as I see it, is that if an implementation has a test mode in
which a random can be specified, then that's potentially a way to create a
hacked version with bad or known keys. But on the other hand, if an
implementation has a test mode in which the randoms can be leaked, then
that's a way to make a hacked version with a covert channel. If I have to
pick between one of those risks, I pick the former.
On the third hand, if we just say "non serviam" to that, then we run the
risk that a hacked version could pass all compliance tests and no one would
know otherwise. I don't like measuring compliance when we aren't specifying
the inputs, either.
I think my opinion (yes, I'm tentative here) is that 2440 is a data format
standard. We already have been wise enough to not get it into the whole
mess of crypto engineering in it. Our goal in advancing this is to make
sure that given a set of inputs, an implementation produces the proper
results. Those inputs may be finished messages that must be decoded, or
they may be plaintext, ciphers, randoms, etc. that should generate a fixed
message. There's an engineering problem in making an implementation
testable and secure. But that's a problem that the implementors have to
solve.
Opinions?
Jon