ietf-mta-filters
[Top] [All Lists]

Re: draft-freed-sieve-in-xml status?

2008-12-20 03:28:23

On Mon, Dec 15, 2008 at 5:18 PM, Ned Freed 
<ned(_dot_)freed(_at_)mrochek(_dot_)com> wrote:
Ned Freed writes:
 is there a compliance test suite?

Not as far as I know.

I wrote the guts of something like that after Prague IETF, where there
was a discussion about advancing sieve to draft. I stopped working on
it that when various people didn't reply to email about it (I did not
feel up to a one-man crusade), but people seem to care more now and
I'll return to it when one of my current drafts passes IESG.

I certainly have no plans to write one or subject our implementation
to any that someone else develops. I doubt if anyone else does
either.

And I do hope you will review the document and run the code.

Not if it's labelled as a compliance test. Again, I have no objections to
unit
tests or sample scripts and use them all the time.

Compliance tests are another matter entirely. I realize evenyone has nothing
but the best intentions here, but you know what's paved with those...

If we release something we call a compliance suite it would be terribly easy
for someone to take advantage of all that nice new copyright boilerplate
that
allows unrestricted use and take the code and turn it into a for-profit
compliance test. It's but one step from there to RFPs requiring passage
of that test.

Really, you cannot know just how radioactive these things can be until
you've
gone through a bunch of them. You may think that such tests are amenable to
a
bright line standard of pass/fail, but that's just not how it ever ends up.
There's always interpretation of results involved, especially when  you're
probing for implementation limits (which these things always seem to do),
and
that leads to, um, let's call them "discussions" between the implementor and
tester.

+1

the economics of Compliance Testing have been the principle cause of
the problems at the JCP. experts have to be employed to create,
maintain and supervise TCKs. however, experience at the JCP has also
shown that it is possible to create open source TCKs through
collaboration which are good enough to achieve reasonable compability
without the cost problems that beset traditional CTs.

i agree that issuing a samples RFC would be very useful especially if
these samples were created with the intention of elucidating
compliance. in other words, describing compliance cases which are not
CTs.

i'm not sure that issuing automatic test code through an RFC would be
a good plan. any code is likely to be tightly couple to the test
framework and langauge used to execute it (eg nUnit in .NET). i think
it would be better to just supply the sample data and a verbal
description of the use case.

i suggest that interested participants from the group set up an
offshore open source project using a liberal license (MIT, say) where
appropriate test fixtures could be developed, allowing easier
automated testing of implementations. compliance tests (in the sense
of tests created with the aim of demonstrating compatibility for
difficult cases) would be included as well as tests aimed at
development.  this would clearly flagged as an unofficial effort aimed
at encouraging the development good implementations (and so the spread
of the specification) rather than as an official part of the standards
effort.

- robert

<Prev in Thread] Current Thread [Next in Thread>