[Top] [All Lists]

Re: draft-freed-sieve-in-xml status?

2008-12-15 00:04:01

On Sun, Dec 14, 2008 at 6:20 PM, Ned Freed 
<ned(_dot_)freed(_at_)mrochek(_dot_)com> wrote:
is there a compliance test suite?

Not as far as I know. I certainly have no plans to write one or subject our
implementation to any that someone else develops. I doubt if anyone else 

The IETF isn't big on compliance test suites as a rule - the overarching 
here is interoperability, not compliance.

however, AFACT the charter of this group does include testing

Unit tests != compliance test suites != interop testing. And the differences
really matter.

And this IMNSHO is a very good thing. in the past I've worked on software,
notably an X.400 implementation, governed by standards that emphasize
compliance testing. What I've observed is that after passing multiple very
extensive compliance test suites, the software still failed to interoperate
worth a damn and required all sorts of tweaks before it could.

there is no silver interoperability bullet :-)

Actually, I sort of disagree. We've held interop test events in the past for
IMAP and SMTP that have been hugely successful in weeding out implementation
problems and making stuff place nice together.

It may not be a silver bullet, but it's surprisingly close. The main problem is
participation, or rather lack of it - getting some of the major vendors to show
up has been tough. And when there are popular implementations out there that
makes no pretense of following the standard - and there are -  it's a real

Now, Sieve presents special problems for the sort of testing that works best
for protocols. Sieve is a language, not a protocol, and that makes it kinda
tough to throw two implementation on the wire and see if they can communicate.
So the charter includes the idea of developing an interop test suite. I'm not
entirely clear how that's going to work, but hopefully it will do so by
building up a suite of scripts covering things various implementations have
found to be problematic and making sure other implementations handle them the
same way. (if I have time I'll toss in some of the unit tests we use.)

all the stuff i help to develop is agile FOSS (TDD and BDD). automated
tests will need to be developed so it's just a question of whether the
tests exist already and can be shared and reused between
implementations. the unit tests themselves are only really effectively
portable between the moderns (.NET, java, python) but it's the
verified example mappings which will take the time.

You now appear to be talking about unit tests. We use agile processes here as
well and unit tests are part of what we do. But these are not at all the same
as compliance tests or interop testing. Compliances tests focus on compliance
with the specification - it's implicit in the name. A proper set of unit
tests are written while looking at an actual implementation, and can look at
known edge cases in an implementation, regression tests to make sure reported
bugs don't appear, and code coverage. This is all great stuff and very

As a result of this and several other experiences I must confess to
considerable cynicism that compliance testing represents a path to
interoperability. In my experience it does not, and has been for the most 
a colossal waste of time.

automated testing has come a very long way in the last decade but i
understand your perspective

GUI testing may have changed  by becoming more automatable (or not - this is
not my area and I don't keep up with the state of the art), but I haven't seen
all that much change in most other areas. Back in the late 70s and early 80s
when I did math modelling software we actually did far more extensive unit
testing and had better code coverage than we do now.

IMO it is a method (but not the only one) of producing reliable
relatively bug free software. i think this is a reasonable
pre-requisite for good interoperability. a good suite should aim to
reduce the numbers of poor implementations which claim compatibility
rather than try to ensure that good implementations interoperate

I'm always cautious about extrapolating from limited data - and we haven't seen
all that much use of scripts being moved from one implementation to another.
But to the extent we have, the problems that have shown up have been interop
issues. Unfortunately there are several Sieve implementations out there that
have chosen to ignore the extensions we've defined and roll their own to do
things the base specification does not cover.


The bottom line, such as it is, is that there's effectively no leeway in the
copyright boilerplate you have to use if you want your stuff published as an
RFC. So I, and I suspect a lot of others, simply go with the flow and use
whatever we're told we have to use. If this is problematic for you, the 
to take that up is on the IPR WG list. It is not within our charter here to
consider such matters in any case.

does the IEFT require copyright assignment?

Yes, there's a thing called the IETF Trust that rights are assigned to.

In fact they're going even further and asking for people to sign an agreement
granting rights to stuff in old RFCs published before copyright assignment
was required. It isn't clear to me how this can handle stuff like the
RFCs I've written with, say. Jon Postel (sadly deceased) as the coauthor, but
again, IANAL and I really try to stay as far away from this as possible.