On Mon, Dec 15, 2008 at 4:04 AM, Ned Freed
<ned(_dot_)freed(_at_)mrochek(_dot_)com> wrote:
On Sun, Dec 14, 2008 at 6:20 PM, Ned Freed
<ned(_dot_)freed(_at_)mrochek(_dot_)com> wrote:
is there a compliance test suite?
Not as far as I know. I certainly have no plans to write one or subject our
implementation to any that someone else develops. I doubt if anyone else
does
either.
The IETF isn't big on compliance test suites as a rule - the overarching
goal
here is interoperability, not compliance.
however, AFACT the charter of this group does include testing
Unit tests != compliance test suites != interop testing. And the differences
really matter.
intention != execution
given the traditional problems that good interoperability and
compliance testing have offered, there's a movement to take another
look at these issues applying the lessons learnt from the experience
of development-centered automated testing.
And this IMNSHO is a very good thing. in the past I've worked on software,
notably an X.400 implementation, governed by standards that emphasize
compliance testing. What I've observed is that after passing multiple very
extensive compliance test suites, the software still failed to interoperate
worth a damn and required all sorts of tweaks before it could.
there is no silver interoperability bullet :-)
Actually, I sort of disagree. We've held interop test events in the past for
IMAP and SMTP that have been hugely successful in weeding out implementation
problems and making stuff place nice together.
It may not be a silver bullet, but it's surprisingly close. The main problem
is
participation, or rather lack of it - getting some of the major vendors to
show
up has been tough. And when there are popular implementations out there that
makes no pretense of following the standard - and there are - it's a real
problem.
interoperability conferences are very useful but expensive, and this
expense disenfranchises
for example, this year m$ paid for serious interoperability testing
(along the lines you suggestsed) in the web services space including
flying FOSS developers around the world. i've heard very good things
about this but most interesting long term movement is that M$ is now
engaging with many of these developers to develop automated test
suites which demonstrate continued interoperability amongst the group.
AIUI the problem is that F2F is very expensive but interoperability is
an ongoing problem. automated testing is now mainstream and the
techiques developed should be applicable more widely.
Now, Sieve presents special problems for the sort of testing that works best
for protocols. Sieve is a language, not a protocol, and that makes it kinda
tough to throw two implementation on the wire and see if they can communicate.
script based protocol testing can - with reasonable ease - be used to
automate this style of functional test
So the charter includes the idea of developing an interop test suite. I'm not
entirely clear how that's going to work, but hopefully it will do so by
building up a suite of scripts covering things various implementations have
found to be problematic and making sure other implementations handle them the
same way. (if I have time I'll toss in some of the unit tests we use.)
one of the problem with this approach to testing is that it does
nothing to help implementors develop the basic function correctly. i
understand that this is the skill in developing traditional compliance
and interoperability suites, and i can understand how good
implementations find this very useful. they do nothing to encourage
new implemetations to get the basics right which IMO is equally
important.
all the stuff i help to develop is agile FOSS (TDD and BDD). automated
tests will need to be developed so it's just a question of whether the
tests exist already and can be shared and reused between
implementations. the unit tests themselves are only really effectively
portable between the moderns (.NET, java, python) but it's the
verified example mappings which will take the time.
You now appear to be talking about unit tests. We use agile processes here as
well and unit tests are part of what we do.
i would prefer to avoid splitting hairs about what exactly is a unit test
But these are not at all the same
as compliance tests or interop testing. Compliances tests focus on compliance
with the specification - it's implicit in the name. A proper set of unit
tests are written while looking at an actual implementation, and can look at
known edge cases in an implementation, regression tests to make sure reported
bugs don't appear, and code coverage. This is all great stuff and very
valuable.
when developing against a specification, a range of automated tests
are valuable. these may well include automated created with unit
testing tools that aren't really best described as unit test (as well
as developer created tests that use other testing tools).
for example, when developing an xml serializer for sieve, i would
expect to create unit tests for each object in the pipeline, BDD
integration tests for partially assembled pipelines and functional
sieve script -> xml document tests. tests in the first category will
only be use internally, the second are only likely to be useful for
libraries sharing the interfaces defined but the functional sieve ->
xml document tests should be useful more generally. IMHO pooling the
third category would be worthwhile.
i agree that this isn't sufficient for a good compliance test suite -
it's unlikely to provide good enough coverage of difficult cases which
a compliance expert would create - but it should provide a reasonable
starting point and reduce the effort involved by allowing experts to
focus exclusively on difficult cases.
As a result of this and several other experiences I must confess to
considerable cynicism that compliance testing represents a path to
interoperability. In my experience it does not, and has been for the most
part
a colossal waste of time.
automated testing has come a very long way in the last decade but i
understand your perspective
GUI testing may have changed by becoming more automatable (or not - this is
not my area and I don't keep up with the state of the art), but I haven't seen
all that much change in most other areas. Back in the late 70s and early 80s
when I did math modelling software we actually did far more extensive unit
testing and had better code coverage than we do now.
automated developer testing (TDD BDD etc) has now mainstreamed.
generally testing toolsets such as xUnit and xMock drove the rise of
FOSS and the (post-)modern languages (java, .NET, python, ruby) over
the last decade. this isn't to say that developers didn't test before
but automated testing and the agile methods it enables have
transformed the development landscape.
IMO it is a method (but not the only one) of producing reliable
relatively bug free software. i think this is a reasonable
pre-requisite for good interoperability. a good suite should aim to
reduce the numbers of poor implementations which claim compatibility
rather than try to ensure that good implementations interoperate
perfectly.
I'm always cautious about extrapolating from limited data - and we haven't
seen
all that much use of scripts being moved from one implementation to another.
But to the extent we have, the problems that have shown up have been interop
issues. Unfortunately there are several Sieve implementations out there that
have chosen to ignore the extensions we've defined and roll their own to do
things the base specification does not cover.
is there any (lightweight) way for implementors to let this group know
about the extensions they've rolled? (other than showing on this list)
<snip>
The bottom line, such as it is, is that there's effectively no leeway in
the
copyright boilerplate you have to use if you want your stuff published as
an
RFC. So I, and I suspect a lot of others, simply go with the flow and use
whatever we're told we have to use. If this is problematic for you, the
place
to take that up is on the IPR WG list. It is not within our charter here to
consider such matters in any case.
does the IEFT require copyright assignment?
Yes, there's a thing called the IETF Trust that rights are assigned to.
copyright assignment should be unnecessary unless the IEFT intends to
sue (which IMO would not be a positive development)
In fact they're going even further and asking for people to sign an agreement
granting rights to stuff in old RFCs published before copyright assignment
was required. It isn't clear to me how this can handle stuff like the
RFCs I've written with, say. Jon Postel (sadly deceased) as the coauthor, but
again, IANAL and I really try to stay as far away from this as possible.
does the IEFT plan to grant back an unlimited exploitation license to
the original authors?
- robert