On Oct 10, 2006, at 10:59 AM, Stefan Manegold wrote:
Hi Ronald,
indeed, make our testing system more robust would be desirable, however, we lack some man power to do so right now --- it's open source, hence, any contribution is more than welcome ;-)
I know the feeling, I have barely enough time to work on OSS projects that I do know more about (maintance of the Mac port of Python and development on PyObjC). I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.
In fact, the test in question are "known to fail" in some cases/on some platform, but not really "expected to fail", i.e., they are indeed bug --- though minor ones that do not have any severe impact on "normal operation".
That's indeed what I meant, bad english on my part. Sometimes the software doesn't do what you'd like it to do (aka contains bugs), but actually fixing the bug isn't high on the TODO list. Lots test harnasses don't have a way to deal with that, which can be annoying when you want to anything that's related to test driven development.
Tests (actually queries) that are "supposed to fail" are handled in our testing system by providing the correct/expected error message as stable reference output.
All tests that are "known to fail" actually do so because some system/situation yields different output than others, i.e., the tests themselves are not robust enough to produce identical or at least determenistic output in all cases. Hence, it's actually the tests that need to be changed, not the testing system.
In the stable release, there should not be any tests that "known to fail", i.e., tests for open bugs; those tests should only be in the development version.
Thanks for the explanations, I assumed that the stable release contained some tests for open bugs.
To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is.
I haven't looked at that yet, the size of the test output directory scared me :-0. Ronald