Hi Ronald, indeed, make our testing system more robust would be desirable, however, we lack some man power to do so right now --- it's open source, hence, any contribution is more than welcome ;-) In fact, the test in question are "known to fail" in some cases/on some platform, but not really "expected to fail", i.e., they are indeed bug --- though minor ones that do not have any severe impact on "normal operation". Tests (actually queries) that are "supposed to fail" are handled in our testing system by providing the correct/expected error message as stable reference output. All tests that are "known to fail" actually do so because some system/situation yields different output than others, i.e., the tests themselves are not robust enough to produce identical or at least determenistic output in all cases. Hence, it's actually the tests that need to be changed, not the testing system. In the stable release, there should not be any tests that "known to fail", i.e., tests for open bugs; those tests should only be in the development version. To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is. Stefan On Tue, Oct 10, 2006 at 10:29:44AM +0200, Ronald Oussoren wrote:
On Oct 5, 2006, at 11:12 PM, Stefan Manegold wrote:
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:
I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:
!ERROR: Testing FAILED SIGNIFICANTLY !
4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output
that looks quite normal some tests are known to fail on some architectures;
Hmm, wouldn't it be more useful to add some more logic to the testing infrastructure to deal with this? Your testing infrastructure seems to suffer from the same problem as a lot of other test harnasses: tests don't have two possible outcomes (pass and fail) but four: tests can not only pass or fail, but can also be expected to fail and do that or not.
Adding support for testcases that are supposed to fail should be fairly easy and would reduce confusion when someone that isn't part the development team runs the testsuite ;-)
Ronald
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |