[Monetdb-developers] Testing code
Hi, I have written some code and now I would like to make changes and still be sure that the new changes are correct. So, I have written some scenarios in mal and put it in a Tests directory. I have read the README file under testing/ directory, but still haven't understood how I could generate the <scenario>.stable.out file. How could that file be generated? Even if I run Mtest.py <mytest>, then Mapprove.py the previously "wrong" output, and then again Mtest.py the test fails. Thanks, Babis
On Tue, Jan 10, 2012 at 08:25:21PM +0100, Sjoerd Mullender wrote:
.. and why / how does it fail? from the README: - A tool Mapprove.py is provided to copy the recent output from $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. Lines starting with a '!' (i.e. error messages) are eliminated during copying. Of course, $TSTTRGDIR/TST.test.{out,err} have to be checked and claimed correct before being approved! Well, that's about it. After running Mtest.py, run Mapprove.py to copy the just generated test output from $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. As Sjoerd pointed out, by default lines starting with a '!', (MonetDB error messages are usually tagged like this) are eliminated during copying, i.e., do not end up in $TSTTRGDIR/TST.test.{out,err}. If your test does generate expected errors (or other lines starting with a '!') that need to be approved as correct output, use Mapprove.py's "-f" option (see also `Mapprove.py --help` for details) to keep these lines. Then, check (by hand --- there is now way to automize this) whether the just generated and approved output in $TSTSRCDIR/TST.stable.{out,err} is indeed correct, and fix (by hand) otherwise. Then, run Mtest.py, again. If Mtest.py reportes a filing test, check the generated html pages to the the differences in detail. Please be aware that Mtest.py basically uses a simple "diff" to compare stable (approved) output and new output, i.e., you test need to produce exactly the same output in a deterministic way with each run to be suitable for Mtest.py. Hope this helps you further. Stefan
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Thanks for the feedback. It appeared that I was running Mtest.py and
mserver5 from different builds which caused the different output. The
html file produced by Mdiff contained only differences in the number
of lines starting with # or =, etc.
Babis
On Tue, Jan 10, 2012 at 10:00 PM, Stefan Manegold
participants (3)
-
Charalampos Nikolaou -
Sjoerd Mullender -
Stefan Manegold