[Monetdb-developers] Testing code
Hi, I have written some code and now I would like to make changes and still be sure that the new changes are correct. So, I have written some scenarios in mal and put it in a Tests directory. I have read the README file under testing/ directory, but still haven't understood how I could generate the <scenario>.stable.out file. How could that file be generated? Even if I run Mtest.py <mytest>, then Mapprove.py the previously "wrong" output, and then again Mtest.py the test fails. Thanks, Babis
On 2012-01-10 19:54, Charalampos Nikolaou wrote:
Hi,
I have written some code and now I would like to make changes and still be sure that the new changes are correct. So, I have written some scenarios in mal and put it in a Tests directory. I have read the README file under testing/ directory, but still haven't understood how I could generate the <scenario>.stable.out file. How could that file be generated? Even if I run Mtest.py <mytest>, then Mapprove.py the previously "wrong" output, and then again Mtest.py the test fails.
Thanks, Babis
In principle that's how it works. However, if there are expected errors in the output, you have to supply the -f (force) option to Mapprove.py. It will normally filter error messages out. -- Sjoerd Mullender
On Tue, Jan 10, 2012 at 08:25:21PM +0100, Sjoerd Mullender wrote:
On 2012-01-10 19:54, Charalampos Nikolaou wrote:
Hi,
I have written some code and now I would like to make changes and still be sure that the new changes are correct. So, I have written some scenarios in mal and put it in a Tests directory. I have read the README file under testing/ directory, but still haven't understood how I could generate the <scenario>.stable.out file. How could that file be generated? Even if I run Mtest.py <mytest>, then Mapprove.py the previously "wrong" output, and then again Mtest.py the test fails.
.. and why / how does it fail? from the README: - A tool Mapprove.py is provided to copy the recent output from $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. Lines starting with a '!' (i.e. error messages) are eliminated during copying. Of course, $TSTTRGDIR/TST.test.{out,err} have to be checked and claimed correct before being approved! Well, that's about it. After running Mtest.py, run Mapprove.py to copy the just generated test output from $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. As Sjoerd pointed out, by default lines starting with a '!', (MonetDB error messages are usually tagged like this) are eliminated during copying, i.e., do not end up in $TSTTRGDIR/TST.test.{out,err}. If your test does generate expected errors (or other lines starting with a '!') that need to be approved as correct output, use Mapprove.py's "-f" option (see also `Mapprove.py --help` for details) to keep these lines. Then, check (by hand --- there is now way to automize this) whether the just generated and approved output in $TSTSRCDIR/TST.stable.{out,err} is indeed correct, and fix (by hand) otherwise. Then, run Mtest.py, again. If Mtest.py reportes a filing test, check the generated html pages to the the differences in detail. Please be aware that Mtest.py basically uses a simple "diff" to compare stable (approved) output and new output, i.e., you test need to produce exactly the same output in a deterministic way with each run to be suitable for Mtest.py. Hope this helps you further. Stefan
Thanks, Babis
In principle that's how it works. However, if there are expected errors in the output, you have to supply the -f (force) option to Mapprove.py. It will normally filter error messages out.
-- Sjoerd Mullender
------------------------------------------------------------------------------ Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev
_______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Thanks for the feedback. It appeared that I was running Mtest.py and
mserver5 from different builds which caused the different output. The
html file produced by Mdiff contained only differences in the number
of lines starting with # or =, etc.
Babis
On Tue, Jan 10, 2012 at 10:00 PM, Stefan Manegold
On Tue, Jan 10, 2012 at 08:25:21PM +0100, Sjoerd Mullender wrote:
On 2012-01-10 19:54, Charalampos Nikolaou wrote:
Hi,
I have written some code and now I would like to make changes and still be sure that the new changes are correct. So, I have written some scenarios in mal and put it in a Tests directory. I have read the README file under testing/ directory, but still haven't understood how I could generate the <scenario>.stable.out file. How could that file be generated? Even if I run Mtest.py <mytest>, then Mapprove.py the previously "wrong" output, and then again Mtest.py the test fails.
.. and why / how does it fail?
from the README:
- A tool Mapprove.py is provided to copy the recent output from $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. Lines starting with a '!' (i.e. error messages) are eliminated during copying. Of course, $TSTTRGDIR/TST.test.{out,err} have to be checked and claimed correct before being approved!
Well, that's about it. After running Mtest.py, run Mapprove.py to copy the just generated test output from $TSTTRGDIR/TST.test.{out,err} to $TSTSRCDIR/TST.stable.{out,err}. As Sjoerd pointed out, by default lines starting with a '!', (MonetDB error messages are usually tagged like this) are eliminated during copying, i.e., do not end up in $TSTTRGDIR/TST.test.{out,err}. If your test does generate expected errors (or other lines starting with a '!') that need to be approved as correct output, use Mapprove.py's "-f" option (see also `Mapprove.py --help` for details) to keep these lines. Then, check (by hand --- there is now way to automize this) whether the just generated and approved output in $TSTSRCDIR/TST.stable.{out,err} is indeed correct, and fix (by hand) otherwise. Then, run Mtest.py, again. If Mtest.py reportes a filing test, check the generated html pages to the the differences in detail. Please be aware that Mtest.py basically uses a simple "diff" to compare stable (approved) output and new output, i.e., you test need to produce exactly the same output in a deterministic way with each run to be suitable for Mtest.py.
Hope this helps you further.
Stefan
Thanks, Babis
In principle that's how it works. However, if there are expected errors in the output, you have to supply the -f (force) option to Mapprove.py. It will normally filter error messages out.
-- Sjoerd Mullender
------------------------------------------------------------------------------ Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev
_______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
------------------------------------------------------------------------------ Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (3)
-
Charalampos Nikolaou -
Sjoerd Mullender -
Stefan Manegold