[MonetDB-users] MonetSQLdb python API

Ronald Oussoren

3 Oct 2006 3 Oct '06

8:27 a.m.

Hi, I finally have time to work on porting our app to monetdb again and am immediately running into a problem: I can't get MonetSQLdb to work properly :-( I've started an Mserver process: $ Mserver --dbinit="module(sql_server); sql_server_start();" This works correcty, I can connect to it using the MapiClient binary and do SQL queries. However, this python code fails: import MonetSQLdb c = MonetSQLdb.connect() It says: !ERROR mapi_reconnect: connect: Connection refused Traceback (most recent call last): File "t.py", line 2, in ? c = MonetSQLdb.connect() File "/opt/monet-4.10/opt/python2.4/lib/python2.4/site-packages/ MonetSQLdb/__init__.py", line 51, in Connect return Connection(**kwargs) File "/opt/monet-4.10/opt/python2.4/lib/python2.4/site-packages/ MonetSQLdb/__init__.py", line 117, in __init__ kwargs['user'], kwargs['password'], kwargs['lang']) File "/opt/monet-4.10/opt/python2.4/lib/python2.4/site-packages/ CMapi.py", line 26, in __init__ raise IOError(self.error_str()) IOError: Setup connection failed Looking into the source code for the python client it should try to connect to port 45123, which the server is listening to. What it actually seems to do is connect to port 42312, but I have no idea why it does so (and hunting this down through several layers of generated code, one of which is swig, is no fun at all). What am I doing wrong? Ronald

Attachments:

smime.p7s (application/pkcs7-signature — 3.5 KB)

Show replies by date

Fabian Groffen

3 Oct 3 Oct

8:32 a.m.

On 03-10-2006 10:27:29 +0200, Ronald Oussoren wrote:

...

Hi,

I finally have time to work on porting our app to monetdb again and am immediately running into a problem: I can't get MonetSQLdb to work properly :-(

I've started an Mserver process:

$ Mserver --dbinit="module(sql_server); sql_server_start();"

This works correcty, I can connect to it using the MapiClient binary and do SQL queries. However, this python code fails:

import MonetSQLdb c = MonetSQLdb.connect()

It says:

!ERROR mapi_reconnect: connect: Connection refused

What version of MonetDB are you using? As far as I know, MonetDB should listen to port 50000 for all languages these days. One reason for your connection refuse can be that you connect to the hostname of your computer, instead of localhost (127.0.0.1). By default the server only listens on the local loopback interface.

...

Looking into the source code for the python client it should try to connect to port 45123, which the server is listening to. What it actually seems to do is connect to port 42312, but I have no idea why it does so (and hunting this down through several layers of generated code, one of which is swig, is no fun at all).

Weird.

Ronald Oussoren

8:42 a.m.

On Oct 3, 2006, at 10:32 AM, Fabian Groffen wrote:

...

On 03-10-2006 10:27:29 +0200, Ronald Oussoren wrote:

...
Hi,

I finally have time to work on porting our app to monetdb again and am immediately running into a problem: I can't get MonetSQLdb to work properly :-(

I've started an Mserver process:

$ Mserver --dbinit="module(sql_server); sql_server_start();"

This works correcty, I can connect to it using the MapiClient binary and do SQL queries. However, this python code fails:

import MonetSQLdb c = MonetSQLdb.connect()

It says:

!ERROR mapi_reconnect: connect: Connection refused

What version of MonetDB are you using? As far as I know, MonetDB should listen to port 50000 for all languages these days. One reason for your connection refuse can be that you connect to the hostname of your computer, instead of localhost (127.0.0.1). By default the server only listens on the local loopback interface.

The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest version (daily snapshot of the release branch) to check if that changes things. Ronald

Fabian Groffen

8:45 a.m.

On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:

...

The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest

Correct, that version is quite old.

...

version (daily snapshot of the release branch) to check if that changes things.

Ok.

Ronald Oussoren

9:55 a.m.

On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:

...

On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:

...
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest

Correct, that version is quite old.

'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about: !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot. BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug. I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python. BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-) BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.

...

...
version (daily snapshot of the release branch) to check if that changes things.

Ok.

Upgrading has solved the problem I was having. I can now perform SQL queries from Python. The MonetSQLdb module doesn't contain a callable named 'Binary'. Is that intentional? This callable is required according to PEP 249. The same is true for the value BINARY which isn't present as well (although it is listed in __all__). SQLObject needs the Binary callable, I'll send a patch when I have a working version of this. Ronald

Fabian Groffen

10:38 a.m.

On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:

...

On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:

...
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:

...
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest

Correct, that version is quite old.

'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

That doesn't feel right.

...

This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.

BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.

There is a default involved here. In order to have monetdb's configure respect your CFLAGS, you have to axe out CFLAGS hackery from the configure script, e.g.: sed -i \ -e 's|CFLAGS="\$CFLAGS \\\$(X_CFLAGS)"||' \ configure || echo "Fixing configure failed!" Then configure with --disable-debug --disable-optimize

...

I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.

That's fine and should work as expected (--prefix=/opt/monetdb-${PV})

...

BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)

I think that's what PYTHONPATH is for, but I leave that to the python experts. If your solution is to write in the non-prefix to make some sort of alias, then it won't work, as you don't have to be root. Maybe you don't even want to do that. (Using another version of python, for example.)

...

BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.

I think slotting is not supported, and that the best way out of there is to just install into different prefixes.

...

Upgrading has solved the problem I was having. I can now perform SQL queries from Python.

That's great.

...

The MonetSQLdb module doesn't contain a callable named 'Binary'. Is that intentional? This callable is required according to PEP 249. The same is true for the value BINARY which isn't present as well (although it is listed in __all__). SQLObject needs the Binary callable, I'll send a patch when I have a working version of this.

I have no clue on this. Maybe some Python expert has.

Ronald Oussoren

10:53 a.m.

On Oct 3, 2006, at 12:38 PM, Fabian Groffen wrote:

...

On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:

...
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:

...
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:

...
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest

Correct, that version is quite old.

'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

That doesn't feel right.

That's what I thought as well ;-)

...

...
This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.

BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.

There is a default involved here. In order to have monetdb's configure respect your CFLAGS, you have to axe out CFLAGS hackery from the configure script, e.g.:

sed -i \ -e 's|CFLAGS="\$CFLAGS \\\$(X_CFLAGS)"||' \ configure || echo "Fixing configure failed!"

Then configure with --disable-debug --disable-optimize

Is this needed at all when I just want to compile with '-g'? -- disable-optimize --enable-debug seems to work correctly for this. I mentioned the need for --disable-optimize because most other programs with a configure script seem to have no problems with having both optimization and debugging enabled (GCC can compile code like that, even if debugging optimized code can be quite confusing).

...

...
I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.

That's fine and should work as expected (--prefix=/opt/monetdb-${PV})

It does, I was trying to avoid questions about why I wasn't using the RPMs.

...

...
BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)

I think that's what PYTHONPATH is for, but I leave that to the python experts. If your solution is to write in the non-prefix to make some sort of alias, then it won't work, as you don't have to be root. Maybe you don't even want to do that. (Using another version of python, for example.)

PYTHONPATH works as well, but I prefer .pth files unless I want to use multiple versions. Anyway, this seems to be a bug in the install target, I didn't notice this before but the python files are installed in ${monetPrefix}${pythonPrefix} instead of just $ {pythonPrefix}, MonetSQLdb.__file__ is '/opt/monet-4.12.1/opt/ python2.4/lib/python2.4/site-packages/MonetSQLdb/__init__.pyc' on my system. Just to be anal: you cannot share extensions between multiple versions of python. It happens to work in most cases, but python 2.x and 2.y (x != y) aren't binary compatible.

...

...
BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.

I think slotting is not supported, and that the best way out of there is to just install into different prefixes.

I did, but had both prefixes on LD_LIBRARY_PATH (that is, I wasn't paying attentention) Thanks for your help, Ronald

Stefan Manegold

11:22 a.m.

On Tue, Oct 03, 2006 at 12:53:37PM +0200, Ronald Oussoren wrote:

...

On Oct 3, 2006, at 12:38 PM, Fabian Groffen wrote:

...
On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:

...
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:

...
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:

...
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest

Correct, that version is quite old.

'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?

...

...
That doesn't feel right.

That's what I thought as well ;-)

...
...
This is on a linux system (SLES9). I aborted 'make check' because it didn't seem to make any progress except printing this error message a lot.

BTW. I also had to suply both '--enable-debug' and '--disable- optimize' to the configure script, without the latter configure claimed I couldn't have both --enable-optimze and --enable-debug.

that's absolutely correct and IMHO desired behavior: the default for the release is "--enable-optimze", and --enable-optimze & --enable-debug do conflict; hence, if you want/need --enable-debug, you also must switch off optimization (--disable-optimize).

...

...
There is a default involved here. In order to have monetdb's configure respect your CFLAGS, you have to axe out CFLAGS hackery from the configure script, e.g.:

sed -i \ -e 's|CFLAGS="\$CFLAGS \\\$(X_CFLAGS)"||' \ configure || echo "Fixing configure failed!"

this is IMHO only required if you really need to explicitly proved CFLAGS with configure; --disable-optimize, --enable-debug do (should) not require such hack(s)...

...

...
Then configure with --disable-debug --disable-optimize

Is this needed at all when I just want to compile with '-g'? -- disable-optimize --enable-debug seems to work correctly for this. I mentioned the need for --disable-optimize because most other programs with a configure script seem to have no problems with having both optimization and debugging enabled (GCC can compile code like that, even if debugging optimized code can be quite confusing).

gcc can (to a certain extend) compile with both debugging and optimization (at least -O2) enabled (I'm not sure whether it stil works with our "aggressive" optimization options in case of --enable-optimize, i.e., "-O6 -fomit-frame-pointer -finline-functions -malign-loops=4 -malign-jumps=4 -malign-functions=4 -fexpensive-optimizations -funroll-all-loops -funroll-loops -frerun-cse-after-loop -frerun-loop-opt"...) However, when using gdb on gcc compiled code with -g -Ox, (x>0), practice shows that things do not work as you expect them to work (e.g., source tracing not necessarily aligned with actual exection, etc.); hence, we consider only "-g -O0" as "save" option for (indeed) debugable code...

...

...
...
I'm building from source because I want to install into /opt/ monet-4.12.1 and link to a custom build of python.

That's fine and should work as expected (--prefix=/opt/monetdb-${PV})

It does, I was trying to avoid questions about why I wasn't using the RPMs.

...
...
BTW2. If monetdb is installed in a different prefix than python the python bindings won't end up on the default search path of the python interpreter. I've added a .pth file to site-packages to fix that, maybe 'make install' should do that automaticly. Building/installing using distutils would be even better :-)

I think that's what PYTHONPATH is for, but I leave that to the python experts. If your solution is to write in the non-prefix to make some sort of alias, then it won't work, as you don't have to be root. Maybe you don't even want to do that. (Using another version of python, for example.)

PYTHONPATH works as well, but I prefer .pth files unless I want to use multiple versions. Anyway, this seems to be a bug in the install target, I didn't notice this before but the python files are installed in ${monetPrefix}${pythonPrefix} instead of just $ {pythonPrefix}, MonetSQLdb.__file__ is '/opt/monet-4.12.1/opt/ python2.4/lib/python2.4/site-packages/MonetSQLdb/__init__.pyc' on my system.

Just to be anal: you cannot share extensions between multiple versions of python. It happens to work in most cases, but python 2.x and 2.y (x != y) aren't binary compatible.

...
...
BTW3. The version info for the shared libraries is incorrect, which means you cannot have two versions of monetdb installed at the same time, I got a Mserver 4.12.1 that claimed it was 4.10 and didn't work at all. I guess this is "don't do that" territory ;-). Removing the older installation solved this.

If Mserver claims to be 4.10, then it indeed is 4.10 --- to be more precise, it at least loads the 4.10 libmonet.so for the "known" location; in case you install both 4.12.1 & 4.10, you need to make sure that they are installed in different locations (prefixes), and your PATH (& LD_LIBRARY_PATH) point indeed to the version you intend to use.

...

...
I think slotting is not supported, and that the best way out of there is to just install into different prefixes.

I did, but had both prefixes on LD_LIBRARY_PATH (that is, I wasn't paying attentention)

Ok, then probably when calling the 4.12.1 Mserver, your 4.10 libdir was first in your LD_LIBRARY_PATH --- amaising that it did load the wrong libmonet at all --- this might also explain the make check problems you mentioned above ... Stefan

...

Thanks for your help, Ronald

Ronald Oussoren

11:59 a.m.

On Oct 3, 2006, at 1:22 PM, Stefan Manegold wrote:

...

On Tue, Oct 03, 2006 at 12:53:37PM +0200, Ronald Oussoren wrote:

...
On Oct 3, 2006, at 12:38 PM, Fabian Groffen wrote:

...
On 03-10-2006 11:55:52 +0200, Ronald Oussoren wrote:

...
On Oct 3, 2006, at 10:45 AM, Fabian Groffen wrote:

...
On 03-10-2006 10:42:24 +0200, Ronald Oussoren wrote:

...
The machine I was testing on is running 4.10.3 and is definitely listening on port 45123 and only that. I'll upgrade to the latest

Correct, that version is quite old.

'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?

This was with the wrong monetdb present, but the problem also exists with a correct installation. I managed to catch the head of the failure as well: Mtest.py: ERROR: 'Mtimeout -timeout 60 Mserver "--config=/home/ xsupport/src/monet/build/conf/MonetDB.conf" --debug=10 --set "monet_mod_path=/home/xsupport/src/monet/build/src/modules/ plain/.libs:/home/xsupport/src/monet/build/src/modules/plain:/home/ xsupport/src/monet/build/src/modules/contrib/.libs:/home/xsupport/src/ monet/build/src/modules/contrib:/home/xsupport/src/monet/build/src/ modules/calibrator/.libs:/home/xsupport/src/monet/build/src/modules/ calibrator:/home/xsupport/src/monet/build/src/modules/mnetcdf/.libs:/ home/xsupport/src/monet/build/src/modules/mnetcdf:/home/xsupport/src/ monet/build/src/mapi/.libs:/home/xsupport/src/monet/build/src/mapi" -- set "gdk_dbfarm=/home/xsupport/src/monet/build/dbfarm" --set "sql_logdir=/home/xsupport/src/monet/build/log" --set mapi_port=37305 --set monet_prompt= --trace --dbname=mTests` failed: !WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: moduleClient: module(builtin) load error. !ERROR: moduleClient: module(arith) load error. !ERROR: moduleClient: module(str) load error. !ERROR: moduleClient: module(constant) load error. !ERROR: moduleClient: module(bat) load error. !ERROR: moduleClient: module(algebra) load error. !ERROR: moduleClient: module(sys) load error. !ERROR: moduleClient: module(trans) load error. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. Something completely different: how do I coax the python DB-API implementation out of autocommit mode? import MonetSQLdb conn = MonetSQLdb.connect() cursor = conn.cursor() conn.commit() # -> raises RuntimeError Any updates you do through 'conn' are also instantationious, e.g. no transactions. I can use the 'START TRANSACTION' query to explictly start a transaction, but I then have to keep track of whether I've used 'START TRANSACTION' on one of the active cursors because executing START TRANSACTION inside a transaction seems to break that transaction. The effect I'm looking after is that of the interactive MapiClient when autocommit is disabled: you don't have to start transactions explicitly but are always in one. Ronald

Fabian Groffen

12:03 p.m.

On 03-10-2006 13:59:16 +0200, Ronald Oussoren wrote:

...

Something completely different: how do I coax the python DB-API implementation out of autocommit mode?

import MonetSQLdb conn = MonetSQLdb.connect() cursor = conn.cursor() conn.commit() # -> raises RuntimeError

Any updates you do through 'conn' are also instantationious, e.g. no transactions. I can use the 'START TRANSACTION' query to explictly start a transaction, but I then have to keep track of whether I've used 'START TRANSACTION' on one of the active cursors because executing START TRANSACTION inside a transaction seems to break that transaction.

The effect I'm looking after is that of the interactive MapiClient when autocommit is disabled: you don't have to start transactions explicitly but are always in one.

Isn't there a conn.setAutoCommit() or something? Switching off auto-commit, is done through a control command Xautocommit 0 (or 1 to enable it again). It is different from START TRANSACTION in that sense that upon a COMMIT the auto-commit is retained (switched off).

Ronald Oussoren

12:34 p.m.

On Oct 3, 2006, at 2:03 PM, Fabian Groffen wrote:

...

On 03-10-2006 13:59:16 +0200, Ronald Oussoren wrote:

...
Something completely different: how do I coax the python DB-API implementation out of autocommit mode?

import MonetSQLdb conn = MonetSQLdb.connect() cursor = conn.cursor() conn.commit() # -> raises RuntimeError

Any updates you do through 'conn' are also instantationious, e.g. no transactions. I can use the 'START TRANSACTION' query to explictly start a transaction, but I then have to keep track of whether I've used 'START TRANSACTION' on one of the active cursors because executing START TRANSACTION inside a transaction seems to break that transaction.

The effect I'm looking after is that of the interactive MapiClient when autocommit is disabled: you don't have to start transactions explicitly but are always in one.

Isn't there a conn.setAutoCommit() or something? Switching off auto-commit, is done through a control command Xautocommit 0 (or 1 to enable it again). It is different from START TRANSACTION in that sense that upon a COMMIT the auto-commit is retained (switched off).

I did find a mapi_setAutocommit function in MapiLib that isn't exposed through the DB-API interface. I've added a setAutocommit method to MonetSQLdb.Connection that calls the mapi method of the same name and that does work. Thanks again, Ronald

...

---------------------------------------------------------------------- --- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php? page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Stefan Manegold

12:13 p.m.

...

This was with the wrong monetdb present, but the problem also exists with a correct installation.

I managed to catch the head of the failure as well:

Mtest.py: ERROR: 'Mtimeout -timeout 60 Mserver "--config=/home/ xsupport/src/monet/build/conf/MonetDB.conf" --debug=10 --set "monet_mod_path=/home/xsupport/src/monet/build/src/modules/ plain/.libs:/home/xsupport/src/monet/build/src/modules/plain:/home/ xsupport/src/monet/build/src/modules/contrib/.libs:/home/xsupport/src/ monet/build/src/modules/contrib:/home/xsupport/src/monet/build/src/ modules/calibrator/.libs:/home/xsupport/src/monet/build/src/modules/ calibrator:/home/xsupport/src/monet/build/src/modules/mnetcdf/.libs:/ home/xsupport/src/monet/build/src/modules/mnetcdf:/home/xsupport/src/ monet/build/src/mapi/.libs:/home/xsupport/src/monet/build/src/mapi" -- set "gdk_dbfarm=/home/xsupport/src/monet/build/dbfarm" --set "sql_logdir=/home/xsupport/src/monet/build/log" --set mapi_port=37305 --set monet_prompt= --trace --dbname=mTests` failed:

!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: moduleClient: module(builtin) load error. !ERROR: moduleClient: module(arith) load error. !ERROR: moduleClient: module(str) load error. !ERROR: moduleClient: module(constant) load error. !ERROR: moduleClient: module(bat) load error. !ERROR: moduleClient: module(algebra) load error. !ERROR: moduleClient: module(sys) load error. !ERROR: moduleClient: module(trans) load error. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'.

And this happens on a "clean" machine with a from scratch installation of MonetDB 4.12.1 from CVS, i.e., cd <monetdb-sourcedir> ./bootstrap mkdir <monetdb-builddir> cd <monetdb-builddir> <monetdb-sourcedir>/configure make make check with <monetdb-builddir>=/home/xsupport/src/monet/build/ in your case ? To me it looks as if there are no modules yet/any more(?) in /home/xsupport/src/monet/build/src/modules/plain/ and/or /home/xsupport/src/monet/build/src/modules/plain/.libs/ ... ?? Could you please check that? Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Ronald Oussoren

12:36 p.m.

On Oct 3, 2006, at 2:13 PM, Stefan Manegold wrote:

...

...
This was with the wrong monetdb present, but the problem also exists with a correct installation.

I managed to catch the head of the failure as well:

Mtest.py: ERROR: 'Mtimeout -timeout 60 Mserver "--config=/home/ xsupport/src/monet/build/conf/MonetDB.conf" --debug=10 --set "monet_mod_path=/home/xsupport/src/monet/build/src/modules/ plain/.libs:/home/xsupport/src/monet/build/src/modules/plain:/home/ xsupport/src/monet/build/src/modules/contrib/.libs:/home/xsupport/ src/ monet/build/src/modules/contrib:/home/xsupport/src/monet/build/src/ modules/calibrator/.libs:/home/xsupport/src/monet/build/src/modules/ calibrator:/home/xsupport/src/monet/build/src/modules/mnetcdf/.libs:/ home/xsupport/src/monet/build/src/modules/mnetcdf:/home/xsupport/src/ monet/build/src/mapi/.libs:/home/xsupport/src/monet/build/src/ mapi" -- set "gdk_dbfarm=/home/xsupport/src/monet/build/dbfarm" --set "sql_logdir=/home/xsupport/src/monet/build/log" --set mapi_port=37305 --set monet_prompt= --trace --dbname=mTests` failed:

!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. !ERROR: moduleClient: module(builtin) load error. !ERROR: moduleClient: module(arith) load error. !ERROR: moduleClient: module(str) load error. !ERROR: moduleClient: module(constant) load error. !ERROR: moduleClient: module(bat) load error. !ERROR: moduleClient: module(algebra) load error. !ERROR: moduleClient: module(sys) load error. !ERROR: moduleClient: module(trans) load error. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'. quit(); !ERROR: interpret: no matching MIL operator to 'quit()'.

And this happens on a "clean" machine with a from scratch installation of MonetDB 4.12.1 from CVS, i.e.,

cd <monetdb-sourcedir> ./bootstrap mkdir <monetdb-builddir> cd <monetdb-builddir> <monetdb-sourcedir>/configure make make check

with <monetdb-builddir>=/home/xsupport/src/monet/build/ in your case ?

To me it looks as if there are no modules yet/any more(?) in /home/xsupport/src/monet/build/src/modules/plain/ and/or /home/xsupport/src/monet/build/src/modules/plain/.libs/ ... ??

Could you please check that?

For what it's worth, the installed copy seems to work correctly in this regard: $ /opt/monet-4.12.1/bin/Mserver --dbinit="module(sql_server); sql_server_start();" # Monet Database Server V4.12.1 # Copyright (c) 1993-2006, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB>quit(); I'll have a closer look at the end of the day. Ronald

...

Stefan

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Ronald Oussoren

5 Oct 5 Oct

8:31 a.m.

On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold wrote:

...

...
...
...
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?

I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures: !ERROR: Testing FAILED SIGNIFICANTLY ! 4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output I can send you the test results, I haven't attached them to this message because the zipfile with the results is about 6 MBytes. What I'm doing (on a SuSE Enterprise Linux 9 system on x86): $ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly I'll rebuild without debug and with optimization once I have my application running (which mostly requires some further tweaking of the SQLObject ORM for Python).

Stefan Manegold

9:12 p.m.

On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:

...

On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold wrote:

...
...
...
...
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?

I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:

!ERROR: Testing FAILED SIGNIFICANTLY !

4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output

that looks quite normal some tests are known to fail on some architectures; just compare your result to those of our nightly testing; for "Stable" (MonetDB 4.12.1) http://monetdb.cwi.nl/Development/TestWeb/Stable/index.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/MonetDB/.mTests103/ind... for "Current" (MonetDB 4.13.1) http://monetdb.cwi.nl/Development/TestWeb/Current/index.html http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB/.mTests103/in... If the same tests fail, that's fine (i.e., known to us, and considered "non-crucial at least for the "Stable" release). If different tests fail, please let us know, preferably with the differences/errors produced by those test.

...

I can send you the test results, I haven't attached them to this message because the zipfile with the results is about 6 MBytes.

As said, I don't expect the differences to be "crucial", but if you want a detailed analysis, please feel free to send me your results directly.

...

What I'm doing (on a SuSE Enterprise Linux 9 system on x86):

$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly

looks perfectly fine to me --- except from the make check problem --- I have to admit, though, that I never tried make check with the tar ball (only from CVS source so far) --- I'll later try it with the tar ball (from our "Daily Builds"), too...

...

I'll rebuild without debug and with optimization once I have my application running (which mostly requires some further tweaking of the SQLObject ORM for Python).

Stefan Manegold

10:54 p.m.

On Thu, Oct 05, 2006 at 11:12:52PM +0200, Stefan Manegold wrote:

...

On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:

...
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold wrote:

...
...
...
...
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

[...]

...
What I'm doing (on a SuSE Enterprise Linux 9 system on x86):

$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly

looks perfectly fine to me --- except from the make check problem --- I have to admit, though, that I never tried make check with the tar ball (only from CVS source so far) --- I'll later try it with the tar ball (from our "Daily Builds"), too...

Found it! [For the "insiders":] The problem is that with the source tarballs all .mil parts of the modules have already been extrated from the .mx files and reside readily in source tree. During `make`, they are hence(?) not copied/links to the build tree. RunMtest, however, that is used to run Mtest.py when `make check` is executed, assumes that the modules' .mil files are in the build tree (as they are in case we build from the CVS sources) and sets the monet_mod_path accordingly. Thus, Mserver cannot find the modules' .mil files, and hence no modules. There are three potential solutions: 1) We need to change the Makefiles to ensure that also with the source tarballs, `make` copies the modules' .mil files from the source tree to the build tree (and least in case they are not identical) 2) We patch configure and RumMtest.in so that RunMtest "knows" whether it use in a "from-CVS" context or a "from-tarball" context, and hence uses either builddir or srcdir, respectively, for the directories in the modpath that contain the .mil files of the modules. 3) We replicate the directories in RunMtest.in's modpath to point to both the source tree and the build tree. To be honest, I don't know right now how to do 1) or 2) quickly (and correctly), hence, I'll just try 3) --- see my checkins later tonight ... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Stefan Manegold

6 Oct 6 Oct

9:44 a.m.

Ronald, new source tarballs (and RPMs) for the daily builds of "Stable" release (4.12.1) are available at http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./ With these, `make check` should work fine --- at least it does for me. Please let us know, in case you in case you still have problems or encounter new ones. Stefan

...

On Thu, Oct 05, 2006 at 11:12:52PM +0200, Stefan Manegold wrote:

...
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:

...
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold wrote:

...
...
...
>'make check' for MonetDB-4.12.1 (downloaded today) complains a lot >about: > >!ERROR: interpret: no matching MIL operator to 'quit()'. >quit();

[...]

...
What I'm doing (on a SuSE Enterprise Linux 9 system on x86):

$ tar zxf MonetDB-4.12.1.tar.gz $ mkdir build $ cd build $ ../MonetDB-4.12.1/configure '--with-php=no' '--with-perl=no' '--with-python=/opt/python2.4/bin/python' '--with-python-incdir=/opt/python2.4/include/python2.4' '-with-python-library=/opt/python2.4/lib' '--with-python-libdir=/opt/python2.4/lib/python2.4/site-packages' '--prefix=/opt/monet-4.12.1' '--enable-debug' '--disable-optimize' $ make $ make check # -> this results in the 'quit' error mentioned earlier $ make install $ Mtest.py -r # -> works correctly

Ronald Oussoren

10 Oct 10 Oct

8:22 a.m.

On Oct 6, 2006, at 11:44 AM, Stefan Manegold wrote:

...

Ronald,

new source tarballs (and RPMs) for the daily builds of "Stable" release (4.12.1) are available at http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./

With these, `make check` should work fine --- at least it does for me.

It works for me too. Thanks, Ronald

Ronald Oussoren

8:29 a.m.

On Oct 5, 2006, at 11:12 PM, Stefan Manegold wrote:

...

On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:

...
On Tuesday, October 03, 2006, at 01:22PM, Stefan Manegold wrote:

...
...
...
...
'make check' for MonetDB-4.12.1 (downloaded today) complains a lot about:

!ERROR: interpret: no matching MIL operator to 'quit()'. quit();

does this happen with only MonetDB-4.12.1 present on your system, or with a "legacy" MonetDB-4.10.3 installed as well (cf. BTW3 below)?

I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:

!ERROR: Testing FAILED SIGNIFICANTLY !

4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output

that looks quite normal some tests are known to fail on some architectures;

Hmm, wouldn't it be more useful to add some more logic to the testing infrastructure to deal with this? Your testing infrastructure seems to suffer from the same problem as a lot of other test harnasses: tests don't have two possible outcomes (pass and fail) but four: tests can not only pass or fail, but can also be expected to fail and do that or not. Adding support for testcases that are supposed to fail should be fairly easy and would reduce confusion when someone that isn't part the development team runs the testsuite ;-) Ronald

Stefan Manegold

8:59 a.m.

Hi Ronald, indeed, make our testing system more robust would be desirable, however, we lack some man power to do so right now --- it's open source, hence, any contribution is more than welcome ;-) In fact, the test in question are "known to fail" in some cases/on some platform, but not really "expected to fail", i.e., they are indeed bug --- though minor ones that do not have any severe impact on "normal operation". Tests (actually queries) that are "supposed to fail" are handled in our testing system by providing the correct/expected error message as stable reference output. All tests that are "known to fail" actually do so because some system/situation yields different output than others, i.e., the tests themselves are not robust enough to produce identical or at least determenistic output in all cases. Hence, it's actually the tests that need to be changed, not the testing system. In the stable release, there should not be any tests that "known to fail", i.e., tests for open bugs; those tests should only be in the development version. To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is. Stefan On Tue, Oct 10, 2006 at 10:29:44AM +0200, Ronald Oussoren wrote:

...

On Oct 5, 2006, at 11:12 PM, Stefan Manegold wrote:

...
On Thu, Oct 05, 2006 at 10:31:17AM +0200, Ronald Oussoren wrote:

...
I've tested again with a clean build of the current snapshot (downloaded half an hour ago), and this gives the same error. Oddly enough running 'make install' and then 'Mtest.py -r' does result in a finished test run. I do get a number of failures:

!ERROR: Testing FAILED SIGNIFICANTLY !

4 out of 313 tests could not be executed 1 out of 313 tests produced slightly different output 6 out of 313 tests produced SIGNIFICANTLY different output

that looks quite normal some tests are known to fail on some architectures;

Hmm, wouldn't it be more useful to add some more logic to the testing infrastructure to deal with this? Your testing infrastructure seems to suffer from the same problem as a lot of other test harnasses: tests don't have two possible outcomes (pass and fail) but four: tests can not only pass or fail, but can also be expected to fail and do that or not.

Adding support for testcases that are supposed to fail should be fairly easy and would reduce confusion when someone that isn't part the development team runs the testsuite ;-)

Ronald

Ronald Oussoren

9:17 a.m.

On Oct 10, 2006, at 10:59 AM, Stefan Manegold wrote:

...

Hi Ronald,

indeed, make our testing system more robust would be desirable, however, we lack some man power to do so right now --- it's open source, hence, any contribution is more than welcome ;-)

I know the feeling, I have barely enough time to work on OSS projects that I do know more about (maintance of the Mac port of Python and development on PyObjC). I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.

...

In fact, the test in question are "known to fail" in some cases/on some platform, but not really "expected to fail", i.e., they are indeed bug --- though minor ones that do not have any severe impact on "normal operation".

That's indeed what I meant, bad english on my part. Sometimes the software doesn't do what you'd like it to do (aka contains bugs), but actually fixing the bug isn't high on the TODO list. Lots test harnasses don't have a way to deal with that, which can be annoying when you want to anything that's related to test driven development.

...

Tests (actually queries) that are "supposed to fail" are handled in our testing system by providing the correct/expected error message as stable reference output.

All tests that are "known to fail" actually do so because some system/situation yields different output than others, i.e., the tests themselves are not robust enough to produce identical or at least determenistic output in all cases. Hence, it's actually the tests that need to be changed, not the testing system.

In the stable release, there should not be any tests that "known to fail", i.e., tests for open bugs; those tests should only be in the development version.

Thanks for the explanations, I assumed that the stable release contained some tests for open bugs.

...

To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is.

I haven't looked at that yet, the size of the test output directory scared me :-0. Ronald

Ronald Oussoren

9:36 a.m.

On Oct 10, 2006, at 11:17 AM, Ronald Oussoren wrote:

...

I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.

Speaking of which, what are the contraints w.r.t. supported Python versions? I noticed some code in MonetSQLdb.converters that refers to python 1.5. Is that still needed and if not, what is the minimal version of python that the python bindings should work with? Ronald

Niels Nes

11:02 a.m.

On Tue, Oct 10, 2006 at 11:36:31AM +0200, Ronald Oussoren wrote:

...

On Oct 10, 2006, at 11:17 AM, Ronald Oussoren wrote:

...
I'm working on a patch that brings MonetSQLdb more in line with the DB-API 2.0 specification, but I'm afraid that's probably it for the forseeable future.

Speaking of which, what are the contraints w.r.t. supported Python versions? I noticed some code in MonetSQLdb.converters that refers to python 1.5. Is that still needed and if not, what is the minimal version of python that the python bindings should work with?

Indeed 1.5 is very old. We could go up to 2.3.5 (our debian systems have that). Niels

...

Ronald

...

------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

Stefan Manegold

11:06 a.m.

On Tue, Oct 10, 2006 at 11:17:13AM +0200, Ronald Oussoren wrote:

...

On Oct 10, 2006, at 10:59 AM, Stefan Manegold wrote: [...]

...
To analyse/assess your test results in detail, I'd need to know which tests fail, and what the output resp. the difference to the correct/expected output is.

I haven't looked at that yet, the size of the test output directory scared me :-0.

Ronald

6726

Age (days ago)

6733

Last active (days ago)

List overview

Download

23 comments

4 participants

participants (4)

Fabian Groffen
Niels Nes
Ronald Oussoren
Stefan Manegold