June 2009 - developers-list

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/python/monetdb mapi.py, May2009, 1.4.2.4, 1.4.2.5 mapi25.py, May2009, 1.1.2.4, 1.1.2.5
by Stefan Manegold 13 Jun '09

13 Jun '09

On Fri, Jun 12, 2009 at 01:23:36PM +0000, Gijs Molenaar wrote: > Update of /cvsroot/monetdb/clients/src/python/monetdb > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv13219/monetdb > > Modified Files: > Tag: May2009 > mapi.py mapi25.py > Log Message: > > * Fixed python3.0 errors. Please also run the tests with python3 when you > commit something Just for info: None of the machines in our testing pool runs/provides Python3.0 (as default), yet; nightly testing uses Python 2.4, 2.5, 2.6. Stefan > * Unicode/UTF-8 should be working now, except for a small python3 problem > > * python2.6 uses the mapi25.py now, since this is easier to manage. > mapi25.py will be renamed in the future > > * added unicode and UTF-8 test > > > U mapi.py > Index: mapi.py > =================================================================== > RCS file: /cvsroot/monetdb/clients/src/python/monetdb/mapi.py,v > retrieving revision 1.4.2.4 > retrieving revision 1.4.2.5 > diff -u -d -r1.4.2.4 -r1.4.2.5 > --- mapi.py 29 May 2009 12:01:20 -0000 1.4.2.4 > +++ mapi.py 12 Jun 2009 13:23:34 -0000 1.4.2.5 > @@ -130,7 +130,7 @@ > logging.debug("II: executing command %s" % operation) > > if self.state != STATE_READY: > - raise(ProgrammingError, "Not connected") > + raise ProgrammingError("Not connected") > > self.__putblock(operation) > response = self.__getblock() > @@ -154,24 +154,19 @@ > > if protocol == '9': > algo = challenges[5] > + import hashlib > if algo == 'SHA512': > - import hashlib > - password = hashlib.sha512(password).hexdigest() > + password = hashlib.sha512(password.encode()).hexdigest() > elif algo == 'SHA384': > - import hashlib > - password = hashlib.sha384(password).hexdigest() > + password = hashlib.sha384(password.encode()).hexdigest() > elif algo == 'SHA256': > - import hashlib > - password = hashlib.sha256(password).hexdigest() > + password = hashlib.sha256(password.encode()).hexdigest() > elif algo == 'SHA224': > - import hashlib > - password = hashlib.sha224(password).hexdigest() > + password = hashlib.sha224(password.encode()).hexdigest() > elif algo == 'SHA1': > - import hashlib > - password = hashlib.sha1(password).hexdigest() > + password = hashlib.sha1(password.encode()).hexdigest() > elif algo == 'MD5': > - import hashlib > - password = hashlib.md5(password).hexdigest() > + password = hashlib.md5(password.encode()).hexdigest() > else: > raise NotSupportedError("The %s hash algorithm is not supported" % algo) > elif protocol != "8": > @@ -187,8 +182,8 @@ > elif "MD5" in h: > import hashlib > m = hashlib.md5() > - m.update(password) > - m.update(salt) > + m.update(password.encode()) > + m.update(salt.encode()) > pwhash = "{MD5}" + m.hexdigest() > elif "crypt" in h: > import crypt > @@ -212,28 +207,29 @@ > if length > 0: > result_bytes.write(self.__getbytes(length)) > > - result = result_bytes.getvalue().decode() > + result = result_bytes.getvalue() > logging.debug("RX: %s" % result) > - return result > + return result.decode() > > > def __getbytes(self, bytes): > """Read an amount of bytes from the socket""" > try: > return self.socket.recv(bytes) > - except socket.error(errorStr): > - raise OperationalError(errorStr[1]) > + except socket.error as error_str: > + raise OperationalError(error_str) > > > def __putblock(self, block): > """ wrap the line in mapi format and put it into the socket """ > pos = 0 > last = 0 > + logging.debug("TX: %s" % block) > while not last: > - data = block[pos:MAX_PACKAGE_LENGTH] > + data = block[pos:MAX_PACKAGE_LENGTH].encode() > if len(data) < MAX_PACKAGE_LENGTH: > last = 1 > flag = struct.pack( '<h', ( len(data) << 1 ) + last ) > self.socket.send(flag) > - self.socket.send(bytes(data.encode())) > + self.socket.send(data) > pos += len(data) > > U mapi25.py > Index: mapi25.py > =================================================================== > RCS file: /cvsroot/monetdb/clients/src/python/monetdb/mapi25.py,v > retrieving revision 1.1.2.4 > retrieving revision 1.1.2.5 > diff -u -d -r1.1.2.4 -r1.1.2.5 > --- mapi25.py 29 May 2009 12:01:21 -0000 1.1.2.4 > +++ mapi25.py 12 Jun 2009 13:23:34 -0000 1.1.2.5 > @@ -191,8 +191,8 @@ > elif "MD5" in h: > import hashlib > m = hashlib.md5() > - m.update(password) > - m.update(salt) > + m.update(password.encode()) > + m.update(salt.encode()) > pwhash = "{MD5}" + m.hexdigest() > elif "crypt" in h: > import crypt > > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables unlimited > royalty-free distribution of the report engine for externally facing > server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, May2009, 1.47.2.1, 1.47.2.2
by Stefan Manegold 08 Jun '09

08 Jun '09

On Mon, Jun 08, 2009 at 05:53:42PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv7310 > > Modified Files: > Tag: May2009 > opt_joinpath.mx > Log Message: > Experimental stable code to deal with too large intermediate joins. > It marks them as 'postpone' until at least one other join in the path > has been executed. Multiple variations can be envisioned. > > > U opt_joinpath.mx > Index: opt_joinpath.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v > retrieving revision 1.47.2.1 > retrieving revision 1.47.2.2 > diff -u -d -r1.47.2.1 -r1.47.2.2 > --- opt_joinpath.mx 8 Jun 2009 10:56:25 -0000 1.47.2.1 > +++ opt_joinpath.mx 8 Jun 2009 17:53:40 -0000 1.47.2.2 > @@ -408,7 +408,9 @@ > { > BAT *b; > BUN estimate, e; > - int i, j; > + int i, j, k; > + int *postpone= (int*) GDKzalloc(sizeof(int) *top); > + int postponed=0; > > /* solve the join by pairing the smallest first */ > while (top > 2) { > @@ -426,7 +428,7 @@ > stream_printf(cntxt->fdout,"#joinPath estimate join(%d,%d) %d\n", joins[i]->batCacheid, > joins[i+1]->batCacheid,(int)e); > #endif > - if (e < estimate) { > + if (e < estimate && ( !(postpone[i] && postpone[i+1]) || postponed==top-1)) { > estimate = e; > j = i; > } > @@ -434,10 +436,28 @@ > @- > BEWARE. you may not use a size estimation, because it > may fire a BATproperty check in a few cases. > +In case a join fails, we may try another order first before > +abandoning the task. > @c > b = BATjoin(joins[j], joins[j + 1], BUN_NONE); > - if (b==NULL) > - return NULL; /* something seriously went wrong */ > + if (b==NULL){ > + if ( postpone[j]){ ^^^^^^^^^^^ shouldn't this be (e.g.) postpone[j] && postpone[j+1] i.e., bailout (only) if the exact join has failed before, not if an other join with the same (i.e., join(j-1,j)) has failed before? ... just a spontaneous idea ... Stefan > + GDKfree(postpone); > + return NULL; > + } > + /* clear the GDKerrors */ > + if( cntxt->errbuf ) > + cntxt->errbuf[0]=0; > + postpone[j] = TRUE; > + postpone[j+1] = TRUE; > + postponed++; > + continue; > + } else { > + /* reset the postponed joins */ > + for( k=0; k<top; k++) > + postpone[k]=FALSE; > + postponed = 0; > + } > #ifdef DEBUG_OPT_JOINPATH > stream_printf(cntxt->fdout, "#joinPath %d:= join(%d,%d)\n", > b->batCacheid, joins[j]->batCacheid, joins[j + 1]->batCacheid); > @@ -450,6 +470,7 @@ > for (i = j + 1; i < top; i++) > joins[i] = joins[i + 1]; > } > + GDKfree(postpone); > #ifdef DEBUG_OPT_JOINPATH > stream_printf(cntxt->fdout, "#joinPath final join %d and %d\n", joins[0]->batCacheid, joins[1]->batCacheid); > #endif > > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables unlimited > royalty-free distribution of the report engine for externally facing > server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/BugTracker-2009/Tests primekeyconstraint.SF-2783425.sql, , 1.1, 1.2 primekeyconstraint.SF-2783425.stable.err, , 1.1, 1.2 primekeyconstraint.SF-2783425.stable.out, , 1.1, 1.2 random.SF-2737889.sql, , NONE, 1.1 All, , 1.30, 1.31
by Stefan Manegold 07 Jun '09

07 Jun '09

On Sun, Jun 07, 2009 at 06:18:45PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/sql/src/test/BugTracker-2009/Tests > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv8305 > > Modified Files: > All > Added Files: > primekeyconstraint.SF-2783425.sql > primekeyconstraint.SF-2783425.stable.err > primekeyconstraint.SF-2783425.stable.out FYI: You added the above files this morning to the May2009 branch. (Since then, there has not been any propagation, yet ...) Stefan > random.SF-2737889.sql > Log Message: > Addition of new tests. > > > U All > Index: All > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugTracker-2009/Tests/All,v > retrieving revision 1.30 > retrieving revision 1.31 > diff -u -d -r1.30 -r1.31 > --- All 30 May 2009 04:29:16 -0000 1.30 > +++ All 7 Jun 2009 18:18:43 -0000 1.31 > @@ -61,3 +61,5 @@ > having_min_crash.SF-2795951 > insert_type_exception.SF-2789976 > explain_gives_crash.SF-2741829 > +primekeyconstraint.SF-2783425 > +random.SF-2737889 > > --- NEW FILE: primekeyconstraint.SF-2783425.stable.out --- > stdout of test 'primekeyconstraint.SF-2783425` in directory 'src/test/BugTracker-2009` itself: > > > # 08:02:34 > > # 08:02:34 > mserver5 "--config=/ufs/mk/monet5//Linux/etc/monetdb5.conf" --debug=10 --set gdk_nr_threads=0 --set "monet_mod_path=/ufs/mk/monet5//Linux/lib/MonetDB5:/ufs/mk/monet5//Linux/lib/MonetDB5/lib:/ufs/mk/monet5//Linux/lib/MonetDB5/bin" --set "gdk_dbfarm=/ufs/mk/monet5//Linux/var/MonetDB5/dbfarm" --set mapi_open=true --set xrpc_open=true --set mapi_port=30582 --set xrpc_port=46770 --set monet_prompt= --trace "--dbname=mTests_src_test_BugTracker-2009" --set mal_listing=0 "--dbinit= include sql;" ; echo ; echo Over.. > # 08:02:34 > > > # MonetDB server v5.13.0, based on kernel v1.31.0 > # Serving database 'mTests_src_test_BugTracker-2009', using 4 threads > # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked > # Copyright (c) 1993-July 2008 CWI. > # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved > # Visit http://monetdb.cwi.nl/ for further information > # Listening for connection requests on mapi:monetdb://eir.ins.cwi.nl:30582/ > # MonetDB/SQL module v2.31.0 loaded > > Ready. > #function user.main():void; > # clients.quit(); > #end main; > > Over.. > > # 08:02:34 > > # 08:02:34 > mclient -lsql -umonetdb -Pmonetdb --host=eir --port=30582 > # 08:02:34 > > > [ 1 ] > [ 1 ] > [ 1 ] > [ 1 ] > [ 1 ] > [ 1 ] > [ 1 ] > [ 1 ] > [ 1 ] > % sys.way_nds, sys.way_nds # table_name > % way, idx # name > % int, int # type > % 1, 1 # length > [ 1, 1 ] > [ 1, 1 ] > [ 2, 2 ] > [ 3, 3 ] > [ 4, 3 ] > [ NULL, 3 ] > [ NULL, 3 ] > [ 5, NULL ] > [ 5, NULL ] > % sys.way_nds, sys.way_nds, sys.way_nds # table_name > % L4, way, idx # name > % wrd, int, int # type > % 1, 1, 1 # length > [ 2, 1, 1 ] > [ 2, NULL, 3 ] > [ 2, 5, NULL ] > > # 08:02:34 > > # 08:02:34 > Done. > # 08:02:34 > > > > --- NEW FILE: primekeyconstraint.SF-2783425.stable.err --- > stderr of test 'primekeyconstraint.SF-2783425` in directory 'src/test/BugTracker-2009` itself: > > > # 08:02:34 > > # 08:02:34 > mserver5 "--config=/ufs/mk/monet5//Linux/etc/monetdb5.conf" --debug=10 --set gdk_nr_threads=0 --set "monet_mod_path=/ufs/mk/monet5//Linux/lib/MonetDB5:/ufs/mk/monet5//Linux/lib/MonetDB5/lib:/ufs/mk/monet5//Linux/lib/MonetDB5/bin" --set "gdk_dbfarm=/ufs/mk/monet5//Linux/var/MonetDB5/dbfarm" --set mapi_open=true --set xrpc_open=true --set mapi_port=30582 --set xrpc_port=46770 --set monet_prompt= --trace "--dbname=mTests_src_test_BugTracker-2009" --set mal_listing=0 "--dbinit= include sql;" ; echo ; echo Over.. > # 08:02:34 > > > # builtin opt gdk_arch = 64bitx86_64-unknown-linux-gnu > # builtin opt gdk_version = 1.31.0 > # builtin opt prefix = /ufs/mk/monet5//Linux > # builtin opt exec_prefix = ${prefix} > # builtin opt gdk_dbname = tst > # builtin opt gdk_dbfarm = ${prefix}/var/MonetDB > # builtin opt gdk_debug = 8 > # builtin opt gdk_alloc_map = yes > # builtin opt gdk_vmtrim = yes > # builtin opt monet_admin = adm > # builtin opt monet_prompt = > > # builtin opt monet_welcome = yes > # builtin opt monet_mod_path = ${exec_prefix}/lib/MonetDB > # builtin opt monet_daemon = yes > # builtin opt host = localhost > # builtin opt mapi_port = 50000 > # builtin opt mapi_noheaders = no > # builtin opt mapi_debug = 0 > # builtin opt mapi_clients = 2 > # builtin opt sql_debug = 0 > # builtin opt sql_logdir = sql_logs > # builtin opt xquery_logdir = xquery_logs > # builtin opt standoff_ns = http://monetdb.cwi.nl/standoff > # builtin opt standoff_start = start > # builtin opt standoff_end = end > # config opt prefix = /ufs/mk/monet5//Linux > # config opt config = ${prefix}/etc/monetdb5.conf > # config opt prefix = /ufs/mk/monet5//Linux > # config opt exec_prefix = ${prefix} > # config opt gdk_dbfarm = ${prefix}/var/MonetDB5/dbfarm > # config opt gdk_dbname = demo > # config opt gdk_alloc_map = no > # config opt gdk_embedded = no > # config opt gdk_debug = 0 > # config opt monet_mod_path = ${exec_prefix}/lib/MonetDB5:${exec_prefix}/lib/MonetDB5/lib:${exec_prefix}/lib/MonetDB5/bin > # config opt monet_daemon = no > # config opt monet_welcome = yes > # config opt mero_msglog = ${prefix}/var/log/MonetDB/merovingian.log > # config opt mero_errlog = ${prefix}/var/log/MonetDB/merovingian.log > # config opt mero_pidfile = ${prefix}/var/run/MonetDB/merovingian.pid > # config opt mal_init = ${exec_prefix}/lib/MonetDB5/mal_init.mal > # config opt mal_listing = 2 > # config opt mapi_port = 50000 > # config opt mapi_open = false > # config opt sql_optimizer = inline,remap,evaluate,costModel,coercions,emptySet,aliases,deadcode,constants,commonTerms,joinPath,deadcode,reduce,garbageCollector,dataflow,history,multiplex > # config opt sql_logdir = sql_logs > # cmdline opt config = /ufs/mk/monet5//Linux/etc/monetdb5.conf > # cmdline opt gdk_nr_threads = 0 > # cmdline opt monet_mod_path = /ufs/mk/monet5//Linux/lib/MonetDB5:/ufs/mk/monet5//Linux/lib/MonetDB5/lib:/ufs/mk/monet5//Linux/lib/MonetDB5/bin > # cmdline opt gdk_dbfarm = /ufs/mk/monet5//Linux/var/MonetDB5/dbfarm > # cmdline opt mapi_open = true > # cmdline opt xrpc_open = true > # cmdline opt mapi_port = 30582 > # cmdline opt xrpc_port = 46770 > # cmdline opt monet_prompt = > # cmdline opt gdk_dbname = mTests_src_test_BugTracker-2009 > # cmdline opt mal_listing = 0 > #warning: please don't forget to set your vault key! > #(see /ufs/mk/monet5//Linux/etc/monetdb5.conf) > > # 08:02:34 > > # 08:02:34 > mclient -lsql -umonetdb -Pmonetdb --host=eir --port=30582 > # 08:02:34 > > > MAPI = monetdb@eir:30582 > QUERY = ALTER TABLE way_nds ADD CONSTRAINT pk_way_nds PRIMARY KEY (way, idx); > > > # 08:02:34 > > # 08:02:34 > Done. > # 08:02:34 > > > > --- NEW FILE: random.SF-2737889.sql --- > create table nr (nr int); > insert into nr values (1); > insert into nr values (2); > insert into nr values (3); > insert into nr values (4); > insert into nr values (5); > insert into nr values (6); > insert into nr values (7); > insert into nr values (8); > insert into nr values (9); > insert into nr values (10); > SELECT nr, RAND() FROM nr; > SELECT nr, RAND() AS rnd FROM nr; > drop table nr; > > --- NEW FILE: primekeyconstraint.SF-2783425.sql --- > CREATE TABLE way_nds ( > way int, > idx int, > payload string > ); > > insert into way_nds values(1,1,'abc'); > insert into way_nds values(1,1,'abc'); > insert into way_nds values(2,2,'abc'); > insert into way_nds values(3,3,'abc'); > insert into way_nds values(4,3,'abc'); > insert into way_nds values(null,3,'abc'); > insert into way_nds values(null,3,'abc'); > insert into way_nds values(5,null,'abc'); > insert into way_nds values(5,null,'abc'); > > ALTER TABLE way_nds ADD CONSTRAINT pk_way_nds PRIMARY KEY (way, idx); > > select way,idx from way_nds; > select count(way), way, idx from way_nds group by way, idx having > count(way) > 1; > drop table way_nds; > > > ------------------------------------------------------------------------------ > OpenSolaris 2009.06 is a cutting edge operating system for enterprises > looking to deploy the next generation of Solaris that includes the latest > innovations from Sun and the OpenSource community. Download a copy and > enjoy capabilities such as Networking, Storage and Virtualization. > Go to: http://p.sf.net/sfu/opensolaris-get > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5/merovingian monetdb.c, , 1.12, 1.13
by Martin Kersten 06 Jun '09

06 Jun '09

octopus errors on sql test set down to 50

1 0

[Monetdb-developers] problems with starting the monetDB4 xquery server
by Mtekel 04 Jun '09

04 Jun '09

Hello, I have problems starting the monetDB xquery server under windowsXP sp3. When I click on the link in the start menu, dos window appears but then it closes immediatelly and I can't see what's there (after pausing I can see only the first 6 lines of the program output as described below, then after unpausing it immediatelly quits and closes the window). If I run MXQserver.bat, or Mserver.bat --dbinit="module(pathfinder);" I get only this: # MonetDB Server v4.28.4 # based on GDK v1.28.4 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-2009, MonetDB B.V.. All rights reserved. # Compiled for i686-pc-win32/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah That's all. The server immediatelly ends. However If I run only Mserver.bat, the server "runs" - I get the prompt, but for example I can't connect to the web interface on the port 50000. Does anyone have any advice what to do? How to make it work? Any advice appreciated, Best regards Michael

3 4

[Monetdb-developers] Announcement: New feature release of MonetDB suite
by Sjoerd Mullender 03 Jun '09

03 Jun '09

The MonetDB team at CWI/MonetDB BV are pleased to announce the new "May2009" feature release of the MonetDB suite of programs. This is a feature release, however it is is (mostly) backward compatible with the previous releases (Nov2008 and Feb2009). The two slight incompatibilities are: - The way string data is stored was changed for 64-bit architectures with 32-bit OIDs. There is however an automatic conversion on startup of the server. The biggest problem is, you can't go back to an older version once the new version has run. This incompatibility affects only the Windows 64 bit build (the other 64 bit builds use 64-bit OIDs). - The default location of the write-ahead log (WAL) has changed. This is relevant for both MonetDB5/SQL and MonetDB4/XQuery on all platforms. When the server first starts on an existing database, it will notice that the WAL is missing and will refuse to continue with a message which indicates where the WAL is now expected. You can either use a command line option to tell the server where the WAL is, or you can manually move the WAL to the new location. This latter option is done automatically on Windows when you use the Start menu to start the server. More information on the new release (and the above mentioned incompatibilities) is available at <http://monetdb.cwi.nl/projects/monetdb/Development/Releases/May2009/>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

1 0