October 2009 - developers-list

[Monetdb-developers] Announcement: New Aug2009-SP2 bugfix release of MonetDB suite
by Sjoerd Mullender 01 Nov '09

01 Nov '09

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2009-SP2 bug fix release of the MonetDB suite of programs. This is the second bug fix release for the Aug2009 release. More information on this release can be found at <http://monetdb.cwi.nl/projects/monetdb//Development/Releases/Aug2009/#Aug20…>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_authorize.mx, Nov2009, 1.82.4.1, 1.82.4.2 mal_box.mx, Nov2009, 1.98.4.1, 1.98.4.2 mal_factory.mx, Nov2009, 1.89.6.1, 1.89.6.2 mal_session.mx, Nov2009, 1.200.2.1, 1.200.2.2
by Fabian Groffen 31 Oct '09

31 Oct '09

On 31-10-2009 09:06:11 +0000, Martin Kersten wrote: [snip] > obj->name= GDKstrdup(name); what if the GDKstrdup fails here? > obj->sym= newMalBlk(MAXVARS,STMT_INCREMENT); > obj->val = newGlobalStack(MAXVARS); > + if ( obj->val == NULL) > + showException(MAL,"box.new", MAL_MALLOC_FAIL); I've been looking at mal_sabaoth to deal with GDKmalloc failures, and there is a question to me in cases like this. malloc failed, now the only right way to communicate that to the caller is to throw an exception. However, doing that requires memory too (it GDKmallocs too), so that will most probably fail too. Hence, the throw will die, crash, or even worse, return NULL, which happens to mean MAL_SUCCEED. What should we do here? define a MAL_MALLOC_FAIL constant that we can always return and that's recognised (and hence never freed), or do we redefine MAL_SUCCEED to be non-NULL such that a real NULL means real trouble, and thus an out of memory condition?

2 2

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah/tjc tjc_milprint.c, , 1.21, 1.22
by Stefan Manegold 26 Oct '09

26 Oct '09

On Mon, Oct 26, 2009 at 04:49:49PM +0000, Henning Rode wrote: > Update of /cvsroot/monetdb/pathfinder/modules/pftijah/tjc > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv28326 > > Modified Files: > tjc_milprint.c > Log Message: > try to fix compilation problem In case you refer to this compilation problem http://monetdb.cwi.nl/testing/projects/monetdb/Candidate/pathfinder/.GNU.64… http://www.ins.cwi.nl/~monet/permastore/2009-10-26/Candidate/make-pathfinde… please note that this problem occurs with Candidate testing, i.e., (also) in the Nov2009 release candidate branch, not (only) in the development trunk, where your checkin went ... Stefan > > Index: tjc_milprint.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/tjc/tjc_milprint.c,v > retrieving revision 1.21 > retrieving revision 1.22 > diff -u -d -r1.21 -r1.22 > --- tjc_milprint.c 25 Sep 2009 07:22:33 -0000 1.21 > +++ tjc_milprint.c 26 Oct 2009 16:49:47 -0000 1.22 > @@ -306,11 +306,11 @@ > // for old index management > if (tjc_c->maxfrag < 0) { > TJCPRINTF(MILOUT,"ftindex := ftiName;\n"); > - TJCPRINTF(MILOUT,&tjc_c->milfragBUFF[0]); > + TJCPRINTF(MILOUT,"%s", &tjc_c->milfragBUFF[0]); > } > for (c = 0; c <= tjc_c->maxfrag; c++) { > TJCPRINTF(MILOUT,"ftindex := ftiName + \"%d\";\n", c); > - TJCPRINTF(MILOUT,&tjc_c->milfragBUFF[0]); > + TJCPRINTF(MILOUT,"%s", &tjc_c->milfragBUFF[0]); > milprint_end_frag (tjc_c, tree, c); > } > > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] How to register new "bulk" function with SQL?
by Stefan Manegold 22 Oct '09

22 Oct '09

Hoi Niels (and other SQL experts), I implemented the "bulk" version of ogc constructor function point(x:dbl,y:dbl) :wkb i.e. point(x:bat[:oid,:dbl],y:bat[:oid,:dbl]) :bat[:oid,:wkb] in the geom module (see below). How do I get SQL to use this function, e.g., in a scenario like this: CREATE TABLE coords (x double, y double); CREATE TABLE points (p point); INSERT INTO points SELECT POINT(x,y) FROM coords; For now, EXPLAIN still shows the explicit MAL interation calling the original scalar function: [...] | _67 := bat.new(nil:oid,nil:wkb); | | barrier (_71,_72,_73) := bat.newIterator(_16); | | _75 := algebra.find(_24,_72); | | _77 := geom.point(_73,_75); | | bat.insert(_67,_72,_77); | | redo (_71,_72,_73) := bat.hasMoreElements(_16); | | exit (_71,_72,_73); | | _25:bat[:oid,:wkb] := _67; | [...] Thanks in advance! Stefan ps: Eventually, we should consider implemeting bulk versions of all scalar functions in geom in the same (or a similar) way --- assuming that this might improve the performance ... On Thu, Oct 22, 2009 at 04:15:30PM +0000, Stefan Manegold wrote: > Update of /cvsroot/monetdb/geom/src/monetdb5 > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv25812/geom/src/monetdb5 > > Modified Files: > geom.mx > Log Message: > > implemented bulk version of ogc constructor function > point(x:dbl,y:dbl) :wkb > i.e. > point(x:bat[:oid,:dbl],y:bat[:oid,:dbl]) :bat[:oid,:wkb] > > Open question: > How to make SQL use this bulk function instead of an explicit > MAL iterator over the scalar function, e.g., in queries like > INSERT INTO points SELECT POINT(x, y) FROM coords; > > > Index: geom.mx > =================================================================== > RCS file: /cvsroot/monetdb/geom/src/monetdb5/geom.mx,v > retrieving revision 1.29 > retrieving revision 1.30 > diff -u -d -r1.29 -r1.30 > --- geom.mx 26 Jun 2009 07:37:52 -0000 1.29 > +++ geom.mx 22 Oct 2009 16:15:28 -0000 1.30 > @@ -170,6 +170,10 @@ > address wkbcreatepoint > comment "Construct a point from two geometries"; > > +command point(x:bat[:oid,:dbl],y:bat[:oid,:dbl]) :bat[:oid,:wkb] > +address wkbcreatepoint_bat > +comment "Construct a point-BAT from two geometry-BATs"; > + > @+ ogc information methods > @mal > command X(g:wkb) :dbl > @@ -1010,6 +1014,76 @@ > } > > geom_export str > +wkbcreatepoint_bat(int *out, int *x,int *y); > + > +str > +wkbcreatepoint_bat(int *out, int *ix,int *iy) > +{ > + BAT *bo = NULL, *bx = NULL, *by = NULL; > + dbl *x = NULL, *y = NULL; > + BUN i, o; > + wkb *p = NULL; > + > + if ((bx = BATdescriptor(*ix)) == NULL) { > + throw(MAL, "geom.point", RUNTIME_OBJECT_MISSING); > + } > + if ((by = BATdescriptor(*iy)) == NULL) { > + BBPreleaseref(bx->batCacheid); > + throw(MAL, "geom.point", RUNTIME_OBJECT_MISSING); > + } > + if (!BAThdense(bx) || !BAThdense(by) || bx->hseqbase != by->hseqbase || BATcount(bx) != BATcount(by)) { > + BBPreleaseref(bx->batCacheid); > + BBPreleaseref(by->batCacheid); > + throw(MAL, "geom.point", "both arguments must have dense and aligned heads"); > + } > + > + if ((bo = BATnew(TYPE_void, ATOMindex("wkb"), BATcount(bx))) == NULL) { > + BBPreleaseref(bx->batCacheid); > + BBPreleaseref(by->batCacheid); > + throw(MAL, "geom.point", MAL_MALLOC_FAIL); > + } > + BATseqbase(bo, bx->hseqbase); > + bo->hdense = TRUE; > + bo->hsorted = TRUE; > + bo->H->nonil = TRUE; > + BATkey(bo, TRUE); > + > + x = (dbl*) Tloc(bx, BUNfirst(bx)); > + y = (dbl*) Tloc(by, BUNfirst(bx)); > + for (i = 0, o = BUNlast(bo); i < BATcount(bx); i++, o++) { > + str err = NULL; > + if ((err = wkbcreatepoint(&p, &x[i], &y[i])) != MAL_SUCCEED) { > + BBPreleaseref(bx->batCacheid); > + BBPreleaseref(by->batCacheid); > + BBPreleaseref(bo->batCacheid); > + throw(MAL, "geom.point", err); > + } > + tfastins_nocheck(bo, o, p, Tsize(bo)); > + GDKfree(p); > + p = NULL; > + } > + > + BATsetcount(bo, BATcount(bx)); > + bo->batDirty = TRUE; > + bo->tdense = FALSE; > + bo->tsorted = FALSE; > + bo->T->nonil = FALSE; > + BATkey(BATmirror(bo), FALSE); > + > + BBPreleaseref(bx->batCacheid); > + BBPreleaseref(by->batCacheid); > + BBPkeepref(*out = bo->batCacheid); > + return MAL_SUCCEED; > + > + bunins_failed: > + if (p) GDKfree(p); > + BBPreleaseref(bx->batCacheid); > + BBPreleaseref(by->batCacheid); > + BBPreleaseref(bo->batCacheid); > + throw(MAL, "geom.point", "bunins failed"); > +} > + > +geom_export str > mbroverlaps( bit *out, mbr *b1, mbr *b2); > > str > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

[Monetdb-developers] web-page: Broken link....
by Romulo Goncalves 22 Oct '09

22 Oct '09

Hello, In this page: http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Start-and-Stop.htm… I click on See for a more complete session "VOC demo." and I end up in the same page, I think the right link should be: http://monetdb.cwi.nl/projects/monetdb/SQL/QuickTour/VOC/index.html Regards, Romulo

2 2

[Monetdb-developers] web-page: mclient options
by Romulo Goncalves 22 Oct '09

22 Oct '09

In monetdb web-page I read the VOC tutorial and I see the the following: shell> mclient -lsql -uvoc -Pvoc sql> \< voc_dump.sql Then I went to: http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/MAPI-Client.html#M… And for client options I see: -P passwd | --passwd=passwd password I just tried in my cvs head version and the option -P does not work. Reading some check ins in clients module, it seems we have to use .monetdb file to avoid typing the password. However, the syntax of the file was not specified. I create the file .monetdb with following content: monetdb monetdb And I get the following message: [goncalve@sirius scripts]$ ./mclient current debug sql .monetdb:1: syntax error: monetdb .monetdb:2: syntax error: monetdb password: I think we need some extra information about this new feature ;) Regards, Romulo

3 3

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/mapiclient dump.c, Aug2009, 1.45.2.1, 1.45.2.2
by Milena Ivanova 22 Oct '09

22 Oct '09

Fabian wrote: > Update of /cvsroot/monetdb/clients/src/mapiclient > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv31397 > > Modified Files: > Tag: Aug2009 > dump.c > Log Message: > Stefan dK ran into a 'table is not unique' message, which IMO can only indicate something is seriously fubared > > Would you please write in English? http://www.merriam-webster.com/dictionary/ showed empty result on 'fubared' and International Maritime Organization for the shortening IMO, Thanks, Milena > Index: dump.c > =================================================================== > RCS file: /cvsroot/monetdb/clients/src/mapiclient/dump.c,v > retrieving revision 1.45.2.1 > retrieving revision 1.45.2.2 > diff -u -d -r1.45.2.1 -r1.45.2.2 > --- dump.c 22 Sep 2009 08:26:36 -0000 1.45.2.1 > +++ dump.c 22 Oct 2009 08:37:26 -0000 1.45.2.2 > @@ -342,9 +342,10 @@ > > if (cnt != 1) { > if (cnt == 0) > - fprintf(stderr, "Table %s.%s does not exist.\n", schema, tname); > + fprintf(stderr, "table %s.%s does not exist\n", schema, tname); > else > - fprintf(stderr, "Table %s.%s not unique.\n", schema, tname); > + fprintf(stderr, "table %s.%s is not unique, corrupt catalog?\n", > + schema, tname); > goto bailout; > } > > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

2 2

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/python/monetdb mapi2.py, , NONE, 1.1 mapi.py, , 1.17, 1.18
by Sjoerd Mullender 22 Oct '09

22 Oct '09

Sounds like something that should have gone to the Candidate (Nov2009) branch? Gijs Molenaar wrote: > Update of /cvsroot/monetdb/clients/src/python/monetdb > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv12494 > > Modified Files: > mapi.py > Added Files: > mapi2.py > Log Message: > > Both python2 and python3 can do import monetdb.mapi now. Change is backwards > compatible. > > Previously you needed to do import monetdb.mapi3 in python3. > > > > --- NEW FILE: mapi2.py --- > # The contents of this file are subject to the MonetDB Public License > # Version 1.1 (the "License"); you may not use this file except in > # compliance with the License. You may obtain a copy of the License at > # http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html > # > # Software distributed under the License is distributed on an "AS IS" > # basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > # License for the specific language governing rights and limitations > # under the License. > # > # The Original Code is the MonetDB Database System. > # > # The Initial Developer of the Original Code is CWI. > # Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > # Copyright August 2008-2009 MonetDB B.V. > # All Rights Reserved. > > """ > this is the python 2.* implementation of the mapi API. > > If you use python 3.* you should use mapi3.py > """ > > import socket > import logging > import struct > > from monetdb.monetdb_exceptions import * > > > logger = logging.getLogger("monetdb") > > MAX_PACKAGE_LENGTH = (1024*8)-2 > > MSG_PROMPT = "" > MSG_INFO = "#" > MSG_ERROR = "!" > MSG_Q = "&" > MSG_QTABLE = "&1" > MSG_QUPDATE = "&2" > MSG_QSCHEMA = "&3" > MSG_QTRANS = "&4" > MSG_QPREPARE = "&5" > MSG_QBLOCK = "&6" > MSG_HEADER = "%" > MSG_TUPLE = "[" > MSG_REDIRECT = "^" > > STATE_INIT = 0 > STATE_READY = 1 > > > class Server: > def __init__(self): > self.state = STATE_INIT > self._result = None > > def connect(self, hostname, port, username, password, database, language): > """ connect to a MonetDB database using the mapi protocol""" > > self.hostname = hostname > self.port = port > self.username = username > self.password = password > self.database = database > self.language = language > > self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > > try: > self.socket.connect((hostname, port)) > except socket.error, error: > (error_code, error_str) = error > raise OperationalError(error_str) > > self.__login() > > > def __login(self, iteration=0): > """ Reads challenge from line, generate response and check if > everything is okay """ > > challenge = self.__getblock() > response = self.__challenge_response(challenge) > self.__putblock(response) > prompt = self.__getblock().strip() > > if len(prompt) == 0: > # Empty response, server is happy > pass > elif prompt.startswith(MSG_INFO): > logger.info("II %s" % prompt[1:]) > > elif prompt.startswith(MSG_ERROR): > logger.error(prompt[1:]) > raise DatabaseError(prompt[1:]) > > elif prompt.startswith(MSG_REDIRECT): > # a redirect can contain multiple redirects, for now we only use > # the first > response = prompt.split()[0][1:].split(':') > if response[1] == "merovingian": > logger.debug("II: merovingian proxy, restarting " + > "authenticatiton") > if iteration <= 10: > self.__login(iteration=iteration+1) > else: > raise OperationalError("maximal number of redirects " + > "reached (10)") > > elif response[1] == "monetdb": > self.hostname = response[2][2:] > self.port, self.database = response[3].split('/') > self.port = int(self.port) > logger.info("II: merovingian redirect to monetdb://%s:%s/%s" % > (self.hostname, self.port, self.database)) > self.socket.close() > self.connect(self.hostname, self.port, self.username, > self.password, self.database, self.language) > > else: > logger.error('!' + prompt[0]) > raise ProgrammingError("unknown redirect: %s" % prompt) > > else: > logger.error('!' + prompt[0]) > raise ProgrammingError("unknown state: %s" % prompt) > > self.state = STATE_READY > return True > > > def disconnect(self): > """ disconnect from the monetdb server """ > self.state = STATE_INIT > self.socket.close() > > > def cmd(self, operation): > """ put a mapi command on the line""" > logger.debug("II: executing command %s" % operation) > > if self.state != STATE_READY: > raise(ProgrammingError, "Not connected") > > self.__putblock(operation) > response = self.__getblock() > if not len(response): > return > if response[0] in [MSG_Q, MSG_HEADER, MSG_TUPLE]: > return response > elif response[0] == MSG_ERROR: > raise OperationalError(response[1:]) > else: > raise ProgrammingError("unknown state: %s" % response) > > > def __challenge_response(self, challenge): > """ generate a response to a mapi login challenge """ > challenges = challenge.split(':') > salt, identity, protocol, hashes, endian = challenges[:5] > > password = self.password > > if protocol == '9': > algo = challenges[5] > if algo == 'SHA512': > import hashlib > password = hashlib.sha512(password).hexdigest() > elif algo == 'SHA384': > import hashlib > password = hashlib.sha384(password).hexdigest() > elif algo == 'SHA256': > import hashlib > password = hashlib.sha256(password).hexdigest() > elif algo == 'SHA224': > import hashlib > password = hashlib.sha224(password).hexdigest() > elif algo == 'SHA1': > import hashlib > password = hashlib.sha1(password).hexdigest() > elif algo == 'MD5': > import hashlib > password = hashlib.md5(password).hexdigest() > else: > raise NotSupportedError("The %s hash algorithm is not " + > "supported" % algo) > elif protocol != "8": > raise NotSupportedError("We only speak protocol v8 and v9") > > h = hashes.split(",") > if "SHA1" in h: > import hashlib > s = hashlib.sha1() > s.update(password.encode()) > s.update(salt.encode()) > pwhash = "{SHA1}" + s.hexdigest() > elif "MD5" in h: > import hashlib > m = hashlib.md5() > m.update(password.encode()) > m.update(salt.encode()) > pwhash = "{MD5}" + m.hexdigest() > elif "crypt" in h: > import crypt > pwhash = "{crypt}" + crypt.crypt((password+salt)[:8], salt[-2:]) > else: > pwhash = "{plain}" + password + salt > > return ":".join(["BIG", self.username, pwhash, self.language, > self.database]) + ":" > > > def __getblock(self): > """ read one mapi encoded block """ > result = [] > last = 0 > while not last: > flag = self.__getbytes(2) > if len(flag) != 2: > raise OperationalError("server returned %s bytes, I need 2" % > len(flag)) > > # unpack (little endian short) > unpacked = struct.unpack('<H', flag)[0] > length = unpacked >> 1 > last = unpacked & 1 > logger.debug("II: reading %i bytes" % length) > if length > 0: > result.append(self.__getbytes(length)) > > result_str = "".join(result) > logger.debug("RX: %s" % result_str) > return result_str > > > def __getbytes(self, bytes): > """Read an amount of bytes from the socket""" > try: > return self.socket.recv(bytes) > except socket.error, error: > raise OperationalError(error[1]) > > > def __putblock(self, block): > """ wrap the line in mapi format and put it into the socket """ > pos = 0 > last = 0 > while not last: > data = block[pos:pos+MAX_PACKAGE_LENGTH] > length = len(data) > if length < MAX_PACKAGE_LENGTH: > last = 1 > flag = struct.pack( '<H', ( length << 1 ) + last ) > try: > logger.debug("II: sending %i bytes, last: %s" % (length, last)) > logger.debug("TX: %s" % data) > self.socket.send(flag) > self.socket.send(data) > except socket.error, error: > raise OperationalError(error[1]) > pos += length > > > Index: mapi.py > =================================================================== > RCS file: /cvsroot/monetdb/clients/src/python/monetdb/mapi.py,v > retrieving revision 1.17 > retrieving revision 1.18 > diff -u -d -r1.17 -r1.18 > --- mapi.py 7 Jul 2009 11:15:45 -0000 1.17 > +++ mapi.py 22 Oct 2009 10:25:35 -0000 1.18 > @@ -1,263 +1,10 @@ > -# The contents of this file are subject to the MonetDB Public License > -# Version 1.1 (the "License"); you may not use this file except in > -# compliance with the License. You may obtain a copy of the License at > -# http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html > -# > -# Software distributed under the License is distributed on an "AS IS" > -# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > -# License for the specific language governing rights and limitations > -# under the License. > -# > -# The Original Code is the MonetDB Database System. > -# > -# The Initial Developer of the Original Code is CWI. > -# Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > -# Copyright August 2008-2009 MonetDB B.V. > -# All Rights Reserved. > - > -""" > -this is the python 2.* implementation of the mapi API. > - > -If you use python 3.* you should use mapi3.py > -""" > - > -import socket > -import logging > -import struct > - > -from monetdb.monetdb_exceptions import * > - > - > -logger = logging.getLogger("monetdb") > - > -MAX_PACKAGE_LENGTH = (1024*8)-2 > - > -MSG_PROMPT = "" > -MSG_INFO = "#" > -MSG_ERROR = "!" > -MSG_Q = "&" > -MSG_QTABLE = "&1" > -MSG_QUPDATE = "&2" > -MSG_QSCHEMA = "&3" > -MSG_QTRANS = "&4" > -MSG_QPREPARE = "&5" > -MSG_QBLOCK = "&6" > -MSG_HEADER = "%" > -MSG_TUPLE = "[" > -MSG_REDIRECT = "^" > - > -STATE_INIT = 0 > -STATE_READY = 1 > - > - > -class Server: > - def __init__(self): > - self.state = STATE_INIT > - self._result = None > - > - def connect(self, hostname, port, username, password, database, language): > - """ connect to a MonetDB database using the mapi protocol""" > - > - self.hostname = hostname > - self.port = port > - self.username = username > - self.password = password > - self.database = database > - self.language = language > - > - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > - > - try: > - self.socket.connect((hostname, port)) > - except socket.error, error: > - (error_code, error_str) = error > - raise OperationalError(error_str) > - > - self.__login() > - > - > - def __login(self, iteration=0): > - """ Reads challenge from line, generate response and check if > - everything is okay """ > - > - challenge = self.__getblock() > - response = self.__challenge_response(challenge) > - self.__putblock(response) > - prompt = self.__getblock().strip() > - > - if len(prompt) == 0: > - # Empty response, server is happy > - pass > - elif prompt.startswith(MSG_INFO): > - logger.info("II %s" % prompt[1:]) > - > - elif prompt.startswith(MSG_ERROR): > - logger.error(prompt[1:]) > - raise DatabaseError(prompt[1:]) > - > - elif prompt.startswith(MSG_REDIRECT): > - # a redirect can contain multiple redirects, for now we only use > - # the first > - response = prompt.split()[0][1:].split(':') > - if response[1] == "merovingian": > - logger.debug("II: merovingian proxy, restarting " + > - "authenticatiton") > - if iteration <= 10: > - self.__login(iteration=iteration+1) > - else: > - raise OperationalError("maximal number of redirects " + > - "reached (10)") > - > - elif response[1] == "monetdb": > - self.hostname = response[2][2:] > - self.port, self.database = response[3].split('/') > - self.port = int(self.port) > - logger.info("II: merovingian redirect to monetdb://%s:%s/%s" % > - (self.hostname, self.port, self.database)) > - self.socket.close() > - self.connect(self.hostname, self.port, self.username, > - self.password, self.database, self.language) > - > - else: > - logger.error('!' + prompt[0]) > - raise ProgrammingError("unknown redirect: %s" % prompt) > - > - else: > - logger.error('!' + prompt[0]) > - raise ProgrammingError("unknown state: %s" % prompt) > - > - self.state = STATE_READY > - return True > - > - > - def disconnect(self): > - """ disconnect from the monetdb server """ > - self.state = STATE_INIT > - self.socket.close() > - > - > - def cmd(self, operation): > - """ put a mapi command on the line""" > - logger.debug("II: executing command %s" % operation) > - > - if self.state != STATE_READY: > - raise(ProgrammingError, "Not connected") > - > - self.__putblock(operation) > - response = self.__getblock() > - if not len(response): > - return > - if response[0] in [MSG_Q, MSG_HEADER, MSG_TUPLE]: > - return response > - elif response[0] == MSG_ERROR: > - raise OperationalError(response[1:]) > - else: > - raise ProgrammingError("unknown state: %s" % response) > - > - > - def __challenge_response(self, challenge): > - """ generate a response to a mapi login challenge """ > - challenges = challenge.split(':') > - salt, identity, protocol, hashes, endian = challenges[:5] > - > - password = self.password > - > - if protocol == '9': > - algo = challenges[5] > - if algo == 'SHA512': > - import hashlib > - password = hashlib.sha512(password).hexdigest() > - elif algo == 'SHA384': > - import hashlib > - password = hashlib.sha384(password).hexdigest() > - elif algo == 'SHA256': > - import hashlib > - password = hashlib.sha256(password).hexdigest() > - elif algo == 'SHA224': > - import hashlib > - password = hashlib.sha224(password).hexdigest() > - elif algo == 'SHA1': > - import hashlib > - password = hashlib.sha1(password).hexdigest() > - elif algo == 'MD5': > - import hashlib > - password = hashlib.md5(password).hexdigest() > - else: > - raise NotSupportedError("The %s hash algorithm is not " + > - "supported" % algo) > - elif protocol != "8": > - raise NotSupportedError("We only speak protocol v8 and v9") > - > - h = hashes.split(",") > - if "SHA1" in h: > - import hashlib > - s = hashlib.sha1() > - s.update(password.encode()) > - s.update(salt.encode()) > - pwhash = "{SHA1}" + s.hexdigest() > - elif "MD5" in h: > - import hashlib > - m = hashlib.md5() > - m.update(password.encode()) > - m.update(salt.encode()) > - pwhash = "{MD5}" + m.hexdigest() > - elif "crypt" in h: > - import crypt > - pwhash = "{crypt}" + crypt.crypt((password+salt)[:8], salt[-2:]) > - else: > - pwhash = "{plain}" + password + salt > - > - return ":".join(["BIG", self.username, pwhash, self.language, > - self.database]) + ":" > - > - > - def __getblock(self): > - """ read one mapi encoded block """ > - result = [] > - last = 0 > - while not last: > - flag = self.__getbytes(2) > - if len(flag) != 2: > - raise OperationalError("server returned %s bytes, I need 2" % > - len(flag)) > - > - # unpack (little endian short) > - unpacked = struct.unpack('<H', flag)[0] > - length = unpacked >> 1 > - last = unpacked & 1 > - logger.debug("II: reading %i bytes" % length) > - if length > 0: > - result.append(self.__getbytes(length)) > - > - result_str = "".join(result) > - logger.debug("RX: %s" % result_str) > - return result_str > - > - > - def __getbytes(self, bytes): > - """Read an amount of bytes from the socket""" > - try: > - return self.socket.recv(bytes) > - except socket.error, error: > - raise OperationalError(error[1]) > > +import sys > > - def __putblock(self, block): > - """ wrap the line in mapi format and put it into the socket """ > - pos = 0 > - last = 0 > - while not last: > - data = block[pos:pos+MAX_PACKAGE_LENGTH] > - length = len(data) > - if length < MAX_PACKAGE_LENGTH: > - last = 1 > - flag = struct.pack( '<H', ( length << 1 ) + last ) > - try: > - logger.debug("II: sending %i bytes, last: %s" % (length, last)) > - logger.debug("TX: %s" % data) > - self.socket.send(flag) > - self.socket.send(data) > - except socket.error, error: > - raise OperationalError(error[1]) > - pos += length > +# a ugly hack to support python 2 and 3 at the same time > +(major, minor, micro, level, serial) = sys.version_info > +if (major == 3): > + from monetdb.mapi3 import * > +else: > + from monetdb.mapi2 import * > > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal tablet_sql.mx, , 1.9, 1.10
by Sjoerd Mullender 06 Oct '09

06 Oct '09

Shouldn't this have gone to the Nov2009 branch? Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv8289 > > Modified Files: > tablet_sql.mx > Log Message: > Handle the case that an integer column is fed a string value that > appears to match the null string. > > > U tablet_sql.mx > Index: tablet_sql.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/tablet_sql.mx,v > retrieving revision 1.9 > retrieving revision 1.10 > diff -u -d -r1.9 -r1.10 > --- tablet_sql.mx 2 Oct 2009 14:31:39 -0000 1.9 > +++ tablet_sql.mx 6 Oct 2009 13:33:45 -0000 1.10 > @@ -88,12 +88,21 @@ > #endif > adt = fmt->nildata; > fmt->c[0]->T->nonil = 0; > - } else if ( quote && *s == quote ) { > - s++; /* find the last quote */ > + } else > + if ( quote && *s == quote ) { > + /* strip the quotes when present */ > + s++; > for ( t = e = s; *t ; t++) > if ( *t == quote) e = t; > *e = 0; > adt = fmt->frstr(fmt, fmt->adt, s, e, 0); > + /* The user might have specified a null string escape > + e.g. NULL as '', which should be tested */ > + if (adt == NULL && s == e && fmt->nullstr && > + strncasecmp(s, fmt->nullstr, fmt->null_length+1) == 0){ > + adt = fmt->nildata; > + fmt->c[0]->T->nonil = 0; > + } > } else { > for( e=s; *e; e++) > ; > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry® Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9-12, 2009. Register now! > http://p.sf.net/sfu/devconf > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5 sql_result.mx, , 1.127, 1.128
by Sjoerd Mullender 06 Oct '09

06 Oct '09

Shouldn't this have gone to the Nov2009 branch? Martin Kersten wrote: > Update of /cvsroot/monetdb/sql/src/backends/monet5 > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv7452 > > Modified Files: > sql_result.mx > Log Message: > The loader now properly recognizes decimals read as integer values. > > > U sql_result.mx > Index: sql_result.mx > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/backends/monet5/sql_result.mx,v > retrieving revision 1.127 > retrieving revision 1.128 > diff -u -d -r1.127 -r1.128 > --- sql_result.mx 23 Sep 2009 07:36:21 -0000 1.127 > +++ sql_result.mx 6 Oct 2009 13:30:13 -0000 1.128 > @@ -381,8 +381,17 @@ > } > > len = (*BATatoms[type].atomFromStr)(s, &c->len, (ptr) &c->data); > - if (len == 0 || len != e-s) > + if (len == 0 || len != e-s){ > + /* decimals can be converted to integers when *.000 */ > + if ( s[len++] == '.') > + switch( type ){ > + case TYPE_bte: case TYPE_int: case TYPE_lng: case TYPE_sht: > + while (s[len]=='0') len++; > + if ( s[len] == 0) > + return c->data; > + } > return NULL; > + } > return c->data; > } > > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry® Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9-12, 2009. Register now! > http://p.sf.net/sfu/devconf > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins -- Sjoerd Mullender

1 0