- developers-list - monetdb.org

Re: MonetDB: default - rewrote result parser to make it eat multiple...
by Stefan Manegold 12 Mar '13

12 Mar '13

Dear Gijs, thank you very much for continuously taking care of MonetDB's python interface / API! We very muchappreciate your work and contribution! However, we'd appreciate even more, it you could also keep the stable output up-to-date whenever you change / extend you test or your changes intendedly change the output of your tests. See http://monetdb.cwi.nl/testweb/web/status.php http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&or… http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&or… http://monetdb.cwi.nl/testweb/web/showtestoutput.php?serial=47169:7d1c376e9… http://monetdb.cwi.nl/testweb/web/testchange.php?test=47169:7d1c376e995f/GN… If nothing else, simply go to .../MonetDB/sql/test/mapi/Tests/ and run `Mtest.py` to run (among others) your pyhton test (preferably on a system with both Python2 & python3 installed, and follow the instructions. See also `Mtest.py --help`, `Mapprove.py --help` and http://dev.monetdb.org/hg/MonetDB/file/tip/testing/README Thanks! Stefan On Mon, Mar 11, 2013 at 06:44:43PM +0100, Gijs Molenaar wrote: > Changeset: 39b2905626b8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=39b2905626b8 > Modified Files: > clients/python2/monetdb/sql/cursors.py > clients/python2/test/capabilities.py > clients/python3/monetdb/sql/cursors.py > clients/python3/test/capabilities.py > Branch: default > Log Message: > > rewrote result parser to make it eat multiple resultsets > > > diffs (truncated from 463 to 300 lines): > > diff --git a/clients/python2/monetdb/sql/cursors.py b/clients/python2/monetdb/sql/cursors.py > --- a/clients/python2/monetdb/sql/cursors.py > +++ b/clients/python2/monetdb/sql/cursors.py > @@ -380,133 +380,99 @@ class Cursor(object): > if not block: > block = "" > > - lines = block.split("\n") > - firstline = lines[0] > + for line in block.split("\n"): > + if line.startswith(mapi.MSG_INFO): > + logger.info(line[1:]) > + self.messages.append((Warning, line[1:])) > > - while firstline.startswith(mapi.MSG_INFO): > - logger.info(firstline[1:]) > - self.messages.append((Warning, firstline[1:])) > - lines = lines[1:] > - firstline = lines[0] > + elif line.startswith(mapi.MSG_QTABLE): > + (self.__query_id, rowcount, columns, tuples) = line[2:].split() > > - if firstline.startswith(mapi.MSG_QTABLE): > - (id, rowcount, columns, tuples) = firstline[2:].split() > - columns = int(columns) # number of columns in result > - rowcount = int(rowcount) # total number of rows > - tuples = int(tuples) # number of rows in this set > - rows = [] > + columns = int(columns) # number of columns in result > + self.rowcount = int(rowcount) # total number of rows > + tuples = int(tuples) # number of rows in this set > + self.__rows = [] > > - # set up fields for description > - table_name = [None]*columns > - column_name = [None]*columns > - type_ = [None]*columns > - display_size = [None]*columns > - internal_size = [None]*columns > - precision = [None]*columns > - scale = [None]*columns > - null_ok = [None]*columns > + # set up fields for description > + table_name = [None] * columns > + column_name = [None] * columns > + type_ = [None] * columns > + display_size = [None] * columns > + internal_size = [None] * columns > + precision = [None] * columns > + scale = [None] * columns > + null_ok = [None] * columns > + typesizes = [(0, 0)] * columns > + self.__offset = 0 > + self.lastrowid = None > > - typesizes = [(0,0)]*columns > + elif line.startswith(mapi.MSG_HEADER): > + (data, identity) = line[1:].split("#") > + values = [x.strip() for x in data.split(",")] > + identity = identity.strip() > > - for line in lines[1:]: > - if line.startswith(mapi.MSG_HEADER): > - (data, identity) = line[1:].split("#") > - values = [x.strip() for x in data.split(",")] > - identity = identity.strip() > + if identity == "table_name": > + table_name = values > + elif identity == "name": > + column_name = values > + elif identity == "type": > + type_ = values > + elif identity == "length": > + length = values > + elif identity == "typesizes": > + typesizes = [[int(j) for j in i.split()] for i in values] > + internal_size = [x[0] for x in typesizes] > + for num, typeelem in enumerate(type_): > + if typeelem in ['decimal']: > + precision[num] = typesizes[num][0] > + scale[num] = typesizes[num][1] > + else: > + msg = "unknown header field" > + self.messages.append((InterfaceError, msg)) > + self.__exception_handler(InterfaceError, msg) > > - if identity == "table_name": > - table_name = values # not used > - elif identity == "name": > - column_name = values > - elif identity == "type": > - type_ = values > - elif identity == "length": > - pass # not used > - elif identity == "typesizes": > - typesizes = [[int(j) for j in i.split()] for i in values] > - internal_size = [x[0] for x in typesizes] > - for num, typeelem in enumerate(type_): > - if typeelem in ['decimal']: > - precision[num] = typesizes[num][0] > - scale[num] = typesizes[num][1] > - else: > - self.messages.append((InterfaceError, > - "unknown header field")) > - self.__exception_handler(InterfaceError, > - "unknown header field") > + self.description = list(zip(column_name, type_, display_size, > + internal_size, precision, scale, > + null_ok)) > + self.__offset = 0 > + self.lastrowid = None > > - self.description = list(zip(column_name, type_, > - display_size, internal_size, precision, scale, null_ok)) > > - if line.startswith(mapi.MSG_TUPLE): > - values = self.__parse_tuple(line) > - rows.append(values) > + elif line.startswith(mapi.MSG_TUPLE): > + values = self.__parse_tuple(line) > + self.__rows .append(values) > > - elif line == mapi.MSG_PROMPT: > - self.__query_id = id > - self.__rows = rows > - self.__offset = 0 > + elif line.startswith(mapi.MSG_QBLOCK): > + self.__rows = [] > > - self.rowcount = rowcount > - self.lastrowid = None > - logger.debug("II store result finished") > - return > - > - elif firstline.startswith(mapi.MSG_QBLOCK): > - rows = [] > - for line in lines[1:]: > - if line.startswith(mapi.MSG_TUPLE): > - values = self.__parse_tuple(line) > - rows.append(values) > - elif line == mapi.MSG_PROMPT: > - logger.debug("II store result finished") > - self.__rows = rows > - return > - > - elif firstline.startswith(mapi.MSG_QSCHEMA): > - if lines[1] == mapi.MSG_PROMPT: > + elif line.startswith(mapi.MSG_QSCHEMA): > + self.__offset = 0 > + self.lastrowid = None > self.__rows = [] > - self.__offset = 0 > self.description = None > self.rowcount = -1 > - self.lastrowid = None > - logger.debug("II schema finished") > - return > > - elif firstline.startswith(mapi.MSG_QUPDATE): > - if lines[1] == mapi.MSG_PROMPT: > - (affected, identity) = firstline[2:].split() > + elif line.startswith(mapi.MSG_QUPDATE): > + (affected, identity) = line[2:].split() > + self.__offset = 0 > self.__rows = [] > - self.__offset = 0 > self.description = None > self.rowcount = int(affected) > self.lastrowid = int(identity) > self.__query_id = -1 > - logger.debug("II update finished") > + > + elif line.startswith(mapi.MSG_QTRANS): > + self.__offset = 0 > + self.lastrowid = None > + self.__rows = [] > + self.description = None > + self.rowcount = -1 > + > + elif line.startswith(mapi.MSG_PROMPT): > return > > - elif firstline.startswith(mapi.MSG_ERROR): > - self.__exception_handler(ProgrammingError, firstline[1:]) > - > - elif firstline.startswith(mapi.MSG_QTRANS): > - if lines[1] == mapi.MSG_PROMPT: > - self.__rows = [] > - self.__offset = 0 > - self.description = None > - self.rowcount = -1 > - self.lastrowid = None > - logger.debug("II transaction finished") > - return > - > - elif firstline.startswith(mapi.MSG_PROMPT): > - self.__query_id = -1 > - self.__rows = [] > - self.__offset = 0 > - > - self.rowcount = 0 > - self.lastrowid = None > - logger.debug("II empty response, assuming everything is ok") > - return > + elif line.startswith(mapi.MSG_ERROR): > + self.__exception_handler(ProgrammingError, line[1:]) > > # you are not supposed to be here > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > diff --git a/clients/python2/test/capabilities.py b/clients/python2/test/capabilities.py > --- a/clients/python2/test/capabilities.py > +++ b/clients/python2/test/capabilities.py > @@ -352,4 +352,16 @@ class DatabaseTest(unittest.TestCase): > t = ["list", "test"] > self.assertRaises(ProgrammingError, self.db_module.monetize.convert, t) > self.db_module.monetize.mapping[list] = str > - self.assertEqual(self.db_module.monetize.convert(t), "['list', 'test']") > \ No newline at end of file > + self.assertEqual(self.db_module.monetize.convert(t), "['list', 'test']") > + > + def test_multiple_queries(self): > + table1 = self.new_table_name() > + table2 = table1[:-1] + 'bla"' > + self.cursor.execute("create table %s (a int)" % table1) > + self.cursor.execute("create table %s (a int, b int)" % table2) > + self.cursor.execute("insert into %s VALUES (100)" % table1) > + self.cursor.execute("insert into %s VALUES (50, 50)" % table2) > + self.cursor.execute('select * from %s; select * from %s;' % > + (table1, table2)) > + result = self.cursor.fetchall() > + self.assertEqual(result, [(50, 50)]) > \ No newline at end of file > diff --git a/clients/python3/monetdb/sql/cursors.py b/clients/python3/monetdb/sql/cursors.py > --- a/clients/python3/monetdb/sql/cursors.py > +++ b/clients/python3/monetdb/sql/cursors.py > @@ -381,133 +381,99 @@ class Cursor(object): > if not block: > block = "" > > - lines = block.split("\n") > - firstline = lines[0] > + for line in block.split("\n"): > + if line.startswith(mapi.MSG_INFO): > + logger.info(line[1:]) > + self.messages.append((Warning, line[1:])) > > - while firstline.startswith(mapi.MSG_INFO): > - logger.info(firstline[1:]) > - self.messages.append((Warning, firstline[1:])) > - lines = lines[1:] > - firstline = lines[0] > + elif line.startswith(mapi.MSG_QTABLE): > + (self.__query_id, rowcount, columns, tuples) = line[2:].split() > > - if firstline.startswith(mapi.MSG_QTABLE): > - (id, rowcount, columns, tuples) = firstline[2:].split() > - columns = int(columns) # number of columns in result > - rowcount = int(rowcount) # total number of rows > - tuples = int(tuples) # number of rows in this set > - rows = [] > + columns = int(columns) # number of columns in result > + self.rowcount = int(rowcount) # total number of rows > + tuples = int(tuples) # number of rows in this set > + self.__rows = [] > > - # set up fields for description > - table_name = [None]*columns > - column_name = [None]*columns > - type_ = [None]*columns > - display_size = [None]*columns > - internal_size = [None]*columns > - precision = [None]*columns > - scale = [None]*columns > - null_ok = [None]*columns > + # set up fields for description > + table_name = [None] * columns > + column_name = [None] * columns > + type_ = [None] * columns > + display_size = [None] * columns > + internal_size = [None] * columns > + precision = [None] * columns > + scale = [None] * columns > + null_ok = [None] * columns > + typesizes = [(0, 0)] * columns > + self.__offset = 0 > + self.lastrowid = None > > - typesizes = [(0,0)]*columns > + elif line.startswith(mapi.MSG_HEADER): > + (data, identity) = line[1:].split("#") > + values = [x.strip() for x in data.split(",")] > + identity = identity.strip() > > - for line in lines[1:]: > - if line.startswith(mapi.MSG_HEADER): > - (data, identity) = line[1:].split("#") > - values = [x.strip() for x in data.split(",")] > - identity = identity.strip() > + if identity == "table_name": > + table_name = values > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 2

mapi.connect / mapi_connect (Second Try)
by Stephen P. Morgan 12 Mar '13

12 Mar '13

I wrote an email a few days ago to the effect that I could not get the mapi.connect() MAL statement to work due to the server side complaining that I had not specified the database to use. This morning I ran one of the sample programs included with MonetDB, clients/examples,C/sample0.c (sample0). It seems very similar as it issues a mapi_connect() with the database field set to NULL. It fails in the same way, i.e., MAPI = monetdb@localhost:50000 ACTION= mapi_start_talking ERROR = !monetdbd: please specify a database Should mapi_connect() work with the database field set to NULL? (If so, then the mapi.connect() MAL statement should work, I think.) Is there a bug in the server that insists on the database being specified? If there is a bug, does anyone have a fix for this? It would be greatly appreciated. Thanks, Steve Morgan

3 7

Coredump caused by function SERVERreconnectWithoutAlias() and its verifies fix
by Fang Cao 12 Mar '13

12 Mar '13

Hi, Today when I try to use mapi.connect() and mapi.reconnect() with mserver5, I found such crash when I tried to : fang@fang-ThinkCentre-M91p:~/Work/DB-Farm$ mserver5 --dbpath=/home/fang/Work/DB-Farm # MonetDB 5 server v11.15.1 "Feb2013" # Serving database 'DB-Farm', using 8 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # Found 3.692 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/JAQL module loaded # MonetDB/SQL module loaded >mid:=mapi.reconnect("localhost",50000,"monetdb","monetdb","mal"); Segmentation fault (core dumped) The root cause is located at: SERVERreconnectWithoutAlias(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) { int *key =(int*) getArgReference(stk,pci,0); str *host = (str*) getArgReference(stk,pci,1); int *port = (int*) getArgReference(stk,pci,2); str *username = (str*) getArgReference(stk,pci,4); ===>> should be 3 str *password = (str*) getArgReference(stk,pci,5); ===>> should be 4 str *lang = (str*) getArgReference(stk,pci,6); ===>> should be 5 Since we only transfer 5 input parameters, so for the last one “lang”, of course we will coredump the program when try to check 6th. After the modification, everything is fine now, for both mapi.connect() and mapi.reconnect() The reason why this issue is introduced: pattern reconnect(host:str, port:int, usr:str, passwd:str,lang:str):int address SERVERreconnectWithoutAlias comment "Re-establish connection with a remote mserver."; pattern reconnect(host:str, port:int, db_alias:str, usr:str, passwd:str,lang:str):int address SERVERreconnectAlias comment "Re-establish connection with a remote mserver."; So when to implement SERVERreconnectWithoutAlias(), seems following the style of SERVERreconnectAlias(). The latter one has 6 inputs, but the first one has only 5 inputs. So un-needed db_alias was removed, but forgot to adjust the sequence number. Thanks, Fang

2 1

Re: MonetDB: Feb2013 - opt_support.c & opt_wrapper.c: "coercion" -> ...
by Stefan Manegold 12 Mar '13

12 Mar '13

> Changeset: 30c2d470a73d for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=30c2d470a73d > Modified Files: > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > monetdb5/optimizer/opt_support.c > monetdb5/optimizer/opt_wrapper.c > sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > Branch: Feb2013 > Log Message: > > opt_support.c & opt_wrapper.c: "coercion" -> "coercions", > i.e., using the correct optimizer name "coercions" > to ensure the optimizer is indeed activated/used. > > Approved related changes in test output (modified MAL plans > due to now active coercions optimizer). > > This should fix bug 3151. ^!^^ Typo: this should read bug 3251 > Thanks to Pete Hollobon for reporting and proving the initial fix! > > > Open question: > > Why is an incorrect / non-exiting optimizer name silently ignored?? > > > diffs (truncated from 413 to 300 lines): > > diff --git > a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > --- > a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > +++ > b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > @@ -61,15 +61,15 @@ Ready. > % .explain # table_name > % mal # name > % clob # type > -% 178 # length > +% 153 # length > function user.s2_1{autoCommit=true}():void; > X_2 := sql.mvc(); > - (X_5:bat[:oid,:int] ,X_6:bat[:oid,:int] ,X_7:bat[:oid,:int] ) := > sql.copy_from("sys":str,"ttt":str,"|":str,"\\n":str,nil:str,"null":str,"/:\\tmp/xyz":str,-1:lng,0:lng,0:int); > - X_9 := sql.append(X_2,"sys","ttt","a",X_5); > - X_11 := sql.append(X_9,"sys","ttt","b",X_6); > - X_15 := sql.append(X_11,"sys","ttt","c",X_7); > - X_16 := aggr.count(X_7); > - sql.affectedRows(X_15,X_16,""); > + (X_13:bat[:oid,:int] ,X_14:bat[:oid,:int] ,X_15:bat[:oid,:int] ) > := > sql.copy_from("sys","ttt","|","\\n",nil:str,"null","/:\\tmp/xyz",-1:lng,0:lng,0); > + X_17 := sql.append(X_2,"sys","ttt","a",X_13); > + X_19 := sql.append(X_17,"sys","ttt","b",X_14); > + X_23 := sql.append(X_19,"sys","ttt","c",X_15); > + X_24 := aggr.count(X_15); > + sql.affectedRows(X_23,X_24,""); > end s2_1; > # optimizer.mitosis() > # optimizer.dataflow() > @@ -89,15 +89,15 @@ end s2_1; > % .explain # table_name > % mal # name > % clob # type > -% 178 # length > +% 153 # length > function user.s5_1{autoCommit=true}():void; > X_2 := sql.mvc(); > - (X_5:bat[:oid,:int] ,X_6:bat[:oid,:int] ,X_7:bat[:oid,:int] ) := > sql.copy_from("sys":str,"ttt":str,"|":str,"\\n":str,nil:str,"null":str,"/:\\tmp/xyz":str,-1:lng,0:lng,0:int); > - X_9 := sql.append(X_2,"sys","ttt","a",X_5); > - X_11 := sql.append(X_9,"sys","ttt","b",X_6); > - X_16 := aggr.count(X_7); > - X_15 := sql.append(X_11,"sys","ttt","c",X_7); > - sql.affectedRows(X_15,X_16,""); > + (X_13:bat[:oid,:int] ,X_14:bat[:oid,:int] ,X_15:bat[:oid,:int] ) > := > sql.copy_from("sys","ttt","|","\\n",nil:str,"null","/:\\tmp/xyz",-1:lng,0:lng,0); > + X_17 := sql.append(X_2,"sys","ttt","a",X_13); > + X_19 := sql.append(X_17,"sys","ttt","b",X_14); > + X_24 := aggr.count(X_15); > + X_23 := sql.append(X_19,"sys","ttt","c",X_15); > + sql.affectedRows(X_23,X_24,""); > end s5_1; > # optimizer.mitosis() > # optimizer.dataflow() > diff --git a/monetdb5/optimizer/opt_support.c > b/monetdb5/optimizer/opt_support.c > --- a/monetdb5/optimizer/opt_support.c > +++ b/monetdb5/optimizer/opt_support.c > @@ -144,7 +144,7 @@ struct OPTcatalog { > {"groups", 0, 0, 0, DEBUG_OPT_GROUPS}, > {"aliases", 0, 0, 0, DEBUG_OPT_ALIASES}, > {"cluster", 0, 0, 0, DEBUG_OPT_CLUSTER}, > -{"coercion", 0, 0, 0, DEBUG_OPT_COERCION}, > +{"coercions", 0, 0, 0, DEBUG_OPT_COERCION}, > {"commonTerms", 0, 0, 0, DEBUG_OPT_COMMONTERMS}, > {"compress", 0, 0, 0, DEBUG_OPT_COMPRESS}, > {"constants", 0, 0, 0, DEBUG_OPT_CONSTANTS}, > diff --git a/monetdb5/optimizer/opt_wrapper.c > b/monetdb5/optimizer/opt_wrapper.c > --- a/monetdb5/optimizer/opt_wrapper.c > +++ b/monetdb5/optimizer/opt_wrapper.c > @@ -84,7 +84,7 @@ struct{ > {"aliases", &OPTaliasesImplementation}, > {"centipede", &OPTcentipedeImplementation}, > {"cluster", &OPTclusterImplementation}, > - {"coercion", &OPTcoercionImplementation}, > + {"coercions", &OPTcoercionImplementation}, > {"commonTerms", &OPTcommonTermsImplementation}, > {"compression", &OPTcompressionImplementation}, > {"costModel", &OPTcostModelImplementation}, > diff --git a/sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > b/sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > --- a/sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > +++ b/sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > @@ -27,10 +27,10 @@ Ready. > % .explain # table_name > % mal # name > % clob # type > -% 76 # length > +% 72 # length > function user.s0_1{autoCommit=true}(A0:bte,A1:bte):void; > X_4 := udf.fuse(A0,A1); > - > sql.exportValue(1,".","fuse_single_value":str,"smallint",16,0,6,X_4,""); > + > sql.exportValue(1,".","fuse_single_value","smallint",16,0,6,X_4,""); > end s0_1; > # optimizer.mitosis() > # optimizer.dataflow() > @@ -44,10 +44,10 @@ end s0_1; > % .explain # table_name > % mal # name > % clob # type > -% 71 # length > +% 67 # length > function user.s1_1{autoCommit=true}(A0:sht,A1:sht):void; > X_4 := udf.fuse(A0,A1); > - > sql.exportValue(1,".","fuse_single_value":str,"int",32,0,6,X_4,""); > + sql.exportValue(1,".","fuse_single_value","int",32,0,6,X_4,""); > end s1_1; > #select fuse(1000,2000); > % . # table_name > @@ -59,10 +59,10 @@ end s1_1; > % .explain # table_name > % mal # name > % clob # type > -% 74 # length > +% 70 # length > function user.s2_1{autoCommit=true}(A0:int,A1:int):void; > X_4 := udf.fuse(A0,A1); > - > sql.exportValue(1,".","fuse_single_value":str,"bigint",64,0,6,X_4,""); > + > sql.exportValue(1,".","fuse_single_value","bigint",64,0,6,X_4,""); > end s2_1; > #select fuse(1000000,2000000); > % . # table_name > diff --git a/sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > b/sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > --- a/sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > +++ b/sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > @@ -48,10 +48,10 @@ Ready. > % .explain # table_name > % mal # name > % clob # type > -% 74 # length > +% 70 # length > function user.s0_1{autoCommit=true}(A0:str):void; > X_3 := udf.reverse(A0); > - > sql.exportValue(1,".","reverse_single_value":str,"clob",0,0,4,X_3,""); > + > sql.exportValue(1,".","reverse_single_value","clob",0,0,4,X_3,""); > end s0_1; > #select reverse('MonetDB'); > % . # table_name > @@ -81,7 +81,7 @@ end s0_1; > % .explain # table_name > % mal # name > % clob # type > -% 62 # length > +% 60 # length > function user.s3_1{autoCommit=true}():void; > X_2 := sql.mvc(); > X_3:bat[:oid,:oid] := sql.tid(X_2,"sys","udf_reverse"); > @@ -92,7 +92,7 @@ function user.s3_1{autoCommit=true}():vo > X_15 := algebra.leftfetchjoin(X_3,X_14); > X_16:bat[:oid,:str] := batudf.reverse(X_15); > X_18 := sql.resultSet(1,1,X_16); > - sql.rsColumn(X_18,"sys.","reverse_x":str,"clob",0,0,X_16); > + sql.rsColumn(X_18,"sys.","reverse_x","clob",0,0,X_16); > X_22 := io.stdout(); > sql.exportResult(X_22,X_18); > end s3_1; > diff --git > a/sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > b/sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > --- > a/sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > +++ > b/sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > @@ -27,7 +27,7 @@ Ready. > % .explain # table_name > % mal # name > % clob # type > -% 63 # length > +% 59 # length > function user.s0_1{autoCommit=true}(A0:dbl):void; > X_3 := sql.mvc(); > X_4:bat[:oid,:oid] := sql.tid(X_3,"sys","sf_2596114"); > @@ -38,7 +38,7 @@ function user.s0_1{autoCommit=true}(A0:d > X_16 := algebra.leftfetchjoin(X_4,X_15); > X_17:bat[:oid,:dbl] := batmmath.pow(X_16,A0); > X_19 := sql.resultSet(1,1,X_17); > - sql.rsColumn(X_19,"sys.","power_x":str,"double",53,0,X_17); > + sql.rsColumn(X_19,"sys.","power_x","double",53,0,X_17); > X_24 := io.stdout(); > sql.exportResult(X_24,X_19); > end s0_1; > @@ -46,7 +46,7 @@ end s0_1; > % .explain # table_name > % mal # name > % clob # type > -% 65 # length > +% 61 # length > function user.s1_1{autoCommit=true}():void; > X_2 := sql.mvc(); > X_3:bat[:oid,:oid] := sql.tid(X_2,"sys","sf_2596114"); > @@ -57,7 +57,7 @@ function user.s1_1{autoCommit=true}():vo > X_15 := algebra.leftfetchjoin(X_3,X_14); > X_16:bat[:oid,:dbl] := batcalc.*(X_15,X_15); > X_18 := sql.resultSet(1,1,X_16); > - sql.rsColumn(X_18,"sys.","sql_mul_x":str,"double",53,0,X_16); > + sql.rsColumn(X_18,"sys.","sql_mul_x","double",53,0,X_16); > X_23 := io.stdout(); > sql.exportResult(X_23,X_18); > end s1_1; > diff --git > a/sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > b/sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > --- > a/sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > +++ > b/sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > @@ -34,22 +34,22 @@ function user.s0_1(A0:str):void; > X_2 := sql.mvc(); > X_6 := sql.bind(X_2,"sys","functions","name",0); > X_3:bat[:oid,:oid] := sql.tid(X_2,"sys","functions"); > - X_45 := algebra.likesubselect(X_6,X_3,A0,"":str,false); > + X_45 := algebra.likesubselect(X_6,X_3,A0,"",false); > (X_9,r1_9) := sql.bind(X_2,"sys","functions","name",2); > - X_46 := algebra.likesubselect(r1_9,A0,"":str,false); > + X_46 := algebra.likesubselect(r1_9,A0,"",false); > X_12 := sql.bind(X_2,"sys","functions","name",1); > - X_47 := algebra.likesubselect(X_12,X_3,A0,"":str,false); > - X_14 := sql.subdelta(X_45,X_9,X_46,X_47); > - X_16 := sql.projectdelta(X_14,X_6,X_9,r1_9,X_12); > - X_17 := sql.bind(X_2,"sys","functions","func",0); > - (X_19,r1_28) := sql.bind(X_2,"sys","functions","func",2); > - X_21 := sql.bind(X_2,"sys","functions","func",1); > - X_22 := sql.projectdelta(X_14,X_17,X_19,r1_28,X_21); > - X_23 := sql.resultSet(2,1,X_16); > - sql.rsColumn(X_23,"sys.functions","name","varchar",256,0,X_16); > - sql.rsColumn(X_23,"sys.functions","func","varchar",8196,0,X_22); > - X_32 := io.stdout(); > - sql.exportResult(X_32,X_23); > + X_47 := algebra.likesubselect(X_12,X_3,A0,"",false); > + X_15 := sql.subdelta(X_45,X_9,X_46,X_47); > + X_17 := sql.projectdelta(X_15,X_6,X_9,r1_9,X_12); > + X_18 := sql.bind(X_2,"sys","functions","func",0); > + (X_20,r1_28) := sql.bind(X_2,"sys","functions","func",2); > + X_22 := sql.bind(X_2,"sys","functions","func",1); > + X_23 := sql.projectdelta(X_15,X_18,X_20,r1_28,X_22); > + X_24 := sql.resultSet(2,1,X_17); > + sql.rsColumn(X_24,"sys.functions","name","varchar",256,0,X_17); > + sql.rsColumn(X_24,"sys.functions","func","varchar",8196,0,X_23); > + X_33 := io.stdout(); > + sql.exportResult(X_33,X_24); > end s0_1; > #explain select name,func from functions where name not like > '%optimizers%'; > % .explain # table_name > @@ -60,22 +60,22 @@ function user.s1_1(A0:str):void; > X_2 := sql.mvc(); > X_6 := sql.bind(X_2,"sys","functions","name",0); > X_3:bat[:oid,:oid] := sql.tid(X_2,"sys","functions"); > - X_45 := algebra.likesubselect(X_6,X_3,A0,"":str,true); > + X_45 := algebra.likesubselect(X_6,X_3,A0,"",true); > (X_9,r1_9) := sql.bind(X_2,"sys","functions","name",2); > - X_46 := algebra.likesubselect(r1_9,A0,"":str,true); > + X_46 := algebra.likesubselect(r1_9,A0,"",true); > X_12 := sql.bind(X_2,"sys","functions","name",1); > - X_47 := algebra.likesubselect(X_12,X_3,A0,"":str,true); > - X_14 := sql.subdelta(X_45,X_9,X_46,X_47); > - X_16 := sql.projectdelta(X_14,X_6,X_9,r1_9,X_12); > - X_17 := sql.bind(X_2,"sys","functions","func",0); > - (X_19,r1_28) := sql.bind(X_2,"sys","functions","func",2); > - X_21 := sql.bind(X_2,"sys","functions","func",1); > - X_22 := sql.projectdelta(X_14,X_17,X_19,r1_28,X_21); > - X_23 := sql.resultSet(2,1,X_16); > - sql.rsColumn(X_23,"sys.functions","name","varchar",256,0,X_16); > - sql.rsColumn(X_23,"sys.functions","func","varchar",8196,0,X_22); > - X_32 := io.stdout(); > - sql.exportResult(X_32,X_23); > + X_47 := algebra.likesubselect(X_12,X_3,A0,"",true); > + X_15 := sql.subdelta(X_45,X_9,X_46,X_47); > + X_17 := sql.projectdelta(X_15,X_6,X_9,r1_9,X_12); > + X_18 := sql.bind(X_2,"sys","functions","func",0); > + (X_20,r1_28) := sql.bind(X_2,"sys","functions","func",2); > + X_22 := sql.bind(X_2,"sys","functions","func",1); > + X_23 := sql.projectdelta(X_15,X_18,X_20,r1_28,X_22); > + X_24 := sql.resultSet(2,1,X_17); > + sql.rsColumn(X_24,"sys.functions","name","varchar",256,0,X_17); > + sql.rsColumn(X_24,"sys.functions","func","varchar",8196,0,X_23); > + X_33 := io.stdout(); > + sql.exportResult(X_33,X_24); > end s1_1; > #explain select name,func from functions where name ilike > '%optimizers%'; > % .explain # table_name > @@ -86,22 +86,22 @@ function user.s2_1(A0:str):void; > X_2 := sql.mvc(); > X_6 := sql.bind(X_2,"sys","functions","name",0); > X_3:bat[:oid,:oid] := sql.tid(X_2,"sys","functions"); > - X_45 := algebra.ilikesubselect(X_6,X_3,A0,"":str,false); > + X_45 := algebra.ilikesubselect(X_6,X_3,A0,"",false); > (X_9,r1_9) := sql.bind(X_2,"sys","functions","name",2); > - X_46 := algebra.ilikesubselect(r1_9,A0,"":str,false); > + X_46 := algebra.ilikesubselect(r1_9,A0,"",false); > X_12 := sql.bind(X_2,"sys","functions","name",1); > - X_47 := algebra.ilikesubselect(X_12,X_3,A0,"":str,false); > - X_14 := sql.subdelta(X_45,X_9,X_46,X_47); > - X_16 := sql.projectdelta(X_14,X_6,X_9,r1_9,X_12); > - X_17 := sql.bind(X_2,"sys","functions","func",0); > - (X_19,r1_28) := sql.bind(X_2,"sys","functions","func",2); > - X_21 := sql.bind(X_2,"sys","functions","func",1); > - X_22 := sql.projectdelta(X_14,X_17,X_19,r1_28,X_21); > - X_23 := sql.resultSet(2,1,X_16); > - sql.rsColumn(X_23,"sys.functions","name","varchar",256,0,X_16); > - sql.rsColumn(X_23,"sys.functions","func","varchar",8196,0,X_22); > - X_32 := io.stdout(); > - sql.exportResult(X_32,X_23); > + X_47 := algebra.ilikesubselect(X_12,X_3,A0,"",false); > + X_15 := sql.subdelta(X_45,X_9,X_46,X_47); > + X_17 := sql.projectdelta(X_15,X_6,X_9,r1_9,X_12); > + X_18 := sql.bind(X_2,"sys","functions","func",0); > + (X_20,r1_28) := sql.bind(X_2,"sys","functions","func",2); > + X_22 := sql.bind(X_2,"sys","functions","func",1); > + X_23 := sql.projectdelta(X_15,X_18,X_20,r1_28,X_22); > + X_24 := sql.resultSet(2,1,X_17); > + sql.rsColumn(X_24,"sys.functions","name","varchar",256,0,X_17); > + sql.rsColumn(X_24,"sys.functions","func","varchar",8196,0,X_23); > + X_33 := io.stdout(); > + sql.exportResult(X_33,X_24); > end s2_1; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Running MAL Programs Manually?
by Stephen P. Morgan 11 Mar '13

11 Mar '13

Should it be possible to run MAL programs manually, by reading them from a file? I just tried a simple experiment and got an error message. The experiment: I created a table, "test," with two columns, "field1" and field2." I created a MAL query by issuing the SQL query, "EXPLAIN SELECT * FROM test;" and cut and pasted the output to a file. I then stripped out all of the formatting characters ('|') and extraneous spaces. I then re-entered mclient, choosing the MAL language, and read in the MAL form of the query with "\< junk.mal." The interpreter then complained that two identifiers were not initialized (see output, below). So, is it not possible to issue a MAL query by hand? Or am I missing something? (Again, please forgive me, I am new to MonetDB.) Thanks, Steve Morgan mal>\< junk.mal MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = function user.s1_1{autoCommit=true}():void; barrier X_41 := language.dataflow(); X_2 := sql.mvc(); X_3:bat[:oid,:oid] := sql.tid(X_2,"sys","test"); X_6 := sql.bind(X_2,"sys","test","field1",0); (X_9,r1_9) := sql.bind(X_2,"sys","test","field1",2); X_12 := sql.bind(X_2,"sys","test","field1",1); X_14 := sql.delta(X_6,X_9,r1_9,X_12); X_15 := algebra.leftfetchjoin(X_3,X_14); X_16 := sql.bind(X_2,"sys","test","field2",0); (X_18,r1_22) := sql.bind(X_2,"sys","test","field2",2); X_20 := sql.bind(X_2,"sys","test","field2",1); X_21 := sql.delta(X_16,X_18,r1_22,X_20); X_22 := algebra.leftfetchjoin(X_3,X_21); language.pass(X_3); exit X_41; X_23 := sql.resultSet(2,1,X_15); sql.rsColumn(X_23,"sys.test","field1","int",32,0,X_15); sql.rsColumn(X_23,"sys.test","field2","int",32,0,X_22); X_29 := io.stdout(); sql.exportResult(X_29,X_23); end s1_1; ERROR = !TypeException:user.s1_1[16]:'X_15' may not be used before being initialized !TypeException:user.s1_1[17]:'X_15' may not be used before being initialized !TypeException:user.s1_1[18]:'X_22' may not be used before being initialized

1 1

External data atoms Issues
by Mike Gomez 11 Mar '13

11 Mar '13

Hi, i have been trying to expand data types (both core and external atoms).However, after discussions, i understood that addition of core data types will takes years( if i am not exaggerating).So, i started testing up external gdk atoms (monetdb5/modules/atoms). i am creating new atom type say mytype which is int based as below. this atom is pure int ( there are no bitfields and so on) typedef int mytype; atom mytype:int; in mal i have defined the following mal procedures( and C backends) fromstr(), tostr(), mytype(s:str):mytype, new(s:str):mytype, isnil(v:mytype):bit at this moment,it can successfully create tables and do selects. however, selects are not working as they should. these queries work. create table test(i mytype); insert into test values('453535'); select * from test these don't work. select * from test2 where i >= 5 above query gives error "types mytype(0,0) and tinyint(8,0) are not equal for column 'i'" insert into test2 values(453535); gives types int(32,0) and mytype(0,0) are not equal however, this query works !!!!!! select * from test2 where i >= '1' after testing and too many code peeping for above query, i figured out that the above query converts each element of table to string using tostr, and then compares the strings, and takes decision. so, it looks like there can be performance improvements here. However,my question is HOW do i get the queries working such as select * from test2 where i >= 5 am i missing something?? any pointers may be helpful. Thanks. -Mike

1 0

sabdb protocol V2
by Gijs Molenaar 11 Mar '13

11 Mar '13

Hi MonetDB people, I've noticed that there has been a new version (2) of the sabdb protocol introduced. The sabdb protocol is used to send administrative command to a monetdb server. This breaks the Python API, which I just fixed, I'll commit it later. But I've noticed some strange behavior, it looks like monetdb closes the connection in some cases, not all. Specifically when I issue a status command the connection is closed, but in other cases it is not. I've fixed it now by reconnecting before every command, but I just wanted to verify if that is the right way to go. greetings, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

3 5

mapi.connect() MAL Statement
by Stephen P. Morgan 08 Mar '13

08 Mar '13

I'm in the early stages of trying to write a MAL program to access a remote database using mapi.* modules, mainly to test out the function, but also for future work on distributed query processing. I find that if I issue the mapi,connect() call as follows: mal> ... mal> mid=mapi.connect("localhost",50000,"monetdb","monetdb","mal"); I get the following response: MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = mid:=mapi.connect("localhost",50000,"monetdb","monetdb","mal"); ERROR = !IOException:mapi.connect:Could not connect: monetdbd: please specify a database The "please specify a database" part seems to be coming from tools/merovingian/daemon/client.c, i.e., the connection was attempted but was rejected by monetdbd because a database parameter was not specified. I( noticed that mapi.connect() calls SERVERconnet() which calls SERVERconnectAll() which calls mapi_connect() with NULL as the database parameter. Should SERVERconnect() (i.e., mapi.connect()) be altered to accept another parameter, database, which would be passed through SERVERconnectAll() and mapi_connect()? Or am I missing something altogether about mapi.connect()? P.S. Please excuse me if this is a foolish question. I am new to MonetDB. Thank you, Steve Morgan

1 0

Adding data types (maybe to core), probably BUG
by Mike Gomez 08 Mar '13

08 Mar '13

Hi, i am working on monetdb so as it can be used efficiently by more users(myself included).However, the issue i seem to face ( as maybe others) is the data types. for many weeks now,I have been going through the source code, understanding and learning it.Data types in monetdb can be extended in 2 types, external and core GDK. external method problem ( maybe bug in monetdb) ======================== I have created new data type, say unsigned int 32 bit, using extensions (such as inet,color and so on).However, the bug is monetdb system doesn't use my comparison operation, instead uses the default inbuilt operators. eg. in the uint mal file( MonetDB/monetdb5/modules/atoms/uint.mal), i have declared -------------- atom uint:int; .. ..............skipped source code however, when i sort the table, the system still places negative values ahead, means instead of using my declared comparison algos, it's using internal algos. my question is how do i override that ( or is it even possible to do it). core GDK method =============== After failing that, i thought lets add the core data type uint in the gdk_atoms.c. i modified the parsor(& everything else) and the system recognizes my data type. however,when i run simple insert query such as below, i get insert into test1 values('2'); TypeException:user.s1_1[3]:'calc.uint' undefined in: _5:any := calc.uint(_4:str) however, i have got completely stuck in the MAL operators and language. so, how do i get the algebra, operations and everything right, so it works? Any pointers will help. let me know if something more is required. i am working on this out of my interest,so eventually if i get it to work somehow, the code goes back to project. Thanks for help. mike

2 3

Announcement: New Feb2013-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 07 Mar '13

07 Mar '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP1 bugfix release Bug Fixes * 3228: Python client - mapi.Connection.__getbyes should throw exception when 0 bytes received from socket * 3229: GRANT does not work for UPDATE * 3230: GRANT SELECT priveleges not set in SQL FUNCTIONs * 3233: getting a crash of server with segmentation fault on SLES 11 64 SP2 * 3235: MapiPP.pm unquote '0' as 'undef' * 3236: mclient \d produces no output * 3237: Assertion on BATgroup_internal * 3241: SEGFAULT on string processing when using optimizer.reduce()

1 0