- developers-list - monetdb.org

Re: [Monetdb-developers] MonetDB: gdk-calc - gdk_calc.c, mod_typeswitchloop(): initialize...
by Stefan Manegold 16 Feb '12

16 Feb '12

Standard gcc on Fedora 16; MonetDB configured with --disable-debug --enable-optimize --disable-assert -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | ----- Reply message ----- From: "Sjoerd Mullender" <sjoerd(a)acm.org> To: <monetdb-developers(a)lists.sourceforge.net> Subject: [Monetdb-developers] MonetDB: gdk-calc - gdk_calc.c, mod_typeswitchloop(): initialize... Date: Thu, Feb 16, 2012 09:18 On 2012-02-15 20:12, Stefan Manegold wrote: > Changeset: fc0d3fad41c2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc0d3fad41c2 > Modified Files: > gdk/gdk_calc.c > Branch: gdk-calc > Log Message: > > gdk_calc.c, mod_typeswitchloop(): initialize "nils" (hopefully correctly) Which sucky compiler was this? There is no code path where nils was not initialized. (This also means that the value you choose here doesn't matter.) > > diffs (12 lines): > > diff --git a/gdk/gdk_calc.c b/gdk/gdk_calc.c > --- a/gdk/gdk_calc.c > +++ b/gdk/gdk_calc.c > @@ -6157,7 +6157,7 @@ mod_typeswitchloop(const void *lft, int > void *dst, int tp, BUN cnt, > int abort_on_error, const char *func) > { > - BUN nils; > + BUN nils = BUN_NONE; > > switch (ATOMstorage(tp1)) { > case TYPE_bte: > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] MonetDB: gdk-calc - gdk_calc.c, mod_typeswitchloop(): initialize...
by Sjoerd Mullender 16 Feb '12

16 Feb '12

On 2012-02-15 20:12, Stefan Manegold wrote: > Changeset: fc0d3fad41c2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc0d3fad41c2 > Modified Files: > gdk/gdk_calc.c > Branch: gdk-calc > Log Message: > > gdk_calc.c, mod_typeswitchloop(): initialize "nils" (hopefully correctly) Which sucky compiler was this? There is no code path where nils was not initialized. (This also means that the value you choose here doesn't matter.) > > diffs (12 lines): > > diff --git a/gdk/gdk_calc.c b/gdk/gdk_calc.c > --- a/gdk/gdk_calc.c > +++ b/gdk/gdk_calc.c > @@ -6157,7 +6157,7 @@ mod_typeswitchloop(const void *lft, int > void *dst, int tp, BUN cnt, > int abort_on_error, const char *func) > { > - BUN nils; > + BUN nils = BUN_NONE; > > switch (ATOMstorage(tp1)) { > case TYPE_bte: > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

1 1

Re: [Monetdb-developers] MonetDB: default - More advice on the optimizer template.
by Stefan Manegold 11 Feb '12

11 Feb '12

On Sat, Feb 11, 2012 at 02:06:17PM +0100, Martin Kersten wrote: > > > On 2/11/12 11:03 AM, Stefan Manegold wrote: > >On Wed, Feb 08, 2012 at 10:27:11AM +0100, Martin Kersten wrote: > >>Changeset: 67c12a700166 for MonetDB > >>URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=67c12a700166 > >>Modified Files: > >> monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >>Branch: default > >>Log Message: > >> > >>More advice on the optimizer template. > >> > >> > >>diffs (140 lines): > >> > >>diff --git a/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx b/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >>--- a/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >>+++ b/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >[...] > >>@@ -39,6 +39,8 @@ All Rights Reserved. > >> * i.e., an sql.append() statement that is eventually followed by some other > >> * statement later on in the MAL program that uses the same v0 BAT as > >> * argument as the sql.append() statement does, > >>+ * Do you assume a single re-use of the variable v0? > > > >No. Why? > Use assign-once and use-many-times policy. It can improve parallel > processing > and simplifies scope analysis. v0 is (as far as I know) created (assigned) once (by Niels, or preceeding optimizers). If it is used only once (only by sql_append), my optimizer does not (have to) do anything. Otherwise, it replaces one use v0 (by sql_append) by a view of v0. That's the very purpose of this optimizer. > >>+ * Do you assume a non-nested MAL block ? > > > >Not necessarily. > > > Analysis may become complex if you have something like > > V0:= expr > barrier E1:=expr > V0:= expr2 > exit E1 > now V0 depends on runtime use > > > same holds for > barrier E1:= expr > V0:=expr > exit E1 > z:= f(V0) > > will be flagged as an error because V0 may be uninitialized > > >I must admit, that I do not know how the oprimizer framework handles nested > >MAL blocks, and what an optimizer needs to do to be aware of nested MAL > >blocks and to handle them correctly. > Preferrably the MAL blocks are linear programs (until you reach the > dataflow optimizer). How do I know / see that in my optimizer? Do I have to check for barrier / exit statements / constructs myself? > > > >In the sample optimizer, for now, I'd be fine if there are no > >false-positives, i.e., the optimizer triggers in case it should not trigger > >or in cases it cannot handle correctly. > >I can accept false-negatives, i.e., not triggering in all case it could handle > >correctly. > > > >> * > >> * and transform them into > >> * > >>@@ -52,6 +54,7 @@ All Rights Reserved. > >> * > >> * i.e., handing a BAT view v2 of BAT v0 as argument to the sql.append() > >> * statement, rather than the original BAT v0. > >>+ * My advice, always use new variable names, it may capture some easy to make errors. > > > >I/my optimizer does use new variables for all new statements/results. > >I/my optimizer re-use variable names only for identical results. > > > >> * > >> * As a refinement, patterns like > >> * > >[...] > >>@@ -181,13 +195,17 @@ OPTsql_appendImplementation(Client cntxt > >> pushInstruction(mb, q); > >> q1 = q; > >> i++; > >>- actions++; > >>+ actions++; /* to keep track if anything has been done */ > >> } > >> } > >> > >>- /* look for > >>+ /* look for > >> * v5 := ... v0 ...; > >> */ > >>+ /* an expensive loop, better would be to remember that v0 has a different role. > >>+ * A typical method is to keep a map from variable -> instruction where it was > >>+ * detected. The you can check each assignment for use of v0 > >>+ */ > > > >This is general support functionality. > >Is this already available in the optimizer framework? > I try to use single pass algorithms in the optimizers. > Even in the case of commonterms optimizer, we may have to > traverse the history. This can become a n^2 process > > >If so, where is it and how can I use it? > Mimic how it is done in other optimizers (e.g. opt_reorder). > Typically, a buffer is maintained per variable to keep > optimization properties around. > > >If not, where/how could we add it? > > > >> for (j = i+1; !found&& j< limit; j++) > >> for (k = old[j]->retc; !found&& k< old[j]->argc; k++) > >> found = (getArg(old[j], k) == getArg(p, 5)); > >>@@ -202,6 +220,8 @@ OPTsql_appendImplementation(Client cntxt > >> > >> /* push new v1 := aggr.count( v0 ); unless already available */ > >> if (q1 == NULL) { > >>+ /* use mal_buil.mx primitives q1 = newStmt(mb, aggrRef,countRef); setArgType(mb,q1,TYPE_wrd) */ > >>+ /* it will be added to the block and even my re-use MAL instructions */ > > > >Is this (supposed to be) documentation of the existing code below, > >or rather advice how to implement the below functionality differently? > Use the mal_builder to simplify your code base. > > > > >> q1 = newInstruction(mb,ASSIGNsymbol); > >> getArg(q1,0) = newTmpVariable(mb, TYPE_wrd); > >> setModuleId(q1, aggrRef); > >>@@ -211,6 +231,7 @@ OPTsql_appendImplementation(Client cntxt > >> } > >> > >> /* push new v2 := algebra.slice( v0, 0, v1 ); */ > >>+ /* use mal_buil.mx primitives q1 = newStmt(mb, algebraRef,sliceRef); */ > > > >Is this (supposed to be) documentation of the existing code below, > >or rather advice how to implement the below functionality differently? > > > >> q2 = newInstruction(mb,ASSIGNsymbol); > >> getArg(q2,0) = newTmpVariable(mb, TYPE_any); > >> setModuleId(q2, algebraRef); > >>@@ -240,6 +261,7 @@ OPTsql_appendImplementation(Client cntxt > >> for(i++; i<limit; i++) > >> if (old[i]) > >> pushInstruction(mb, old[i]); > >>+ /* any remaining MAL instruction records are removed */ > >> for(; i<slimit; i++) > >> if (old[i]) > >> freeInstruction(old[i]); > >>@@ -253,6 +275,9 @@ OPTsql_appendImplementation(Client cntxt > >> return actions; > >> } > >> > >>+/* optimizers have to be registered in the optcatalog in opt_support.c. > > > >Why? > SQL needs a place to pick up all optimizers known. You may also have > to extend the optimizer pipeline validity code. > > >If at all possible, I'd prefer to be able to add a new optimizer without the > >need to change existing code ... > yes understood, but you have to patch Makefile.ag, youroptimizer.mx, and > opt_support. Possibly, you may have to extend opt_prelude as well > > > > >>+ * you have to path the file accordingly. > "path" > > ^^^^ > >parse? > > > >What does this mean? What am I supposed to do in detail? > > > >>+ */ > >> @include ../../optimizer/optimizerWrapper.mx > >> @c > >> #include "opt_statistics.h" > >>_______________________________________________ > >>Checkin-list mailing list > >>Checkin-list(a)monetdb.org > >>http://mail.monetdb.org/mailman/listinfo/checkin-list > >> > > > >Thanks! > > > >Stefan > > > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > > -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: [Monetdb-developers] MonetDB: default - More advice on the optimizer template.
by Martin Kersten 11 Feb '12

11 Feb '12

>>> + * you have to path the file accordingly. > "path" >> ^^^^ >> parse? patch

1 0

Re: [Monetdb-developers] MonetDB: default - More advice on the optimizer template.
by Stefan Manegold 11 Feb '12

11 Feb '12

On Wed, Feb 08, 2012 at 10:27:11AM +0100, Martin Kersten wrote: > Changeset: 67c12a700166 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=67c12a700166 > Modified Files: > monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > Branch: default > Log Message: > > More advice on the optimizer template. > > > diffs (140 lines): > > diff --git a/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx b/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > --- a/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > +++ b/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx [...] > @@ -39,6 +39,8 @@ All Rights Reserved. > * i.e., an sql.append() statement that is eventually followed by some other > * statement later on in the MAL program that uses the same v0 BAT as > * argument as the sql.append() statement does, > + * Do you assume a single re-use of the variable v0? No. Why? > + * Do you assume a non-nested MAL block ? Not necessarily. I must admit, that I do not know how the oprimizer framework handles nested MAL blocks, and what an optimizer needs to do to be aware of nested MAL blocks and to handle them correctly. In the sample optimizer, for now, I'd be fine if there are no false-positives, i.e., the optimizer triggers in case it should not trigger or in cases it cannot handle correctly. I can accept false-negatives, i.e., not triggering in all case it could handle correctly. > * > * and transform them into > * > @@ -52,6 +54,7 @@ All Rights Reserved. > * > * i.e., handing a BAT view v2 of BAT v0 as argument to the sql.append() > * statement, rather than the original BAT v0. > + * My advice, always use new variable names, it may capture some easy to make errors. I/my optimizer does use new variables for all new statements/results. I/my optimizer re-use variable names only for identical results. > * > * As a refinement, patterns like > * [...] > @@ -181,13 +195,17 @@ OPTsql_appendImplementation(Client cntxt > pushInstruction(mb, q); > q1 = q; > i++; > - actions++; > + actions++; /* to keep track if anything has been done */ > } > } > > - /* look for > + /* look for > * v5 := ... v0 ...; > */ > + /* an expensive loop, better would be to remember that v0 has a different role. > + * A typical method is to keep a map from variable -> instruction where it was > + * detected. The you can check each assignment for use of v0 > + */ This is general support functionality. Is this already available in the optimizer framework? If so, where is it and how can I use it? If not, where/how could we add it? > for (j = i+1; !found && j < limit; j++) > for (k = old[j]->retc; !found && k < old[j]->argc; k++) > found = (getArg(old[j], k) == getArg(p, 5)); > @@ -202,6 +220,8 @@ OPTsql_appendImplementation(Client cntxt > > /* push new v1 := aggr.count( v0 ); unless already available */ > if (q1 == NULL) { > + /* use mal_buil.mx primitives q1 = newStmt(mb, aggrRef,countRef); setArgType(mb,q1,TYPE_wrd) */ > + /* it will be added to the block and even my re-use MAL instructions */ Is this (supposed to be) documentation of the existing code below, or rather advice how to implement the below functionality differently? > q1 = newInstruction(mb,ASSIGNsymbol); > getArg(q1,0) = newTmpVariable(mb, TYPE_wrd); > setModuleId(q1, aggrRef); > @@ -211,6 +231,7 @@ OPTsql_appendImplementation(Client cntxt > } > > /* push new v2 := algebra.slice( v0, 0, v1 ); */ > + /* use mal_buil.mx primitives q1 = newStmt(mb, algebraRef,sliceRef); */ Is this (supposed to be) documentation of the existing code below, or rather advice how to implement the below functionality differently? > q2 = newInstruction(mb,ASSIGNsymbol); > getArg(q2,0) = newTmpVariable(mb, TYPE_any); > setModuleId(q2, algebraRef); > @@ -240,6 +261,7 @@ OPTsql_appendImplementation(Client cntxt > for(i++; i<limit; i++) > if (old[i]) > pushInstruction(mb, old[i]); > + /* any remaining MAL instruction records are removed */ > for(; i<slimit; i++) > if (old[i]) > freeInstruction(old[i]); > @@ -253,6 +275,9 @@ OPTsql_appendImplementation(Client cntxt > return actions; > } > > +/* optimizers have to be registered in the optcatalog in opt_support.c. Why? If at all possible, I'd prefer to be able to add a new optimizer without the need to change existing code ... > + * you have to path the file accordingly. ^^^^ parse? What does this mean? What am I supposed to do in detail? > + */ > @include ../../optimizer/optimizerWrapper.mx > @c > #include "opt_statistics.h" > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > Thanks! Stefan -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: [Monetdb-developers] MonetDB: jacqueline - mal: allow command calls up to 16 arguments
by Martin Kersten 28 Jan '12

28 Jan '12

at some point commands should become patterns. There is no limitation on the number of arguments per se. On 1/27/12 10:14 PM, Fabian Groffen wrote: > Changeset: a9fa1c35a273 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a9fa1c35a273 > Modified Files: > monetdb5/mal/mal_interpreter.mx > Branch: jacqueline > Log Message: > > mal: allow command calls up to 16 arguments > > future functions will need more arguments in total (in particular > because each json argument or return requires 7 separate variables) > > > diffs (76 lines): > > diff --git a/monetdb5/mal/mal_interpreter.mx b/monetdb5/mal/mal_interpreter.mx > --- a/monetdb5/mal/mal_interpreter.mx > +++ b/monetdb5/mal/mal_interpreter.mx > @@ -2053,6 +2053,72 @@ safeguardStack(Client cntxt, MalBlkPtr m > (ptr)getArgReference(stk, pci, 10), > (ptr)getArgReference(stk, pci, 11)); > break; > + case 13: ret = (str)(*pci->fcn)( > + (ptr)getArgReference(stk, pci, 0), > + (ptr)getArgReference(stk, pci, 1), > + (ptr)getArgReference(stk, pci, 2), > + (ptr)getArgReference(stk, pci, 3), > + (ptr)getArgReference(stk, pci, 4), > + (ptr)getArgReference(stk, pci, 5), > + (ptr)getArgReference(stk, pci, 6), > + (ptr)getArgReference(stk, pci, 7), > + (ptr)getArgReference(stk, pci, 8), > + (ptr)getArgReference(stk, pci, 9), > + (ptr)getArgReference(stk, pci, 10), > + (ptr)getArgReference(stk, pci, 11), > + (ptr)getArgReference(stk, pci, 12)); > + break; > + case 14: ret = (str)(*pci->fcn)( > + (ptr)getArgReference(stk, pci, 0), > + (ptr)getArgReference(stk, pci, 1), > + (ptr)getArgReference(stk, pci, 2), > + (ptr)getArgReference(stk, pci, 3), > + (ptr)getArgReference(stk, pci, 4), > + (ptr)getArgReference(stk, pci, 5), > + (ptr)getArgReference(stk, pci, 6), > + (ptr)getArgReference(stk, pci, 7), > + (ptr)getArgReference(stk, pci, 8), > + (ptr)getArgReference(stk, pci, 9), > + (ptr)getArgReference(stk, pci, 10), > + (ptr)getArgReference(stk, pci, 11), > + (ptr)getArgReference(stk, pci, 12), > + (ptr)getArgReference(stk, pci, 13)); > + break; > + case 15: ret = (str)(*pci->fcn)( > + (ptr)getArgReference(stk, pci, 0), > + (ptr)getArgReference(stk, pci, 1), > + (ptr)getArgReference(stk, pci, 2), > + (ptr)getArgReference(stk, pci, 3), > + (ptr)getArgReference(stk, pci, 4), > + (ptr)getArgReference(stk, pci, 5), > + (ptr)getArgReference(stk, pci, 6), > + (ptr)getArgReference(stk, pci, 7), > + (ptr)getArgReference(stk, pci, 8), > + (ptr)getArgReference(stk, pci, 9), > + (ptr)getArgReference(stk, pci, 10), > + (ptr)getArgReference(stk, pci, 11), > + (ptr)getArgReference(stk, pci, 12), > + (ptr)getArgReference(stk, pci, 13), > + (ptr)getArgReference(stk, pci, 14)); > + break; > + case 16: ret = (str)(*pci->fcn)( > + (ptr)getArgReference(stk, pci, 0), > + (ptr)getArgReference(stk, pci, 1), > + (ptr)getArgReference(stk, pci, 2), > + (ptr)getArgReference(stk, pci, 3), > + (ptr)getArgReference(stk, pci, 4), > + (ptr)getArgReference(stk, pci, 5), > + (ptr)getArgReference(stk, pci, 6), > + (ptr)getArgReference(stk, pci, 7), > + (ptr)getArgReference(stk, pci, 8), > + (ptr)getArgReference(stk, pci, 9), > + (ptr)getArgReference(stk, pci, 10), > + (ptr)getArgReference(stk, pci, 11), > + (ptr)getArgReference(stk, pci, 12), > + (ptr)getArgReference(stk, pci, 13), > + (ptr)getArgReference(stk, pci, 14), > + (ptr)getArgReference(stk, pci, 15)); > + break; > default: > ret = createScriptException(mb, stkpc, MAL, NULL, > "too many arguments for command call"); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

2 1

[Monetdb-developers] Announcement: New Dec2011 Feature release of MonetDB suite
by Sjoerd Mullender 26 Jan '12

26 Jan '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011 feature release SQL * mclient: The csv output format can now also be of the form csv=c and csv+c where c is the column separator. The form with csv+ produces a single header line with column names. * Changed a bug where the sign() function returned the same type as its argument instead of always an INTEGER. * Added a fix for bug #2834, which caused weird (failing) behaviour with PreparedStatements. * Fixed a bug when using default values for interval columns. Bug 2877. * A bug was fixed where deleted rows weren't properly accounted for in all operations. This was bug 2882. * A bug was fixed which caused an update to an internal table to happen too soon. The bug could be observed on a multicore system with a query INSERT INTO t (SELECT * FROM t) when the table t is "large enough". This was bug 2883. Java Module * Implemented getMetaData() method of PreparedStatement. * Fixed an AssertionError for special results from e.g. EXPLAIN queries. * Fixed overflow error when batching large statements, bug #2952 * Resolved a concurrency problem where ResultSet's date-related getters could cause odd stack traces when used by multiple threads at the same time, bug #2950. * JDBC now implements JDBCv4.1 which makes it possible to be built with Java 7 and up. JDBCv4.1 is a maintenance release of JDBCv4, and hence can be compiled with Java 6, although the added methods obviously are not part of the java.sql interfaces. * SQLExceptions thrown now carry a SQLSTATE. Until the server starts sending correct SQLSTATEs for all errors, server originated errors without SQLSTATE are considered generic data exceptions (22000). * Fixed a bug where closing ResultSets and PreparedStatements could lead to errors on concurrent running queries using the same Connection due to a race condition. * Changed version scheme of JDBC driver and MCL jar to be more standard, from monetdb-X.Y-<thing>.jar to monetdb-<thing>-X.Y.jar, bug #2943 * The embedded monet instance capability of MonetConnection was removed. * Bump JDBC version to 2.0 (Liberica). JDBC now implements JDBCv4 which makes it possible to be built with Java 6 and up. Java 5 and before are no longer supported, and can use the 1.x releases of the driver. * INTERVAL columns are now treated as decimals, since they can have sub-second precision. Client Package * Fix making connections with stethoscope to hosts without monetdbd. Bug #2944. * Fixed some bugs in the PHP interface affecting the mapi_ping(), monetdb_insert_id() and auto_commit() functions. Bugs #2936, #2937, #2938. * ODBC: Fixed SQLNumResultCols and SQLDescribeCol to return useful information after a call to SQLPrepare and before any SQLExecute. * mclient: The exact interpretation of the -i (--interactive) option and the - filename argument have changed. The - filename argument means read from standard input and no longer implies that no \ interpretation is done on the input. Instead, \ interpretation is done if either standard input is a terminal, or if the -i option is given. The -i option no longer causes mclient to read from standard input. It only means to do \ interpretation when reading from standard input. Use the - filename argument to read from standard input. Note that if no -s option is specified and no filename arguments are present, mclient still reads from standard input. * ODBC: Improved internal query for SQLSpecialColumns. Before, the query returned all columns taking part in a PRIMARY KEY *and* all columns taking part in a UNIQUE constraint. Now it returns only one or the other set. * ODBC: The database name is now used as the catalog name throughout. Functions that return a catalog name return the database name, and functions that match on catalog name match it with the database name. * ODBC: Implemented an easier way to create a log file of interactions with the ODBC driver. You can now add a connection attribute "LOGFILE=filename" to the connection string parameter of SQLBrowseConnect and SQLDriverConnect, and to the relevant part of the Windows registry or odbc.ini file. This value is only used if there is no environment variable ODBCDEBUG. * Quoting of object names for mclient's \d command is now more flexible and consistent with standard SQL quoting rules, bug #2846. * ODBC: Implemented SQLColumnPrivileges function. * Install new program, mnc, which provides netcat functionality, but based on MonetDB's communication libraries * Report full URI to database instead of just database when available in mclient. * The time format of the timer output can now be controlled with an optional argument to the -i option. ms, s and m force the time to be formatted as milliseconds, seconds or minutes + seconds respectively. * ODBC: Implemented a workaround in SQLTables for bug 2908. * Small improvement to mclient's table rendering for tables without any rows. Previously, the column names in the header could be squeezed to very small widths, degrading readability. * Python DB API connect() function now supports PEP 249-style arguments user and host, bug #2901 * mclient now checks the result of encoding conversions using the iconv library. * Fixed a source of crashes in mclient when a query on the command line using the -s option is combined with input on standard input (e.g. in the construct mclient -s 'COPY INTO t FROM STDIN ...' < file.csv). * Fixed bug 2897 where slow (network) reads could cause blocks to not be fully read in one go, causing errors in the subsequent use of those blocks. With thanks to Rémy Chibois. * Fixed a bug in mclient where processing queries from files could result in ghost empty results to be reported in the output * Fixed Perl DBD rowcount for larger results, bug #2889 * Perl: We now distinguish properly between TABLE and GLOBAL TEMPORARY (the latter are recognized by being in the "tmp" schema). * Perl: fixed a bunch of syntax errors. This fixes bug 2884. With thanks to Rémy Chibois. * Perl: Fixed DBD::monetdb table_info and tabletype_info. This fixes bug 2885. With thanks to Rémy Chibois. * mclient: fix display of varchar columns with only NULL values. * Fixed a bug in mclient/msqldump where an internal error occurred during dump when there are GLOBAL TEMPORARY tables. MonetDB5 Server * A couple of memory leaks have been plugged. * Introduced new variable max_clients that allows to define how many concurrent connections are allowed to be made against the database. * mal_interpreter.mx Protect against concurrent exceptions If multiple parallel blocks access the exception variables and perform GDKfree on old messages, then we may enter a case that one thread attempts a second free. Simple lock-based protection is the first line of defense. * bpm and partitions have been moved to the attic. It is replaced by the partition optimizer, still under construction. * The MAL debugger next/step operations semantics have been fixed. The profiler also now clearly shows entrance/exit of a MAL function. * The dataflow optimizer uses a less strict side-effect test for BAT new operations and better recognizes trivial plans. * Fixed a problem where MAL variables weren't properly cleared before reuse of the data strucutre. This problem could cause the data flow scheduler to generate dependencies between instructions that didn't actually exist, which in turn could cause circular dependencies among instructions with deadlock as a result. Bugs 2865 and 2888. Merovingian * Fixed crash in monetdb when an invalid property was retrieved using the get command, bug #2953. * monetdb get output is now grouped by database instead of by property * Unlike in previous releases, 'monetdbd get mserver' now returns the path to the mserver5 binary only for dbfarms that have a running monetdbd, instead of only for those that are not served by a monetdbd. * Add nclients property that controls the maximum number of concurrent clients allowed to access the database * Starting monetdbd without any arguments or without dbfarm is no longer supported. A dbfarm now must be provided for each command. * The control passphrase has been turned into a hash of the password, for a more stronger authorisation model as used by mclients. On upgrade, any existing passphrase is converted to the hashed version of the passphrase. * The monetdbd controlport option has been removed in favour of a boolean control option. On upgrade, when controlport was set to a non-zero value and a passphrase was set, control is set to true, or false otherwise. * The monetdbd discoveryport option has been removed in favour of a boolean discovery option. On upgrade, when discoveryport was set to a non-zero value, discovery is set to true, or false otherwise. * Fixed a bug where monetdbd's socket files from /tmp were removed when a second monetdbd was attempted to be started using the same port. * Resolved problem where monetdbd would terminate abnormally when databases named 'control', 'discovery' or 'merovingian' were stopped. * monetdbd get status now also reports the version of the running monetdbd * Improved response time of 'monetdb start' when the database fails to start. * Fixed a bug in monetdbd where starting a failing database could incorrectly be reported as a 'running but dead' database. * To avoid confusion, all occurrences of merovingian were changed into monetdbd for error messages sent to a client. MonetDB Common * Fixed memory detection on 32-bits Solaris systems with more memory than can be addressed in 32-bits mode, bug #2935 Bug Fixes

1 0

Re: [Monetdb-developers] MonetDB: default - Plug memory leak.
by Stefan Manegold 14 Jan '12

14 Jan '12

(on my 64-bit Fedora 14 desktop) this checkin seems cause several tests to fail or even crash: testing status with changeset 42821:3967ec4a8aef: monetdb5: !ERROR: Testing FAILED SIGNIFICANTLY (4 out of 494 tests failed) 1 out of 494 tests ran into timeout 3 out of 494 tests produced SIGNIFICANTLY different output sql: !ERROR: Testing FAILED SIGNIFICANTLY (4 out of 1161 tests failed) 15 out of 1161 tests could not be executed 1 out of 1161 tests produced slightly different output 3 out of 1161 tests produced SIGNIFICANTLY different output geom: !ERROR: Testing FAILED SIGNIFICANTLY (1 out of 16 tests failed) 1 out of 16 tests resulted in a crash testing status with changeset 42825:f15583fdc322 monetdb5: !ERROR: Testing FAILED SIGNIFICANTLY (21 out of 494 tests failed) 1 out of 494 tests ran into timeout 20 out of 494 tests produced SIGNIFICANTLY different output sql: !ERROR: Testing FAILED SIGNIFICANTLY (909 out of 1161 tests failed) 248 out of 1161 tests could not be executed 6 out of 1161 tests ran into timeout 900 out of 1161 tests resulted in a crash 3 out of 1161 tests produced SIGNIFICANTLY different output geom: !ERROR: Testing FAILED SIGNIFICANTLY (16 out of 16 tests failed) 16 out of 16 tests resulted in a crash I have not yet looked into the details. Stefan On Fri, Jan 13, 2012 at 06:08:12PM +0100, Sjoerd Mullender wrote: > Changeset: f15583fdc322 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=f15583fdc322 > Modified Files: > clients/Tests/exports.stable.out > gdk/gdk.mx > gdk/gdk_atoms.mx > monetdb5/mal/mal_atom.c > monetdb5/mal/mal_interpreter.mx > monetdb5/mal/mal_resolve.mx > monetdb5/modules/atoms/blob.c > monetdb5/modules/atoms/blob.mal > monetdb5/modules/mal/Tests/inspect05.stable.out > Branch: default > Log Message: > > Plug memory leak. > Variables of types other than the standard built-in types could not be > freed since the interpreter didn't know what to do. We now have a > new function pointer in BATatoms that the interpreter can use to > destroy an atom. Typically the function pointer will be NULL, but for > e.g. BLOBs it should be GDKfree. This function pointer can be set > using the new > command destroy() address XXX; > construct in MAL. > > > diffs (278 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -954,6 +954,7 @@ str BLOBblob_blob(blob **d, blob **s); > str BLOBblob_fromstr(blob **b, str *d); > void BLOBconvert(blob *b, int direction); > void BLOBdel(Heap *h, var_t *index); > +void BLOBdestroy(blob *b); > str BLOBeoln(char *src, char *end); > str BLOBfromblob(str *retval, blob **b); > str BLOBfromidx(str *retval, blob **binp, int *index); > diff --git a/gdk/gdk.mx b/gdk/gdk.mx > --- a/gdk/gdk.mx > +++ b/gdk/gdk.mx > @@ -2125,6 +2125,7 @@ typedef struct { > void (*atomConvert) (ptr v, int direction); > int (*atomFix) (ptr atom); > int (*atomUnfix) (ptr atom); > + void (*atomDestroy) (ptr atom); > > /* varsized atom-only ADT functions */ > var_t (*atomPut) (Heap *, var_t *off, ptr src); > diff --git a/gdk/gdk_atoms.mx b/gdk/gdk_atoms.mx > --- a/gdk/gdk_atoms.mx > +++ b/gdk/gdk_atoms.mx > @@ -2350,64 +2350,44 @@ atomDesc BATatoms[MAXATOMS] = { > (int (*)(ptr, ptr)) lngCmp, > (BUN (*)(ptr)) lngHash, 0, > #endif > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"bit", TYPE_chr, 1, sizeof(bit), sizeof(bit), 0, 0, (ptr) &chr_nil, > (int (*)(str, int *, ptr *)) bitFromStr, (int (*)(str *, int *, ptr)) bitToStr, > (void *(*)(void *, stream *, size_t)) bitRead, (int (*)(void *, stream *, size_t)) bitWrite, > (int (*)(ptr, ptr)) chrCmp, > (BUN (*)(ptr)) chrHash, 0, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"chr", TYPE_chr, 1, sizeof(chr), sizeof(chr), 0, 0, (ptr) &chr_nil, > (int (*)(str, int *, ptr *)) chrFromStr, (int (*)(str *, int *, ptr)) chrToStr, > (void *(*)(void *, stream *, size_t)) chrRead, (int (*)(void *, stream *, size_t)) chrWrite, > (int (*)(ptr, ptr)) chrCmp, > (BUN (*)(ptr)) chrHash, 0, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"bte", TYPE_bte, 1, sizeof(bte), sizeof(bte), 0, 0, (ptr) &bte_nil, > (int (*)(str, int *, ptr *)) bteFromStr, (int (*)(str *, int *, ptr)) bteToStr, > (void *(*)(void *, stream *, size_t)) bteRead, (int (*)(void *, stream *, size_t)) bteWrite, > (int (*)(ptr, ptr)) bteCmp, > (BUN (*)(ptr)) bteHash, 0, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"sht", TYPE_sht, 1, sizeof(sht), sizeof(sht), 0, 0, (ptr) &sht_nil, > (int (*)(str, int *, ptr *)) shtFromStr, (int (*)(str *, int *, ptr)) shtToStr, > (void *(*)(void *, stream *, size_t)) shtRead, (int (*)(void *, stream *, size_t)) shtWrite, > (int (*)(ptr, ptr)) shtCmp, > (BUN (*)(ptr)) shtHash, (void (*)(ptr, int)) shtConvert, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"BAT", TYPE_int, 1, sizeof(bat), sizeof(bat), 0, 0, (ptr) &int_nil, > (int (*)(str, int *, ptr *)) batFromStr, (int (*)(str *, int *, ptr)) batToStr, > (void *(*)(void *, stream *, size_t)) batRead, (int (*)(void *, stream *, size_t)) batWrite, > (int (*)(ptr, ptr)) intCmp, > (BUN (*)(ptr)) intHash, (void (*)(ptr, int)) intConvert, > (int (*)(ptr)) batFix, (int (*)(ptr)) batUnfix, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0}, > {"int", TYPE_int, 1, sizeof(int), sizeof(int), 0, 0, (ptr) &int_nil, > (int (*)(str, int *, ptr *)) intFromStr, (int (*)(str *, int *, ptr)) intToStr, > (void *(*)(void *, stream *, size_t)) intRead, (int (*)(void *, stream *, size_t)) intWrite, > (int (*)(ptr, ptr)) intCmp, > (BUN (*)(ptr)) intHash, (void (*)(ptr, int)) intConvert, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"oid", > #if SIZEOF_OID == SIZEOF_INT > TYPE_int, 1, sizeof(oid), sizeof(oid), 0, 0, (ptr) &int_nil, > @@ -2422,10 +2402,7 @@ atomDesc BATatoms[MAXATOMS] = { > (int (*)(ptr, ptr)) lngCmp, > (BUN (*)(ptr)) lngHash, (void (*)(ptr, int)) lngConvert, > #endif > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"wrd", > #if SIZEOF_WRD == SIZEOF_INT > TYPE_int, 1, sizeof(wrd), sizeof(wrd), 0, 0, (ptr) &int_nil, > @@ -2440,10 +2417,7 @@ atomDesc BATatoms[MAXATOMS] = { > (int (*)(ptr, ptr)) lngCmp, > (BUN (*)(ptr)) lngHash, (void (*)(ptr, int)) lngConvert, > #endif > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"ptr", > #if SIZEOF_VOID_P == SIZEOF_INT > TYPE_ptr, 1, sizeof(ptr), sizeof(ptr), 0, 0, (ptr) &ptr_nil, > @@ -2458,43 +2432,31 @@ atomDesc BATatoms[MAXATOMS] = { > (int (*)(ptr, ptr)) lngCmp, > (BUN (*)(ptr)) lngHash, (void (*)(ptr, int)) lngConvert, > #endif > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"flt", TYPE_flt, 1, sizeof(flt), sizeof(flt), 0, 0, (ptr) &flt_nil, > (int (*)(str, int *, ptr *)) fltFromStr, (int (*)(str *, int *, ptr)) fltToStr, > (void *(*)(void *, stream *, size_t)) fltRead, (int (*)(void *, stream *, size_t)) fltWrite, > (int (*)(ptr, ptr)) fltCmp, > (BUN (*)(ptr)) intHash, (void (*)(ptr, int)) intConvert, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"dbl", TYPE_dbl, 1, sizeof(dbl), sizeof(dbl), 0, 0, (ptr) &dbl_nil, > (int (*)(str, int *, ptr *)) dblFromStr, (int (*)(str *, int *, ptr)) dblToStr, > (void *(*)(void *, stream *, size_t)) dblRead, (int (*)(void *, stream *, size_t)) dblWrite, > (int (*)(ptr, ptr)) dblCmp, > (BUN (*)(ptr)) lngHash, (void (*)(ptr, int)) lngConvert, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"lng", TYPE_lng, 1, sizeof(lng), sizeof(lng), 0, 0, (ptr) &lng_nil, > (int (*)(str, int *, ptr *)) lngFromStr, (int (*)(str *, int *, ptr)) lngToStr, > (void *(*)(void *, stream *, size_t)) lngRead, (int (*)(void *, stream *, size_t)) lngWrite, > (int (*)(ptr, ptr)) lngCmp, > (BUN (*)(ptr)) lngHash, (void (*)(ptr, int)) lngConvert, > - 0, 0, > - 0, 0, > - 0, 0, > - 0, 0}, > + 0, 0, 0, 0, 0, 0, 0, 0, 0}, > {"str", TYPE_str, 1, sizeof(var_t), sizeof(var_t), 0, 1, (ptr) str_nil, > (int (*)(str, int *, ptr *)) strFromStr, (int (*)(str *, int *, ptr)) strToStr, > (void *(*)(void *, stream *, size_t)) strRead, (int (*)(void *, stream *, size_t)) strWrite, > (int (*)(ptr, ptr)) strCmp, > (BUN (*)(ptr)) strHash, 0, > - 0, 0, > + 0, 0, GDKfree, > (var_t (*)(Heap *, var_t *, ptr)) strPut, 0, > (int (*)(ptr)) strLen, strHeap, > (void (*)(Heap *, int)) 0, 0}, > diff --git a/monetdb5/mal/mal_atom.c b/monetdb5/mal/mal_atom.c > --- a/monetdb5/mal/mal_atom.c > +++ b/monetdb5/mal/mal_atom.c > @@ -118,6 +118,11 @@ int malAtomProperty(MalBlkPtr mb, InstrP > setAtomName(pci); > return 1; > } > + if (idcmp("destroy", name) == 0 && pci->argc == 1) { > + BATatoms[tpe].atomDestroy = (void (*)(void *))pci->fcn; > + setAtomName(pci); > + return 1; > + } > break; > case 'c': > if (idcmp("cmp", name) == 0 && pci->argc == 1) { > diff --git a/monetdb5/mal/mal_interpreter.mx b/monetdb5/mal/mal_interpreter.mx > --- a/monetdb5/mal/mal_interpreter.mx > +++ b/monetdb5/mal/mal_interpreter.mx > @@ -2442,9 +2442,7 @@ void garbageElement(Client cntxt, ValPtr > v->val.sval = NULL; > } > v->len = 0; > - return; > - } > - if (v->vtype == TYPE_bat) { > + } else if (v->vtype == TYPE_bat) { > /* > * @- > * All operations are responsible to properly set the > @@ -2466,6 +2464,8 @@ void garbageElement(Client cntxt, ValPtr > if (cntxt && cntxt->flags & bigfootFlag) > updateBigFoot(cntxt, bid, FALSE); > BBPdecref(bid, TRUE); > + } else if (BATatoms[v->vtype].atomDestroy) { > + (*BATatoms[v->vtype].atomDestroy)(v->val.pval); > } > } > /* > diff --git a/monetdb5/mal/mal_resolve.mx b/monetdb5/mal/mal_resolve.mx > --- a/monetdb5/mal/mal_resolve.mx > +++ b/monetdb5/mal/mal_resolve.mx > @@ -529,7 +529,8 @@ findFunctionType(Module scope, MalBlkPtr > isaBatType(@1) || > findGDKtype(@1) == TYPE_str || > (!isPolyType(@1) && @1 < TYPE_any && > - @1 >= 0 && ATOMstorage(@1) == TYPE_str)) { > + @1 >= 0 && (ATOMstorage(@1) == TYPE_str || > + BATatoms[(a)1].atomDestroy))) { > getInstrPtr(@3, 0)->gc |= GARBAGECONTROL; > setVarCleanup(mb, getArg(p, @2)); > p->gc |= GARBAGECONTROL; > @@ -783,7 +784,9 @@ void typeChecker(Module scope, MalBlkPtr > int tpe = getArgType(mb,p,k); > if (findGDKtype(tpe) == TYPE_bat || > findGDKtype(tpe) == TYPE_str || > - (!isPolyType(tpe) && tpe< TYPE_any && ATOMstorage(tpe) == TYPE_str)) > + (!isPolyType(tpe) && tpe< TYPE_any && > + (ATOMstorage(tpe) == TYPE_str || > + BATatoms[tpe].atomDestroy))) > setVarCleanup(mb,getArg(p,k)); > } > } > diff --git a/monetdb5/modules/atoms/blob.c b/monetdb5/modules/atoms/blob.c > --- a/monetdb5/modules/atoms/blob.c > +++ b/monetdb5/modules/atoms/blob.c > @@ -68,6 +68,7 @@ blob_export int BLOBnitems(int *ret, blo > blob_export int BLOBget(Heap *h, int *bun, int *l, blob **val); > blob_export blob * BLOBread(blob *a, stream *s, size_t cnt); > blob_export int BLOBwrite(blob *a, stream *s, size_t cnt); > +blob_export void BLOBdestroy(blob *b); > > blob_export str BLOBblob_blob(blob **d, blob **s); > blob_export str BLOBblob_fromstr(blob **b, str *d); > @@ -741,6 +742,12 @@ BLOBheap(Heap *heap, size_t capacity) > blob_heap(heap, capacity); > } > > +void > +BLOBdestroy(blob *b) > +{ > + GDKfree(b); > +} > + > var_t > BLOBput(Heap *h, var_t *bun, blob *val) > { > diff --git a/monetdb5/modules/atoms/blob.mal b/monetdb5/modules/atoms/blob.mal > --- a/monetdb5/modules/atoms/blob.mal > +++ b/monetdb5/modules/atoms/blob.mal > @@ -31,6 +31,7 @@ command put() address BLOBput; > command del() address BLOBdel; > command length() address BLOBlength; > command heap() address BLOBheap; > +command destroy() address BLOBdestroy; > > command blob(s:blob):blob > address BLOBblob_blob > diff --git a/monetdb5/modules/mal/Tests/inspect05.stable.out b/monetdb5/modules/mal/Tests/inspect05.stable.out > --- a/monetdb5/modules/mal/Tests/inspect05.stable.out > +++ b/monetdb5/modules/mal/Tests/inspect05.stable.out > @@ -3399,6 +3399,7 @@ end main; > [ "blob", "command", "blob", "(s:str):blob ", "BLOBblob_fromstr;" ] > [ "blob", "command", "blob", "(s:blob):blob ", "BLOBblob_blob;" ] > [ "#convert", "command", "blob", "():void ", "BLOBconvert;" ] > +[ "#destroy", "command", "blob", "():void ", "BLOBdestroy;" ] > [ "#del", "command", "blob", "():void ", "BLOBdel;" ] > [ "#fromstr", "command", "blob", "():void ", "BLOBfromstr;" ] > [ "#heap", "command", "blob", "():void ", "BLOBheap;" ] > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

Re: [Monetdb-developers] MonetDB: sciql - Add two new functions to create an sql_exp* for...
by Fabian Groffen 12 Jan '12

12 Jan '12

On 11-01-2012 23:22:23 +0100, Jennie Zhang wrote: > Changeset: d42756602492 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=d42756602492 > Modified Files: > sql/server/rel_exp.c > sql/server/rel_exp.h > Branch: sciql > Log Message: > > Add two new functions to create an sql_exp* for flt and dbl type atoms. > > This change should be propagated to the default branch, but how? Not propagated, but backported. You can use hg transplant --log d42756602492 on the default branch.

1 0

[Monetdb-developers] Testing code
by Charalampos Nikolaou 10 Jan '12

10 Jan '12

Hi, I have written some code and now I would like to make changes and still be sure that the new changes are correct. So, I have written some scenarios in mal and put it in a Tests directory. I have read the README file under testing/ directory, but still haven't understood how I could generate the <scenario>.stable.out file. How could that file be generated? Even if I run Mtest.py <mytest>, then Mapprove.py the previously "wrong" output, and then again Mtest.py the test fails. Thanks, Babis

3 3