February 2012 - developers-list

[Monetdb-developers] New Apr2012 stable branch created
by Sjoerd Mullender 29 Feb '12

29 Feb '12

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Apr2012. So in each of your stable checkout directories you can do hg pull hg update -rApr2012 to update to this new branch. -- Sjoerd Mullender

1 0

[Monetdb-developers] Announcement: New Dec2011-SP1 Bugfix release of MonetDB suite
by Fabian Groffen 29 Feb '12

29 Feb '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011-SP1 bugfix release Build Environment * Fixed some of the package names for Debian/Ubuntu. Packages for libraries should contain the major number of the library version number. This was not always the case. * On Debian and Ubuntu, install Python modules in dist-packages instead of site-packages. This fixed bug 2997. SQL * Fixed a crash that happened at the end of a database upgrade to the Dec2011 database scheme. The crash happened during cleanup after the database was upgraded, so it was merely inconvenient. * Stripped off implementation-specific parts from error messages before they get presented to the user. Java Module * Fixed bug in DatabaseMetaData.getSchemas() method that caused an SQL error when called with catalog and schema argument. * Resolved a bug where JDBC and Control connections could terminate abruptly with 'Connection closed' messages MonetDB5 Server * Fixed potential crash by dealing better with non-standard types. Merovingian * Fixed a bug in the multiplex-funnel where certain clients would abort on responses for update queries. * Fixed problem where version and mserver properties for monetdbd were not always successfully retrieved. Bug #2982. * Fixed problem where shutdown of monetdbd would lead to shutting down database 'control' which does not exist. Bug #2983. * Fixed issue causing (harmless) 'error reading from control channel' messages. Bug #2984. * Resolved problem where remote start/stop/etc. commands with monetdb would report error 'OK'. Bug #2984. Bug Fixes * 2862: foreign key self references cause violation upon alter table add constraint... * 2933: "TypeException: algebra.join undefined" when evaluating EXISTS (SELECT ...) predicate * 2946: Invalid BAT if left join condition has only columns from right table * 2964: LEFT OUTER JOIN with constants returns errors * 2969: Result precision issue with Decimal data type * 2973: Date coercion functionality tests * 2979: misc merovingian control problems * 2982: monetdbd status and mserver properties have random values * 2983: monetdbd attempts to stop non-existing database 'control' * 2984: monetdb create/release/start/stop returns error "OK" for remote connections * 2995: Test geom/sql/Tests/example.sql crashes since changeset 6399077ff8a4 * 2997: monetdb-python packages installed in wrong dir on Ubuntu 10.04 LTS (lucid) * 3004: Incorrect evaluation of conditions in WHERE part of SQL statement * 3009: Segmentation fault on simple query * 3013: calc.and is not yet defined for dbl * 3014: dayofweek comment is wrong in mtime.mx:700 * 3017: rel_bin.c:611: exp_bin: Assertion `0' failed. with between on multicolumns * 3020: Assertion error in debugging mode * 3034: INNER JOIN with Subquery produces mal errors * 3041: WHERE condition ignored after 2 LEFT JOINs with subqueries

1 0

Re: [Monetdb-developers] MonetDB: default - make sure we reset the point_query flag
by Sjoerd Mullender 23 Feb '12

23 Feb '12

Would this also be appropriate for the Dec2011 branch? On 2012-02-23 14:22, Niels Nes wrote: > Changeset: 656d66d54802 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=656d66d54802 > Modified Files: > sql/backends/monet5/sql.mx > Branch: default > Log Message: > > make sure we reset the point_query flag > > > diffs (11 lines): > > diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx > --- a/sql/backends/monet5/sql.mx > +++ b/sql/backends/monet5/sql.mx > @@ -1694,6 +1694,7 @@ sqlcleanup(mvc *c, int err) > if (err < 0) > c->session->status = err; > c->label = 0; > + c->point_query = 0; > scanner_query_processed(&(c->scanner)); > return err; > } > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

2 2

Re: [Monetdb-developers] MonetDB: sciql - rel_select.c, rel_arraytiling(): we do not (yet...
by Ying Zhang 23 Feb '12

23 Feb '12

On Thu, Feb 23, 2012 at 11:06:55AM +0100, Stefan Manegold wrote: > Changeset: 869144f7d945 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=869144f7d945 > Modified Files: > sql/server/rel_select.c > Branch: sciql > Log Message: > > rel_select.c, rel_arraytiling(): we do not (yet?) support tiling on subset of dimensions No, it's not yet supported. > > > diffs (12 lines): > > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -1910,6 +1910,8 @@ rel_arraytiling(mvc *sql, sql_rel **rel, > idx_exps = tile_def->data.lval->h->next->data.lval; > if (dlist_length(idx_exps) > a->ndims) > return sql_error(sql, 02, "SELECT: #dimensions (%d) in array tiling larger than #dimensions (%d) in the array", dlist_length(idx_exps), a->ndims); > + if (dlist_length(idx_exps) < a->ndims) > + return sql_error(sql, 02, "SELECT: #dimensions (%d) in array tiling fewer than #dimensions (%d) in the array --- not supported (yet?)", dlist_length(idx_exps), a->ndims); > if (dlist_length(idx_exps) > ARRAY_TILING_MAX_DIMS) > return sql_error(sql, 02, "SELECT: TODO: array tiling over >%d dimensions", ARRAY_TILING_MAX_DIMS); > for (n = idx_exps->h; n; n = n->next) { > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

1 0

Re: [Monetdb-developers] MonetDB: default - test opt_sql_append: trying to upgrade to new...
by Martin Kersten 18 Feb '12

18 Feb '12

The new optimizer infrastructure returns the name of the pipe, not its content. The definitions can be obtained from the system view: select * from optimizers(); can be used to get the expanded form. On 2/18/12 6:11 PM, Stefan Manegold wrote: > Changeset: 62a52aaf4241 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=62a52aaf4241 > Modified Files: > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.sql > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > Branch: default > Log Message: > > test opt_sql_append: trying to upgrade to new optimizer framework > > However, setting SQL variable "optimizer" to a MAL snippert > rather than an optimizer pipeline name does not seem to work (yet?) ... > > > diffs (39 lines): > > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.sql b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.sql > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.sql > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.sql > @@ -1,7 +1,9 @@ > create table ttt (a int, b int, c int); > select optimizer; > explain copy into ttt from '/tmp/xyz'; > -set optimizer = substring(optimizer,0,length(optimizer)-length('garbageCollector')) || 'sql_append,garbageCollector'; > +declare opt_pipe_name string; set opt_pipe_name = ( select optimizer ); > +declare opt_pipe_def string; set opt_pipe_def = ( select def from optimizers() where name = opt_pipe_name ); > +set optimizer = substring(opt_pipe_def,0,length(opt_pipe_def)-length('optimizer.garbageCollector();')) || 'optimizer.sql_append();optimizer.garbageCollector();'; > select optimizer; > explain copy into ttt from '/tmp/xyz'; > drop table ttt; > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > @@ -49,8 +49,8 @@ Ready. > % . # table_name > % single_value # name > % varchar # type > -% 188 # length > -[ "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,accumulators,garbageCollector" ] > +% 12 # length > +[ "default_pipe" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > @@ -69,8 +69,8 @@ end s2_1; > % . # table_name > % single_value # name > % varchar # type > -% 199 # length > -[ "inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,accumulators,sql_append,garbageCollector" ] > +% 452 # length > +[ "optimizer.inline();optimizer.remap();optimizer.evaluate();optimizer.costModel();optimizer.coercions();optimizer.emptySet();optimizer.aliases();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.dataflow();optimizer.history();optimizer.multiplex();optimizer.accumulators();optimizer.sql_append();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

2 1

[Monetdb-developers] How to embed MonetDB
by Florin Ciodaru 17 Feb '12

17 Feb '12

Hi all, First I want to congratulate all of you for this work. Now, we are trying to build a "real-time" like application that on one side periodically receives sets of matrices and on the other side receives query to apply on this matrices - Each set of matrices can have thousands of elements o each of them belongs to a certain class, hence groups of them will need to be inserted in the same monetdb table o each matrix has ~20 columns and the number of rows can grow to 128k - The queries can be o One-time o or , continuous (to be applied on each new set of matrices) Looking for stream processing engine, we reached DataCell and MonetDB. We downloaded the Dec2011 source code from Mercurial depot. Unfortunately, we could not yet figure out how to bring DataCell up and running, although we successfully built it. Maybe you can gives us more pointers than what already exists on the site. In parallel we are trying to experiment with MonetDB functionality as a library to be embedded in our own host (e.g. windows application). The basic functions we want from this library are - Fast low level API's to create tables and populate them with values - High level API's to execute SQL queries For the insertions we figure out the API's in sql_storage.h are close to what we want. However we could not yet find API's that allow us to execute SQL queries (we did find SQLstatementIntern but cannot yet figure out how to retrieve the results of query execution). Your help will be really appreciated, Florin

1 0

Re: [Monetdb-developers] MonetDB: gdk-calc - gdk_calc.c, mod_typeswitchloop(): initialize...
by Stefan Manegold 16 Feb '12

16 Feb '12

Standard gcc on Fedora 16; MonetDB configured with --disable-debug --enable-optimize --disable-assert -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | ----- Reply message ----- From: "Sjoerd Mullender" <sjoerd(a)acm.org> To: <monetdb-developers(a)lists.sourceforge.net> Subject: [Monetdb-developers] MonetDB: gdk-calc - gdk_calc.c, mod_typeswitchloop(): initialize... Date: Thu, Feb 16, 2012 09:18 On 2012-02-15 20:12, Stefan Manegold wrote: > Changeset: fc0d3fad41c2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc0d3fad41c2 > Modified Files: > gdk/gdk_calc.c > Branch: gdk-calc > Log Message: > > gdk_calc.c, mod_typeswitchloop(): initialize "nils" (hopefully correctly) Which sucky compiler was this? There is no code path where nils was not initialized. (This also means that the value you choose here doesn't matter.) > > diffs (12 lines): > > diff --git a/gdk/gdk_calc.c b/gdk/gdk_calc.c > --- a/gdk/gdk_calc.c > +++ b/gdk/gdk_calc.c > @@ -6157,7 +6157,7 @@ mod_typeswitchloop(const void *lft, int > void *dst, int tp, BUN cnt, > int abort_on_error, const char *func) > { > - BUN nils; > + BUN nils = BUN_NONE; > > switch (ATOMstorage(tp1)) { > case TYPE_bte: > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] MonetDB: gdk-calc - gdk_calc.c, mod_typeswitchloop(): initialize...
by Sjoerd Mullender 16 Feb '12

16 Feb '12

On 2012-02-15 20:12, Stefan Manegold wrote: > Changeset: fc0d3fad41c2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc0d3fad41c2 > Modified Files: > gdk/gdk_calc.c > Branch: gdk-calc > Log Message: > > gdk_calc.c, mod_typeswitchloop(): initialize "nils" (hopefully correctly) Which sucky compiler was this? There is no code path where nils was not initialized. (This also means that the value you choose here doesn't matter.) > > diffs (12 lines): > > diff --git a/gdk/gdk_calc.c b/gdk/gdk_calc.c > --- a/gdk/gdk_calc.c > +++ b/gdk/gdk_calc.c > @@ -6157,7 +6157,7 @@ mod_typeswitchloop(const void *lft, int > void *dst, int tp, BUN cnt, > int abort_on_error, const char *func) > { > - BUN nils; > + BUN nils = BUN_NONE; > > switch (ATOMstorage(tp1)) { > case TYPE_bte: > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

1 1

Re: [Monetdb-developers] MonetDB: default - More advice on the optimizer template.
by Stefan Manegold 11 Feb '12

11 Feb '12

On Sat, Feb 11, 2012 at 02:06:17PM +0100, Martin Kersten wrote: > > > On 2/11/12 11:03 AM, Stefan Manegold wrote: > >On Wed, Feb 08, 2012 at 10:27:11AM +0100, Martin Kersten wrote: > >>Changeset: 67c12a700166 for MonetDB > >>URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=67c12a700166 > >>Modified Files: > >> monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >>Branch: default > >>Log Message: > >> > >>More advice on the optimizer template. > >> > >> > >>diffs (140 lines): > >> > >>diff --git a/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx b/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >>--- a/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >>+++ b/monetdb5/extras/mal_optimizer_template/opt_sql_append.mx > >[...] > >>@@ -39,6 +39,8 @@ All Rights Reserved. > >> * i.e., an sql.append() statement that is eventually followed by some other > >> * statement later on in the MAL program that uses the same v0 BAT as > >> * argument as the sql.append() statement does, > >>+ * Do you assume a single re-use of the variable v0? > > > >No. Why? > Use assign-once and use-many-times policy. It can improve parallel > processing > and simplifies scope analysis. v0 is (as far as I know) created (assigned) once (by Niels, or preceeding optimizers). If it is used only once (only by sql_append), my optimizer does not (have to) do anything. Otherwise, it replaces one use v0 (by sql_append) by a view of v0. That's the very purpose of this optimizer. > >>+ * Do you assume a non-nested MAL block ? > > > >Not necessarily. > > > Analysis may become complex if you have something like > > V0:= expr > barrier E1:=expr > V0:= expr2 > exit E1 > now V0 depends on runtime use > > > same holds for > barrier E1:= expr > V0:=expr > exit E1 > z:= f(V0) > > will be flagged as an error because V0 may be uninitialized > > >I must admit, that I do not know how the oprimizer framework handles nested > >MAL blocks, and what an optimizer needs to do to be aware of nested MAL > >blocks and to handle them correctly. > Preferrably the MAL blocks are linear programs (until you reach the > dataflow optimizer). How do I know / see that in my optimizer? Do I have to check for barrier / exit statements / constructs myself? > > > >In the sample optimizer, for now, I'd be fine if there are no > >false-positives, i.e., the optimizer triggers in case it should not trigger > >or in cases it cannot handle correctly. > >I can accept false-negatives, i.e., not triggering in all case it could handle > >correctly. > > > >> * > >> * and transform them into > >> * > >>@@ -52,6 +54,7 @@ All Rights Reserved. > >> * > >> * i.e., handing a BAT view v2 of BAT v0 as argument to the sql.append() > >> * statement, rather than the original BAT v0. > >>+ * My advice, always use new variable names, it may capture some easy to make errors. > > > >I/my optimizer does use new variables for all new statements/results. > >I/my optimizer re-use variable names only for identical results. > > > >> * > >> * As a refinement, patterns like > >> * > >[...] > >>@@ -181,13 +195,17 @@ OPTsql_appendImplementation(Client cntxt > >> pushInstruction(mb, q); > >> q1 = q; > >> i++; > >>- actions++; > >>+ actions++; /* to keep track if anything has been done */ > >> } > >> } > >> > >>- /* look for > >>+ /* look for > >> * v5 := ... v0 ...; > >> */ > >>+ /* an expensive loop, better would be to remember that v0 has a different role. > >>+ * A typical method is to keep a map from variable -> instruction where it was > >>+ * detected. The you can check each assignment for use of v0 > >>+ */ > > > >This is general support functionality. > >Is this already available in the optimizer framework? > I try to use single pass algorithms in the optimizers. > Even in the case of commonterms optimizer, we may have to > traverse the history. This can become a n^2 process > > >If so, where is it and how can I use it? > Mimic how it is done in other optimizers (e.g. opt_reorder). > Typically, a buffer is maintained per variable to keep > optimization properties around. > > >If not, where/how could we add it? > > > >> for (j = i+1; !found&& j< limit; j++) > >> for (k = old[j]->retc; !found&& k< old[j]->argc; k++) > >> found = (getArg(old[j], k) == getArg(p, 5)); > >>@@ -202,6 +220,8 @@ OPTsql_appendImplementation(Client cntxt > >> > >> /* push new v1 := aggr.count( v0 ); unless already available */ > >> if (q1 == NULL) { > >>+ /* use mal_buil.mx primitives q1 = newStmt(mb, aggrRef,countRef); setArgType(mb,q1,TYPE_wrd) */ > >>+ /* it will be added to the block and even my re-use MAL instructions */ > > > >Is this (supposed to be) documentation of the existing code below, > >or rather advice how to implement the below functionality differently? > Use the mal_builder to simplify your code base. > > > > >> q1 = newInstruction(mb,ASSIGNsymbol); > >> getArg(q1,0) = newTmpVariable(mb, TYPE_wrd); > >> setModuleId(q1, aggrRef); > >>@@ -211,6 +231,7 @@ OPTsql_appendImplementation(Client cntxt > >> } > >> > >> /* push new v2 := algebra.slice( v0, 0, v1 ); */ > >>+ /* use mal_buil.mx primitives q1 = newStmt(mb, algebraRef,sliceRef); */ > > > >Is this (supposed to be) documentation of the existing code below, > >or rather advice how to implement the below functionality differently? > > > >> q2 = newInstruction(mb,ASSIGNsymbol); > >> getArg(q2,0) = newTmpVariable(mb, TYPE_any); > >> setModuleId(q2, algebraRef); > >>@@ -240,6 +261,7 @@ OPTsql_appendImplementation(Client cntxt > >> for(i++; i<limit; i++) > >> if (old[i]) > >> pushInstruction(mb, old[i]); > >>+ /* any remaining MAL instruction records are removed */ > >> for(; i<slimit; i++) > >> if (old[i]) > >> freeInstruction(old[i]); > >>@@ -253,6 +275,9 @@ OPTsql_appendImplementation(Client cntxt > >> return actions; > >> } > >> > >>+/* optimizers have to be registered in the optcatalog in opt_support.c. > > > >Why? > SQL needs a place to pick up all optimizers known. You may also have > to extend the optimizer pipeline validity code. > > >If at all possible, I'd prefer to be able to add a new optimizer without the > >need to change existing code ... > yes understood, but you have to patch Makefile.ag, youroptimizer.mx, and > opt_support. Possibly, you may have to extend opt_prelude as well > > > > >>+ * you have to path the file accordingly. > "path" > > ^^^^ > >parse? > > > >What does this mean? What am I supposed to do in detail? > > > >>+ */ > >> @include ../../optimizer/optimizerWrapper.mx > >> @c > >> #include "opt_statistics.h" > >>_______________________________________________ > >>Checkin-list mailing list > >>Checkin-list(a)monetdb.org > >>http://mail.monetdb.org/mailman/listinfo/checkin-list > >> > > > >Thanks! > > > >Stefan > > > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > > -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

Re: [Monetdb-developers] MonetDB: default - More advice on the optimizer template.
by Martin Kersten 11 Feb '12

11 Feb '12

>>> + * you have to path the file accordingly. > "path" >> ^^^^ >> parse? patch

1 0