May 2010 - developers-list

[Monetdb-developers] Basics about MonetDB
by 叶亮 01 Jun '11

01 Jun '11

Hi, I'm a new comer with some questions about basic conceptions of MonetDB. 1. The executor I've ever found the executor of postgreSQL at src\backend\executor, but cannot make sure where it is like that in MonetDB src. MonetDB\MonetDB\src\gdk or MonetDB5-server\MonetDB5\src\mal ? 3. MAL MAL (MonetDB Assembly Language) is a script tool only for MonetDB or a common script language that can be applied to other softwares. Why it is a must in MonetDB ? 4. .mx files In MonetDB, .mx files are compiled to make .h and .c files generated.(in this way ?) Why is this routines ? (not just coding in .h and .c files directly) Yel ---- Best Regards richardroky(a)gmail.com

8 10

[Monetdb-developers] encoding of strings returned by Python client
by Mark Bucciarelli 28 Jun '10

28 Jun '10

Any chance the Python client could be modified to return strings in Unicode instead of UTF-8? Seems like a better fit for Python; for example, that's how Python's sqlite3 module behaves. An alternative approach is to expose a set_encoding method on the connection that let's you specify the encoding of all text fields returned. If backward compatibility is an issue, the latter idea seems better even though it adds a knob. Any comments? (Is the maintainer of the Python adapter active even on this list? I didn't see many Python-specific posts in the archives.) Thanks, m

3 12

[Monetdb-developers] mx tool status and future
by Alex Dubov 01 Jun '10

01 Jun '10

Greetings. I understand, that this issue was probably discussed before, but I couldn't quite find a way to search the list archive. It appears to me, that mx version 3, found on savannah.nongnu.org, is somewhat out of sync with the one used for monetdb (@mal and @include tags are not supported by it; adding the former was trivial, while I've set the later to no-op, as included *.mx files are nowhere to be found in the monetdb source tree). The mx doc formatter itself is also quite underdeveloped, compared to what we came to expect from systems like doxygen or sphinx. In light of the above, I wonder, are there any plans to switch to a different code documentation system (doxygen?) or is the mx tool intended for future development?

4 3

Re: [Monetdb-developers] MonetDB: default - Remove the C flag -Wno-uninitialized for Mac ...
by Stefan Manegold 20 May '10

20 May '10

Good point. Thanks! I don't expect any problems with this change, mostly because we don't have any MacOS X machine for testing, anymore; only one for buildbot --- once that's running again,we'll see ... Assuming that compilation works for you on a "modern" MacOS X (Snow Leopard), I assume that the mentioned bison.simple problems were related/specific to an older version of MacOS X, only. (MacOS X was never a primary develoment platform for us, that why disabling -Wuninitialized (by enabling -Wno-uninitialized) on MacOS X used to be - "acceptable" for us; obviously, with you using MacOS X as a primary development platform, -Wno-uninitialized should indeed not be used - (gloablly) ...) Stefan On Thu, May 20, 2010 at 06:20:02PM +0200, Jan Rittinger wrote: > Changeset: 892fc9b323aa for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=892fc9b323aa > Modified Files: > buildtools/conf/MonetDB.m4 > Branch: default > Log Message: > > Remove the C flag -Wno-uninitialized for Mac operating systems. > This helps to detect uninitialized variables on OS X. > > (Please undo the fix in case it breaks anything.) > > > diffs (12 lines): > > diff -r b181da69e3ba -r 892fc9b323aa buildtools/conf/MonetDB.m4 > --- a/buildtools/conf/MonetDB.m4 Thu May 20 17:50:15 2010 +0200 > +++ b/buildtools/conf/MonetDB.m4 Thu May 20 18:18:33 2010 +0200 > @@ -811,7 +811,7 @@ > dnl generates "X_CFLAGS =" in the generated Makefile. > NO_X_CFLAGS='X_CFLAGS' > ;; > - *-solaris*|*-darwin*|*-aix*) > + *-solaris*|*-aix*) > dnl In some cases, there is a (possibly) uninitialized > dnl variable in bison.simple ... |-( > X_CFLAGS="$X_CFLAGS -Wno-uninitialized" > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

[Monetdb-developers] MonetDB/SQL: bulk-loading large data sets & adding key constraints
by Stefan Manegold 15 May '10

15 May '10

Dear all, derived from recent experiments and intensive testing, here are some recommendations for bulk-loading large data sets (in particular those larger than your available main memory) in MonetDB/SQL and adding constraints, i.e., primary and foreign keys. First of all, bulk loading should be done from (possibly gzip or bzip2 compressed) CSV files using the SQL `COPY INTO` command; cf., http://monetdb.cwi.nl/SQL/Documentation/Data-Manipulation.html In case your tables come with primary or foreign key constraints, it is recommended to omit these constrains when initially creating the tables. Then bulk-load (`COPY INTO`) your data. Once the data is loaded add the key constraints one by one using the SQL `ALTER TABLE ... ADD CONSTRAINT ...` command. Note, though, that the current Feb2010 release of MonetDB/SQL might experience some performance problems when adding constraints on data sets larger than main memory on multi-core machines. The reason it a non-optimal query translation and plan parallelization. For the time being, i.e., until we manage to fix this problem properly, it is recommended to resort to the optimizer pipeline of the Nov2009 release (for insiders, this effectively means disabling the "mitosis" optimizer), (only) while adding the key constraints. To do so, simply activate the Nov2009 optimizer pipeline via SET optimizer='nov2009_pipe'; just before adding the key constraints via `ALTER TABLE ... ADD CONSTRAINTS` and then restore the default optimizer pipeline via SET optimizer='default_pipe'; once you're done with adding constraints. For TPCH scale factor 100 (100 GB) on a 8 GB 4-core machine, this reduces the total time to add all key constraints from ~1.5h to ~1h. For SSB scale factor 100 (100 GB) on a 8 GB 4-core machine, this reduces the total time to add all key constraints from ~4h to ~1h. Hope this helps. Please don't hesitate to comment or ask questions. We'll let you know, one we managed to fix this problem properly. Kind regards, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

Re: [Monetdb-developers] MonetDB: Jun2010 - Fix compilation on Windows
by Stefan Manegold 13 May '10

13 May '10

On Thu, May 13, 2010 at 10:02:14AM +0200, Fabian Groffen wrote: > Changeset: db9f1c910dbb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=db9f1c910dbb > Modified Files: > MonetDB5/src/mal/mal_linker.mx > Branch: Jun2010 > Log Message: > > Fix compilation on Windows > > .\..\..\..\src\mal\mal_linker.mx(62): catastrophic error: could not open > source file "dirent.h" > > Seems Sabaoth can use struct dirent without including dirent.h as well, > so I guess some gdk header takes care of this. I just removed dirent.h > include. > FYI: MonetDB/src/gdk/gdk.mx-#ifdef HAVE_DIRENT_H MonetDB/src/gdk/gdk.mx:# include <dirent.h> MonetDB/src/gdk/gdk.mx:# define NAMLEN(dirent) strlen((dirent)->d_name) MonetDB/src/gdk/gdk.mx-#else MonetDB/src/gdk/gdk.mx:# define dirent direct MonetDB/src/gdk/gdk.mx:# define NAMLEN(dirent) (dirent)->d_namlen MonetDB/src/gdk/gdk.mx-# ifdef HAVE_SYS_NDIR_H MonetDB/src/gdk/gdk.mx-# include <sys/ndir.h> MonetDB/src/gdk/gdk.mx-# endif MonetDB/src/gdk/gdk.mx-# ifdef HAVE_SYS_DIR_H MonetDB/src/gdk/gdk.mx-# include <sys/dir.h> MonetDB/src/gdk/gdk.mx-# endif MonetDB/src/gdk/gdk.mx-# ifdef HAVE_NDIR_H MonetDB/src/gdk/gdk.mx-# include <ndir.h> MonetDB/src/gdk/gdk.mx-# endif MonetDB/src/gdk/gdk.mx-#endif Stefan > > diffs (11 lines): > > diff -r e1c42586a988 -r db9f1c910dbb MonetDB5/src/mal/mal_linker.mx > --- a/MonetDB5/src/mal/mal_linker.mx Wed May 12 16:58:04 2010 +0200 > +++ b/MonetDB5/src/mal/mal_linker.mx Thu May 13 10:02:00 2010 +0200 > @@ -59,7 +59,6 @@ > #endif > > #include <sys/types.h> /* opendir */ > -#include <dirent.h> /* opendir */ > > /* #define DEBUG_MAL_LINKER */ > #define MONET64 1 > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

Re: [Monetdb-developers] MonetDB: Jun2010 - include sql is no longer in the arguments
by Stefan Manegold 10 May '10

10 May '10

Oops ;-) Sorry, missed that ... Thanks for watching & fixing! Stefan On Mon, May 10, 2010 at 08:16:01AM +0200, Fabian Groffen wrote: > Changeset: 2716b35c7688 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2716b35c7688 > Modified Files: > sql/src/backends/monet5/merovingian/merovingian_forkmserver.c > Branch: Jun2010 > Log Message: > > include sql is no longer in the arguments > > > diffs (12 lines): > > diff -r a901acf756bb -r 2716b35c7688 sql/src/backends/monet5/merovingian/merovingian_forkmserver.c > --- a/sql/src/backends/monet5/merovingian/merovingian_forkmserver.c Mon May 10 08:06:52 2010 +0200 > +++ b/sql/src/backends/monet5/merovingian/merovingian_forkmserver.c Mon May 10 08:15:50 2010 +0200 > @@ -226,7 +226,7 @@ > str master = NULL; > str slave = NULL; > str pipeline = NULL; > - str argv[27]; /* for the exec arguments */ > + str argv[26]; /* for the exec arguments */ > confkeyval *ckv, *kv; > int c = 0; > > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

[Monetdb-developers] About the use of hg transplant
by Fabian Groffen 09 May '10

09 May '10

For those using hg transplant to backport changes. Using the -m flag of transplant causes a merge commit to occur. This means two heads (could be branches) are merged into one. For some not yet fully understood reason this causes a hg merge to perform a large batch of changes, probably recursively performing the full changes of the branch that was transplanted from. I think for backports it is better not to use -m, because it artificially says you moved over everything that was supposed to be moved over. However, in fact you didn't. Just don't make a merge commit. I found out that this is easy to deal with in the propagation, since transplant can be used in the following fashion: % cvs transplant --log -b Jun2010 -p 456afbaaf416 -p 0ee83fdb0e97 -m 63f8c7be221a applying 63f8c7be221a 35864:63f8c7be221a merged at cfbfaf940bb2 % Which in this case means: ignore changesets 456afbaaf416 and 0ee83fdb0e97, and at changeset 63f8c7be221a pretend a merge was done, which happens to be correct. (Actually, without the -p and -m opts, transplant will ask per changeset what to do with it, so it seems better suited to do regulated propagation merges.)

2 2

Re: [Monetdb-developers] MonetDB: default - Revert the changes of May 04 to recover perfo...
by Stefan Manegold 09 May '10

09 May '10

In case the changes of May 04 indeed caused the reported performance degradation, I assume this checkin should then also be back-ported to the Jun2010 branch that was created on May 06 and hence also contains the problematic changes of May 04, right? Once confirmed and I can back-port this checkin. Stefan On Sat, May 08, 2010 at 08:30:35PM +0200, Martin.Kersten(a)cwi.nl wrote: > Changeset: e9c1206c5008 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e9c1206c5008 > Modified Files: > MonetDB5/src/mal/mal_interpreter.mx > clients/src/python/test/capabilities.py > clients/src/python/test/dbapi20.py > geom/src/sql/geom.sql > Branch: default > Log Message: > > Revert the changes of May 04 to recover performance. > > > diffs (76 lines): > > diff -r dd777c76a3f9 -r e9c1206c5008 MonetDB5/src/mal/mal_interpreter.mx > --- a/MonetDB5/src/mal/mal_interpreter.mx Sat May 08 09:16:12 2010 +0200 > +++ b/MonetDB5/src/mal/mal_interpreter.mx Sat May 08 20:30:23 2010 +0200 > @@ -341,7 +341,7 @@ > > /* pass the new debug mode to the caller */ > if (stk->cmd && env && stk->cmd!='f') env->cmd = stk->cmd; > - if ( garbageControl(getInstrPtr(mb,0)) ) > + if ( !stk->keepAlive && garbageControl(getInstrPtr(mb,0)) ) > garbageCollector(cntxt, mb,stk, env != stk); > @:endProfile(stk)@ > return ret; > @@ -361,15 +361,19 @@ > MalStkPtr stk, MalStkPtr env, InstrPtr pcicaller) > { > str ret; > + int keepAlive; > > if (stk == NULL) > throw(MAL, "mal.interpreter", MAL_STACK_FAIL); > + keepAlive = stk->keepAlive; > if(env && stk && stk->cmd!='f') stk->cmd = env->cmd ; > > ret = runMALsequence(cntxt, mb, startpc, stoppc, stk, env, pcicaller); > > /* pass the new debug mode to the caller */ > if (env && stk->cmd!='f') env->cmd = stk->cmd; > + if ( keepAlive == 0 && garbageControl(getInstrPtr(mb,0)) ) > + garbageCollector(cntxt, mb,stk,env!=stk); > return ret; > } > @- > @@ -1336,7 +1340,6 @@ > { > DataFlow flow = NULL; > str ret = MAL_SUCCEED; > - int keepalive; > > #ifdef DEBUG_FLOW > stream_printf(GDKstdout, "runMALdataflow for block %d - %d\n", startpc,stoppc); > @@ -1387,9 +1390,6 @@ > flow->stop = stoppc; > flow->mb = mb; > flow->stk = stk; > - /* avoid the garbage collection call at the end of interpreter loop */ > - keepalive = flow->stk->keepAlive; > - flow->stk->keepAlive= TRUE; > > flow->status = (char*) GDKzalloc((stoppc-startpc+1)); > flow->blocked = (char*) GDKzalloc(sizeof(char)*mb->vtop); > @@ -1397,7 +1397,6 @@ > mal_unset_lock(mal_contextLock, "runMALdataflow"); > > ret = DFLOWscheduler(flow); > - flow->stk->keepAlive= keepalive; > GDKfree(flow->status); > GDKfree(flow->blocked); > GDKfree(flow->inuse); > @@ -1764,7 +1763,7 @@ > sbackup[i]= 0; > backup[i]= 0; > garbage[i]= -1; > - if (stk->stk[getArg(pci,i)].vtype == TYPE_bat && getEndOfLife(mb,getArg(pci,i)) == stkpc && isNotUsedIn(pci,i+1,getArg(pci,i)) && stk->keepAlive == 0){ > + if (stk->stk[getArg(pci,i)].vtype == TYPE_bat && getEndOfLife(mb,getArg(pci,i)) == stkpc && isNotUsedIn(pci,i+1,getArg(pci,i)) ){ > garbage[i]= getArg(pci,i); > #ifdef DEBUG_GC > stream_printf(GDKstdout,"GC %d %s prep\n",getArg(pci,i), getArgName(mb,pci,i)); > @@ -2345,8 +2344,6 @@ > int k; > ValPtr v; > > - if ( stk->keepAlive) > - return; > #ifdef STACKTRACE > stream_printf(cntxt->fdout,"#--->stack before garbage collector\n"); > printStack(cntxt->fdout,mb,stk,0); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

3 2

Re: [Monetdb-developers] commit mail subject
by Fabian Groffen 07 May '10

07 May '10

On 07-05-2010 18:38:42 +0200, Stefan Manegold wrote: > On Fri, May 07, 2010 at 06:31:21PM +0200, Jan Rittinger wrote: > > Hi all, > > > > I think the subject format of the checkin message mail could be improved by changing the format from 'MonetDB: <log-message> (<branch>)' or sometimes 'MonetDB: <partial log-message>...' to 'MonetDB (<branch>): <log-message>' or 'MonetDB (<branch>): <partial log-message>...'. > > FYI: > > Mercurial uses the first line of checkin messages as summary in (e.g.) logs, > etc. and as email subject. > > It depends on the committer how useful this first line (or "partial > log-message") in isolation, and hence, the summary and email subject is > --- in particular for other --- but that holds for the complete checkin > message, too ... If the first line is too long (like a full page) the line is truncated by the script, guess that's what Jan refers too. Makes sense from a mail point of view. > (Sjoerd pointed this out in his Mercurial intorduction during MADAM and in > his MonetDB-Mercirial document.) > > > In short I vote for switching the order of <log-message> and <branch> in the subject. > > Fabian patched the Mercurial setup to add the brnach to the checkin email > subject. I guess, he can also prepend instead of append it. > > Would be fine by me. (In fact, I like your suggestion ;-)) Ok, that makes the score already 2 then :)

2 3