- developers-list - monetdb.org

[Monetdb-developers] Parallel insertions in monetdb table - some records do not insert
by kss ram 16 Jun '10

16 Jun '10

Hello Everyone! Brief Overview of the issue 1. Sample Data that erlang program tries to insert in monetdb {data, [{"a","string"}, {"b","string"}], %% a,b are column names [ [{"a","a1"}, {"b","b1"}], %% key, value pair of the data - {columnname, data} [{"a","a2"}, {"b","b2"}], [{"a","a3"}, {"b","b3"}], [{"a","a4"}, {"b","b4"}], [{"a","a5"}, {"b","b5"}], [{"a","a6"}, {"b","b6"}], [{"a","a7"}, {"b","b7"}], [{"a","a8"}, {"b","b8"}], [{"a","a9"}, {"b","b9"}], [{"a","a10"}, {"b","b10"}], [{"a","a11"}, {"b","b11"}] ], {"no_of_records","12"} } 2. no of threads/processes that would do parallel inserts would be determined by the thread size i.e how many of records the thread would insert, if say 5 records then 3 threads, if 3 records then 4 threads. 3. in order to connect to monetdb from erlang, the monetdb ODBC driver is specified in ODBCConfig utility of ubuntu, driver file is /usr/lib/libMonetODBC.so 4. threads are successfully created and the ODBC module quits successfully after inserting in the erlang program, but when we look in the monetdb table, some of the rows get missed, for example row from 1 to 11 get inserted without row 3 or 4 getting inserted. could this be a problem with the ODBC driver?, am I missing any configuration?, any help would be deeply appreciated. Thank you, Ram.

3 2

Re: [Monetdb-developers] MonetDB: default - fix compilation failure opt_mapreduce.mx:141:...
by Sjoerd Mullender 10 Jun '10

10 Jun '10

Why not fix this "properly" by changing the type of cnt (and the function return value and several other variables)? At the very least you could have an assert to check that the value fits in an int. On 2010-06-10 10:54, Fabian Groffen wrote: > Changeset: 820a7637cf5e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=820a7637cf5e > Modified Files: > MonetDB5/src/optimizer/opt_mapreduce.mx > Branch: default > Log Message: > > fix compilation failure opt_mapreduce.mx:141: '=' : conversion from 'size_t' to 'int', possible loss of data > > > diffs (12 lines): > > diff -r c7d1cd0d0bbb -r 820a7637cf5e MonetDB5/src/optimizer/opt_mapreduce.mx > --- a/MonetDB5/src/optimizer/opt_mapreduce.mx Wed Jun 09 15:30:45 2010 +0200 > +++ b/MonetDB5/src/optimizer/opt_mapreduce.mx Thu Jun 10 10:54:23 2010 +0200 > @@ -138,7 +138,7 @@ > return 0; > } > cloud = BATdescriptor(bid); > - cnt = BATcount(cloud); > + cnt = (int)BATcount(cloud); > BBPreleaseref(bid); /* we're done with it */ > return(cnt); > } > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] MonetDB: default - Add test to prevent warning
by Stefan Manegold 01 Jun '10

01 Jun '10

Arjen, is this a fix that should also be include in the upcoming Jun2010 release? If so, it would need to be back-ported to the Jun2010 branch ... Stefan On Tue, Jun 01, 2010 at 11:40:20AM +0200, Arjen de Rijke wrote: > Changeset: 74ee41649425 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=74ee41649425 > Modified Files: > clients/src/php/native/lib/php_mapi.inc > Branch: default > Log Message: > > Add test to prevent warning > > > diffs (15 lines): > > diff -r 80afe14ca780 -r 74ee41649425 clients/src/php/native/lib/php_mapi.inc > --- a/clients/src/php/native/lib/php_mapi.inc Tue Jun 01 10:02:37 2010 +0200 > +++ b/clients/src/php/native/lib/php_mapi.inc Tue Jun 01 11:37:25 2010 +0200 > @@ -768,8 +768,10 @@ > global $connection_pool; > > $connections = array(); > - foreach($connection_pool as $conn) { > + if ($connection_pool !== null) { > + foreach($connection_pool as $conn) { > $connections[] = $conn["id"]; > + } > } > > $id = hash("sha1", time()); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

[Monetdb-developers] mx tool status and future
by Alex Dubov 01 Jun '10

01 Jun '10

Greetings. I understand, that this issue was probably discussed before, but I couldn't quite find a way to search the list archive. It appears to me, that mx version 3, found on savannah.nongnu.org, is somewhat out of sync with the one used for monetdb (@mal and @include tags are not supported by it; adding the former was trivial, while I've set the later to no-op, as included *.mx files are nowhere to be found in the monetdb source tree). The mx doc formatter itself is also quite underdeveloped, compared to what we came to expect from systems like doxygen or sphinx. In light of the above, I wonder, are there any plans to switch to a different code documentation system (doxygen?) or is the mx tool intended for future development?

4 3

Re: [Monetdb-developers] MonetDB: default - Remove the C flag -Wno-uninitialized for Mac ...
by Stefan Manegold 20 May '10

20 May '10

Good point. Thanks! I don't expect any problems with this change, mostly because we don't have any MacOS X machine for testing, anymore; only one for buildbot --- once that's running again,we'll see ... Assuming that compilation works for you on a "modern" MacOS X (Snow Leopard), I assume that the mentioned bison.simple problems were related/specific to an older version of MacOS X, only. (MacOS X was never a primary develoment platform for us, that why disabling -Wuninitialized (by enabling -Wno-uninitialized) on MacOS X used to be - "acceptable" for us; obviously, with you using MacOS X as a primary development platform, -Wno-uninitialized should indeed not be used - (gloablly) ...) Stefan On Thu, May 20, 2010 at 06:20:02PM +0200, Jan Rittinger wrote: > Changeset: 892fc9b323aa for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=892fc9b323aa > Modified Files: > buildtools/conf/MonetDB.m4 > Branch: default > Log Message: > > Remove the C flag -Wno-uninitialized for Mac operating systems. > This helps to detect uninitialized variables on OS X. > > (Please undo the fix in case it breaks anything.) > > > diffs (12 lines): > > diff -r b181da69e3ba -r 892fc9b323aa buildtools/conf/MonetDB.m4 > --- a/buildtools/conf/MonetDB.m4 Thu May 20 17:50:15 2010 +0200 > +++ b/buildtools/conf/MonetDB.m4 Thu May 20 18:18:33 2010 +0200 > @@ -811,7 +811,7 @@ > dnl generates "X_CFLAGS =" in the generated Makefile. > NO_X_CFLAGS='X_CFLAGS' > ;; > - *-solaris*|*-darwin*|*-aix*) > + *-solaris*|*-aix*) > dnl In some cases, there is a (possibly) uninitialized > dnl variable in bison.simple ... |-( > X_CFLAGS="$X_CFLAGS -Wno-uninitialized" > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

[Monetdb-developers] MonetDB/SQL: bulk-loading large data sets & adding key constraints
by Stefan Manegold 15 May '10

15 May '10

Dear all, derived from recent experiments and intensive testing, here are some recommendations for bulk-loading large data sets (in particular those larger than your available main memory) in MonetDB/SQL and adding constraints, i.e., primary and foreign keys. First of all, bulk loading should be done from (possibly gzip or bzip2 compressed) CSV files using the SQL `COPY INTO` command; cf., http://monetdb.cwi.nl/SQL/Documentation/Data-Manipulation.html In case your tables come with primary or foreign key constraints, it is recommended to omit these constrains when initially creating the tables. Then bulk-load (`COPY INTO`) your data. Once the data is loaded add the key constraints one by one using the SQL `ALTER TABLE ... ADD CONSTRAINT ...` command. Note, though, that the current Feb2010 release of MonetDB/SQL might experience some performance problems when adding constraints on data sets larger than main memory on multi-core machines. The reason it a non-optimal query translation and plan parallelization. For the time being, i.e., until we manage to fix this problem properly, it is recommended to resort to the optimizer pipeline of the Nov2009 release (for insiders, this effectively means disabling the "mitosis" optimizer), (only) while adding the key constraints. To do so, simply activate the Nov2009 optimizer pipeline via SET optimizer='nov2009_pipe'; just before adding the key constraints via `ALTER TABLE ... ADD CONSTRAINTS` and then restore the default optimizer pipeline via SET optimizer='default_pipe'; once you're done with adding constraints. For TPCH scale factor 100 (100 GB) on a 8 GB 4-core machine, this reduces the total time to add all key constraints from ~1.5h to ~1h. For SSB scale factor 100 (100 GB) on a 8 GB 4-core machine, this reduces the total time to add all key constraints from ~4h to ~1h. Hope this helps. Please don't hesitate to comment or ask questions. We'll let you know, one we managed to fix this problem properly. Kind regards, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

Re: [Monetdb-developers] MonetDB: Jun2010 - Fix compilation on Windows
by Stefan Manegold 13 May '10

13 May '10

On Thu, May 13, 2010 at 10:02:14AM +0200, Fabian Groffen wrote: > Changeset: db9f1c910dbb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=db9f1c910dbb > Modified Files: > MonetDB5/src/mal/mal_linker.mx > Branch: Jun2010 > Log Message: > > Fix compilation on Windows > > .\..\..\..\src\mal\mal_linker.mx(62): catastrophic error: could not open > source file "dirent.h" > > Seems Sabaoth can use struct dirent without including dirent.h as well, > so I guess some gdk header takes care of this. I just removed dirent.h > include. > FYI: MonetDB/src/gdk/gdk.mx-#ifdef HAVE_DIRENT_H MonetDB/src/gdk/gdk.mx:# include <dirent.h> MonetDB/src/gdk/gdk.mx:# define NAMLEN(dirent) strlen((dirent)->d_name) MonetDB/src/gdk/gdk.mx-#else MonetDB/src/gdk/gdk.mx:# define dirent direct MonetDB/src/gdk/gdk.mx:# define NAMLEN(dirent) (dirent)->d_namlen MonetDB/src/gdk/gdk.mx-# ifdef HAVE_SYS_NDIR_H MonetDB/src/gdk/gdk.mx-# include <sys/ndir.h> MonetDB/src/gdk/gdk.mx-# endif MonetDB/src/gdk/gdk.mx-# ifdef HAVE_SYS_DIR_H MonetDB/src/gdk/gdk.mx-# include <sys/dir.h> MonetDB/src/gdk/gdk.mx-# endif MonetDB/src/gdk/gdk.mx-# ifdef HAVE_NDIR_H MonetDB/src/gdk/gdk.mx-# include <ndir.h> MonetDB/src/gdk/gdk.mx-# endif MonetDB/src/gdk/gdk.mx-#endif Stefan > > diffs (11 lines): > > diff -r e1c42586a988 -r db9f1c910dbb MonetDB5/src/mal/mal_linker.mx > --- a/MonetDB5/src/mal/mal_linker.mx Wed May 12 16:58:04 2010 +0200 > +++ b/MonetDB5/src/mal/mal_linker.mx Thu May 13 10:02:00 2010 +0200 > @@ -59,7 +59,6 @@ > #endif > > #include <sys/types.h> /* opendir */ > -#include <dirent.h> /* opendir */ > > /* #define DEBUG_MAL_LINKER */ > #define MONET64 1 > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

Re: [Monetdb-developers] MonetDB: Jun2010 - include sql is no longer in the arguments
by Stefan Manegold 10 May '10

10 May '10

Oops ;-) Sorry, missed that ... Thanks for watching & fixing! Stefan On Mon, May 10, 2010 at 08:16:01AM +0200, Fabian Groffen wrote: > Changeset: 2716b35c7688 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=2716b35c7688 > Modified Files: > sql/src/backends/monet5/merovingian/merovingian_forkmserver.c > Branch: Jun2010 > Log Message: > > include sql is no longer in the arguments > > > diffs (12 lines): > > diff -r a901acf756bb -r 2716b35c7688 sql/src/backends/monet5/merovingian/merovingian_forkmserver.c > --- a/sql/src/backends/monet5/merovingian/merovingian_forkmserver.c Mon May 10 08:06:52 2010 +0200 > +++ b/sql/src/backends/monet5/merovingian/merovingian_forkmserver.c Mon May 10 08:15:50 2010 +0200 > @@ -226,7 +226,7 @@ > str master = NULL; > str slave = NULL; > str pipeline = NULL; > - str argv[27]; /* for the exec arguments */ > + str argv[26]; /* for the exec arguments */ > confkeyval *ckv, *kv; > int c = 0; > > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

[Monetdb-developers] About the use of hg transplant
by Fabian Groffen 09 May '10

09 May '10

For those using hg transplant to backport changes. Using the -m flag of transplant causes a merge commit to occur. This means two heads (could be branches) are merged into one. For some not yet fully understood reason this causes a hg merge to perform a large batch of changes, probably recursively performing the full changes of the branch that was transplanted from. I think for backports it is better not to use -m, because it artificially says you moved over everything that was supposed to be moved over. However, in fact you didn't. Just don't make a merge commit. I found out that this is easy to deal with in the propagation, since transplant can be used in the following fashion: % cvs transplant --log -b Jun2010 -p 456afbaaf416 -p 0ee83fdb0e97 -m 63f8c7be221a applying 63f8c7be221a 35864:63f8c7be221a merged at cfbfaf940bb2 % Which in this case means: ignore changesets 456afbaaf416 and 0ee83fdb0e97, and at changeset 63f8c7be221a pretend a merge was done, which happens to be correct. (Actually, without the -p and -m opts, transplant will ask per changeset what to do with it, so it seems better suited to do regulated propagation merges.)

2 2

Re: [Monetdb-developers] MonetDB: default - Revert the changes of May 04 to recover perfo...
by Stefan Manegold 09 May '10

09 May '10

In case the changes of May 04 indeed caused the reported performance degradation, I assume this checkin should then also be back-ported to the Jun2010 branch that was created on May 06 and hence also contains the problematic changes of May 04, right? Once confirmed and I can back-port this checkin. Stefan On Sat, May 08, 2010 at 08:30:35PM +0200, Martin.Kersten(a)cwi.nl wrote: > Changeset: e9c1206c5008 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e9c1206c5008 > Modified Files: > MonetDB5/src/mal/mal_interpreter.mx > clients/src/python/test/capabilities.py > clients/src/python/test/dbapi20.py > geom/src/sql/geom.sql > Branch: default > Log Message: > > Revert the changes of May 04 to recover performance. > > > diffs (76 lines): > > diff -r dd777c76a3f9 -r e9c1206c5008 MonetDB5/src/mal/mal_interpreter.mx > --- a/MonetDB5/src/mal/mal_interpreter.mx Sat May 08 09:16:12 2010 +0200 > +++ b/MonetDB5/src/mal/mal_interpreter.mx Sat May 08 20:30:23 2010 +0200 > @@ -341,7 +341,7 @@ > > /* pass the new debug mode to the caller */ > if (stk->cmd && env && stk->cmd!='f') env->cmd = stk->cmd; > - if ( garbageControl(getInstrPtr(mb,0)) ) > + if ( !stk->keepAlive && garbageControl(getInstrPtr(mb,0)) ) > garbageCollector(cntxt, mb,stk, env != stk); > @:endProfile(stk)@ > return ret; > @@ -361,15 +361,19 @@ > MalStkPtr stk, MalStkPtr env, InstrPtr pcicaller) > { > str ret; > + int keepAlive; > > if (stk == NULL) > throw(MAL, "mal.interpreter", MAL_STACK_FAIL); > + keepAlive = stk->keepAlive; > if(env && stk && stk->cmd!='f') stk->cmd = env->cmd ; > > ret = runMALsequence(cntxt, mb, startpc, stoppc, stk, env, pcicaller); > > /* pass the new debug mode to the caller */ > if (env && stk->cmd!='f') env->cmd = stk->cmd; > + if ( keepAlive == 0 && garbageControl(getInstrPtr(mb,0)) ) > + garbageCollector(cntxt, mb,stk,env!=stk); > return ret; > } > @- > @@ -1336,7 +1340,6 @@ > { > DataFlow flow = NULL; > str ret = MAL_SUCCEED; > - int keepalive; > > #ifdef DEBUG_FLOW > stream_printf(GDKstdout, "runMALdataflow for block %d - %d\n", startpc,stoppc); > @@ -1387,9 +1390,6 @@ > flow->stop = stoppc; > flow->mb = mb; > flow->stk = stk; > - /* avoid the garbage collection call at the end of interpreter loop */ > - keepalive = flow->stk->keepAlive; > - flow->stk->keepAlive= TRUE; > > flow->status = (char*) GDKzalloc((stoppc-startpc+1)); > flow->blocked = (char*) GDKzalloc(sizeof(char)*mb->vtop); > @@ -1397,7 +1397,6 @@ > mal_unset_lock(mal_contextLock, "runMALdataflow"); > > ret = DFLOWscheduler(flow); > - flow->stk->keepAlive= keepalive; > GDKfree(flow->status); > GDKfree(flow->blocked); > GDKfree(flow->inuse); > @@ -1764,7 +1763,7 @@ > sbackup[i]= 0; > backup[i]= 0; > garbage[i]= -1; > - if (stk->stk[getArg(pci,i)].vtype == TYPE_bat && getEndOfLife(mb,getArg(pci,i)) == stkpc && isNotUsedIn(pci,i+1,getArg(pci,i)) && stk->keepAlive == 0){ > + if (stk->stk[getArg(pci,i)].vtype == TYPE_bat && getEndOfLife(mb,getArg(pci,i)) == stkpc && isNotUsedIn(pci,i+1,getArg(pci,i)) ){ > garbage[i]= getArg(pci,i); > #ifdef DEBUG_GC > stream_printf(GDKstdout,"GC %d %s prep\n",getArg(pci,i), getArgName(mb,pci,i)); > @@ -2345,8 +2344,6 @@ > int k; > ValPtr v; > > - if ( stk->keepAlive) > - return; > #ifdef STACKTRACE > stream_printf(cntxt->fdout,"#--->stack before garbage collector\n"); > printStack(cntxt->fdout,mb,stk,0); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

3 2