- developers-list - monetdb.org

Re: MonetDB: default - Report correct 100% sample size
by Niels Nes 16 Aug '13

16 Aug '13

On Fri, Aug 16, 2013 at 09:56:49PM +0200, Martin Kersten wrote: > Changeset: afa6da5dd7bd for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=afa6da5dd7bd > Modified Files: > sql/backends/monet5/sql_statistics.c > Branch: default > Log Message: > > Report correct 100% sample size > > > diffs (12 lines): > > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -169,7 +169,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > } > snprintf(query,8192, > "insert into sys.statistics values('%s','%s','%s','%s',%d,now(),"LLFMT","LLFMT","LLFMT","LLFMT",'%s','%s',%s);", > - b->name, bt->name, bc->name, c->type.type->sqlname,width, samplesize, sz, uniq, nils, minval,maxval, sorted?"true":"false"); > + b->name, bt->name, bc->name, c->type.type->sqlname,width, (samplesize?samplesize:sz), sz, uniq, nils, minval,maxval, sorted?"true":"false"); > #ifdef DEBUG_SQL_STATISTICS > mnstr_printf(cntxt->fdout,"%s\n",dquery); > mnstr_printf(cntxt->fdout,"%s\n",query); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list Martin Volgens mij wil je de volgende verzameling van analyze calls: analyze([opt_sample_size]) -- full db analyze(schema, [opt_sample_size]) -- full schema analyze(schema, table, [opt_sample_size]) -- full table analyze(schema, table, column, [opt_sample_size]) -- full table ie dus niet analyze(table, ...) voor die is het onduidelijk welk schema. Niels -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

Re: MonetDB: Feb2013 - New thread group per dataflow block
by Stefan Manegold 16 Aug '13

16 Aug '13

Martin, this checkins appears to make several tests run into timeouts; I have not yet been able to analyze why ... A complete Mtest.py run results in * 142 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 * 6 out of 1863 tests ran into timeout * sql/benchmarks/ATIS/load * sql/benchmarks/arno/insert_ATOM * sql/benchmarks/arno/insert_BOND * sql/benchmarks/arno_flt/init * sql/benchmarks/moa/load * sql/benchmarks/wisconsin/load 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 while with the version before this checkin, it results in 41 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 Best, Stefan ----- Original Message ----- > Changeset: 3669ddd28bf0 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3669ddd28bf0 > Modified Files: > monetdb5/mal/mal_dataflow.c > Branch: Feb2013 > Log Message: > > New thread group per dataflow block > The centralized worker thread group could lead to an unacceptable > situation. If a user is heavily processing complex queries, then > no other user could even log into the system, for its MAL statements > ended up at the end of the shared queues. > > The problem has been resolved by introducing a thread group per dataflow > block. This may lead to a large number of processes, whose resources > are managed by the OS. > > It solves bug 3258 > > > diffs (235 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -78,12 +78,12 @@ typedef struct DATAFLOW { > int *nodes; /* dependency graph nodes */ > int *edges; /* dependency graph */ > MT_Lock flowlock; /* lock to protect the above */ > + queue *todo; /* pending instructions */ > queue *done; /* instructions handled */ > + int threads; /* worker threads active */ > + MT_Id workers[THREADS]; > } *DataFlow, DataFlowRec; > > -static MT_Id workers[THREADS]; > -static queue *todo = 0; /* pending instructions */ > - > /* > * Calculate the size of the dataflow dependency graph. > */ > @@ -138,7 +138,6 @@ q_destroy(queue *q) > static void > q_enqueue_(queue *q, FlowEvent d) > { > - assert(d); > if (q->last == q->size) { > q->size <<= 1; > q->data = GDKrealloc(q->data, sizeof(FlowEvent) * q->size); > @@ -214,7 +213,6 @@ q_dequeue(queue *q) > */ > > MT_lock_unset(&q->l, "q_dequeue"); > - assert(r); > return r; > } > > @@ -239,14 +237,15 @@ q_dequeue(queue *q) > static void > DFLOWworker(void *t) > { > - DataFlow flow; > + DataFlow flow = (DataFlow) t; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > + MT_Id id = MT_getpid(); > + int last = 0; > Thread thr; > str error = 0; > > int i; > - lng usec = 0; > + //lng usec = 0; > > thr = THRnew("DFLOWworker"); > > @@ -254,8 +253,10 @@ DFLOWworker(void *t) > GDKerrbuf[0] = 0; > while (1) { > if (fnxt == 0) > - fe = q_dequeue(todo); > + fe = q_dequeue(flow->todo); > else fe = fnxt; > + if ( fe == 0) > + break; > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -266,20 +267,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > + //usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - if (todo->last == 0) > + if (flow->todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo, fe); > + q_requeue(flow->todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > - PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > + PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= "SZFMT" claim= " > LLFMT "," LLFMT " %s\n", > fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > @@ -330,12 +331,15 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - if (todo->last == 0) > + if (flow->todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > + //else > + //MALresourceFairness(NULL, NULL, usec); > } > } > + for( i = 0; i< flow->threads; i++) > + if ( flow->workers[i] == id) > + flow->workers[i] = 0; > GDKfree(GDKerrbuf); > GDKsetbuf(0); > THRdel(thr); > @@ -349,22 +353,51 @@ DFLOWworker(void *t) > * The workers are assembled in a local table to enable debugging. > */ > static void > -DFLOWinitialize(void) > +DFLOWinitialize(DataFlow flow, int size) > { > - int i, limit; > + int i; > > - MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + MT_lock_init(&flow->flowlock, "DFLOWworker"); > + flow->todo = q_create(size); > + flow->done = q_create(size); > + flow->threads = GDKnr_threads ? GDKnr_threads :1; > + for (i = 0; i < flow->threads; i++){ > + MT_create_thread(&flow->workers[i], DFLOWworker, (void *) flow, > MT_THR_JOINABLE); > + /* upon failure of starting threads we reduce the count */ > + if ( flow->workers[i]== 0){ > + flow->threads --; > + i--; > + } > } > - todo = q_create(2048); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - for (i = 0; i < limit; i++) > - MT_create_thread(&workers[i], DFLOWworker, (void *) &workers[i], > MT_THR_JOINABLE); > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > } > > +static str > +DFLOWfinalize(DataFlow flow) > +{ > + int i, cnt= flow->threads, runs =0; > + > + for( i = 0; i< cnt; i++) > + q_enqueue(flow->todo, 0); > + /* time out when threads are already killed */ > + do{ > + runs++; > + cnt = 0; > + MT_sleep_ms(1); > + for( i = 0; i < flow->threads; i++) > + cnt += flow->workers[i] ==0; > + } while( cnt != flow->threads && runs <5000); > + > + if ( runs == 5000) > + throw(MAL,"dataflow","Timeout on thread termination"); > + GDKfree(flow->status); > + GDKfree(flow->edges); > + GDKfree(flow->nodes); > + q_destroy(flow->done); > + q_destroy(flow->todo); > + MT_lock_destroy(&flow->flowlock); > + GDKfree(flow); > + return MAL_SUCCEED; > +} > /* > * The dataflow administration is based on administration of > * how many variables are still missing before it can be executed. > @@ -518,7 +551,7 @@ DFLOWscheduler(DataFlow flow) > for (j = p->retc; j < p->argc; j++) > fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > #endif > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -543,7 +576,7 @@ DFLOWscheduler(DataFlow flow) > if (flow->status[i].blocks == 1 ) { > flow->status[i].state = DFLOWrunning; > flow->status[i].blocks--; > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > PARDEBUG > mnstr_printf(GDKstdout, "#enqueue pc=%d claim= " LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } else { > @@ -579,13 +612,11 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > assert(stoppc > startpc); > > - /* check existence of workers */ > - if (workers[0] == 0) > - DFLOWinitialize(); > - assert(workers[0]); > - assert(todo); > - > flow = (DataFlow)GDKzalloc(sizeof(DataFlowRec)); > + > + DFLOWinitialize(flow, stoppc- startpc +1); > + assert(flow->todo); > + assert(flow->done); > > flow->cntxt = cntxt; > flow->mb = mb; > @@ -596,9 +627,6 @@ runMALdataflow(Client cntxt, MalBlkPtr m > flow->start = startpc + 1; > flow->stop = stoppc; > > - MT_lock_init(&flow->flowlock, "DFLOWworker"); > - flow->done = q_create(stoppc- startpc+1); > - > flow->status = (FlowEvent)GDKzalloc((stoppc - startpc + 1) * > sizeof(FlowEventRec)); > size = DFLOWgraphSize(mb, startpc, stoppc); > size += stoppc - startpc; > @@ -608,11 +636,9 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > ret = DFLOWscheduler(flow); > > - GDKfree(flow->status); > - GDKfree(flow->edges); > - GDKfree(flow->nodes); > - q_destroy(flow->done); > - MT_lock_destroy(&flow->flowlock); > - GDKfree(flow); > + if( ret == MAL_SUCCEED) > + ret = DFLOWfinalize(flow); > + else (void) > + DFLOWfinalize(flow); > return ret; > } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Kernel Function-level Profiling
by ajonwa ajonwa 15 Aug '13

15 Aug '13

Hello All, I am new to MonetDB and would like to profile the MonetDB kernel to find out the functions that are most frequently executed while it runs TPCH. So far, I have tried compiling with "--enable-profile" but it seems that gmon.out files are not generated. Is there a recommend way of doing function level profiling in MonetDB? Thanks, Geoffrey

2 1

Re: MonetDB: SciQL-2 - handle non-rectangular array tiles
by Stefan Manegold 15 Aug '13

15 Aug '13

Hoi Jennie, > Changeset: 58e31783ae0f for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=58e31783ae0f > Modified Files: > monetdb5/modules/mal/array.mx > sql/server/rel_dump.c > sql/server/rel_exp.c > sql/server/rel_select.c > sql/test/sciql/Tests/teleios_noa_bsm.sql > Branch: SciQL-2 > Log Message: > > handle non-rectangular array tiles > > - accept either interval tile range, or point tile ragnes, no mix of them > - multiple tile ranges are appended to e_column->f, thus e_column->f list can > have more than 3 items. > - to distinguish a point range from an interval range: a point range only has > one expr., while an interval range has three Thanks! > array.mx: > - save the values for the error message, since the lists are freed in > AGGR_CLEANUP Nice catch ;-) --- Thanks for fixing! ... and sorry for overlooking it myself ;-) > - disable the check on (max - min + 1 = size), since it only holds for > stepsize = 1. > maybe we should just remove this check, since we don't have step size in > the > tiling aggr impl. functions, do we? IMHO, our current tiled aggregation implementation inherently assumes (and thus requires) that all dimensions of the given array have step-size == 1. Since ARRAYtiles*() does not get the step sizes of the given array as parameters, the check (max - min + 1 = size) was/is IMHO the only way to check/ensure that we are indeed working on an array where all dimensions have step-size == 1. > teleios_noa_bsm.sql: > now we can get rid of the workarounds > > more tests needed... Indeed --- I'll resume me test tonight or tomorrow ... Best, Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

UTF-8 support for column names?
by Satoshi Nagayasu 12 Aug '13

12 Aug '13

Hi, Is there any plan to support UTF-8 charactors for column names (or object identifiers) in MonetDB? I think it's not supported so far. Right? I'm trying to port my database from a row-store one to MonetDB. Now, we are using UTF-8 charactors (Japanese) in column names, so we need to convert them manually. Obviously, it is possible, but if MonetDB can support UTF-8 charactors in column name, it would be easier to get porting our database to MonetDB. And I found the related thread and feature request on this. [MonetDB-users] 答复: Does Monetdb supports UTF-8 characters when I use 'as' clause? http://mail.monetdb.org/pipermail/users-list/2012-February/005397.html Bug 3010 - Does not currently support non-ASCII names in identifiers http://bugs.monetdb.org/show_bug.cgi?id=3010#c2 Is there any particluar plan? Is it difficult to implement? Regards, -- Satoshi Nagayasu <snaga(a)uptime.jp> Uptime Technologies, LLC. http://www.uptime.jp

1 0

Re: MonetDB repository problem!
by Stefan Manegold 07 Aug '13

07 Aug '13

In sort, there is a slight problem with a server side check to prevent users from pushing in unresolved merge conflicts. The problem materializes such that one can checkin such conflicts, but then not (easily) remove them, again. Having you .hgrc up to date as "requested", in particular the whitespace hook [hooks] pretxncommit.whitespace = python:<path-to-check_whitespace-directory>/check_whitespace.py:hook prevent the checkin of unresolved unresolved merge conflicts (already) on the client side. (We'll fully fix the server side once Sjoerd is back from vacation.) Stefan ----- Original Message ----- > Hi stefan, > > can you please share some more information about the problem? how can > configurations of local files on the client side affect the server/main > repository? I am asking for purely academic reasons. Is there some kind of > security problem in hg? > > ----- Original Message ----- > > Dear all, > > > > the MonetDB Mercurial (HG) repository should be working, again. > > > > IMPORTANT NOTE: > > > > To prevent further problems, it is mandatory that you do consciously read > > http://www.monetdb.org/Assets/MonetDB-Mercurial.html > > and follow all instructions carefully. > > > > In particular, your .hgrc must contain all of the "Important Configuration" > > mention in above document. > > > > You must double-check your .hgrc before your next hg pull, hg commit or hg > > push, > > which ever comes first. > > > > Thanks! > > Stefan > > > > ----- Original Message ----- > > > Dear all, > > > > > > there seems to be a problem with the MonetDB HG (Mercurial) repository. > > > Please refrain from pushing to it until we have sorted-out the problem. > > > > > > Thanks! > > > Stefan > > > > -- > > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > > > > > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

MonetDB repository problem!
by Stefan Manegold 06 Aug '13

06 Aug '13

Dear all, there seems to be a problem with the MonetDB HG (Mercurial) repository. Please refrain from pushing to it until we have sorted-out the problem. Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 1

Re: MonetDB: Feb2013 - also batxml can be handled by mitosis/mergetable
by Sjoerd Mullender 31 Jul '13

31 Jul '13

Sorry, not allowed in the release branch: this changes the ABI. On 2013-07-31 09:47, Niels Nes wrote: > Changeset: 8300b5c5bd97 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8300b5c5bd97 > Modified Files: > monetdb5/optimizer/opt_prelude.c > monetdb5/optimizer/opt_prelude.h > monetdb5/optimizer/opt_support.c > Branch: Feb2013 > Log Message: > > also batxml can be handled by mitosis/mergetable > > > diffs (41 lines): > > diff --git a/monetdb5/optimizer/opt_prelude.c b/monetdb5/optimizer/opt_prelude.c > --- a/monetdb5/optimizer/opt_prelude.c > +++ b/monetdb5/optimizer/opt_prelude.c > @@ -42,6 +42,7 @@ str boxRef; > str batstrRef; > str batmtimeRef; > str batmmathRef; > +str batxmlRef; > str bbpRef; > str tidRef; > str deltaRef; > @@ -301,6 +302,7 @@ void optimizerInit(void){ > batstrRef = putName("batstr",6); > batmtimeRef = putName("batmtime",8); > batmmathRef = putName("batmmath",8); > + batxmlRef = putName("batxml",6); > bbpRef = putName("bbp",3); > tidRef = putName("tid",3); > deltaRef = putName("delta",5); > diff --git a/monetdb5/optimizer/opt_prelude.h b/monetdb5/optimizer/opt_prelude.h > --- a/monetdb5/optimizer/opt_prelude.h > +++ b/monetdb5/optimizer/opt_prelude.h > @@ -41,6 +41,7 @@ opt_export str boxRef; > opt_export str batstrRef; > opt_export str batmtimeRef; > opt_export str batmmathRef; > +opt_export str batxmlRef; > opt_export str bbpRef; > opt_export str tidRef; > opt_export str deltaRef; > diff --git a/monetdb5/optimizer/opt_support.c b/monetdb5/optimizer/opt_support.c > --- a/monetdb5/optimizer/opt_support.c > +++ b/monetdb5/optimizer/opt_support.c > @@ -827,6 +827,7 @@ int isMapOp(InstrPtr p){ > (getModuleId(p)== batmtimeRef) || > (getModuleId(p)== batstrRef) || > (getModuleId(p)== batmmathRef) || > + (getModuleId(p)== batxmlRef) || > (getModuleId(p)== mkeyRef); > } > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 4

Re: MonetDB: SciQL-2 - forked new SciQL-2 branch off Feb2013 branch:
by Ying Zhang 25 Jul '13

25 Jul '13

On Jul 25, 2013, at 23:12, Stefan Manegold wrote: > Changeset: b1131b028770 for MonetDB > ... > Branch: SciQL-2 > Log Message: > > forked new SciQL-2 branch off Feb2013 branch: Thanks! > > The SciQL-2 branch has been initialized by applying the differences > between tag before_new_JOIN_implementation (changeset 334140294fb2) of the default branch > and tag SciQL-2_source (changeset c05174470f48) of the sciql branch > onto the Feb2013 branch (tag SciQL-2_root / changeset a2665f0d1fc5), > i.e., the SciQL-2 branch differs from the Feb2013 branch > just like the sciql branch differs from the default banch. > > While this excercise was aimed at improving the stability of the SciQL code base > (by eliminating potentially not yet stabelized development code), > the segfaults that triggered this excercise (as reported internally) > do unfortunately still occur also with the SciQL-2 branch ... I'll look into this a.s.a.p., but it's better to work based on a stable branch anyway. Jennie

1 0

(no subject)
by Ketan Sharma 22 Jul '13

22 Jul '13

Hi all, I'm Ketan and I'm new to Monet DB. I have successfully installed MonetDB. I have a table of size ~200MB which is to be inserted into another MonetDB Table. But while inserting data using SELECT command, I am getting the following error: #================================================ GDK reported error. BATfetchjoin(tmpr_2257,tmp_ 2241) does not hit always (|bn|=0 != 10=|l|) => can't use fetchjoin. #================================================ Please do help me in getting this error solved. Thanks.

3 5