- developers-list - monetdb.org

Kernel Function-level Profiling
by ajonwa ajonwa 15 Aug '13

15 Aug '13

Hello All, I am new to MonetDB and would like to profile the MonetDB kernel to find out the functions that are most frequently executed while it runs TPCH. So far, I have tried compiling with "--enable-profile" but it seems that gmon.out files are not generated. Is there a recommend way of doing function level profiling in MonetDB? Thanks, Geoffrey

2 1

Re: MonetDB: SciQL-2 - handle non-rectangular array tiles
by Stefan Manegold 15 Aug '13

15 Aug '13

Hoi Jennie, > Changeset: 58e31783ae0f for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=58e31783ae0f > Modified Files: > monetdb5/modules/mal/array.mx > sql/server/rel_dump.c > sql/server/rel_exp.c > sql/server/rel_select.c > sql/test/sciql/Tests/teleios_noa_bsm.sql > Branch: SciQL-2 > Log Message: > > handle non-rectangular array tiles > > - accept either interval tile range, or point tile ragnes, no mix of them > - multiple tile ranges are appended to e_column->f, thus e_column->f list can > have more than 3 items. > - to distinguish a point range from an interval range: a point range only has > one expr., while an interval range has three Thanks! > array.mx: > - save the values for the error message, since the lists are freed in > AGGR_CLEANUP Nice catch ;-) --- Thanks for fixing! ... and sorry for overlooking it myself ;-) > - disable the check on (max - min + 1 = size), since it only holds for > stepsize = 1. > maybe we should just remove this check, since we don't have step size in > the > tiling aggr impl. functions, do we? IMHO, our current tiled aggregation implementation inherently assumes (and thus requires) that all dimensions of the given array have step-size == 1. Since ARRAYtiles*() does not get the step sizes of the given array as parameters, the check (max - min + 1 = size) was/is IMHO the only way to check/ensure that we are indeed working on an array where all dimensions have step-size == 1. > teleios_noa_bsm.sql: > now we can get rid of the workarounds > > more tests needed... Indeed --- I'll resume me test tonight or tomorrow ... Best, Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

UTF-8 support for column names?
by Satoshi Nagayasu 12 Aug '13

12 Aug '13

Hi, Is there any plan to support UTF-8 charactors for column names (or object identifiers) in MonetDB? I think it's not supported so far. Right? I'm trying to port my database from a row-store one to MonetDB. Now, we are using UTF-8 charactors (Japanese) in column names, so we need to convert them manually. Obviously, it is possible, but if MonetDB can support UTF-8 charactors in column name, it would be easier to get porting our database to MonetDB. And I found the related thread and feature request on this. [MonetDB-users] 答复: Does Monetdb supports UTF-8 characters when I use 'as' clause? http://mail.monetdb.org/pipermail/users-list/2012-February/005397.html Bug 3010 - Does not currently support non-ASCII names in identifiers http://bugs.monetdb.org/show_bug.cgi?id=3010#c2 Is there any particluar plan? Is it difficult to implement? Regards, -- Satoshi Nagayasu <snaga(a)uptime.jp> Uptime Technologies, LLC. http://www.uptime.jp

1 0

Re: MonetDB repository problem!
by Stefan Manegold 07 Aug '13

07 Aug '13

In sort, there is a slight problem with a server side check to prevent users from pushing in unresolved merge conflicts. The problem materializes such that one can checkin such conflicts, but then not (easily) remove them, again. Having you .hgrc up to date as "requested", in particular the whitespace hook [hooks] pretxncommit.whitespace = python:<path-to-check_whitespace-directory>/check_whitespace.py:hook prevent the checkin of unresolved unresolved merge conflicts (already) on the client side. (We'll fully fix the server side once Sjoerd is back from vacation.) Stefan ----- Original Message ----- > Hi stefan, > > can you please share some more information about the problem? how can > configurations of local files on the client side affect the server/main > repository? I am asking for purely academic reasons. Is there some kind of > security problem in hg? > > ----- Original Message ----- > > Dear all, > > > > the MonetDB Mercurial (HG) repository should be working, again. > > > > IMPORTANT NOTE: > > > > To prevent further problems, it is mandatory that you do consciously read > > http://www.monetdb.org/Assets/MonetDB-Mercurial.html > > and follow all instructions carefully. > > > > In particular, your .hgrc must contain all of the "Important Configuration" > > mention in above document. > > > > You must double-check your .hgrc before your next hg pull, hg commit or hg > > push, > > which ever comes first. > > > > Thanks! > > Stefan > > > > ----- Original Message ----- > > > Dear all, > > > > > > there seems to be a problem with the MonetDB HG (Mercurial) repository. > > > Please refrain from pushing to it until we have sorted-out the problem. > > > > > > Thanks! > > > Stefan > > > > -- > > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > > > > > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

MonetDB repository problem!
by Stefan Manegold 06 Aug '13

06 Aug '13

Dear all, there seems to be a problem with the MonetDB HG (Mercurial) repository. Please refrain from pushing to it until we have sorted-out the problem. Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 1

Re: MonetDB: Feb2013 - also batxml can be handled by mitosis/mergetable
by Sjoerd Mullender 31 Jul '13

31 Jul '13

Sorry, not allowed in the release branch: this changes the ABI. On 2013-07-31 09:47, Niels Nes wrote: > Changeset: 8300b5c5bd97 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8300b5c5bd97 > Modified Files: > monetdb5/optimizer/opt_prelude.c > monetdb5/optimizer/opt_prelude.h > monetdb5/optimizer/opt_support.c > Branch: Feb2013 > Log Message: > > also batxml can be handled by mitosis/mergetable > > > diffs (41 lines): > > diff --git a/monetdb5/optimizer/opt_prelude.c b/monetdb5/optimizer/opt_prelude.c > --- a/monetdb5/optimizer/opt_prelude.c > +++ b/monetdb5/optimizer/opt_prelude.c > @@ -42,6 +42,7 @@ str boxRef; > str batstrRef; > str batmtimeRef; > str batmmathRef; > +str batxmlRef; > str bbpRef; > str tidRef; > str deltaRef; > @@ -301,6 +302,7 @@ void optimizerInit(void){ > batstrRef = putName("batstr",6); > batmtimeRef = putName("batmtime",8); > batmmathRef = putName("batmmath",8); > + batxmlRef = putName("batxml",6); > bbpRef = putName("bbp",3); > tidRef = putName("tid",3); > deltaRef = putName("delta",5); > diff --git a/monetdb5/optimizer/opt_prelude.h b/monetdb5/optimizer/opt_prelude.h > --- a/monetdb5/optimizer/opt_prelude.h > +++ b/monetdb5/optimizer/opt_prelude.h > @@ -41,6 +41,7 @@ opt_export str boxRef; > opt_export str batstrRef; > opt_export str batmtimeRef; > opt_export str batmmathRef; > +opt_export str batxmlRef; > opt_export str bbpRef; > opt_export str tidRef; > opt_export str deltaRef; > diff --git a/monetdb5/optimizer/opt_support.c b/monetdb5/optimizer/opt_support.c > --- a/monetdb5/optimizer/opt_support.c > +++ b/monetdb5/optimizer/opt_support.c > @@ -827,6 +827,7 @@ int isMapOp(InstrPtr p){ > (getModuleId(p)== batmtimeRef) || > (getModuleId(p)== batstrRef) || > (getModuleId(p)== batmmathRef) || > + (getModuleId(p)== batxmlRef) || > (getModuleId(p)== mkeyRef); > } > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 4

Re: MonetDB: SciQL-2 - forked new SciQL-2 branch off Feb2013 branch:
by Ying Zhang 25 Jul '13

25 Jul '13

On Jul 25, 2013, at 23:12, Stefan Manegold wrote: > Changeset: b1131b028770 for MonetDB > ... > Branch: SciQL-2 > Log Message: > > forked new SciQL-2 branch off Feb2013 branch: Thanks! > > The SciQL-2 branch has been initialized by applying the differences > between tag before_new_JOIN_implementation (changeset 334140294fb2) of the default branch > and tag SciQL-2_source (changeset c05174470f48) of the sciql branch > onto the Feb2013 branch (tag SciQL-2_root / changeset a2665f0d1fc5), > i.e., the SciQL-2 branch differs from the Feb2013 branch > just like the sciql branch differs from the default banch. > > While this excercise was aimed at improving the stability of the SciQL code base > (by eliminating potentially not yet stabelized development code), > the segfaults that triggered this excercise (as reported internally) > do unfortunately still occur also with the SciQL-2 branch ... I'll look into this a.s.a.p., but it's better to work based on a stable branch anyway. Jennie

1 0

(no subject)
by Ketan Sharma 22 Jul '13

22 Jul '13

Hi all, I'm Ketan and I'm new to Monet DB. I have successfully installed MonetDB. I have a table of size ~200MB which is to be inserted into another MonetDB Table. But while inserting data using SELECT command, I am getting the following error: #================================================ GDK reported error. BATfetchjoin(tmpr_2257,tmp_ 2241) does not hit always (|bn|=0 != 10=|l|) => can't use fetchjoin. #================================================ Please do help me in getting this error solved. Thanks.

3 5

Performance fixes
by Alistair Sutherland 17 Jul '13

17 Jul '13

Hi, I've recently been conducting some performance testing of MonetDB with a variety of servers (from 4 core -8GB RAM to 20 core-64GB RAM) with various data sizes, in an attempt to gain a better understanding of how MonetDB scales. During the performance tests it became obvious that much of the processing was IO bound due to: 1) Columns being unmapped from memory overly aggressively (even when there was plenty of memory still available). 2) The constant mapping/unmapping of memory mapped bat files for intermediate results. I've attached a patch which attempts to address both issues. The first patch to (gdk_utils.c) is to update the memory limit at which the GDKvmtrim kicks in to be 80% memory usage. The second patch (gdk_heap.c) limits the number of mmap/munmap calls via the existing 'heap caching' mechanism which was not working at all! In addition to fixing up the caching code, I've also wired in the heap cache into the case where extending a malloced heap results in a swap over to memory mapped storage. After applying the patches I was seeing approximately 40% performance improvements (your mileage may vary!) If the changes are deemed to be useful, how do I go about getting them accepted into the MonetDB source repository? Thanks, Alistair Sutherland ________________________________

2 2

Re: MonetDB: DVframework - fixed bug. Three subselect instructions r...
by Ying Zhang 16 Jul '13

16 Jul '13

Hai Yagiz, Thanks for fixed bugs! :) Is this check-in supposed to fix some of the demo queries? I tried the query (on a freshly created repo.) SELECT channel, sample_time, sample_value FROM mseed.dataview WHERE station = 'ISK' AND sample_time > '2010-01-12T22:15:00.000' AND sample_time < '2010-01-12T22:18:00.000'; But it still gives the error: From the recursive call: MALException:sql.delta:Object not found Is this expected or did I forget something? Jennie On Jul 16, 2013, at 15:49, Yagiz Kargin wrote: > Changeset: 95bc831eb9fb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=95bc831eb9fb > Modified Files: > monetdb5/extras/dvf/dvf.c > monetdb5/extras/dvf/opt_dvf.c > Branch: DVframework > Log Message: > > fixed bug. Three subselect instructions resulting from selection predicate on sample_time are now separately taken care of. > > > diffs (152 lines): > > diff --git a/monetdb5/extras/dvf/dvf.c b/monetdb5/extras/dvf/dvf.c > --- a/monetdb5/extras/dvf/dvf.c > +++ b/monetdb5/extras/dvf/dvf.c > @@ -77,7 +77,7 @@ str plan_modifier(Client cntxt, MalBlkPt > BATiter fli; > > int run_dataflow_opt = 0; > - int run_recycle_opt = 1; > + int run_recycle_opt = 0; > > VarRecord low, high; > > @@ -343,8 +343,9 @@ str plan_modifier(Client cntxt, MalBlkPt > /* adjust variable lifetimes */ > malGarbageCollector(mb); > > - /*chkProgram(cntxt->fdout, cntxt->nspace, mb); > - printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > + /*chkProgram(cntxt->fdout, cntxt->nspace, mb);*/ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_EXPLAIN); */ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > > /* relocate the startpc, the instruction to proceed with the execution. Because it might be changed by the optimizers. */ > for (i = 0; i < limit; i++) > diff --git a/monetdb5/extras/dvf/opt_dvf.c b/monetdb5/extras/dvf/opt_dvf.c > --- a/monetdb5/extras/dvf/opt_dvf.c > +++ b/monetdb5/extras/dvf/opt_dvf.c > @@ -119,6 +119,9 @@ OPTdvfImplementation(Client cntxt, MalBl > int i, limit, which_column, actions = 0; > int last_bind_return_var_id = -1; > int last_data_tid_return_var_id = -1; > + int last_insert_bind_return_var_id = -1; > + int last_subselect_return_var_id = -1; > + int last_update_bind_second_return_var_id = -1; > > stk = stk; //to escape 'unused' parameter error. > pci = pci; //to escape 'unused' parameter error. > @@ -452,35 +455,107 @@ OPTdvfImplementation(Client cntxt, MalBl > actions++; > } > else if((state == 2 || state == 3) && > - ((getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 7 && > p->retc == 2 && > strcmp(getVarConstant(mb, getArg(p, 3)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 4)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 6)).val.ival == 2) || > - (getModuleId(p) == sqlRef && > + getVarConstant(mb, getArg(p, 6)).val.ival == 2) > + { > + last_update_bind_second_return_var_id = getArg(p, 1); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 6 && > p->retc == 1 && > strcmp(getVarConstant(mb, getArg(p, 2)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 3)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 5)).val.ival == 1))) > + getVarConstant(mb, getArg(p, 5)).val.ival == 1) > + { > + last_insert_bind_return_var_id = getArg(p, 0); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + p->argc == 8 && > + p->retc == 1 && > + getArg(p, 1) == last_bind_return_var_id) > + { > + last_subselect_return_var_id = getArg(p, 0); > + > + r = newInstruction(mb, ASSIGNsymbol); > + setModuleId(r, algebraRef); > + setFunctionId(r, subselectRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, getArg(p, 3)); > + r = pushArgument(mb, r, getArg(p, 4)); > + r = pushArgument(mb, r, getArg(p, 5)); > + r = pushArgument(mb, r, getArg(p, 6)); > + r = pushArgument(mb, r, getArg(p, 7)); > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + (p->argc == 7 || p->argc == 8) && > + p->retc == 1 && > + (getArg(p, 1) == last_update_bind_second_return_var_id || getArg(p, 1) == last_insert_bind_return_var_id)) > { > removeInstruction(mb, p); > i--; > actions++; > } > else if((state == 3) && > - (getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == projectdeltaRef && > p->argc == 6 && > p->retc == 1 && > - last_bind_return_var_id == getArg(p, 2))) > + last_bind_return_var_id == getArg(p, 2)) > + { > + r = newInstruction(mb, ASSIGNsymbol); > + > + if(getArg(p, 1) == last_data_tid_return_var_id) > + { > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + else > + { > + setModuleId(r, algebraRef); > + setFunctionId(r, leftfetchjoinRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 3) && > + getModuleId(p) == sqlRef && > + getFunctionId(p) == subdeltaRef && > + p->argc == 6 && > + p->retc == 1 && > + last_subselect_return_var_id == getArg(p, 1)) > { > r = newInstruction(mb, ASSIGNsymbol); > r = pushReturn(mb, r, getArg(p, 0)); > - r = pushArgument(mb, r, last_bind_return_var_id); > + r = pushArgument(mb, r, last_subselect_return_var_id); > > insertInstruction(mb, r, i+1); > removeInstruction(mb, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

2 1