March 2015 - developers-list

Re: MonetDB: default - Defense against non-existing MAL primitives
by Stefan Manegold 20 Mar '15

20 Mar '15

FYI, test clients/R/Tests/survey.R still fails with "Server says '!TypeException:user.s94_2[40]:'batcalc.*' undefined in: X_168:bat[:oid,:hge] := batcalc.*(X_146,X_166);'." i.e., the check whether the requested new signature exists does not seem to work, yet. cf., http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=55075:ca15cab457b8&or… Stefan ----- Original Message ----- > Changeset: 4165dc34b2bb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4165dc34b2bb > Modified Files: > monetdb5/mal/mal_resolve.c > monetdb5/mal/mal_resolve.h > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Defense against non-existing MAL primitives > and further limit the rewrite to the simpliest > and most expensive case encountered. > > > diffs (244 lines): > > diff --git a/monetdb5/mal/mal_resolve.c b/monetdb5/mal/mal_resolve.c > --- a/monetdb5/mal/mal_resolve.c > +++ b/monetdb5/mal/mal_resolve.c > @@ -115,7 +115,6 @@ findFunctionType(stream *out, Module sco > Symbol s; > InstrPtr sig; > int i, k, unmatched = 0, s1; > - /* int foundbutwrong=0; */ > int polytype[MAXTYPEVAR]; > int returns[256]; > int *returntype = NULL; > @@ -396,7 +395,6 @@ findFunctionType(stream *out, Module sco > } > } > if (s1 < 0) { > - /* if(getSignature(s)->token !=PATTERNsymbol) foundbutwrong++; */ > s = s->peer; > continue; > } > @@ -445,7 +443,6 @@ findFunctionType(stream *out, Module sco > * typed. This should be reflected in the symbol table. > */ > s1 = returntype[0]; /* for those interested */ > - /* foundbutwrong = 0; */ > /* > * If the call refers to a polymorphic function, we clone it > * to arrive at a bounded instance. Polymorphic patterns and > @@ -485,12 +482,6 @@ findFunctionType(stream *out, Module sco > * arguments, but that clashes with one of the target variables. > */ > wrapup: > - /* foundbutwrong has not been changed, commented out code above > - if (foundbutwrong && !silent) { > - showScriptException(out, mb, getPC(mb, p), TYPE, > - "type conflict in assignment"); > - } > - */ > if (returntype && returntype != returns) > GDKfree(returntype); > return -3; > @@ -819,13 +810,17 @@ chkTypes(stream *out, Module s, MalBlkPt > > /* > * Type checking an individual instruction is dangerous, > - * because it ignores data flow and declarations issues. > - * It is only to be used in isolated cases. > + * because it ignores data flow and variable declarations. > */ > -void > +int > chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr p) > { > - typeChecker(out, s, mb, p, FALSE); > + int olderrors= mb->errors; > + int error; > + typeChecker(out, s, mb, p, TRUE); > + error = mb->errors; > + mb->errors = olderrors; > + return error; > } > > void > diff --git a/monetdb5/mal/mal_resolve.h b/monetdb5/mal/mal_resolve.h > --- a/monetdb5/mal/mal_resolve.h > +++ b/monetdb5/mal/mal_resolve.h > @@ -20,7 +20,7 @@ > #define MAXTYPEVAR 10 > > mal_export void chkProgram(stream *out, Module s, MalBlkPtr mb); > -mal_export void chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > +mal_export int chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > mal_export void chkTypes(stream *out, Module s, MalBlkPtr mb, int silent); > mal_export void typeChecker(stream *out, Module scope, MalBlkPtr mb, > InstrPtr p, int silent); > mal_export int fcnBinder(stream *out, Module scope, MalBlkPtr mb, InstrPtr > p); > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -18,8 +18,9 @@ typedef struct{ > int fromtype; > int totype; > int src; > -/* not used, yet !?? > +/* not used, yet !?? Indeed > int digits; > + int fromscale; > int scale; > */ > } Coercion; > @@ -47,58 +48,60 @@ coercionOptimizerStep(MalBlkPtr mb, int > return 0; > } > > -/* Check coercions for numeric types that can be handled with smaller ones. > +/* Check coercions for numeric types towards :hge that can be handled with > smaller ones. > * For now, limit to +,-,/,*,% hge expressions > - * To be extended to deal with math calls as well. > + * Not every combination may be available in the MAL layer, which calls > + * for a separate type check before fixing it. > + * Superflous coercion statements will be garbagecollected later on in the > pipeline > */ > static void > -coercionOptimizerCalcStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerCalcStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > - int r, a, b; > + int r, a, b, varid; > > + r = getColumnType(getVarType(mb, getArg(p,0))); > +#ifdef HAVE_HGE > + if ( r != TYPE_hge) > + return; > +#endif > if( getModuleId(p) != batcalcRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == plusRef || getFunctionId(p) == minusRef || > getFunctionId(p) == mulRef || getFunctionId(p) == divRef || > *getFunctionId(p) =='%') || p->argc !=3) > return; > > - r = getColumnType(getVarType(mb, getArg(p,0))); > - switch(r){ > - case TYPE_bte: > - case TYPE_sht: > - case TYPE_int: > - case TYPE_lng: > -#ifdef HAVE_HGE > - case TYPE_hge: > -#endif > - break; > - case TYPE_dbl: > - case TYPE_flt: > - /* to be determined */ > - default: > - return; > - } > a = getColumnType(getVarType(mb, getArg(p,1))); > b = getColumnType(getVarType(mb, getArg(p,2))); > - if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) /*digit/scale test as well*/ > + if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) > + { > + varid = getArg(p,1); > getArg(p,1) = coerce[getArg(p,1)].src; > - if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) /*digit/scale test as well*/ > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[1] = varid; > + } > + if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) > + { > + varid = getArg(p,2); > getArg(p,2) = coerce[getArg(p,2)].src; > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[2] = varid; > + } > return; > } > > static void > -coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerAggrStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > int r, k; > > + (void) cntxt; > + > if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > return; > > r = getColumnType(getVarType(mb, getArg(p,0))); > k = getArg(p,1); > - // check the digits/scale > if( r == TYPE_dbl && coerce[k].src ) > getArg(p,1) = coerce[getArg(p,1)].src; > return; > @@ -124,35 +127,33 @@ OPTcoercionImplementation(Client cntxt,M > if (getModuleId(p) == NULL) > continue; > /* Downscale the type, avoiding hge storage when lng would be sufficient. > - * The code template can be extended to handle other downscale options as > well > */ > #ifdef HAVE_HGE > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == hgeRef > && p->retc == 1 > - && ( p->argc == 2 > - || ( p->argc == 5 > - && isVarConstant(mb,getArg(p,1)) > - && getArgType(mb,p,1) == TYPE_int > - && isVarConstant(mb,getArg(p,3)) > - && getArgType(mb,p,3) == TYPE_int > - && isVarConstant(mb,getArg(p,4)) > - && getArgType(mb,p,4) == TYPE_int > - /* from-scale == to-scale, i.e., no scale change */ > - && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) > getVarValue(mb, getArg(p,4)) ) ) ){ > + && ( p->argc == 5 > + && isVarConstant(mb,getArg(p,1)) > + && getArgType(mb,p,1) == TYPE_int > + && isVarConstant(mb,getArg(p,3)) > + && getArgType(mb,p,3) == TYPE_int > + && isVarConstant(mb,getArg(p,4)) > + && getArgType(mb,p,4) == TYPE_int > + /* from-scale == to-scale, i.e., no scale change */ > + && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) getVarValue(mb, > getArg(p,4)) ) ){ > k = getArg(p,0); > coerce[k].pc= i; > coerce[k].totype= TYPE_hge; > coerce[k].src= getArg(p,2); > coerce[k].fromtype= getColumnType(getArgType(mb,p,2)); > -/* not used, yet !?? > - if (p->argc == 5) { > - coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > - coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > - } > +/* not used, yet !?? indeed > + coerce[k].fromscale= getVarConstant(mb,getArg(p,1)).val.ival; > + coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > + coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > */ > } > #endif > +/* > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == dblRef > && p->retc == 1 > @@ -160,7 +161,7 @@ OPTcoercionImplementation(Client cntxt,M > || ( p->argc == 3 > && isVarConstant(mb,getArg(p,1)) > && getArgType(mb,p,1) == TYPE_int > - /* to-scale == 0, i.e., no scale change */ > + //to-scale == 0, i.e., no scale change > && *(int*) getVarValue(mb, getArg(p,1)) == 0 ) ) ) { > k = getArg(p,0); > coerce[k].pc= i; > @@ -168,8 +169,9 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].src= getArg(p,1 + (p->argc ==3)); > coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > } > - coercionOptimizerAggrStep(mb, i, coerce); > - coercionOptimizerCalcStep(mb, i, coerce); > +*/ > + coercionOptimizerAggrStep(cntxt,mb, i, coerce); > + coercionOptimizerCalcStep(cntxt,mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > actions += k; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 6

Re: MonetDB: default - Minor textual changes to remote tests
by Stefan Manegold 20 Mar '15

20 Mar '15

> Changeset: ca15cab457b8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ca15cab457b8 > Modified Files: > monetdb5/modules/mal/Tests/All > monetdb5/modules/mal/Tests/remote07.mal > monetdb5/modules/mal/Tests/remote88.mal > Branch: default > Log Message: > > Minor textual changes to remote tests > > > diffs (104 lines): > > diff --git a/monetdb5/modules/mal/Tests/All b/monetdb5/modules/mal/Tests/All > --- a/monetdb5/modules/mal/Tests/All > +++ b/monetdb5/modules/mal/Tests/All > @@ -36,6 +36,7 @@ remote02 > HAVE_GEOM?remote03 > remote04 > remote06 > +# should be run as 'mclient -d demo -l mal remote07.mal For what it's worth: MAL tests with extension ".malC" (instead of ".mal") are automatically run trough "mclient -lmal ..." (instead of "mserver5 ...") by Mtest.py; cf., dev.monetdb.org/hg/MonetDB/file/tip/testing/README i.e., `hg rename monetdb5/modules/mal/Tests/remote07.mal monetdb5/modules/mal/Tests/remote07.malC` might be considered ... Stefan > remote08 > remote09 > remote11 [...] -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Avoid coercion in aggr.subavg
by Stefan Manegold 16 Mar '15

16 Mar '15

According to Mtest, this makes TPCh Q1 return a wrong result. I suspect because this now indeed removes a scale-changing cast. Stefan ----- Original Message ----- > Changeset: 9779801a1482 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9779801a1482 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Avoid coercion in aggr.subavg > Another example where we can avoid coercions in most cases > > > diffs (43 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -84,6 +84,24 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > return; > } > > +static void > +coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +{ > + InstrPtr p = getInstrPtr(mb,i); > + int r, k; > + > + if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > + if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > + return; > + > + r = getColumnType(getVarType(mb, getArg(p,0))); > + k = getArg(p,1); > + // check the digits/scale > + if( r == TYPE_dbl && coerce[k].src ) > + getArg(p,1) = coerce[getArg(p,1)].src; > + return; > +} > + > int > OPTcoercionImplementation(Client cntxt,MalBlkPtr mb, MalStkPtr stk, InstrPtr > pci) > { > @@ -117,6 +135,14 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > } > #endif > + if ( getModuleId(p) == batcalcRef && getFunctionId(p) == dblRef && p->retc > == 1 && ((p->argc == 3 && isVarConstant(mb,getArg(p,1))) || p->argc ==2) ){ > + k = getArg(p,0); > + coerce[k].pc= i; > + coerce[k].totype= TYPE_dbl; > + coerce[k].src= getArg(p,1 + (p->argc ==3)); > + coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > + } > + coercionOptimizerAggrStep(mb, i, coerce); > coercionOptimizerCalcStep(mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - coercionOptimizerCalcStep(): fixing changeset...
by Stefan Manegold 16 Mar '15

16 Mar '15

IMHO this new coersion optimization as introduced in changeset 5786f1be12bb only works correctly for decimal casts (coersions) that do not change the scale (number of decimals), i.e., when the first and last argument (i.e., input and output number of decimals, respectively) are identical. This sanity check is currently missing in the code. Stefan ----- Original Message ----- > Changeset: 289a5a038774 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=289a5a038774 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > coercionOptimizerCalcStep(): fixing changeset 71bfaf7e9841: also handle > TYPE_hge > to re-activate the intended behavior of changeset 5786f1be12bb > > OPEN ISSUE / QUESTION: > > This optimizer now (only) handles (i.e., removes) decimal (up-)casts, > but no plain/pure integer casts at all. > > (a) Shouldn't we also handle (remove) plain/pure integer casts? > > (b) Can we always safely remove decimal (up-)casts without altering > semantics? > > > diffs (13 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -65,6 +65,9 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > case TYPE_sht: > case TYPE_int: > case TYPE_lng: > +#ifdef HAVE_HGE > + case TYPE_hge: > +#endif > break; > case TYPE_dbl: > case TYPE_flt: > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

About the Join Algorithm in MonetDB
by Zhiwei Fan 13 Mar '15

13 Mar '15

Dear Developers of MonetDB, ?I really appreciate your excellent work on MonetDB. I am a student in University of Wisconsin Madison and now I am doing research on processing machine learning in the column-oriented Database since we have found a way that could possibly improve the performance of learning process if we use data stored in column rather than data stored in row. For the research purpose, we need to know the detail of the join algorithm in column-store database in order to analyze the cost of the process. And monetDB, as we know, has done a very good job in this part, so I hope you could tell us about the details of the join algorithms implemented in the monetDB, that would be really helpful and we would really appreciate it! Thanks. Best Regards Zhiwei Fan ?

1 0

MAL property for thread-unsafe function
by Roberto Cornacchia 05 Mar '15

05 Mar '15

Hello, I have a function definition command batstemmer.stem(terms:bat[:oid,:str], stemmer_name:str):bat[:oid,:str] address CMDbatstem comment "Wrapper for snowball stemmer"; which internally uses the snowball stemmer (http://snowball.tartarus.org/) When the bat to be stemmed is large enough, mitosis will split it into chunks and call the function "stem" on each chunk, possibly in parallel. Problem is, the snowball stemmer implementation appears to be thread-unsafe, which causes a SIGSEGV. Indeed, using the no_mitosis_pipe solves the issue. However, this solution is suboptimal. Another solution I found is to mark the mal signature as {unsafe}. This works, although it does something a bit silly: it splits the table into chunks, then repacks everything, and finally runs my function on the re-packed bat (basically wasting effort on a useless split + repack). Now, my question is: is there a more focussed property to use? {unsafe} implies thread-unsafe, but it is actually stronger than that. For example, it also implies that there might be side-effects. Therefore, the result cannot be recycled. In my case, instead, the result is perfectly safe to be reused. Thanks for any tip. Roberto

2 2