- developers-list - monetdb.org

MonetDB, data loss in restart server
by Luciano Sasso 02 Apr '15

02 Apr '15

Hello, I have used the MonetDB in production for some time, and I have nothing to complain about, are tables with more than 30 columns and 30 million records, do analysis through cubes in Pentaho / Saiku. Have restarted the server several times, and never had problems with loss of data.But today, after performing a query that was consuming a lot of machine, stopped the database, and when I started, data from some columns were blank. How can it, will the data were still uncommitted? when performing a query the MonetDB changes the data or reorganized? which may have occurred, I am concerned. -- Luciano Sasso Vieira Gestor - Inteligência de Negócios luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 9706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

2 1

Re: MonetDB: default - Defense against non-existing MAL primitives
by Stefan Manegold 20 Mar '15

20 Mar '15

FYI, test clients/R/Tests/survey.R still fails with "Server says '!TypeException:user.s94_2[40]:'batcalc.*' undefined in: X_168:bat[:oid,:hge] := batcalc.*(X_146,X_166);'." i.e., the check whether the requested new signature exists does not seem to work, yet. cf., http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=55075:ca15cab457b8&or… Stefan ----- Original Message ----- > Changeset: 4165dc34b2bb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4165dc34b2bb > Modified Files: > monetdb5/mal/mal_resolve.c > monetdb5/mal/mal_resolve.h > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Defense against non-existing MAL primitives > and further limit the rewrite to the simpliest > and most expensive case encountered. > > > diffs (244 lines): > > diff --git a/monetdb5/mal/mal_resolve.c b/monetdb5/mal/mal_resolve.c > --- a/monetdb5/mal/mal_resolve.c > +++ b/monetdb5/mal/mal_resolve.c > @@ -115,7 +115,6 @@ findFunctionType(stream *out, Module sco > Symbol s; > InstrPtr sig; > int i, k, unmatched = 0, s1; > - /* int foundbutwrong=0; */ > int polytype[MAXTYPEVAR]; > int returns[256]; > int *returntype = NULL; > @@ -396,7 +395,6 @@ findFunctionType(stream *out, Module sco > } > } > if (s1 < 0) { > - /* if(getSignature(s)->token !=PATTERNsymbol) foundbutwrong++; */ > s = s->peer; > continue; > } > @@ -445,7 +443,6 @@ findFunctionType(stream *out, Module sco > * typed. This should be reflected in the symbol table. > */ > s1 = returntype[0]; /* for those interested */ > - /* foundbutwrong = 0; */ > /* > * If the call refers to a polymorphic function, we clone it > * to arrive at a bounded instance. Polymorphic patterns and > @@ -485,12 +482,6 @@ findFunctionType(stream *out, Module sco > * arguments, but that clashes with one of the target variables. > */ > wrapup: > - /* foundbutwrong has not been changed, commented out code above > - if (foundbutwrong && !silent) { > - showScriptException(out, mb, getPC(mb, p), TYPE, > - "type conflict in assignment"); > - } > - */ > if (returntype && returntype != returns) > GDKfree(returntype); > return -3; > @@ -819,13 +810,17 @@ chkTypes(stream *out, Module s, MalBlkPt > > /* > * Type checking an individual instruction is dangerous, > - * because it ignores data flow and declarations issues. > - * It is only to be used in isolated cases. > + * because it ignores data flow and variable declarations. > */ > -void > +int > chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr p) > { > - typeChecker(out, s, mb, p, FALSE); > + int olderrors= mb->errors; > + int error; > + typeChecker(out, s, mb, p, TRUE); > + error = mb->errors; > + mb->errors = olderrors; > + return error; > } > > void > diff --git a/monetdb5/mal/mal_resolve.h b/monetdb5/mal/mal_resolve.h > --- a/monetdb5/mal/mal_resolve.h > +++ b/monetdb5/mal/mal_resolve.h > @@ -20,7 +20,7 @@ > #define MAXTYPEVAR 10 > > mal_export void chkProgram(stream *out, Module s, MalBlkPtr mb); > -mal_export void chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > +mal_export int chkInstruction(stream *out, Module s, MalBlkPtr mb, InstrPtr > p); > mal_export void chkTypes(stream *out, Module s, MalBlkPtr mb, int silent); > mal_export void typeChecker(stream *out, Module scope, MalBlkPtr mb, > InstrPtr p, int silent); > mal_export int fcnBinder(stream *out, Module scope, MalBlkPtr mb, InstrPtr > p); > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -18,8 +18,9 @@ typedef struct{ > int fromtype; > int totype; > int src; > -/* not used, yet !?? > +/* not used, yet !?? Indeed > int digits; > + int fromscale; > int scale; > */ > } Coercion; > @@ -47,58 +48,60 @@ coercionOptimizerStep(MalBlkPtr mb, int > return 0; > } > > -/* Check coercions for numeric types that can be handled with smaller ones. > +/* Check coercions for numeric types towards :hge that can be handled with > smaller ones. > * For now, limit to +,-,/,*,% hge expressions > - * To be extended to deal with math calls as well. > + * Not every combination may be available in the MAL layer, which calls > + * for a separate type check before fixing it. > + * Superflous coercion statements will be garbagecollected later on in the > pipeline > */ > static void > -coercionOptimizerCalcStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerCalcStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > - int r, a, b; > + int r, a, b, varid; > > + r = getColumnType(getVarType(mb, getArg(p,0))); > +#ifdef HAVE_HGE > + if ( r != TYPE_hge) > + return; > +#endif > if( getModuleId(p) != batcalcRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == plusRef || getFunctionId(p) == minusRef || > getFunctionId(p) == mulRef || getFunctionId(p) == divRef || > *getFunctionId(p) =='%') || p->argc !=3) > return; > > - r = getColumnType(getVarType(mb, getArg(p,0))); > - switch(r){ > - case TYPE_bte: > - case TYPE_sht: > - case TYPE_int: > - case TYPE_lng: > -#ifdef HAVE_HGE > - case TYPE_hge: > -#endif > - break; > - case TYPE_dbl: > - case TYPE_flt: > - /* to be determined */ > - default: > - return; > - } > a = getColumnType(getVarType(mb, getArg(p,1))); > b = getColumnType(getVarType(mb, getArg(p,2))); > - if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) /*digit/scale test as well*/ > + if ( a == r && coerce[getArg(p,1)].src && coerce[getArg(p,1)].fromtype < r > ) > + { > + varid = getArg(p,1); > getArg(p,1) = coerce[getArg(p,1)].src; > - if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) /*digit/scale test as well*/ > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[1] = varid; > + } > + if ( b == r && coerce[getArg(p,2)].src && coerce[getArg(p,2)].fromtype < r > ) > + { > + varid = getArg(p,2); > getArg(p,2) = coerce[getArg(p,2)].src; > + if ( chkInstruction(NULL, cntxt->nspace, mb, p)) > + p->argv[2] = varid; > + } > return; > } > > static void > -coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +coercionOptimizerAggrStep(Client cntxt, MalBlkPtr mb, int i, Coercion > *coerce) > { > InstrPtr p = getInstrPtr(mb,i); > int r, k; > > + (void) cntxt; > + > if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > return; > > r = getColumnType(getVarType(mb, getArg(p,0))); > k = getArg(p,1); > - // check the digits/scale > if( r == TYPE_dbl && coerce[k].src ) > getArg(p,1) = coerce[getArg(p,1)].src; > return; > @@ -124,35 +127,33 @@ OPTcoercionImplementation(Client cntxt,M > if (getModuleId(p) == NULL) > continue; > /* Downscale the type, avoiding hge storage when lng would be sufficient. > - * The code template can be extended to handle other downscale options as > well > */ > #ifdef HAVE_HGE > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == hgeRef > && p->retc == 1 > - && ( p->argc == 2 > - || ( p->argc == 5 > - && isVarConstant(mb,getArg(p,1)) > - && getArgType(mb,p,1) == TYPE_int > - && isVarConstant(mb,getArg(p,3)) > - && getArgType(mb,p,3) == TYPE_int > - && isVarConstant(mb,getArg(p,4)) > - && getArgType(mb,p,4) == TYPE_int > - /* from-scale == to-scale, i.e., no scale change */ > - && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) > getVarValue(mb, getArg(p,4)) ) ) ){ > + && ( p->argc == 5 > + && isVarConstant(mb,getArg(p,1)) > + && getArgType(mb,p,1) == TYPE_int > + && isVarConstant(mb,getArg(p,3)) > + && getArgType(mb,p,3) == TYPE_int > + && isVarConstant(mb,getArg(p,4)) > + && getArgType(mb,p,4) == TYPE_int > + /* from-scale == to-scale, i.e., no scale change */ > + && *(int*) getVarValue(mb, getArg(p,1)) == *(int*) getVarValue(mb, > getArg(p,4)) ) ){ > k = getArg(p,0); > coerce[k].pc= i; > coerce[k].totype= TYPE_hge; > coerce[k].src= getArg(p,2); > coerce[k].fromtype= getColumnType(getArgType(mb,p,2)); > -/* not used, yet !?? > - if (p->argc == 5) { > - coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > - coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > - } > +/* not used, yet !?? indeed > + coerce[k].fromscale= getVarConstant(mb,getArg(p,1)).val.ival; > + coerce[k].digits= getVarConstant(mb,getArg(p,3)).val.ival; > + coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > */ > } > #endif > +/* > if ( getModuleId(p) == batcalcRef > && getFunctionId(p) == dblRef > && p->retc == 1 > @@ -160,7 +161,7 @@ OPTcoercionImplementation(Client cntxt,M > || ( p->argc == 3 > && isVarConstant(mb,getArg(p,1)) > && getArgType(mb,p,1) == TYPE_int > - /* to-scale == 0, i.e., no scale change */ > + //to-scale == 0, i.e., no scale change > && *(int*) getVarValue(mb, getArg(p,1)) == 0 ) ) ) { > k = getArg(p,0); > coerce[k].pc= i; > @@ -168,8 +169,9 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].src= getArg(p,1 + (p->argc ==3)); > coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > } > - coercionOptimizerAggrStep(mb, i, coerce); > - coercionOptimizerCalcStep(mb, i, coerce); > +*/ > + coercionOptimizerAggrStep(cntxt,mb, i, coerce); > + coercionOptimizerCalcStep(cntxt,mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > actions += k; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 6

Re: MonetDB: default - Minor textual changes to remote tests
by Stefan Manegold 20 Mar '15

20 Mar '15

> Changeset: ca15cab457b8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ca15cab457b8 > Modified Files: > monetdb5/modules/mal/Tests/All > monetdb5/modules/mal/Tests/remote07.mal > monetdb5/modules/mal/Tests/remote88.mal > Branch: default > Log Message: > > Minor textual changes to remote tests > > > diffs (104 lines): > > diff --git a/monetdb5/modules/mal/Tests/All b/monetdb5/modules/mal/Tests/All > --- a/monetdb5/modules/mal/Tests/All > +++ b/monetdb5/modules/mal/Tests/All > @@ -36,6 +36,7 @@ remote02 > HAVE_GEOM?remote03 > remote04 > remote06 > +# should be run as 'mclient -d demo -l mal remote07.mal For what it's worth: MAL tests with extension ".malC" (instead of ".mal") are automatically run trough "mclient -lmal ..." (instead of "mserver5 ...") by Mtest.py; cf., dev.monetdb.org/hg/MonetDB/file/tip/testing/README i.e., `hg rename monetdb5/modules/mal/Tests/remote07.mal monetdb5/modules/mal/Tests/remote07.malC` might be considered ... Stefan > remote08 > remote09 > remote11 [...] -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Avoid coercion in aggr.subavg
by Stefan Manegold 16 Mar '15

16 Mar '15

According to Mtest, this makes TPCh Q1 return a wrong result. I suspect because this now indeed removes a scale-changing cast. Stefan ----- Original Message ----- > Changeset: 9779801a1482 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9779801a1482 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > Avoid coercion in aggr.subavg > Another example where we can avoid coercions in most cases > > > diffs (43 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -84,6 +84,24 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > return; > } > > +static void > +coercionOptimizerAggrStep(MalBlkPtr mb, int i, Coercion *coerce) > +{ > + InstrPtr p = getInstrPtr(mb,i); > + int r, k; > + > + if( getModuleId(p) != aggrRef || getFunctionId(p) == 0) return; > + if( ! (getFunctionId(p) == subavgRef ) || p->argc !=6) > + return; > + > + r = getColumnType(getVarType(mb, getArg(p,0))); > + k = getArg(p,1); > + // check the digits/scale > + if( r == TYPE_dbl && coerce[k].src ) > + getArg(p,1) = coerce[getArg(p,1)].src; > + return; > +} > + > int > OPTcoercionImplementation(Client cntxt,MalBlkPtr mb, MalStkPtr stk, InstrPtr > pci) > { > @@ -117,6 +135,14 @@ OPTcoercionImplementation(Client cntxt,M > coerce[k].scale= getVarConstant(mb,getArg(p,4)).val.ival; > } > #endif > + if ( getModuleId(p) == batcalcRef && getFunctionId(p) == dblRef && p->retc > == 1 && ((p->argc == 3 && isVarConstant(mb,getArg(p,1))) || p->argc ==2) ){ > + k = getArg(p,0); > + coerce[k].pc= i; > + coerce[k].totype= TYPE_dbl; > + coerce[k].src= getArg(p,1 + (p->argc ==3)); > + coerce[k].fromtype= getColumnType(getArgType(mb,p,1 + (p->argc ==3))); > + } > + coercionOptimizerAggrStep(mb, i, coerce); > coercionOptimizerCalcStep(mb, i, coerce); > if (getModuleId(p)==calcRef && p->argc == 2) { > k= coercionOptimizerStep(mb, i, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - coercionOptimizerCalcStep(): fixing changeset...
by Stefan Manegold 16 Mar '15

16 Mar '15

IMHO this new coersion optimization as introduced in changeset 5786f1be12bb only works correctly for decimal casts (coersions) that do not change the scale (number of decimals), i.e., when the first and last argument (i.e., input and output number of decimals, respectively) are identical. This sanity check is currently missing in the code. Stefan ----- Original Message ----- > Changeset: 289a5a038774 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=289a5a038774 > Modified Files: > monetdb5/optimizer/opt_coercion.c > Branch: default > Log Message: > > coercionOptimizerCalcStep(): fixing changeset 71bfaf7e9841: also handle > TYPE_hge > to re-activate the intended behavior of changeset 5786f1be12bb > > OPEN ISSUE / QUESTION: > > This optimizer now (only) handles (i.e., removes) decimal (up-)casts, > but no plain/pure integer casts at all. > > (a) Shouldn't we also handle (remove) plain/pure integer casts? > > (b) Can we always safely remove decimal (up-)casts without altering > semantics? > > > diffs (13 lines): > > diff --git a/monetdb5/optimizer/opt_coercion.c > b/monetdb5/optimizer/opt_coercion.c > --- a/monetdb5/optimizer/opt_coercion.c > +++ b/monetdb5/optimizer/opt_coercion.c > @@ -65,6 +65,9 @@ coercionOptimizerCalcStep(MalBlkPtr mb, > case TYPE_sht: > case TYPE_int: > case TYPE_lng: > +#ifdef HAVE_HGE > + case TYPE_hge: > +#endif > break; > case TYPE_dbl: > case TYPE_flt: > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

About the Join Algorithm in MonetDB
by Zhiwei Fan 13 Mar '15

13 Mar '15

Dear Developers of MonetDB, ?I really appreciate your excellent work on MonetDB. I am a student in University of Wisconsin Madison and now I am doing research on processing machine learning in the column-oriented Database since we have found a way that could possibly improve the performance of learning process if we use data stored in column rather than data stored in row. For the research purpose, we need to know the detail of the join algorithm in column-store database in order to analyze the cost of the process. And monetDB, as we know, has done a very good job in this part, so I hope you could tell us about the details of the join algorithms implemented in the monetDB, that would be really helpful and we would really appreciate it! Thanks. Best Regards Zhiwei Fan ?

1 0

MAL property for thread-unsafe function
by Roberto Cornacchia 05 Mar '15

05 Mar '15

Hello, I have a function definition command batstemmer.stem(terms:bat[:oid,:str], stemmer_name:str):bat[:oid,:str] address CMDbatstem comment "Wrapper for snowball stemmer"; which internally uses the snowball stemmer (http://snowball.tartarus.org/) When the bat to be stemmed is large enough, mitosis will split it into chunks and call the function "stem" on each chunk, possibly in parallel. Problem is, the snowball stemmer implementation appears to be thread-unsafe, which causes a SIGSEGV. Indeed, using the no_mitosis_pipe solves the issue. However, this solution is suboptimal. Another solution I found is to mark the mal signature as {unsafe}. This works, although it does something a bit silly: it splits the table into chunks, then repacks everything, and finally runs my function on the re-packed bat (basically wasting effort on a useless split + repack). Now, my question is: is there a more focussed property to use? {unsafe} implies thread-unsafe, but it is actually stronger than that. For example, it also implies that there might be side-effects. Therefore, the result cannot be recycled. In my case, instead, the result is perfectly safe to be reused. Thanks for any tip. Roberto

2 2

Re: MonetDB: default - Progress bar: Ignore comments in trace output
by Ying Zhang 24 Feb '15

24 Feb '15

Hai Hannes, Compiling default branch complains about "‘profiler_querystart’ may be used uninitialized": libtool: compile: gcc -DHAVE_CONFIG_H -I. -I/ufs/zhang/monet/default/clients/mapiclient -I../.. -I/ufs/zhang/monet/default/clients/mapiclient -I../mapilib -I/ufs/zhang/monet/default/clients/mapiclient/../mapilib -I../../common/options -I/ufs/zhang/monet/default/clients/mapiclient/../../common/options -I../../common/stream -I/ufs/zhang/monet/default/clients/mapiclient/../../common/stream -I../R/MonetDB.R/src -I/ufs/zhang/monet/default/clients/mapiclient/../R/MonetDB.R/src -DLIBMCUTIL -O3 -pipe -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_FORTIFY_SOURCE=2 -D_REENTRANT -c /ufs/zhang/monet/default/clients/mapiclient/prompt.c -fPIC -DPIC -o .libs/libmcutil_la-prompt.o In file included from /ufs/zhang/monet/default/clients/mapiclient/profiler-dummy.c:5:0: /ufs/zhang/monet/default/clients/mapiclient/../R/MonetDB.R/src/profiler.c: In function ‘profiler_thread’: /ufs/zhang/monet/default/clients/mapiclient/../R/MonetDB.R/src/profiler.c:219:59 error: ‘profiler_querystart’ may be used uninitialized in this function [-Werror=maybe-uninitialized] if (profiler_msgs_expect > 0 && (profiler_tsms() - profiler_querystart) > 500) { ^ This seems to only happen if I use the --enable-optimize --disable-debug configure options. Can I just initialise profiler_querystart with 0? Thanks! Jennie > On Feb 20, 2015, at 18:39, Hannes Muehleisen <commits(a)monetdb.org> wrote: > > Changeset: cde6ad120a72 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=cde6ad120a72 > Modified Files: > clients/R/MonetDB.R/src/profiler.c > Branch: default > Log Message: > > Progress bar: Ignore comments in trace output > > > diffs (13 lines): > > diff --git a/clients/R/MonetDB.R/src/profiler.c b/clients/R/MonetDB.R/src/profiler.c > --- a/clients/R/MonetDB.R/src/profiler.c > +++ b/clients/R/MonetDB.R/src/profiler.c > @@ -177,6 +177,9 @@ static void* profiler_thread(void* param > recvd = read(profiler_socket, buf, sizeof(buf)); > if (recvd > 0) { > buf[recvd] = 0; > + if (buf[0]== '#') { > + continue; > + } > mapi_line_split(buf, elems, TRACE_NCOLS); > if (strncmp(elems[TRACE_COL_STATEFL], "done", 4) != 0) { > continue; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 2

Re: MonetDB: default - mclient progress meter part one, mal statemen...
by Stefan Manegold 24 Feb '15

24 Feb '15

This fails to compile, at least when BUILDDIR != SOURCEDIR: ======== make[5]: Entering directory `/net/rome.ins.cwi.nl/export/scratch1/manegold/Monet/HG/default/builds/_/clients/mapiclient' /bin/sh ../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../.. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../mapilib -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../mapilib -I../../common/options -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/options -I../../common/stream -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/stream -I../R/MonetDB.R/src/ -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../R/MonetDB.R/src/ -DLIBMCUTIL -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_REENTRANT -c -o libmcutil_la-dump.lo `test -f 'dump.c' || echo '/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/'`dump.c libtool: compile: gcc -DHAVE_CONFIG_H -I. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../.. -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient -I../mapilib -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../mapilib -I../../common/options -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/options -I../../common/stream -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../../common/stream -I../R/MonetDB.R/src/ -I/ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/../R/MonetDB.R/src/ -DLIBMCUTIL -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_REENTRANT -c /ufs/manegold/_/Monet/HG/default/source/MonetDB/clients/mapiclient/dump.c -fPIC -DPIC -o .libs/libmcutil_la-dump.o cc1: error: ../R/MonetDB.R/src/: No such file or directory [-Werror] cc1: all warnings being treated as errors make[5]: *** [libmcutil_la-dump.lo] Error 1 make[5]: Leaving directory `/net/rome.ins.cwi.nl/export/scratch1/manegold/Monet/HG/default/builds/_/clients/mapiclient' ======== Stefan ----- Original Message ----- > Changeset: 18471e6cd453 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=18471e6cd453 > Added Files: > clients/mapiclient/mapisplit-dummy.c > clients/mapiclient/profiler-dummy.c > Modified Files: > clients/mapiclient/Makefile.ag > clients/mapiclient/mclient.c > Branch: default > Log Message: > > mclient progress meter part one, mal statement progress > > > diffs (148 lines): > > diff --git a/clients/mapiclient/Makefile.ag b/clients/mapiclient/Makefile.ag > --- a/clients/mapiclient/Makefile.ag > +++ b/clients/mapiclient/Makefile.ag > @@ -16,7 +16,7 @@ > # All Rights Reserved. > > MTSAFE > -INCLUDES = ../mapilib ../../common/options ../../common/stream > $(READLINE_INCS) > +INCLUDES = ../mapilib ../../common/options ../../common/stream > ../R/MonetDB.R/src/ $(READLINE_INCS) > > lib_mcutil = { > NOINST > @@ -24,7 +24,7 @@ lib_mcutil = { > } > > bin_mclient = { > - SOURCES = mclient.c ReadlineTools.c ReadlineTools.h > + SOURCES = mclient.c ReadlineTools.c ReadlineTools.h mapisplit-dummy.c > profiler-dummy.c > LIBS = libmcutil ../mapilib/libmapi \ > ../../common/stream/libstream \ > $(READLINE_LIBS) \ > diff --git a/clients/mapiclient/mapisplit-dummy.c > b/clients/mapiclient/mapisplit-dummy.c > new file mode 100644 > --- /dev/null > +++ b/clients/mapiclient/mapisplit-dummy.c > @@ -0,0 +1,5 @@ > +/* > +This refers to ../R/MonetDB.R/src/mapisplit.c . We cannot have this file > here, > +as the R connector needs to be able to compile stand-alone. > +*/ > +#include "mapisplit.c" > \ No newline at end of file > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -78,6 +78,8 @@ > #define S_ISREG(m) (((m) & S_IFMT) == S_IFREG) > #endif > > +#include "profiler.h" > + > enum modes { > MAL, > SQL, > @@ -1934,14 +1936,13 @@ doFileBulk(Mapi mid, FILE *fp) > } > } > timerResume(); > - > if (hdl == NULL) { > hdl = mapi_query_prep(mid); > CHECK_RESULT(mid, hdl, buf, continue, buf); > } > > assert(hdl != NULL); > - > + profiler_arm(); > mapi_query_part(hdl, buf, length); > CHECK_RESULT(mid, hdl, buf, continue, buf); > > @@ -2722,6 +2723,8 @@ doFile(Mapi mid, const char *file, int u > > if (hdl == NULL) { > timerStart(); > + profiler_arm(); > + > hdl = mapi_query_prep(mid); > CHECK_RESULT(mid, hdl, buf, continue, buf); > } else > @@ -2861,6 +2864,7 @@ usage(const char *prog, int xit) > fprintf(stderr, " -w nr | --width=nr for pagination\n"); > fprintf(stderr, " -D | --dump create an SQL dump\n"); > fprintf(stderr, " -N | --inserts use INSERT INTO > statements when dumping\n"); > + fprintf(stderr, " -P | --progress show progress bar\n"); > fprintf(stderr, "The file argument can be - for stdin\n"); > exit(xit); > } > @@ -2887,6 +2891,7 @@ main(int argc, char **argv) > int interactive = 0; > int has_fileargs = 0; > int option_index = 0; > + int progress = 0; > int settz = 1; > int autocommit = 1; /* autocommit mode default on */ > struct stat statb; > @@ -2912,6 +2917,7 @@ main(int argc, char **argv) > {"pager", 1, 0, '|'}, > #endif > {"port", 1, 0, 'p'}, > + {"progress", 0, 0, 'P'}, > {"rows", 1, 0, 'r'}, > {"statement", 1, 0, 's'}, > #if 0 > @@ -2970,7 +2976,7 @@ main(int argc, char **argv) > #if 0 > "t" > #endif > - "w:r:p:s:Xu:vzH?", > + "w:r:p:s:Xu:vzHP?", > long_options, &option_index)) != -1) { > switch (c) { > case 0: > @@ -3114,6 +3120,9 @@ main(int argc, char **argv) > case 'z': > settz = 0; > break; > + case 'P': > + progress = 1; > + break; > case '?': > /* a bit of a hack: look at the option that the > * current `c' is based on and see if we recognize > @@ -3226,6 +3235,16 @@ main(int argc, char **argv) > } > } > > + // switch on progress bars > + if (mode == SQL && progress) { > + char* buf = malloc(100); > + int port = profiler_start(); > + sprintf(buf, "CALL profiler_openstream('127.0.0.1', %d)", port); > + mapi_query(mid, buf); > + sprintf(buf, "CALL profiler_stethoscope(0)"); > + mapi_query(mid, buf); > + } > + > /* give the user a welcome message with some general info */ > if (!has_fileargs && command == NULL && isatty(fileno(stdin))) { > char *lang; > @@ -3296,6 +3315,7 @@ main(int argc, char **argv) > /* execute from command-line, need interactive to know whether > * to keep the mapi handle open */ > timerStart(); > + profiler_arm(); > c = doRequest(mid, command); > timerEnd(); > } > diff --git a/clients/mapiclient/profiler-dummy.c > b/clients/mapiclient/profiler-dummy.c > new file mode 100644 > --- /dev/null > +++ b/clients/mapiclient/profiler-dummy.c > @@ -0,0 +1,5 @@ > +/* > +This refers to ../R/MonetDB.R/src/profiler.c . We cannot have this file > here, > +as the R connector needs to be able to compile stand-alone. > +*/ > +#include "profiler.c" > \ No newline at end of file > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

About merovingian and the MySQL protocol
by Fabian Groffen 23 Feb '15

23 Feb '15

Hiya all, A long time ago, I examined the MySQL client-server protocol, to understand why it is performing so well. This resulted in the idea (back then) to create a MySQL server implementation in merovingian, that proxied commands to mserver/mapi calls. This to possibly gain from performance benefits, but obviously more to open up the world of MySQL clients that exist to MonetDB. A minimalistic start was added to the code [1], but nothing more than that ever emerged. Years later, I ended up at a similar quest, one where I had to implement the MySQL server protocol and perform the rituals to make a MySQL client believe it is dealing with a proper server [2]. Objective is different here (some will recognise merovingian's multiplex-funnel feature), but the codepath is largely re-usable for the purpose of the above. Being rusty with the MonetDB code-base, not having followed any of its developments lately, whilst on this island in the sun, I'm wondering if it makes sense to implement the first at all, or whether it is better to simply remove the if-deffed code, for it is incomplete. Opinions? Best, Fabian [1] http://dev.monetdb.org/hg/MonetDB/file/fb8e50da2917/tools/merovingian/daemo… [2] https://github.com/grobian/mysqlpq

1 0