- developers-list - monetdb.org

Re: MonetDB: default - Addition of the candidatelist property
by Sjoerd Mullender 07 Oct '15

07 Oct '15

opt_candidates.c and opt_candidates.h are missing. On 07/10/15 14:33, Martin Kersten wrote: > Changeset: c6e5763417d9 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c6e5763417d9 > Modified Files: > clients/Tests/exports.stable.out > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > monetdb5/mal/mal.h > monetdb5/mal/mal_instruction.c > monetdb5/mal/mal_instruction.h > monetdb5/mal/mal_parser.c > monetdb5/mal/mal_type.h > monetdb5/optimizer/Makefile.ag > monetdb5/optimizer/opt_pipes.c > monetdb5/optimizer/opt_prelude.c > monetdb5/optimizer/opt_prelude.h > monetdb5/optimizer/opt_pushselect.c > monetdb5/optimizer/opt_remoteQueries.c > monetdb5/optimizer/opt_support.h > monetdb5/optimizer/opt_wrapper.c > monetdb5/optimizer/optimizer.mal > sql/backends/monet5/UDF/Tests/udf-fuse.stable.out > sql/backends/monet5/UDF/Tests/udf-reverse.stable.out > sql/backends/monet5/sql.mal > sql/backends/monet5/sql_gencode.c > sql/benchmarks/ssbm/Tests/01-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/02-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/03-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/04-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/05-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/06-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/07-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/08-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/09-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/10-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/11-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/12-explain.stable.out.int128 > sql/benchmarks/ssbm/Tests/13-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/01-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/02-explain.stable.out > sql/benchmarks/tpch/Tests/03-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/04-explain.stable.out > sql/benchmarks/tpch/Tests/05-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/06-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/07-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/08-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/09-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/10-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/11-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/12-explain.stable.out > sql/benchmarks/tpch/Tests/12-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/13-explain.stable.out > sql/benchmarks/tpch/Tests/13-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/14-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/15-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/16-explain.stable.out > sql/benchmarks/tpch/Tests/16-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/17-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/18-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/19-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/20-explain.stable.out.int128 > sql/benchmarks/tpch/Tests/21-explain.stable.out > sql/benchmarks/tpch/Tests/21-explain.stable.out.32bit > sql/benchmarks/tpch/Tests/22-explain.stable.out.int128 > sql/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > sql/test/BugTracker-2010/Tests/LIMIT_OFFSET_big-endian.Bug-2622.stable.out > sql/test/BugTracker-2010/Tests/group-by_ordered_column.Bug-2564.stable.out > sql/test/BugTracker-2010/Tests/group-by_ordered_column.Bug-2564.stable.out.32bit > sql/test/BugTracker-2011/Tests/crash_on_alias.Bug-2798.stable.out.int128 > sql/test/BugTracker-2011/Tests/func_iter_vs_bulk.Bug-2826.stable.out > sql/test/BugTracker-2012/Tests/rewrite_like_into_likesubselect.Bug-3179.stable.out > sql/test/BugTracker-2013/Tests/between.Bug-3259.stable.out.int128 > sql/test/BugTracker-2014/Tests/manifold.Bug-3556.stable.out > sql/test/BugTracker/Tests/explain.SF-1739353.stable.out > sql/test/BugTracker/Tests/jdbc_no_debug.SF-1739356.stable.out > sql/test/Tests/setoptimizer.stable.err > sql/test/Tests/setoptimizer.stable.out > sql/test/Tests/setoptimizer.stable.out.Windows > sql/test/mergetables/Tests/mergequery.stable.out > Branch: default > Log Message: > > Addition of the candidatelist property > MAL variables can have a candidate list flag, which is set by > a cheap optimizer at the start of the pipe. > This property leads to different rendering of the variables. > > > diffs (truncated from 5982 to 300 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1559,6 +1559,7 @@ str MTIMEtzone_tostr(str *s, const tzone > str MTIMEunix_epoch(timestamp *ret); > void OPTaliasRemap(InstrPtr p, int *alias); > int OPTaliasesImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > +int OPTcandidatesImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > int OPTcoercionImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > int OPTcommonTermsImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > int OPTconstantsImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > @@ -2186,6 +2187,7 @@ str ifthenelseRef; > str ilikeRef; > str ilikeselectRef; > str ilikesubselectRef; > +str ilikethetasubselectRef; > str ilikeuselectRef; > void initHeartbeat(void); > void initLibraries(void); > @@ -2257,6 +2259,7 @@ str levenshteinbasic_impl(int *result, s > str likeRef; > str likeselectRef; > str likesubselectRef; > +str likethetasubselectRef; > str likeuselectRef; > str listRef; > str loadLibrary(str modulename, int flag); > @@ -2587,6 +2590,7 @@ str subsliceRef; > str subsortRef; > str subsumRef; > str subthetajoinRef; > +str subuniqueRef; > str sumRef; > int tableProp; > str takeOid(oid id, str *val); > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.err > @@ -30,16 +30,16 @@ stderr of test 'opt_sql_append` in direc > # 22:58:57 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=rome" "--port=36275" > # 22:58:57 > > > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from '\tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: \tmp/xyz > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from 'a:\tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: a:\tmp/xyz > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from '\tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: \tmp/xyz > -MAPI = (monetdb) /var/tmp/mtest-28239/.s.monetdb.30808 > +MAPI = (monetdb) /var/tmp/mtest-21015/.s.monetdb.39072 > QUERY = explain copy into ttt from 'Z:/tmp/xyz'; > ERROR = !COPY INTO: filename must have absolute path: Z:/tmp/xyz > > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out > @@ -55,8 +55,8 @@ Ready. > % .L # table_name > % def # name > % clob # type > -% 491 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > +% 514 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > @@ -89,8 +89,8 @@ end user.s4_1; > % .L # table_name > % def # name > % clob # type > -% 514 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > +% 537 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > diff --git a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > --- a/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > +++ b/monetdb5/extras/mal_optimizer_template/Tests/opt_sql_append.stable.out.single > @@ -55,8 +55,8 @@ Ready. > % .L # table_name > % def # name > % clob # type > -% 491 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > +% 514 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > @@ -87,8 +87,8 @@ end user.s4_1; > % .L # table_name > % def # name > % clob # type > -% 514 # length > -[ "optimizer.inline();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > +% 537 # length > +[ "optimizer.inline();optimizer.candidates();optimizer.remap();optimizer.costModel();optimizer.coercions();optimizer.evaluate();optimizer.aliases();optimizer.pushselect();optimizer.mitosis();optimizer.mergetable();optimizer.deadcode();optimizer.aliases();optimizer.constants();optimizer.commonTerms();optimizer.joinPath();optimizer.reorder();optimizer.deadcode();optimizer.reduce();optimizer.matpack();optimizer.dataflow();optimizer.querylog();optimizer.multiplex();optimizer.generator();optimizer.sql_append();optimizer.garbageCollector();" ] > #explain copy into ttt from '/tmp/xyz'; > % .explain # table_name > % mal # name > diff --git a/monetdb5/mal/mal.h b/monetdb5/mal/mal.h > --- a/monetdb5/mal/mal.h > +++ b/monetdb5/mal/mal.h > @@ -120,7 +120,8 @@ mal_export void mal_exit(void); > #define VAR_CLEANUP 16 > #define VAR_INIT 32 > #define VAR_USED 64 > -#define VAR_DISABLED 128 /* used for comments and scheduler */ > +#define VAR_CLIST 128 /* Candidate list variable */ > +#define VAR_DISABLED 256 /* used for comments and scheduler */ > > /* type check status is kept around to improve type checking efficiency */ > #define TYPE_ERROR -1 > diff --git a/monetdb5/mal/mal_instruction.c b/monetdb5/mal/mal_instruction.c > --- a/monetdb5/mal/mal_instruction.c > +++ b/monetdb5/mal/mal_instruction.c > @@ -653,7 +653,7 @@ getVarName(MalBlkPtr mb, int i) > nme = mb->var[i]->name; > > if (nme == 0 || *nme =='_') { > - snprintf(buf, PATHLENGTH, "%c_%d", REFMARKER, mb->var[i]->tmpindex); > + snprintf(buf, PATHLENGTH, "%c_%d", refMarker(mb,i), mb->var[i]->tmpindex); > nme = mb->var[i]->name = GDKstrdup(buf); > } > return nme; > diff --git a/monetdb5/mal/mal_instruction.h b/monetdb5/mal/mal_instruction.h > --- a/monetdb5/mal/mal_instruction.h > +++ b/monetdb5/mal/mal_instruction.h > @@ -86,6 +86,10 @@ > #define setVarConstant(M,I) ((M)->var[I]->flags |= VAR_CONSTANT) > #define isVarConstant(M,I) ((M)->var[I]->flags & VAR_CONSTANT) > > +#define clrVarCList(M,I) ((M)->var[I]->flags &= ~VAR_CLIST) > +#define setVarCList(M,I) ((M)->var[I]->flags |= VAR_CLIST) > +#define isVarCList(M,I) ((M)->var[I]->flags & VAR_CLIST) > + > #define getVarConstant(M,I) ((M)->var[I]->value) > #define getVarValue(M,I) VALget(&(M)->var[I]->value) > > diff --git a/monetdb5/mal/mal_parser.c b/monetdb5/mal/mal_parser.c > --- a/monetdb5/mal/mal_parser.c > +++ b/monetdb5/mal/mal_parser.c > @@ -116,7 +116,9 @@ initParser(void) > static int > idLength(Client cntxt) > { > - str s, t; > + str s,t; > + int len = 0; > + > skipSpace(cntxt); > s = CURRENT(cntxt); > t = s; > @@ -128,9 +130,15 @@ idLength(Client cntxt) > s[0] = REFMARKER; > /* prepare escape of temporary names */ > s++; > - while (idCharacter2[(int) (*s)]) > + while (len < IDLENGTH && idCharacter2[(int) (*s)]){ > s++; > - return (int) (s - t); > + len++; > + } > + if( len == IDLENGTH) > + // skip remainder > + while (idCharacter2[(int) (*s)]) > + s++; > + return (int) (s-t);; > } > > /* Simple type identifiers can not be marked with a type variable. */ > diff --git a/monetdb5/mal/mal_type.h b/monetdb5/mal/mal_type.h > --- a/monetdb5/mal/mal_type.h > +++ b/monetdb5/mal/mal_type.h > @@ -15,6 +15,8 @@ > #define malVARG " malVARG" > #define TMPMARKER '_' > #define REFMARKER 'X' > +#define REFMARKERC 'C' > +#define refMarker(M,I) (isVarCList(M,I)? 'C':'X') > > #define newBatType(H,T) (1<<16 | (((TYPE_oid & 0377) <<8) | (T & 0377) )) > #define isaBatType(X) ((1<<16) & (X) && (X)!= TYPE_any) > diff --git a/monetdb5/optimizer/Makefile.ag b/monetdb5/optimizer/Makefile.ag > --- a/monetdb5/optimizer/Makefile.ag > +++ b/monetdb5/optimizer/Makefile.ag > @@ -21,6 +21,7 @@ lib_optimizer = { > opt_aliases.c opt_aliases.h \ > opt_coercion.c opt_coercion.h \ > opt_commonTerms.c opt_commonTerms.h \ > + opt_candidates.c opt_candidates.h \ > opt_constants.c opt_constants.h \ > opt_costModel.c opt_costModel.h \ > opt_dataflow.c opt_dataflow.h \ > diff --git a/monetdb5/optimizer/opt_pipes.c b/monetdb5/optimizer/opt_pipes.c > --- a/monetdb5/optimizer/opt_pipes.c > +++ b/monetdb5/optimizer/opt_pipes.c > @@ -46,6 +46,7 @@ static struct PIPELINES { > */ > {"minimal_pipe", > "optimizer.inline();" > + "optimizer.candidates();" > "optimizer.remap();" > "optimizer.deadcode();" > "optimizer.multiplex();" > @@ -63,6 +64,7 @@ static struct PIPELINES { > */ > {"default_pipe", > "optimizer.inline();" > + "optimizer.candidates();" > "optimizer.remap();" > "optimizer.costModel();" > "optimizer.coercions();" > @@ -99,6 +101,7 @@ static struct PIPELINES { > {"no_mitosis_pipe", > "optimizer.inline();" > "optimizer.remap();" > + "optimizer.candidates();" > "optimizer.costModel();" > "optimizer.coercions();" > "optimizer.evaluate();" > @@ -132,6 +135,7 @@ static struct PIPELINES { > */ > {"sequential_pipe", > "optimizer.inline();" > + "optimizer.candidates();" > "optimizer.remap();" > "optimizer.costModel();" > "optimizer.coercions();" > diff --git a/monetdb5/optimizer/opt_prelude.c b/monetdb5/optimizer/opt_prelude.c > --- a/monetdb5/optimizer/opt_prelude.c > +++ b/monetdb5/optimizer/opt_prelude.c > @@ -114,10 +114,10 @@ str kdifferenceRef; > str languageRef; > str leftfetchjoinRef; > str leftfetchjoinPathRef; > -str likeselectRef; > -str ilikeselectRef; > -str likeuselectRef; > -str ilikeuselectRef; > +str likesubselectRef; > +str likethetasubselectRef; > +str ilikesubselectRef; > +str ilikethetasubselectRef; > str likeRef; > str ilikeRef; > str not_likeRef; > @@ -211,6 +211,7 @@ str subavgRef; > str subsortRef; > str takeRef; > str not_uniqueRef; > +str subuniqueRef; > str unlockRef; > str unpackRef; > str unpinRef; > @@ -357,10 +358,8 @@ void optimizerInit(void) > languageRef= putName("language",8); > leftfetchjoinRef = putName("leftfetchjoin",13); > leftfetchjoinPathRef = putName("leftfetchjoinPath",17); > - likeselectRef = putName("like_select",11); > - ilikeselectRef = putName("ilike_select",12); > - likeuselectRef = putName("like_uselect",12); > - ilikeuselectRef = putName("ilike_uselect",13); > + likesubselectRef = putName("likesubselect",13); > + ilikesubselectRef = putName("ilikesubselect",14); > listRef = putName("list",4); > likeRef = putName("like",4); > ilikeRef = putName("ilike",5); > @@ -455,6 +454,7 @@ void optimizerInit(void) > takeRef= putName("take",5); > timestampRef = putName("timestamp", 9); > not_uniqueRef= putName("not_unique",10); > + subuniqueRef= putName("subunique",9); > unlockRef= putName("unlock",6); > unpackRef = putName("unpack",6); > unpinRef = putName("unpin",5); > @@ -462,7 +462,9 @@ void optimizerInit(void) > subselectRef = putName("subselect",9); > thetasubselectRef = putName("thetasubselect",14); > likesubselectRef = putName("likesubselect",13); > + likethetasubselectRef = putName("likethetasubselect",18); > ilikesubselectRef = putName("ilikesubselect",14); > + ilikethetasubselectRef = putName("ilikethetasubselect",19); > vectorRef = putName("vector",6); > zero_or_oneRef = putName("zero_or_one",11); > userRef = putName("user",4); > diff --git a/monetdb5/optimizer/opt_prelude.h b/monetdb5/optimizer/opt_prelude.h > --- a/monetdb5/optimizer/opt_prelude.h > +++ b/monetdb5/optimizer/opt_prelude.h > @@ -117,10 +117,6 @@ opt_export str ilikeselectRef; > opt_export str likeuselectRef; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

MonetDB type system documentation
by Stefan Manegold 07 Oct '15

07 Oct '15

Dear all, since there is ever so often confusion about the (admittedly not entirely trivial) type system of MonetDB, and about which type to use for what to ensure correct semantics and portability, I felt free to collect some of the crucial information on the MonetDB wiki. Under https://www.monetdb.org/wiki/Main_Page following links "3.1 Software development" and then "MonetDB type system" gets you to https://www.monetdb.org/wiki/MonetDB_type_system I hope this helps to clarify some details. In particular mind the "Type Rules" given there (with any of your MonetDB-related code that you eventually want to see in some future MonetDB release). Your cooperation is highly appreciated! In case you have any questions about correct and proper type usage in MonetDB, please do not hesitate to ask Sjoerd or myself (preferably via the MonetDB developers mailing list, if not in person). Enjoy! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Getting a column type for result column
by Dimitar Nedev 03 Oct '15

03 Oct '15

Dear co-developers, How does one get the type of a column given a 'res_col'. I mean a humanly readable one, in other words get the value that determines the type and how they map to data types. Cheers, Dimitar

2 2

Server crash when trying convert timestamp to string with incorrect format
by Sherzod Mutalov 01 Oct '15

01 Oct '15

Hello Developers, MonetDB server crashes when trying to execute timestamp_to_str function with incorrect format string. SQL: SELECT timestamp_to_str(current_timestamp, '%Q'); I send bug report, https://www.monetdb.org/bugzilla/show_bug.cgi?id=3816 But, also I need QUARTER extraction from date values by SQL server, like "extract(quarter from date '2015-10-01')" = 4 Now, I am using this "((extract(month from {0}) - 1) / 3 + 1)" -- Best regards, Sherzod mailto:shmutalov@gmail.com

1 0

Some annoying bugs with date and number operations
by Sherzod Mutalov 01 Oct '15

01 Oct '15

Hello Developers, Firstly, I must say thank you for releasing MonetDb Jul2015. Secondly, Problem 1: Following query crashes the mserver5: SELECT ~0; -- here can be any number or numeric column There is NULL dereference problem in rel_select.c file at line 3915 (rel_binop function). dl->next->next is null, but rel_value_exp receives dl->next->next->data.sym as argument. How correctly use bitwise NOT operator with SQL? Problem 2: Following queries produces incorrect results: SELECT ( (current_timestamp - str_to_timestamp('2015-09-30', '%Y-%m-%d'))/86400 , BIGINT); SELECT CONVERT( CONVERT( (current_timestamp - str_to_timestamp('2015-09-30', '%Y-%m-%d')) , BIGINT) /86400 , BIGINT); SELECT (current_timestamp - str_to_timestamp('2015-09-30', '%Y-%m-%d'))/86400.0; Server info: # MonetDB 5 server v11.21.5 "Jul2015" # Serving database 'demo', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 15.916 GiB available main-memory. -- Best regards, Sherzod mailto:shmutalov@gmail.com

2 2

Inserting in 'raw' mode??
by Vijay Krishna 15 Sep '15

15 Sep '15

Hi, I was analysing the Insert performance of MonetDB... I am trying to optimize a process where in a single tuple has to be inserted into a table of 1200 columns, which takes about 7.5 seconds presently... The query has values for only 5 columns out of 1200, yet the speed is still the same.. Is it possible to reduce this time even?? While digging on this, I came across this thread.. https://www.monetdb.org/pipermail/users-list/2005-August/000139.html *What is the insert speed relatively to other DBs? > * > Using SQL, with the current version we won't compete very well. We get > much more acceptable speeds in 'raw' mode, where lots of the parsing > overhead is removed. > > Can anyone please help me understand what is the 'raw' mode that is explained in this thread?? I am using Oct-2014 version on a CentOS machine with 128 GB RAM... Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

2 1

Why doesn't MonetDB exploit SIMD feature?
by Xu,Wenjian 15 Sep '15

15 Sep '15

Hi, We know that MonetDB excels in application where the database hot-set can be largely held in main-memory, since the physical operators (e.g., hashjoin, sort) are highly optimized for main-memory. But it is strange to me that these operators do not utilize SIMD which a performance-critical feature of modern CPUs. For example, MonetDB use *timsort* as its underlying stable sort algorithm (do_ssort() in gdk/gdk_ssort_impl.h), but I cannot find any SIMD instructions there. Why doesn't MonetDB exploit SIMD feature? or did I miss something? In paper 'Vectorwise: Beyond Column Stores', the authors claim that X100/Vectorwise use SIMD instructions. So my follow-up question is that whether any of X100/Vectorwise's technologies/features has been integrated into MonetDB's code base. Thank you. Best regards, Xu, Wenjian

1 0

DbVisualizer support for MonetDB
by Robert Koch 14 Sep '15

14 Sep '15

Hi Folks, DbVisualizer is my preferred GUI to MonetDB. DbVis offers capability to write own database profiles for specific DB's. Today I committed a MonetDB database profile for DbVisualizer. https://github.com/dbvisualizer/dbvis-database-profiles/tree/master/monetdb Anyone familiar to DbVis can import and/or improve this profile. There's a free DbVis version, but the Pro offers much more features, which make life easier. BTW, I'm not working for them. :-) Please feel free to write me your desired features by mail or directly into the MD-file inside the monetdb-directory. If possible, I will implement it inte the profile. Best regards Mario

1 0

Announcement: New Jul2015 Feature release of MonetDB suite
by Sjoerd Mullender 09 Sep '15

09 Sep '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015 feature release MonetDB5 Server * Implemented batcalc.min and batcalc.max. Made calc.min and calc.max generic so that no other implementations are needed. * Removed function batcalc.ifthen. Build Environment * Upgraded the license to the Mozilla Public License Version 2.0. Client Package * Added a new output format to mclient: --format=expanded (or -fx). In this format, column values are shown in full and below each other. MonetDB Common * BUNtvar and BUNhvar macros no longer work for TYPE_void columns. * Changed interfaces of a lot of GDK-level functions. When they modify a BAT, don't return the same BAT or NULL, but instead return GDK_SUCCEED or GDK_FAIL. * Changed a bunch of hash-related functions to work on the tail column. The functions that have been changed to work on the tail column are: BAThash, BATprepareHash, HASHgonebad, HASHins, and HASHremove. Java Module * Improved JDBC driver to not throw NullPointerException anymore when calling isNullable() or getPrecision() or getScale() or getColumnDisplaySize() or getSchemaName() or getTableName() or getColumnClassName() on a ResultSetMetaData object. * We now compile the Java classes using the latest Java 1.8 version, and we tell it to compile for Java 1.7. SQL * Differences between time, timestamp, and date values now return properly typed interval types (second or month intervals) instead of integers. * Added support for 128-bit integers (called HUGEINT) on platforms that support this. * Removed support for the mseed library. * Removed support for RDF. * Removed DataCell. It was experimental code that was never enabled. Bug Fixes * 2618: Implement master slave scheme * 3361: constants as MAL function parameters prevent intermediate reuse * 3364: Cannot set role back to a user's default role * 3365: Unable to grant object privileges while having a non-default current_role * 3440: Sequence type errors * 3449: mserver crash on start - Freebsd 10 amd64 * 3476: Cannot revoke object access * 3496: autocompletion table names does not work correctly * 3556: when 2 multiplexed functions in MAL plan, only one is mapped correctly to bat<mod>.function primitive * 3564: Request: add support for postgresql specific scalar function: split_part(string text, delimiter text, field int) * 3625: SIGSEGV because mat array can overrun in opt_mergetable.c * 3627: SQRT in CASE does not work as of Oct2014 * 3654: configure --enable-fits requires extra commands after creating a database instance * 3673: mclient 'expanded' row formatter * 3674: Obfuscate event tracing * 3679: No error is given when incorrect timezone value is specified for a timetz column * 3686: Wrong associativity of multiply/divide * 3702: Filter function not found if created in a user schema * 3708: wrong scoping for cross-schema view references * 3716: alter table my_merge_table drop table t1; crashes mserver5 with Segmentation fault * 3724: Wrong size calculation in BATsubjoin * 3730: SAMPLE function not sampling randomly * 3732: memory leak (of InstrRecord) in opt_mergetable * 3733: "(TRUE OR <Exp>) AND <Exp>" is evaluated incorrectly * 3735: python connection with unix_socket * 3736: crash if mclient disconnects abruptly during a query * 3738: Database inconsistency when using savepoint * 3739: CASE statements do not handle NULLs in the IN () operator properly * 3740: select epoch(now()); types timestamptz(7,0) and bigint(64,0) are not equal * 3742: Division By Zero * 3744: cast to int gives different results for decimal than double * 3747: joins fail in the presence of nulls * 3748: Missing META-INF/services/java.sql.Driver in JDBC package * 3753: Hang on json field parsing * 3754: select from a REMOTE TABLE referring local table crashes mserver5 * 3756: column type conversion sticks to subsequent queries * 3758: "COPY INTO ..." doesn't work, if executing from 2 processes concurrently. * 3759: select data from "sys"."rejects" returns unexpected error and when next select data from "sys"."sessions" causes an assertion failure in mal_interpreter.c:646. * 3760: SQL parser has problem with (position of) a scalar subquery in a SELECT-list * 3761: SQL executor has problem with (position of) a subquery in a SELECT-list. Inconsistent behavior. * 3763: JDBC PreparedStatement for a table with 14 Foreign Keys crashing the Database * 3764: DROPping multiple users causes a crash * 3765: Re-granting a revoked privilege does not work * 3766: VIEW not visible if created under a different schema * 3767: CREATE TEMP TABLE using "LIKE" incorrectly handled * 3769: SIGSEGV when combining a cast/column alias with a UNION ALL view * 3770: combined conditions on declared table in User Defined Function definition crashes monetdb * 3771: Owner of the schema loses rights if assumes the monetdb role. * 3772: Any user can grant a role. * 3773: quantile(col, 0) and quantile(col, 1) fail * 3774: mclient is unaware of merge tables and remote tables * 3775: COPY INTO: Backslash preceding field separator kills import * 3778: Crash on remote table schema mismatch * 3779: server crashes on MAX() on SELECT DISTINCT something combo * 3783: Behavioural change in Jul2015 for 'timestamp minus timestamp' * 3784: Assertion failed: (bn->batCapacity >= cnt), function BAT_scanselect, file gdk_select.c, line 1008. * 3785: sum(interval) causes overflow in conversion to bte * 3786: ResultSet.close() never sends Xclose to free resources * 3787: "b and g must be aligned" from complex group/union query * 3791: HEAPextend: failed to extend to 2420077101056

1 1

Install Monetdb
by HENGKI EFFENDI 01 Sep '15

01 Sep '15

Hi, I try to install Monetdb in Linux RH, when configure there's error message : configure: error: MonetDB/SQL requires bison Please help me, what should i do. Regards Hengki

2 1