- developers-list - monetdb.org

histogram
by Roberto Cornacchia 21 Mar '16

21 Mar '16

In a C UDF, looping on a [:oid,:str:] bat, I'm tokenizing each string tail into a str array: BATloop(..) { str *token_array = /* create a str array by tokenizing the str tail of this BUN */ /* append a histogram of token_array to result */ } The token array is expected to be in the range of 10 - 10K short strings (1 to 10 bytes). Question: To get the histogram, I'd turn the token_array into a BAT b, then use BATgroup: BATgroup(&gn, NULL, &hn, b, NULL, NULL, NULL); Do you see a more efficient way? Is there a group/histogram primitive implemented that works directly on arrays rather than BATs? Thanks, Roberto

1 0

Error after compiling monetdb
by Shmagi Kavtaradze 18 Mar '16

18 Mar '16

I am trying to write a C User Defined Function. It works in Neatbeans. What I want is, to take value from cells, insert them in text or char array and compare each corresponding array position to each other. As values I will have "1"s and "0s". E.g "101010100011". THe code I wrote in udf.c: /* Comparing bag of words */ char BitAnd(char Cell1[], char Cell2[]) { Cell1[10000]; Cell2[10000]; int counter; float result; int strlen = sizeof(Cell1)/sizeof(Cell1[0])-1; int i; for (i=0; i < strlen; i++) { if (Cell1[i] == Cell2[i]) { counter++; } } result = (float)counter / strlen; } /* MAL wrapper */ char UDFBitAnd(char Cell1[], char Cell2[]) { /* assert calling sanity */ assert(Cell1[] != NULL && Cell2[] != NULL); return MAL_SUCCEED; } In udf.mal I have: command BitAnd(Cell1:chr,Cell2:chr):flt address UDFBitAnd comment "Compare bag of words"; In 80_udf.sql: -- Comparing bag of words create function BitAnd(Cell1[] char, Cell2[] char) returns float external name udf.BitAnd; I compiled Monetdb, but when I try to connect with "monetdb" to db it gives me error. In the log file I found: MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^Type identifier expected MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^',' expected MSG merovingian[8780]: database 'voc' (18857) has exited with exit status 0 MSG merovingian[8780]: database 'voc' has shut down ERR merovingian[8780]: client error: database 'voc' started up, but failed to open up a communication channel

1 0

Adapting BATs
by Till Kolditz 18 Mar '16

18 Mar '16

Hi there, I'm just new to the list, so hello to everyone :-) I'm currently trying to find out how to adapt BATs in MonetDB for a scientific project. What I want to do is to add an encoding to data as some kind of a native DB feature. I.e. that a BAT stores data in some encoded fashion to detect certain kinds of errors that alter data in memory. By that, talking e.g. about an int BAT (column), the data width would increase e.g. from 4 to 5 Bytes (just as an example) where the additional byte stores the coding information. As I see it there are for a first shot two possibilities to do that: 1) Take one atom (e.g. int) and adapt all BAT / atom access code which touches int BATs. 2) Add some kind of new atom which is handled in a specific way. Since I have just started looking into MonetDB I now hope for a few hints to decide whether one of these approaches is sensible and whether there is maybe a simpler way. Thanks a lot in advance for any helpful comments :-) Best Till Kolditz -- Dipl.-Inf. Till Kolditz wiss. Mitarbeiter Technische Universität Dresden Fakultät Informatik Lehrstuhl Datenbanken 01062 Dresden Tel.: +49 (351) 463-38077 Fax: +49 (351) 463-38259 E-Mail: Till.Kolditz(a)tu-dresden.de jExam Group www: http://www.jexam.de Mail: till.kolditz(a)jexam.de

1 0

New Jun2016 stable branch created
by Sjoerd Mullender 14 Mar '16

14 Mar '16

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jun2016. So in each of your stable checkout directories you can do hg pull hg update -rJun2016 to update to this new branch. -- Sjoerd Mullender

1 0

Re: MonetDB: leftmart - trying to fix Windows compilation: do not "i...
by Lefteris 08 Mar '16

08 Mar '16

Stefan, this needs to be inlined because it is used many times. I will deal with compilation issues later on. This is a dev branch and we are running different experiments every night. But with this change if measured times are different I would not know if it is because of other changes or because of the inline statement removal. I would appreciate if you did not make any changes in this branch at the moment. thank you, lefteris On Mon, Mar 7, 2016 at 8:34 PM, Stefan Manegold < commits+Stefan.Manegold=cwi.nl(a)monetdb.org> wrote: > Changeset: ea1aed8beff8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ea1aed8beff8 > Modified Files: > gdk/gdk_bloom.c > Branch: leftmart > Log Message: > > trying to fix Windows compilation: do not "inline" BLOOMask() > > On Windows, compilation fails with > failed invocation: cl -GF -W3 -WX -MD -nologo -Ox -Zi -Oi -I. -I.\.. > -I.\..\..\gdk -I..\common\options -I.\..\..\gdk\..\common\options > -I..\common\stream -I.\..\..\gdk\..\common\stream -I..\common\utils > -I.\..\..\gdk\..\common\utils -I.\..\..\buildtools\conf -DHAVE_ICONV > -IC:\Libraries\iconv-1.11.1.win64-vs2010\include -DHAVE_LIBATOMIC_OPS > -IC:\Libraries\atomic_ops-7.4.2 -DNDEBUG -LD -Felibbat.dll gdk_batop.obj > gdk_select.obj gdk_search.obj gdk_hash.obj gdk_tm.obj gdk_orderidx.obj > gdk_align.obj gdk_bbp.obj gdk_heap.obj gdk_utils.obj gdk_atoms.obj > gdk_qsort.obj gdk_ssort.obj gdk_storage.obj gdk_bat.obj gdk_delta.obj > gdk_cross.obj gdk_system.obj gdk_value.obj gdk_posix.obj gdk_logger.obj > gdk_sample.obj gdk_calc.obj gdk_aggr.obj gdk_group.obj gdk_imprints.obj > gdk_bloom.obj gdk_join.obj gdk_project.obj gdk_unique.obj gdk_firstn.obj > libbat.res /link ..\common\options\monet_options.obj > ..\common\stream\libstream.lib ..\common\utils\mutils.obj > ..\common\utils\prompt.obj wsock32.lib Ws2_32.l > ib C:\Libraries\zlib-1.2.8.win64-vs2010\lib\zdll.lib > C:\Libraries\bzip2-1.0.6.win64-vs2010\lib\libbz2.lib psapi.lib > Creating library libbat.lib and object libbat.exp > gdk_join.obj : error LNK2019: unresolved external symbol BLOOMask > referenced in function hashjoin > libbat.dll : fatal error LNK1120: 1 unresolved externals > > > Maybe, the fact that BLOOMask() is "inline"d triggers this problem ... > > > diffs (11 lines): > > diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > --- a/gdk/gdk_bloom.c > +++ b/gdk/gdk_bloom.c > @@ -240,7 +240,6 @@ BATbloom(BAT *b) > return GDK_SUCCEED; > } > > -inline > int BLOOMask(BUN v, Bloomfilter *bloom) > { > BUN hv,mv,x,y,z; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

3 2

Re: MonetDB: leftmart - silence some compilers compaining about prin...
by Stefan Manegold 07 Mar '16

07 Mar '16

Hi Lefteris, the portable format string that we ussually use for size_t in MonetDB is "SZFMT" Best, Stefan ----- On Mar 7, 2016, at 2:38 PM, lsidir commits+lsidir=cwi.nl(a)monetdb.org wrote: > Changeset: a76605fb0b33 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a76605fb0b33 > Modified Files: > gdk/gdk_bloom.c > Branch: leftmart > Log Message: > > silence some compilers compaining about printing a size_t as unsigned int or > long > > > diffs (12 lines): > > diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > --- a/gdk/gdk_bloom.c > +++ b/gdk/gdk_bloom.c > @@ -183,7 +183,7 @@ BATbloom(BAT *b) > > ALGODEBUG fprintf(stderr, "#BATbloom(b=%s#" BUNFMT ") %s: " > "create bloom filter: mbits = " BUNFMT ", ratio = " BUNFMT > - ", kfunc = %d, bytes = " BUNFMT "\n", BATgetId(b), > + ", kfunc = %d, bytes = %lu\n", BATgetId(b), > BATcount(b), b->T->heap.filename, > bloom->mbits, bloom->mbits/BATcount(b), > bloom->kfunc, bloom->bytes); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Monetdb database crashes
by Souri Ratnaparkhi 04 Mar '16

04 Mar '16

Hello Team, Hello monetdb team, We have a table with 126 columns and approx. 94000 rows. We are trying to load approx 2500 records from temporary table into main table using query “insert into select” query. The database crashes upon query execution. No specific error is created in log file. When tried with 1500 records, it works. We also created another table with same DDL, that table is empty & inserted all the records from temp successfully without any issue. What could be the reason? Is 126 cols an issue? Please help. We are stuck. (btw: We have machine about 200GB of hard disk & 16 GB RAM (14 GB free)) Thank you [image: Description: Description: Description: Description: logo] * Souri Ratnaparkhi United Software Associates Pvt. Ltd. Pune, India Mob: 8975677255* * Phone (India Lines): (91) 020-25445550 **www.usain.com* <http://www.usain.com/>

2 1

Error delete rows on table
by Luciano Sasso 24 Feb '16

24 Feb '16

Hi, I have two tables , deleted records of the two tables , then tried to delete again in Table 1 , an error occurred ( could not get print) , and in Table 2 records erased, again show in Table 2. Everything was running on a SQL console ( NetBeans ) and auto commit on. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow query
by Luciano Sasso 24 Feb '16

24 Feb '16

Hi, I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, 4 of number type, others string type. A simple "select count(*) from table", run on 90 minutes. Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. While monet run the query, command top, show 42G of virtual memory. Its normal ?, any idea ? thanks. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Re: MonetDB: default - use same optimizer pipe for inlined queries.
by Stefan Manegold 03 Feb '16

03 Feb '16

cwi@achtbaan:~/manegold/source$ hg clone http://dev.monetdb.org/hg/MonetDB/ Het programma ‘hg’ is momenteel niet geïnstalleerd. Vraag uw beheerder om het pakket 'mercurial' te installeren om 'hg' uit te kunnen voeren nou, dan maar wget de tarball - werkt maar: cwi@achtbaan:~/manegold/build$ ../source/MonetDB-11.21.13/configure --enable-optimize --prefix=/home/cwi/manegold/prefix 2>&1 | tee config.out checking build system type... powerpc64le-unknown-linux-gnu checking host system type... powerpc64le-unknown-linux-gnu checking target system type... powerpc64le-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether UID '1001' is supported by ustar format... yes checking whether GID '1001' is supported by ustar format... yes checking how to create a ustar tar archive... gnutar checking for pkg-config... no configure: error: pkg-config is required to configure MonetDB ... jammer ... Stefan ----- On Feb 3, 2016, at 10:45 AM, Niels Nes commits+niels=cwi.nl(a)monetdb.org wrote: > Changeset: 039a2b238d13 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=039a2b238d13 > Modified Files: > sql/backends/monet5/sql_optimizer.c > sql/backends/monet5/sql_optimizer.h > sql/backends/monet5/sql_scenario.c > Branch: default > Log Message: > > use same optimizer pipe for inlined queries. > > > diffs (107 lines): > > diff --git a/sql/backends/monet5/sql_optimizer.c > b/sql/backends/monet5/sql_optimizer.c > --- a/sql/backends/monet5/sql_optimizer.c > +++ b/sql/backends/monet5/sql_optimizer.c > @@ -215,8 +215,8 @@ addOptimizers(Client c, MalBlkPtr mb, ch > addtoMalBlkHistory(mb, "getStatistics"); > } > > -void > -addQueryToCache(Client c) > +str > +optimizeQuery(Client c) > { > MalBlkPtr mb; > backend *be; > @@ -226,13 +226,12 @@ addQueryToCache(Client c) > assert(be && be->mvc); /* SQL clients should always have their state set */ > pipe = getSQLoptimizer(be->mvc); > > - insertSymbol(c->nspace, c->curprg); > trimMalBlk(c->curprg->def); > c->blkmode = 0; > mb = c->curprg->def; > chkProgram(c->fdout, c->nspace, mb); > #ifdef _SQL_OPTIMIZER_DEBUG > - mnstr_printf(GDKout, "ADD QUERY TO CACHE\n"); > + mnstr_printf(GDKout, "Optimize query\n"); > printFunction(GDKout, mb, 0, LIST_MAL_ALL); > #endif > /* > @@ -250,23 +249,34 @@ addQueryToCache(Client c) > if (msg != MAL_SUCCEED) > GDKfree(msg); /* ignore error */ > } > - return; > + return NULL; > } > addOptimizers(c, mb, pipe); > msg = optimizeMALBlock(c, mb); > - if (msg != MAL_SUCCEED) { > - showScriptException(c->fdout, mb, 0, MAL, "%s", msg); > - GDKfree(msg); > - return; > - } > + if (msg) > + return msg; > > /* time to execute the optimizers */ > if (c->debug) > optimizerCheck(c, mb, "sql.baseline", -1, 0); > #ifdef _SQL_OPTIMIZER_DEBUG > - mnstr_printf(GDKout, "ADD optimized QUERY TO CACHE\n"); > + mnstr_printf(GDKout, "End Optimize Query\n"); > printFunction(GDKout, mb, 0, LIST_MAL_ALL); > #endif > + return NULL; > +} > + > +void > +addQueryToCache(Client c) > +{ > + str msg = NULL; > + > + insertSymbol(c->nspace, c->curprg); > + msg = optimizeQuery(c); > + if (msg != MAL_SUCCEED) { > + showScriptException(c->fdout, c->curprg->def, 0, MAL, "%s", msg); > + GDKfree(msg); > + } > } > > /* > diff --git a/sql/backends/monet5/sql_optimizer.h > b/sql/backends/monet5/sql_optimizer.h > --- a/sql/backends/monet5/sql_optimizer.h > +++ b/sql/backends/monet5/sql_optimizer.h > @@ -12,6 +12,7 @@ > > //#define _SQL_OPTIMIZER_DEBUG > > +sql5_export str optimizeQuery(Client c); > sql5_export void addQueryToCache(Client c); > sql5_export str SQLoptimizer(Client c); > sql5_export void SQLsetAccessMode(Client c); > diff --git a/sql/backends/monet5/sql_scenario.c > b/sql/backends/monet5/sql_scenario.c > --- a/sql/backends/monet5/sql_scenario.c > +++ b/sql/backends/monet5/sql_scenario.c > @@ -1263,17 +1263,20 @@ recompilequery: > > chkTypes(c->fdout, c->nspace, c->curprg->def, TRUE); /* resolve types */ > if (opt) { > + str msg = optimizeQuery(c); > + /* > MalBlkPtr mb = c->curprg->def; > > trimMalBlk(mb); > chkProgram(c->fdout, c->nspace, mb); > addOptimizers(c, mb, "default_pipe"); // TODO change to active pipe! > msg = optimizeMALBlock(c, mb); > + */ > if (msg != MAL_SUCCEED) { > sqlcleanup(m, err); > goto finalize; > } > - c->curprg->def = mb; > + //c->curprg->def = mb; > } > //printFunction(c->fdout, c->curprg->def, 0, LIST_MAL_ALL); > /* we know more in this case than chkProgram(c->fdout, c->nspace, > c->curprg->def); */ > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0