- developers-list - monetdb.org

Performance fixes
by Alistair Sutherland 17 Jul '13

17 Jul '13

Hi, I've recently been conducting some performance testing of MonetDB with a variety of servers (from 4 core -8GB RAM to 20 core-64GB RAM) with various data sizes, in an attempt to gain a better understanding of how MonetDB scales. During the performance tests it became obvious that much of the processing was IO bound due to: 1) Columns being unmapped from memory overly aggressively (even when there was plenty of memory still available). 2) The constant mapping/unmapping of memory mapped bat files for intermediate results. I've attached a patch which attempts to address both issues. The first patch to (gdk_utils.c) is to update the memory limit at which the GDKvmtrim kicks in to be 80% memory usage. The second patch (gdk_heap.c) limits the number of mmap/munmap calls via the existing 'heap caching' mechanism which was not working at all! In addition to fixing up the caching code, I've also wired in the heap cache into the case where extending a malloced heap results in a swap over to memory mapped storage. After applying the patches I was seeing approximately 40% performance improvements (your mileage may vary!) If the changes are deemed to be useful, how do I go about getting them accepted into the MonetDB source repository? Thanks, Alistair Sutherland ________________________________

2 2

Re: MonetDB: DVframework - fixed bug. Three subselect instructions r...
by Ying Zhang 17 Jul '13

17 Jul '13

Hai Yagiz, Thanks for fixed bugs! :) Is this check-in supposed to fix some of the demo queries? I tried the query (on a freshly created repo.) SELECT channel, sample_time, sample_value FROM mseed.dataview WHERE station = 'ISK' AND sample_time > '2010-01-12T22:15:00.000' AND sample_time < '2010-01-12T22:18:00.000'; But it still gives the error: From the recursive call: MALException:sql.delta:Object not found Is this expected or did I forget something? Jennie On Jul 16, 2013, at 15:49, Yagiz Kargin wrote: > Changeset: 95bc831eb9fb for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=95bc831eb9fb > Modified Files: > monetdb5/extras/dvf/dvf.c > monetdb5/extras/dvf/opt_dvf.c > Branch: DVframework > Log Message: > > fixed bug. Three subselect instructions resulting from selection predicate on sample_time are now separately taken care of. > > > diffs (152 lines): > > diff --git a/monetdb5/extras/dvf/dvf.c b/monetdb5/extras/dvf/dvf.c > --- a/monetdb5/extras/dvf/dvf.c > +++ b/monetdb5/extras/dvf/dvf.c > @@ -77,7 +77,7 @@ str plan_modifier(Client cntxt, MalBlkPt > BATiter fli; > > int run_dataflow_opt = 0; > - int run_recycle_opt = 1; > + int run_recycle_opt = 0; > > VarRecord low, high; > > @@ -343,8 +343,9 @@ str plan_modifier(Client cntxt, MalBlkPt > /* adjust variable lifetimes */ > malGarbageCollector(mb); > > - /*chkProgram(cntxt->fdout, cntxt->nspace, mb); > - printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > + /*chkProgram(cntxt->fdout, cntxt->nspace, mb);*/ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_EXPLAIN); */ > + /*printFunction(cntxt->fdout,mb, 0, LIST_MAL_ALL); */ > > /* relocate the startpc, the instruction to proceed with the execution. Because it might be changed by the optimizers. */ > for (i = 0; i < limit; i++) > diff --git a/monetdb5/extras/dvf/opt_dvf.c b/monetdb5/extras/dvf/opt_dvf.c > --- a/monetdb5/extras/dvf/opt_dvf.c > +++ b/monetdb5/extras/dvf/opt_dvf.c > @@ -119,6 +119,9 @@ OPTdvfImplementation(Client cntxt, MalBl > int i, limit, which_column, actions = 0; > int last_bind_return_var_id = -1; > int last_data_tid_return_var_id = -1; > + int last_insert_bind_return_var_id = -1; > + int last_subselect_return_var_id = -1; > + int last_update_bind_second_return_var_id = -1; > > stk = stk; //to escape 'unused' parameter error. > pci = pci; //to escape 'unused' parameter error. > @@ -452,35 +455,107 @@ OPTdvfImplementation(Client cntxt, MalBl > actions++; > } > else if((state == 2 || state == 3) && > - ((getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 7 && > p->retc == 2 && > strcmp(getVarConstant(mb, getArg(p, 3)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 4)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 6)).val.ival == 2) || > - (getModuleId(p) == sqlRef && > + getVarConstant(mb, getArg(p, 6)).val.ival == 2) > + { > + last_update_bind_second_return_var_id = getArg(p, 1); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == sqlRef && > getFunctionId(p) == bindRef && > p->argc == 6 && > p->retc == 1 && > strcmp(getVarConstant(mb, getArg(p, 2)).val.sval, getVarConstant(mb, getArg(old[i1], 2)).val.sval) == 0 && > strstr(getVarConstant(mb, getArg(p, 3)).val.sval, data_table_identifier) != NULL && > - getVarConstant(mb, getArg(p, 5)).val.ival == 1))) > + getVarConstant(mb, getArg(p, 5)).val.ival == 1) > + { > + last_insert_bind_return_var_id = getArg(p, 0); > + removeInstruction(mb, p); > + i--; > + actions++; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + p->argc == 8 && > + p->retc == 1 && > + getArg(p, 1) == last_bind_return_var_id) > + { > + last_subselect_return_var_id = getArg(p, 0); > + > + r = newInstruction(mb, ASSIGNsymbol); > + setModuleId(r, algebraRef); > + setFunctionId(r, subselectRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, getArg(p, 3)); > + r = pushArgument(mb, r, getArg(p, 4)); > + r = pushArgument(mb, r, getArg(p, 5)); > + r = pushArgument(mb, r, getArg(p, 6)); > + r = pushArgument(mb, r, getArg(p, 7)); > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 2 || state == 3) && > + getModuleId(p) == algebraRef && > + getFunctionId(p) == subselectRef && > + (p->argc == 7 || p->argc == 8) && > + p->retc == 1 && > + (getArg(p, 1) == last_update_bind_second_return_var_id || getArg(p, 1) == last_insert_bind_return_var_id)) > { > removeInstruction(mb, p); > i--; > actions++; > } > else if((state == 3) && > - (getModuleId(p) == sqlRef && > + getModuleId(p) == sqlRef && > getFunctionId(p) == projectdeltaRef && > p->argc == 6 && > p->retc == 1 && > - last_bind_return_var_id == getArg(p, 2))) > + last_bind_return_var_id == getArg(p, 2)) > + { > + r = newInstruction(mb, ASSIGNsymbol); > + > + if(getArg(p, 1) == last_data_tid_return_var_id) > + { > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + else > + { > + setModuleId(r, algebraRef); > + setFunctionId(r, leftfetchjoinRef); > + r = pushReturn(mb, r, getArg(p, 0)); > + r = pushArgument(mb, r, getArg(p, 1)); > + r = pushArgument(mb, r, last_bind_return_var_id); > + } > + > + insertInstruction(mb, r, i+1); > + removeInstruction(mb, p); > + > + actions += 2; > + } > + else if((state == 3) && > + getModuleId(p) == sqlRef && > + getFunctionId(p) == subdeltaRef && > + p->argc == 6 && > + p->retc == 1 && > + last_subselect_return_var_id == getArg(p, 1)) > { > r = newInstruction(mb, ASSIGNsymbol); > r = pushReturn(mb, r, getArg(p, 0)); > - r = pushArgument(mb, r, last_bind_return_var_id); > + r = pushArgument(mb, r, last_subselect_return_var_id); > > insertInstruction(mb, r, i+1); > removeInstruction(mb, p); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list

2 1

Re: MonetDB: default - Initialize variables with a fake value so tha...
by Stefan Manegold 11 Jul '13

11 Jul '13

Thank you very much, Sjoerd! However, this is not yet sufficient to get this compiled (with optimization enabled): .../monetdb5/optimizer/opt_json.c:54:6: error: 'j' may be used uninitialized in this function [-Werror=maybe-uninitialized] However, simply initializing variable j when it is declared might not be the correct solution since it depends on itself: j = getArg(q,j); ^ ^ [...] pushArgument(mb,q,j); ^ Martin, can you please double-check your code and fix (also) the initilization / use of variable j according to the intended functionality? Thanks! Stefan ----- Original Message ----- > Changeset: fc7c72bf8a56 for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fc7c72bf8a56 > Modified Files: > monetdb5/optimizer/opt_json.c > Branch: default > Log Message: > > Initialize variables with a fake value so that this can be compiled. > > > diffs (12 lines): > > diff --git a/monetdb5/optimizer/opt_json.c > b/monetdb5/optimizer/opt_json.c > --- a/monetdb5/optimizer/opt_json.c > +++ b/monetdb5/optimizer/opt_json.c > @@ -30,7 +30,7 @@ int > OPTjsonImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, > InstrPtr pci) > { > int i, j, limit, slimit; > - int bu,br,bj; > + int bu = 0, br = 0, bj = 0; > str nme; > InstrPtr p,q; > int actions = 0; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

code pattern and code generate
by 张磊 04 Jul '13

04 Jul '13

hi all, I found that in GDK of MonetDB, many source code like .h/.c is generated from .mx file when compiling, who can tell me which module control this function? -- Best Wishes! zhanglei

2 1

abou different performaces between SQL and MAL
by Yanchen 03 Jul '13

03 Jul '13

Hi, May I have aquestion about MonetDB performance on TPC-H benchmark? I do not know why I got very different performances when I run 01.sql in TPC-H and the explained 01.mal in mal domain. The average execution time for sql against SF100 scale factor is 19.94s, and the time for mal is 34.7. I also tried to use stethoscope to see the detail, seems like mal stop for several seconds before the last line code. Please let me know if you have any thoughts about that. Thank you so much in advance. Best, Lou I am using Monetdb 11.15.3(Feb. 2013). My mserver5 information is as followed: MonetDB 5 server v11.15.3 "Feb2013-SP1" (64-bit, 64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 188.9GiB available memory, 32 available cpu cores Libraries: libpcre: 8.30 2012-02-04 (compiled with 8.30) openssl: OpenSSL 1.0.1c 10 May 2012 (compiled with OpenSSL 1.0.1c 10 May 2012) libxml2: 2.8.0 (compiled with 2.8.0) Compiled by: root@LLL-MMMDB2 (x86_64-unknown-linux-gnu) Compilation: gcc -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code Linking : /usr/bin/ld -m elf_x86_64

2 1

Problem running web app using latest jdbc driver
by Casey Gum 03 Jul '13

03 Jul '13

I have a RESTful web service that stores its information in MonetDB. I have to run in a Java 1.6 environment. The latest version of the jdbc driver says that it is for both 1.6 and 1.7. I'm getting the dependency resolved through maven using the clojars repository * <repository>* * <id>monetdb</id>* * <url>http://repository.pentaho.org/artifactory/repo</url>* * <snapshots>* * <enabled>false</enabled>* * </snapshots>* * </repository>* My dependency block looks like this: * <dependency>* * <groupId>monetdb</groupId>* * <artifactId>monetdb-jdbc</artifactId>* * <version>2.9</version>* * </dependency>* Everything worked when I was running v2.8 and getting my dependency resolved from the pentaho repository, but when I get the dependency from clojars, I get the following error: *java.lang.UnsupportedClassVersionError: nl/cwi/monetdb/jdbc/MonetDriver : Unsupported major.minor version 51.0* * * Two questions: 1. Should the 2.9 version work under Java 1.6? 2. Is there some setting I need to put in my pom.xml file so that it runs in "1.6 compatibility mode" or something? Thanks, Casey Gum

2 1

MonetDB configured with "--program-suffix" does not start DBs
by Benjamin Reilly 22 Jun '13

22 Jun '13

Hi all, I downloaded a copy of the 11.15.7 (Feb2013 SP2) source and compiled and installed it. I also have a copy of the same version of MonetDB installed from elsewhere (I'm not sure where now--sorry). Now, when I run the "old" version of Monet (the one I didn't just compile) and create a DB (`monetdb create test`), release it, and start it, everything works fine. The log looks likes this: ... MSG merovingian[32695]: starting database 'test', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) MSG control[32695]: (local): started 'test' MSG test[32705]: arguments: /usr/local/bin/mserver5 ...<snip>... ... However, when I run with my compiled version (using `configure --program-suffix=-vanilla`), starting a created DB does not call mserver5-vanilla, but rather monetdbd-vanilla, which is the incorrect command, and thus the database cannot start. (If I manually run mserver5-vanilla, it works fine.) The log looks like this: MSG merovingian[2616]: starting database 'test', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) MSG test[2625]: arguments: /usr/local/bin/monetdbd-vanilla ...<snip>... MSG test[2625]: usage: monetdbd [ command [ command-options ] ] <dbfarm> MSG test[2625]: where command is one of: MSG test[2625]: create, start, stop, get, set, version or help MSG test[2625]: use the help command to get help for a particular command MSG test[2625]: The dbfarm to operate on must always be given to MSG test[2625]: monetdbd explicitly. ERR test[2625]: monetdbd: unknown command: --dbpath=/home/breilly/db/farm-vanilla/test MSG merovingian[2616]: database 'test' (2625) has exited with exit status 1 ERR control[2616]: (local): failed to fork mserver: database 'test' appears to shut itself down after starting, check monetdbd's logfile for possible hints I tried just now to compile without the suffix, but in a different directory (i.e. `configure --prefix=$HOME/apps`), and it worked fine, as with the first version above. I'm not sure why the one with the suffix fails in the way it does: it's not like it tries to find "mserver5" (without the suffix) and fails. Instead, it successfully finds a real command with the suffix, but it's the wrong one. I have also tried simply symlinking to my suffixed program with non-suffixed name (i.e. `ln -s $HOME/apps/bin/monetdbd-vanilla $HOME/apps/bin/monetdb`, for each of monetdbd, monetdb, mclient, and mserver5), and this also works. However, it does not run the correct version of mserver5: it runs /usr/local/bin/mserver5, rather than my local copy at $HOME/apps/bin/mserver5. Is there some environment variable that I'm not setting correctly? ($HOME/apps/bin is in my PATH). Please let me know if there is any more information that I can provide for you. Thanks in advance for your help. Regards, Ben Reilly

2 1

Re: Monetdb - production users
by Baskar Duraikannu 20 Jun '13

20 Jun '13

Thanks. May I know size of your database? --- Original Message --- From: "Brandon Jackson" <usbrandon(a)gmail.com> Sent: June 20, 2013 12:09 PM To: "Communication channel for developers of the MonetDB suite." <developers-list(a)monetdb.org> Subject: Re: Monetdb - production users We are running MonetDB in our enterprise in a production environment. It works well. There are nuances. We do a lot of bulk loading, so we picked a time during the day to shut MonetDB down and restart it. The reason, memory fragmentation, does occur and in our experience would cause the database to fail and lock up. It will stop accepting connections. Our database health gradually and constantly decreases. Although I can say we have never received wrong answers from the database or had any obvious effects from poor database health. We periodically recreate the entire database to get the health of the database back up to 100%. Essentially we use MonetDB as a fast query database and not the primary database for maintaining the integrity of our data. There is no repair utility for the database. We're still learning how and if we should report issues like this because it happens gradually and the log has some splat marks " *** glibc detected *** /usr/bin/mserver5: corrupted doubly-linked list: 0x00007fa12411f4d0 *** and *** glibc detected *** /user/bin/mserver5: double free or corruption (!prev): 0x00007fa12411f4d0 *** in it. Like anything else, I think you learn to live with whatever tools you are using and play into the strengths and avoid the weaknesses, reporting them and hoping they can get patched up eventually. Brandon On Thu, Jun 20, 2013 at 9:29 AM, Baskar Duraikannu < baskar.duraikannu(a)outlook.com> wrote: > I would like to know whether any enterprise is running MonetDB outside > research community. Please help. > > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/developers-list > > _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Monetdb - production users
by Baskar Duraikannu 20 Jun '13

20 Jun '13

I would like to know whether any enterprise is running MonetDB outside research community. Please help.

2 1

Buffer Manager - MonetDB
by Baskar Duraikannu 20 Jun '13

20 Jun '13

Good morning. I recently installed and tested monetdb with a sample dataset and it performed really well. Then I came across the below posthttp://stackoverflow.com/questions/7435235/is-it-worth-trying-monetdb Above post talks about non-availability of buffer manager in MonetDB. Has this changed?

2 1