August 2016 - users-list

Concurrent query execution issue
by Stefano Fioravanzo 24 Aug '16

24 Aug '16

Hello, I am trying to execute in parallel a few queries in MonetDB. I have read in the documentation that Monet uses the OCC transaction model so read operations should not block the table. I have 3 CREATE TABLE queries that read from the same source table, I launch 3 different mclient processes and launch one query on each client. The queries execute in parallel as expected, I can see them running in sys.queue(). The first query that completes, creates the table and commits fine. When the other two queries finish their job I can see the message “operation successful” but right after the client prints "COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead”. What am I doing wrong? Apart from the newly created table, I did non write anything to the db so it should work fine. Any suggestion would be much appreciated, Thanks, Stefano

2 2

Run two MonetDB databases simoultaneously in Windows
by Pablo S 21 Aug '16

21 Aug '16

I'm trying to run two MonetDB databases simoultaneously in Windows. I named the two databases mydb and mydb2. I changed the database name in M5server.bat to mydb, and added -database=mydb to mclient.bat Then I copied the M5server.bat file and changed the farm to mydb2. I also copied the mclient.bat file and added -database=mydb2. I run the two server bats to start the two databases and that seems to work fine (no errors). Then I run the first mclient.bat and it connects to the server with no issues. But when I run the copied mclient.bat and try to connect using user/password monetdb/monetdb, I get the following error and cannot connect: request for database 'mydb2', but this is database 'mydb', did you mean to connect to monetdbd instead? How to run two databases simoultaneously in Windows? How to connect to two databases at the same time?

3 2

Program contains errors
by Stefano Fioravanzo 19 Aug '16

19 Aug '16

Hello, I will briefly recap the problem I have already reported a few weeks ago: I am using monetdb default branch (I did not compile the stable version because I need embedded python) and I have a large table (~80 GB). A simple query like SELECT * FROM large_table WHERE field=x, fails. mserver5 reports this error: mserver5: gdk_select.c:867: fullscan_int: Assertion `cnt < (bn)->S.capacity' failed. It seems that monet thinks it doesn’t have enough space to do a full scan of the table, maybe? (The machine has 800GB of free disk space and 30GB of RAM - Ubuntu 14.04). I was using a default branch of december 2015, I have tried to use the latest default branch (4/08/16), recompiled, installed but nothing, always the same error. So, as suggested, I have compiled the latest stable release (Jun2016-SP1). Now even a query which does not require monet to read the whole table (like SELECT * FROM large_table LIMIT 10 - This query worked fine in the default branch version) fails with ‘Program contains error’ message by client. Selecting a single column of the table works just fine. Moreover, with the stable release, the same issue arises with the second biggest table of the database (~35 GB). [Here you can find the configuration of MonetDB: http://pastie.org/private/r8gtssg944aqwnigrysa] All queries on all the other tables in the database (they are all much smaller than these two) work fine. I do not understand if the issue is in the size of table or somewhere else. How can I know what ‘error’ the message ‘program contains errors’ is referring to? NOTE: At the beginning, using the default compiled branch everything worked just fine! Every query on my large_table worked as expected. Then I had to recreate the database from scratch so I deleted the dbfarm and created a new one. That’s when this headache started… At this point I don’t know what to do, any help would be MUCH appreciated. Thanks, Stefano

4 7

synchronize data from mysql performance
by wang.tong＠shomop.com 19 Aug '16

19 Aug '16

Hi, I try to synchronize data from mysql to monetdb. Steps: 1. Read the modified rows from mysql table periodically. 2. Delete the rows in monetdb by primary key. (To avoid 'PRIMARY KEY constraint violated' error.) The sql is like 'delete from a where id in (1, 2, 3)'. 3. Insert the rows into monetdb by 'copy into ... from stdin'. I find step 2 may be too slow. When there are 2 million rows, delete 10,000 rows takes about 1 minute. When there are 30 million rows, delete 10,000 rows takes about 20 minute. I have to do join on the 30 million rows table, so I cannot split it. I try to upgrade the server from 2 core 8 G memory to 8 core 32 G memory, but it has no effect. It seems the limitation is CPU, and the deleted operation can only use 1 core. How can I optimize the performance? Is there any better approach to synchronize data? Thanks in advance! OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64 mserver5 --version MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 31.0GiB available memory, 8 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 wang.tong(a)shomop.com

2 2

database was killed by signal SIGSEGV
by 王彤 19 Aug '16

19 Aug '16

Hi, I try to synchronize data from mysql to monetdb. Total rows is about 15 million. I insert the rows in batch. Each batch has about 10000 rows. I use 'copy into from stdin' as in http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java . The virtual memory continuously grows. When it grows to about 80G, the server is killed by signal SIGSEGV. The database size on disk is about 30GB. Below is the server infomation and error log. Any help would be greatly appreciated! OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64 mserver5 --version MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved Visit http://www.monetdb.org/ for further information Found 7.4GiB available memory, 2 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 in /data/dbfarm/merovingian.log 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=98280928,virtual memory in use=3779503584] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=98280928,vm=3779503584] 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46953 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.153.193.249:55915 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=185008352,virtual memory in use=3840344288] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=185008352,vm=3840344288] 2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46963 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm 2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) fails, try to free up space [memory in use=99455712,virtual memory in use=3766391520] 2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) result [mem=99455712,vm=3766391520] 2016-08-18 17:29:04 MSG merovingian[10194]: database 'shomopcrm' (10246) was killed by signal SIGSEGV 2016-08-18 17:29:04 MSG merovingian[10194]: starting database 'shomopcrm', up min/avg/max: 17s/15h/5d, crash average: 1.00 0.90 0.40 (23-11=12) 2016-08-18 17:29:05 MSG shomopcrm[984]: arguments: /usr/local/bin/mserver5 --dbpath=/data/dbfarm/shomopcrm --set merovingian_uri=mapi:monetdb://jstu87d9w6zj4u:30001/shomopcrm --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/dbfarm/shomopcrm/.mapi.sock --set monet_vault_key=/data/dbfarm/shomopcrm/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB 5 server v11.23.7 "Jun2016-SP1" 2016-08-18 17:29:05 MSG shomopcrm[984]: # Serving database 'shomopcrm', using 2 threads 2016-08-18 17:29:05 MSG shomopcrm[984]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-08-18 17:29:05 MSG shomopcrm[984]: # Found 7.390 GiB available main-memory. 2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) 1993-July 2008 CWI. 2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved 2016-08-18 17:29:05 MSG shomopcrm[984]: # Visit http://www.monetdb.org/ for further information 2016-08-18 17:29:05 MSG shomopcrm[984]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock 2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB/SQL module loaded in /var/log/messages, there is an error: kernel: mserver5[981]: segfault at 18 ip 00007fe10f845a89 sp 00007fcf38e79760 error 4 in lib_sql.so[7fe10f817000+175000] wang.tong(a)shomop.com

2 1

Distributed Query Processing questions
by Anton Kravchenko 18 Aug '16

18 Aug '16

Hi there, I have few questions on merge/remote tables. Are there known MonetDB users that use(d) merge/remote tables in production? Are there official MonetDB performance benchmarks for merge/remote vs local tables? Thank you, Anton

2 1

distributed query processing example
by Anton Kravchenko 16 Aug '16

16 Aug '16

Hi there, I am trying to follow a distributed query processing example from https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/DistributedQuery… and here is my experience: Welcome to mclient, the MonetDB/SQL interactive terminal (Jun2016-SP1) Database: MonetDB v11.23.7 (Jun2016-SP1), 'mapi:monetdb://hnode3.secdev.local:52018/db3' Type \q to quit, \? for a list of available commands auto commit mode: on sql>CREATE MERGE TABLE mt1 (t int); operation successful (2.367ms) sql>CREATE REMOTE TABLE t1 (i int) on 'mapi:monetdb://hnode1.secdev.local:52016/db1'; operation successful (1.287ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (4.694ms) *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *Connection lost* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *illegal input, not a JSON header (got '')* *sql>select count(*) from t1;* *an error occurred on connection: Connection lost* *sql>select count(*) from t1;* *Connection lost* sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (508.682ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (6.227ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (5.065ms) sql>select * from t1; +------+ | i | +======+ | 11 | | 13 | +------+ 2 tuples (4.178ms) sql> P.S. Is there something wrong with my monetdb cluster setup or is it a real bug? On hnode3 (CentOS 6.8 machine): [akravchenko@hnode3 ~]$ monetdbd -v MonetDB Database Server v1.7 (Jun2016-SP1) [akravchenko@hnode3 ~]$ monetdb discover location hnode3.secdev.local:52018* mapi:monetdb://hnode3.secdev.local:52018/db3 [akravchenko@hnode3 ~]$ monetdb status name state health remarks db3 R 20h 100% 11m mapi:monetdb://hnode3.secdev.local:52018/db3 [akravchenko@hnode3 ~]$ monetdbd get all hnode3-dbfarm property value hostname hnode3.secdev.local dbfarm hnode3-dbfarm status monetdbd[18052] 1.7 (Jun2016-SP1) is serving this dbfarm mserver /usr/bin/mserver5 logfile hnode3-dbfarm/merovingian.log pidfile hnode3-dbfarm/merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 52018 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.52018 controlsock /tmp/.s.merovingian.52018 and on hnode1 (CentOS 6.8 machine): [akravchenko@hnode1 ~]$ monetdb -v MonetDB Database Server Toolkit v1.1 (Jun2016-SP1) [akravchenko@hnode1 ~]$ monetdb discover location hnode1.secdev.local:52016* mapi:monetdb://hnode1.secdev.local:52016/db1 [akravchenko@hnode1 ~]$ monetdb status name state health remarks db1 R 4m 15% 32m mapi:monetdb://hnode1.secdev.local:52016/db1 [akravchenko@hnode1 ~]$ monetdbd get all hnode1-dbfarm/ property value hostname hnode1.secdev.local dbfarm hnode1-dbfarm/ status monetdbd[1355] 1.7 (Jun2016-SP1) is serving this dbfarm mserver /usr/bin/mserver5 logfile hnode1-dbfarm//merovingian.log pidfile hnode1-dbfarm//merovingian.pid sockdir /tmp listenaddr 0.0.0.0 port 52016 exittimeout 60 forward proxy discovery yes discoveryttl 600 control yes passphrase {SHA512}a73f1d86383446438ac64f56e15ada38b41fbb18f029d2181723aeb2acac6a831f60e5fdbd64ac2c8c70e035dd44cbbe3b45565ef2d58feb2821a2078c7fad35 mapisock /tmp/.s.monetdb.52016 controlsock /tmp/.s.merovingian.52016 [akravchenko@hnode1 ~]$ Thank you, Anton

1 0

Add a new optimizer pipeline.
by Xu,Wenjian 10 Aug '16

10 Aug '16

Hi all, MonetDB version: 11.23.7 Ubuntu version: 14.04 I created a new optimizer pipeline by adding an additional optimization step after 'default_pipe', i.e.: '*my_customized_pipe*'='optimizer.inline();optimizer.candidates(); optimizer.remap();optimizer.costModel();optimizer.coercions(); optimizer.evaluate();optimizer.pushselect();optimizer.aliases(); optimizer.mitosis();optimizer.mergetable();optimizer.deadcode(); optimizer.aliases();optimizer.constants();optimizer.commonTerms(); optimizer.projectionpath();optimizer.deadcode();optimizer.reorder(); optimizer.reduce();optimizer.matpack();optimizer.dataflow(); optimizer.querylog();optimizer.multiplex();optimizer.generator(); optimizer.profiler();optimizer.garbageCollector(); *optimizer.customizedstep()*' What I have done is: 1) Create an empty optimizer step: -- 1.1 add *opt_customizedstep.c/h* in package *monetdb/optimizer/**, leave function body *OPTcustomizedstepImplementation() *empty*, *i.e. with one statement **return 0** * -- *1.2 register the step by modifying *opt_wrapper.c *and* optimizer.mal* 2) Add the definition of the new pipeline in *opt_pipes.c* 3) Modify the configuration file to compile the added source files. After re-build the system, I can enable the new optimizer pipeline by *set optimizer='my_customized_pipe'*. However, I have two issues: i) using EXPLAIN statement, I found the MAL plan different from the one generated by 'default_pipe' (which should be the same since I did nothing in the added step?). Specifically, the plan seems to be not affected by *optimizer.reorder() *step, i.e., bat.new() and bat.append() instructions appear in the end of the plan. ii) then I entered mdb mode by DEBUG statement. But it seems that some command did not work as before. For example, *help* command does not display all commands available; *optimizer* command does not display actions performed by each optimizer step. What's wrong with my procedure to add a new optimizer pipeline? Any comment would be helpful. Best regards, Wenjian

3 3

[Low level] Understanding duplicate entries in delta BAT (delete BAT)
by Frédéric Jolliton 08 Aug '16

08 Aug '16

Hi, As per MonetDB Solution suggestion, we are asking here about this topic. We are studying how MonetDB store its database on disk. So far we were able to understand pretty much all the files (BAT, journal, BBP.dir, heap, hash, imprints, and so on). We have custom tools to decode and dump them (experimental at this point). There is one thing however that is unclear: we thought that "delta BAT", which are the BAT assigned to an SQL tables to list the OIDs of rows that were deleted (named "D_<schema>_<table>" internaly), would contains only *uniques* values (since you can never delete twice the same row). But on several databases that we are running, we found that some of theses BATs contain duplicates. Lot of duplicates actually. Especially on D_sys__columns and D_sys__tables BATs (respectively for the sys._columns and sys._tables system tables). Some databases do not have this "issue" (while they all have the same schema, and process the same kind of data as the other ones). Can someone explain if duplicates are expected in theses BATs ? Here is an excerpt of D_sys__tables: # hexdump -C 02/210.tail 00000000 2e 00 00 00 00 00 00 00 2f 00 00 00 00 00 00 00 |......../.......| 00000010 30 00 00 00 00 00 00 00 31 00 00 00 00 00 00 00 |0.......1.......| 00000020 32 00 00 00 00 00 00 00 33 00 00 00 00 00 00 00 |2.......3.......| 00000030 34 00 00 00 00 00 00 00 35 00 00 00 00 00 00 00 |4.......5.......| 00000040 36 00 00 00 00 00 00 00 37 00 00 00 00 00 00 00 |6.......7.......| 00000050 42 00 00 00 00 00 00 00 43 00 00 00 00 00 00 00 |B.......C.......| 00000060 44 00 00 00 00 00 00 00 45 00 00 00 00 00 00 00 |D.......E.......| 00000070 46 00 00 00 00 00 00 00 47 00 00 00 00 00 00 00 |F.......G.......| 00000080 48 00 00 00 00 00 00 00 49 00 00 00 00 00 00 00 |H.......I.......| 00000090 4a 00 00 00 00 00 00 00 4b 00 00 00 00 00 00 00 |J.......K.......| 000000a0 4c 00 00 00 00 00 00 00 4d 00 00 00 00 00 00 00 |L.......M.......| 000000b0 4e 00 00 00 00 00 00 00 4f 00 00 00 00 00 00 00 |N.......O.......| 000000c0 50 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 |P.......P.......| 000000d0 51 00 00 00 00 00 00 00 50 00 00 00 00 00 00 00 |Q.......P.......| 000000e0 51 00 00 00 00 00 00 00 52 00 00 00 00 00 00 00 |Q.......R.......| 000000f0 50 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00 |P.......Q.......| 00000100 52 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00 |R.......S.......| 00000110 50 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00 |P.......Q.......| 00000120 52 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00 |R.......S.......| ... You can clearly see duplicates OID (the first one being 0x50 at 0xc8). Its entry in BBP.dir (split into sections): 136 32 tmp_210 tmpr_210 02/210 610523782 2 171807 0 0 171807 172032 0 0 0 0 void 0 1 1793 0 0 0 0 0 1000651 0 0 0 oid 8 0 1024 24 25 27 1 46 773603235 1374456 1376256 1 We are using: MonetDB 5 server v11.21.14 (64-bit, 64-bit oids, 128-bit integers). -- Frédéric Jolliton Sécuractive

4 3

Does R run in a sandbox in MonetDB/R?
by Pablo S 07 Aug '16

07 Aug '16

MonetDB allows to create R functions that run embedded in the database. For example, you can do CREATE FUNCTION rapi00() RETURNS TABLE (d INTEGER) LANGUAGE R { seq(1,10); }; Now, I need my users to create dynamically these user defined functions by entering R statements on a screen; the application will create the functions in the database and run them. Problem is that R allows to run operating system level commands, such as list.files(path=".") or even file.remove("some_file") Will MonetDB allow these external interactions with the operating system, or it runs in a sandbox?

1 0