Hi,
I try to synchronize data from mysql to monetdb. Total rows is about 15 million. I insert the rows in batch. Each batch has about 10000 rows. I use 'copy into from stdin' as in http://dev.monetdb.org/hg/MonetDB/file/tip/java/example/SQLcopyinto.java .
The virtual memory continuously grows. When it grows to about 80G, the server is killed by signal SIGSEGV.
The database size on disk is about 30GB.
Below is the server infomation and error log. Any help would be greatly appreciated!
OS version: centos-release-7-0.1406.el7.centos.2.5.x86_64
mserver5 --version
MonetDB 5 server v11.23.7 "Jun2016-SP1" (64-bit, 64-bit oids, 128-bit integers)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved
Visit http://www.monetdb.org/ for further information
Found 7.4GiB available memory, 2 available cpu cores
Libraries:
libpcre: 8.32 2012-11-30 (compiled with 8.32)
openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips 11 Feb 2013)
libxml2: 2.9.1 (compiled with 2.9.1)
Compiled by: root@jstu87d9w6zj4u (x86_64-unknown-linux-gnu)
Compilation: gcc -g -O2
Linking : /usr/bin/ld -m elf_x86_64
in /data/dbfarm/merovingian.log
2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=98280928,virtual memory in use=3779503584]
2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=98280928,vm=3779503584]
2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46953 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm
2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying
2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.153.193.249:55915 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm
2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying
2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) fails, try to free up space [memory in use=185008352,virtual memory in use=3840344288]
2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(38862848) result [mem=185008352,vm=3840344288]
2016-08-18 17:29:04 MSG merovingian[10194]: proxying client 10.173.87.201:46963 for database 'shomopcrm' to mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock?database=shomopcrm
2016-08-18 17:29:04 MSG merovingian[10194]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying
2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) fails, try to free up space [memory in use=99455712,virtual memory in use=3766391520]
2016-08-18 17:29:04 ERR shomopcrm[10246]: #GDKmmap(51838976) result [mem=99455712,vm=3766391520]
2016-08-18 17:29:04 MSG merovingian[10194]: database 'shomopcrm' (10246) was killed by signal SIGSEGV
2016-08-18 17:29:04 MSG merovingian[10194]: starting database 'shomopcrm', up min/avg/max: 17s/15h/5d, crash average: 1.00 0.90 0.40 (23-11=12)
2016-08-18 17:29:05 MSG shomopcrm[984]: arguments: /usr/local/bin/mserver5 --dbpath=/data/dbfarm/shomopcrm --set merovingian_uri=mapi:monetdb://jstu87d9w6zj4u:30001/shomopcrm --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data/dbfarm/shomopcrm/.mapi.sock --set monet_vault_key=/data/dbfarm/shomopcrm/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes
2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB 5 server v11.23.7 "Jun2016-SP1"
2016-08-18 17:29:05 MSG shomopcrm[984]: # Serving database 'shomopcrm', using 2 threads
2016-08-18 17:29:05 MSG shomopcrm[984]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked
2016-08-18 17:29:05 MSG shomopcrm[984]: # Found 7.390 GiB available main-memory.
2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) 1993-July 2008 CWI.
2016-08-18 17:29:05 MSG shomopcrm[984]: # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved
2016-08-18 17:29:05 MSG shomopcrm[984]: # Visit http://www.monetdb.org/ for further information
2016-08-18 17:29:05 MSG shomopcrm[984]: # Listening for UNIX domain connection requests on mapi:monetdb:///data/dbfarm/shomopcrm/.mapi.sock
2016-08-18 17:29:05 MSG shomopcrm[984]: # MonetDB/SQL module loaded
in /var/log/messages, there is an error:
kernel: mserver5[981]: segfault at 18 ip 00007fe10f845a89 sp 00007fcf38e79760 error 4 in lib_sql.so[7fe10f817000+175000]