- users-list - monetdb.org

Re: How to do profiling in MAL interface?
by Constantine Feng 11 Apr '16

11 Apr '16

Hi Martin, Thanks. stethoscope does do some work. But since it is for online monitoring, it is inconvenient for offline analysis of trace profiles. After reading more docs on the web, I find Tachograph comes closest to what I want. A minor problem is that the schema of the generated files (.trace, .csv, .json) is not given. Somehow I have to guess. And seems that each file format covers a different subset of attributes. - Neo > > ------------------------------ > > Message: 2 > Date: Sat, 9 Apr 2016 08:17:15 +0200 > From: Martin Kersten <martin.kersten(a)cwi.nl> > To: users-list(a)monetdb.org > Subject: Re: How to do profiling in MAL interface? > Message-ID: <57089E6B.7050504(a)cwi.nl> > Content-Type: text/plain; charset=windows-1252; format=flowed > > You better consider using the stethoscope for doing this kind of analysis. > The documentation need some updates. > > > On 09/04/16 06:07, Constantine Feng wrote: > > Hi all, > > > > I know I can use the TRACE SQL command. But I want to do it in the > "-lmal" interface (because I massaged the MAL plans a bit to test my > research ideas). > > > > Particularly, I find the information given on this website ( > https://www.monetdb.org/Documentation/Manuals/MonetDB/Modules/profiler ) > doesn't match the source code. I can't find the profiler.activate() or > profiler.deactivate(), for example. > > > > > > mal>manual.help("profiler.*"); > > profiler.cpuload > > profiler.cpustats > > profiler.cleanup > > profiler.closeStream > > profiler.dumpTrace > > profiler.getSystemTime > > profiler.getUserTime > > profiler.getDiskWrites > > profiler.getDiskReads > > profiler.getTrace > > profiler.noop > > profiler.openStream > > profiler.reset > > profiler.setHeartbeat > > profiler.stop > > profiler.stethoscope > > profiler.tomograph > > > > > > Version: (Jul2015-SP3). Build from source. > > > > - Neo > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > >

1 0

Announcement: New Jul2015-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 11 Apr '16

11 Apr '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP4 bugfix release MonetDB Common * Fixed a bug that caused various instances where old data returned or where crashes occurred. The problem was that internally data wasn't always marked dirty when it was being changed, causing later processing to not deal with the changed data correctly. Bug Fixes * 3905: MonetDB doesn't handle ANY/SOME/ALL operator correctly * 3929: R aggregate not recognized when using 3 or more parameters * 3965: Not possible to quote/escape single quote character in the name of the file to load. * 3968: Missing double use of column names

1 0

Can not enable python
by Shmagi Kavtaradze 10 Apr '16

10 Apr '16

I created function in python, but when tried to call, I receive : Embedded Python has not been enabled. Start server with --set embedded_py=true I stopped farm. used mserver5 --set embedded_py=true and got in mserver5 console output: # MonetDB/SQL module loaded # MonetDB/Python module loaded Then I started farm and still getting same error. Then I did this: monetdb stop dbname monetdb set embedpy=true dbname monetdb start dbname Still getting same. Any ideas what I am doing wrong?

2 2

Fwd: Function does not complete
by Shmagi Kavtaradze 09 Apr '16

09 Apr '16

Thanks for answer. I have bits of size 50, so I need bigint/long long int. Problem is in while loop. When I use bitand(a,b) and numbers of digits of a and b both are 18-19, the system enters endless loop I think. Bigint maximum value is 19 digits long.I have mistakes in code when using unsigned int and I changed, but same problem. As I know from data types page, monetdb uses only signed integers, so I use only signed ones. Any ideas why does it hang on 18-19 digits and not on 1-17 digits long numbers?

1 0

How to do profiling in MAL interface?
by Constantine Feng 09 Apr '16

09 Apr '16

Hi all, I know I can use the TRACE SQL command. But I want to do it in the "-lmal" interface (because I massaged the MAL plans a bit to test my research ideas). Particularly, I find the information given on this website ( https://www.monetdb.org/Documentation/Manuals/MonetDB/Modules/profiler ) doesn't match the source code. I can't find the profiler.activate() or profiler.deactivate(), for example. mal>manual.help("profiler.*"); profiler.cpuload profiler.cpustats profiler.cleanup profiler.closeStream profiler.dumpTrace profiler.getSystemTime profiler.getUserTime profiler.getDiskWrites profiler.getDiskReads profiler.getTrace profiler.noop profiler.openStream profiler.reset profiler.setHeartbeat profiler.stop profiler.stethoscope profiler.tomograph Version: (Jul2015-SP3). Build from source. - Neo

2 1

Controlling --dbextras Using MonetDB(D)
by Alex Manaev 09 Apr '16

09 Apr '16

Hi, I use wrapper shell in such case. May be useful for you. $ mv /usr/bin/mserver5 /usr/bin/mserver5.bin $ vim /usr/bin/mserver5 #!/usr/bin/env bash exec /usr/bin/mserver5.bin --set gdk_vmtrim yes --set gdk_vm_maxsize 200000000000 --dbextra=/mnt/data/monetdb/extra $@ Alexey Manaev, Head of software development CJSC "RUTUBE"

3 4

monetdb crash as start : Logger_new: BBPrename to sql_snapshots_bid failed
by Pierre-Adrien Coustillas 08 Apr '16

08 Apr '16

Hello, after a server crash : # mclient comptage_cookie_telerama monetdbd: internal error while starting mserver, please refer to the logs # cat merovingian.log 2016-04-08 14:28:33 MSG merovingian[13582]: starting database 'comptage_cookie_telerama', up min/avg/max: 28m/4w/10w, crash average: 1.00 1.00 1.00 (246-6=240) 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/comptage_cookie_telerama --set merovingian_uri=mapi:monetdb://groupelemonde-crm:50000/comptage_cookie_telerama --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/comptage_cookie_telerama/.mapi.sock --set monet_vault_key=/var/lib/monetdb/comptage_cookie_telerama/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # MonetDB 5 server v11.21.13 "Jul2015-SP2" 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Serving database 'comptage_cookie_telerama', using 8 threads 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Found 64.000 GiB available main-memory. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Copyright (c) 1993-July 2008 CWI. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Visit http://www.monetdb.org/ for further information 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/comptage_cookie_telerama/.mapi.sock 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16350]: # Start processing logs sql/sql_logs version 52200 2016-04-08 14:28:34 MSG merovingian[13582]: database 'comptage_cookie_telerama' (16350) has exited with exit status 1 2016-04-08 14:28:34 ERR merovingian[13582]: client error: database 'comptage_cookie_telerama' appears to shut itself down after starting, check monetdbd's logfile for possible hints 2016-04-08 14:28:34 MSG merovingian[13582]: starting database 'comptage_cookie_telerama', up min/avg/max: 28m/4w/10w, crash average: 1.00 1.00 1.00 (247-6=241) 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: arguments: /usr/bin/mserver5 --dbpath=/var/lib/monetdb/comptage_cookie_telerama --set merovingian_uri=mapi:monetdb://groupelemonde-crm:50000/comptage_cookie_telerama --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/comptage_cookie_telerama/.mapi.sock --set monet_vault_key=/var/lib/monetdb/comptage_cookie_telerama/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # MonetDB 5 server v11.21.13 "Jul2015-SP2" 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Serving database 'comptage_cookie_telerama', using 8 threads 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Found 64.000 GiB available main-memory. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Copyright (c) 1993-July 2008 CWI. 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Visit http://www.monetdb.org/ for further information 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/lib/monetdb/comptage_cookie_telerama/.mapi.sock 2016-04-08 14:28:34 MSG comptage_cookie_telerama[16354]: # Start processing logs sql/sql_logs version 52200 2016-04-08 14:28:34 MSG merovingian[13582]: database 'comptage_cookie_telerama' (16354) has exited with exit status 1 2016-04-08 14:28:39 ERR merovingian[13582]: client error: database 'comptage_cookie_telerama' appears to shut itself down after starting, check monetdbd's logfile for possible hints # cat .gdk_lock USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ LOGON USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ !FATAL: Logger_new: BBPrename to sql_snapshots_bid failed USR=107 PID=16354 TIME=Fri Apr 8 14:28:34 2016 @ LOGOFF do you have an idea ? Thank, Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

1 0

any ideas why CPU usage would never go over 25%?
by Anthony Damico 08 Apr '16

08 Apr '16

hi, we have a powerful windows server with 200GB RAM and 40 cores. i've been using it for a few weeks now, and mserver.exe never seems to use more than 10 of the cores. mserver does say "using 40 threads" so they're all available is it just rare for any SQL query to use more than 10 cores at once, or is it possible i've misconfigured something? thanks

1 0

Function does not complete
by Shmagi Kavtaradze 07 Apr '16

07 Apr '16

When I run the own written function it runs and does no exit. I was waiting for 5-6 hours, checked dstats, no disk writes, only 1 core works and no memory usage. It is like stuck and can not exit. It happens when I use function separately for each variable, without callingm for example: while( *a ){ countA += *a & 1; *a >>= 1; } while( *b ){ countB += *b & 1; *b >>= 1; } while( c ){ countC += c & 1; c >>= 1; } but when I call functions like this monetdb works fine: int countSetBits1(unsigned int n){ unsigned int count = 0; while(n){ count += n & 1; n >>= 1; } return count; } countA = countSetBits1(*a); countB = countSetBits1(*b); countC = countSetBits1(c); I am not strong in C or monetdb internals, but in Netbeans both codes work well. The entire code : str UDFbitanddec(flt *ret, lng *a, lng *b) { int c = 0; int countA = 0; int countB = 0; int countC = 0; int maxCount = 0; unsigned int iA = *a; unsigned int iB = *b; c = iA & iB; while( iA ){ countA += iA & 1; iA >>= 1; } while( iB ){ countB += iB & 1; iB >>= 1; } while( c ){ countC += c & 1; c >>= 1; } if (countA >= countB) { maxCount = countA; } else { maxCount = countB; } *ret = (float)countC /maxCount; return MAL_SUCCEED; }

2 1

Exception Error
by Erie, Rob 07 Apr '16

07 Apr '16

I am in the process of evaluating databases for analysis at my organization. I have installed MonetDB and loaded some data. I was attempting to compare results both in performance and accuracy between MySQL and MonetDB. I have been receiving Overflow in Calculation errors from MonetDB. I am a total novice with MonetDB. Here is the information about my system: Red Hat Enterprise Linux Server release 7.2 (Maipo) (Linux 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux) MySQL Server version: 5.7.11-log MySQL Community Server Java version "1.8.0_74"; Java(TM) SE Runtime Environment (build 1.8.0_74-b02); Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode). Apache Tomcat 7.0.54 Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP3) Database: MonetDB v11.21.17 (Jul2015-SP3), 'mapi:monetdb://ip-10-153-0-5.ec2.internal:50000/set_mktng' Issue Description: I would like to know the correlation between two variables group by a variable. SQL: select milageRange, sys.corr(numberOfTheseParts,avgHomeValue) FROM aPartsByZipRange INNER JOIN avgHomeValueZipPeriod ON zipCode = zip where numberOfTheseParts <> 0 and avgHomeValue <>0 group by milageRange; Error: TypeException:user.s78_2[44]:'aggr.subcorr' undefined in: X_75:bat[:any,:int] := aggr.subcorr(X_72:bat[:oid,:int],X_74:bat[:oid,:int],X_67:bat[:oid,:oid],r1_67:bat[:oid,:oid],true:bit); program contains errors -- -------------------------------------------------- This e-mail transmission contains information intended only for the use of the recipient(s) named above. Further, it contains information that may be privileged and confidential. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this message (including any attachments) is strictly prohibited. If you have received this e-mail in error, please notify the sender by reply e-mail and then delete this message from your mail system. Thank you for your compliance.

3 4