- users-list - monetdb.org

COPY INTO with ordered column
by Sebastian Dorok 09 Aug '13

09 Aug '13

Hi all, I have some slow queries that I want to accelerate by utilizing information about sorting. My table definition: create table genomics.grch36 (site bigint, base char(1)); I use COPY INTO to populate the table. The source file size is 40GB and contains more than 3 billion rows. I can guarantee that the data in file is ordered by site. I query the data like this: sql>select base from genomics.grch36 where site between 10000 and 10010; +------+ | base | +======+ | N | | T | | A | | A | | C | | C | | C | | T | | A | | A | | C | +------+ 11 tuples (5m 23s) I think 5 minutes seem too much for this query. Primary key or an index on 'site' don't work or at least aren't recognized in query execution. I think MonetDB would benefit when knowing that the data is ordered by site. Wouldn't it? Because I know that the data is ordered by 'site', how can I tell MonetDB about this fact to speed up the query? Any suggestions? Regards, Sebastian

2 6

Monetdb performance improvement with more RAM
by Anshuman Kumar 08 Aug '13

08 Aug '13

Hello All, We are evaluating monetdb for use as a tick database. We are storing all the data into a single table. Our table has a column for timestamp and a few other fields. We wish to run queries against it, the most useful of which is to return all entries within a given timestamp range. This query takes a long time to execute for even a 15 minute interval (about 40 to 60 seconds). We have noted that the system RAM usage goes up to the maximum (almost 8GB) while we run such queries on monetdb. We wanted to understand whether installing more RAM (say, we upgrade to 16 GB) will result in a significant performance improvement for such queries on monetdb. Also, if this is the case, we would like to know, how much RAM is monetdb optimised for? The following is the specification of our server: Intel Xeon X3430 2.4Ghz 64Bit 8Gb Ram 500Gb HDD - Raid0 Running on Windows Server 2008 R2. We are using the 20130212 build of monetdb. Thank you, Anshuman

2 1

Local, searchable mail archive
by Brandon Jackson 08 Aug '13

08 Aug '13

Would anyone be able to recommend an approach or software on either Windows or Linux that would enable me to replicate the MonetDB mail archives to my local machine to make it easer to search for answers and study topics before posting questions asked before. Even if I could get an export of the whole thing in some format that be consumed by a mail client. I did read that mailman is capable of such archives, but it seems to require a real administrative user and password to create the archive. I wouldn't expect a frequent completely up to date copy of the archive, just a backup that is closer to now and stays reasonably up to date (few months) Thanks for any suggestions. Brandon Sent from my iPhone

1 0

Re:monetdb lost column info when queryring
by integrity 08 Aug '13

08 Aug '13

sql>select * from kbms_logserver_msg t where command = 'CmdWxInfo' and logdate = '2013-08-06' and functionret <> '0' and servertime > '1375757230' limit 10; +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ | logdate | command | commandtype | functionret | costtime | servertime | serverip | serverport | clientip | clientport | clienttype | +============+===========+==============+=============+==========+============+===============+============+================+============+============+ | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757231 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757244 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3030 | 1375757257 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757962 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3030 | 1375757970 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 60 | 1375760693 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 60 | 1375760693 | 10.187.143.80 | 7000 | 10.153.130.228 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 59 | 1375760693 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 63 | 1375760872 | 10.187.143.80 | 7000 | 10.153.130.229 | | | | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 54 | 1375760872 | 10.187.143.80 | 7000 | 10.153.130.229 | | | +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ 10 tuples (55.709ms) sql>select * from kbms_logserver_msg t where command = 'CmdWxInfo' and logdate = '2013-08-06' and functionret <> '0' and servertime = '1375757231' limit 10; +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ | logdate | command | commandtype | functionret | costtime | servertime | serverip | serverport | clientip | clientport | clienttype | +============+===========+==============+=============+==========+============+===============+============+================+============+============+ | 2013-08-06 | CmdWxInfo | getwxinfoall | -535 | 3032 | 1375757231 | 10.187.143.80 | 7000 | 10.153.130.228 | 44328 | | +------------+-----------+--------------+-------------+----------+------------+---------------+------------+----------------+------------+------------+ 1 tuple (52.761ms) in the first query,when servertime > '1375757230' limit 10;clientport are all NULL. while in the second query, servertime = '1375757231' ,clientport appeard again. it proves that the clientport exist but just didn't appear in the query. Thanks, Meng

4 5

Determining Storage Consumption
by Sebastian Dorok 08 Aug '13

08 Aug '13

Hi all, I'm trying to figure out how much storage MonetDB (Feb2013-SP2) uses when populated with my data on a fresh database. Here is my table definition: create table genomics.grch36 (site bigint, base char(1)); I'm populating the table via COPY INTO. The source file size is 40GB and contains more than 3 billion rows. Following the suggestions on http://www.monetdb.org/Documentation/Userguide/diskspace, I get following result for the SQL query: sql>select * from storage() where "table" = 'grch36'; +----------+--------+--------+--------+----------+------------+-----------+-------------+----------+---------+--------+ | schema | table | column | type | location | count | typewidth | columnsize | heapsize | indices | sorted | +==========+========+========+========+==========+============+===========+=============+==========+=========+========+ | genomics | grch36 | site | bigint | 12/1252 | 3137459420 | 8 | 25099675360 | 0 | 0 | false | | genomics | grch36 | base | char | 12/1247 | 3137459420 | 1 | 6274918840 | 131072 | 0 | false | +----------+--------+--------+--------+----------+------------+-----------+-------------+----------+---------+--------+ I assume that 'columnsize' refers to the storage demand. I'm I right? Based on the numbers given, I assume a storage need of round about 31 GB (without any compression). Calling 'du -h' on my database directory confirms this assumption. 4,0K ./bat/14 44K ./bat/10 108K ./bat/07 476K ./bat/01 30G ./bat/12 256K ./bat/02 4,0K ./bat/03 4,0K ./bat/HC 4,0K ./bat/16 4,0K ./bat/17 12K ./bat/13 4,0K ./bat/LEFTOVERS 28K ./bat/BACKUP 4,0K ./bat/15 4,0K ./bat/11 12K ./bat/04 30G ./bat 8,0K ./sql_logs/sql 12K ./sql_logs 4,0K ./box 30G . But why is the 'columnsize' of the 'base' column something about 6GB whereas 3 billion times 1 Byte (for a 1 char) is 3GB? For the 'site' column this calculation works! Another question that comes into my mind: The number of distinct values in the 'base' column is 5. Thus, assuming a ideal dictionary encoding 3 Bit would be sufficient to encode all values. Since MonetDB uses dictionary compression, can it utilize this fact? Regards, Sebastian

1 0

SQL - syntax error in DELETE ALL FROM statement
by Radovan Bičiště 08 Aug '13

08 Aug '13

Hi, It is me again. :( I'm trying to "truncate" according to http://www.monetdb.org/Documentation/SQLreference/Updates using DELETE ALL FROM statement. But what I get is syntax error. :( ceos@lux-analytics:~$ mclient -h localhost -d zkdev1 -u drmaxtst -p 50000 password: Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013-SP3) Database: MonetDB v11.15.11 (Feb2013-SP3), 'mapi:monetdb://lux-analytics:50000/zkdev1' Type \q to quit, \? for a list of available commands auto commit mode: on sql>delete all from crmkarta; syntax error, unexpected ALL, expecting FROM in: "delete all" sql> Thank you for any hints. Best regards, Radovan -- __________________________ Radovan Bičiště

3 5

rpm for monetdb
by Sanjoy Bose 07 Aug '13

07 Aug '13

Hi, I am new to monetdb and considering using it for our Analytics need. Is there any rpm available for Centos 5 / Centos 6 RedHat 5 / Redhat 6 for its latest release or do I have to build from source ? Thanks, Sanjoy

4 6

SQL - temporary table data inside stored procedure
by Radovan Bičiště 07 Aug '13

07 Aug '13

Hello, I must be doing something horribly wrong. I have a stored procedure where I have temporary table declared, something like: DECLARE TABLE t1tmp (id bigint); Then in the same procedure I do INSERT INTO t1tmp SELECT .....; When I do Count(*) from temporary table in the procedure it returns 0. Fun starts after procedure called. When call is done temporary table is full of data as expected. But it seems the data is nowhere to be found when I look for them inside the procedure. Am I missing something again? Thanks for any hints, Radovan TEST CASE: CREATE PROCEDURE test1() BEGIN DECLARE TABLE t1tmp (bigint); INSERT INTO t1tmp SELECT id FROM t1; DECLARE v_cont BIGINT; -- returns 0 SELECT COUNT(*) INTO v_count FROM t1tmp; END; CALL test1; -- actually returns the data SELECT * FROM t1tmp; -- __________________________ Radovan Bičiště ceos data s.r.o. třída SNP 402/48 500 03 Hradec Králové Czech Republic

2 2

Data Vault:SyntaxException
by integrity 06 Aug '13

06 Aug '13

hi all, What's the correct way to use FIT DATA VAULT in mal? I got the following errors and still havent try out the right syntax ...... [svomhpc@localhost /home/data2]$>/home/data1/opt/MonetDB.fits/bin/mclient -lmal -dmydb Welcome to mclient, the MonetDB interactive terminal (unreleased) Type \q to quit, \? for a list of available commands mal>?fits.attach() pattern fits.attach(fname:str):void address FITSattach; comment Open a FITS file and return catalog of the table HDUs mal>call fits.attach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = call fits.attach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); ERROR = !SyntaxException:parseError:call fits.attach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); !SyntaxException:parseError: ^';' expected mal> mal> mal>call fitsattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = call fitsattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); ERROR = !SyntaxException:parseError:call fitsattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); !SyntaxException:parseError: ^';' expected mal> mal>call FITSattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); MAPI = (monetdb) /tmp/.s.monetdb.50000 QUERY = call FITSattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); ERROR = !SyntaxException:parseError:call FITSattach('/home/data3/gwacdata/20130111_177d550752_60d851283-530.fit'); !SyntaxException:parseError: ^';' expected Thanks! Meng

4 15

Re: How to use Data Vault with FITS files?
by integrity 05 Aug '13

05 Aug '13

Hi Ying, The thesis doesn't have instructions on how to use it....or it doesn't describe it very clearly maybe. Bests, Meng ------------------ Original ------------------ From: "Ying Zhang"<Y.Zhang(a)cwi.nl>; Date: Mon, Aug 5, 2013 04:18 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Subject: Re: How to use Data Vault with FITS files? Hai Meng, I already sent you the master thesis on FITS data vault. Didn't you receive it? Regards, Jennie On Mon, Aug 05, 2013 at 03:52:50PM +0800, integrity wrote: > Hi all, > > Could you give a tutorial of Data vault with FITS files, i searched the > moentdb.org and google, but didn't find the step-by-step tutorials. > > Thanks! > Meng > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1