- users-list - monetdb.org

Re: users-list Digest, Vol 29, Issue 14
by bhuvan Mitra 20 Jan '15

20 Jan '15

Thank you stefan. Is there any way through which I can get to know about those implementations or buying it from actian is the only way? Actually I want to contribute to c-store, open source column oriented database in this compression/decompression area. This answer from you will be very worthy to me. Thanks in advance. On Mon, Jan 19, 2015 at 9:29 PM, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: users-list Digest, Vol 29, Issue 12 (Stefan Manegold) > 2. Next release? (g4(a)novadsp.com) > 3. Re: Not utilising Mutithreads (shamsul hassan) > 4. Re: Next release? (Sjoerd Mullender) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 19 Jan 2015 14:58:13 +0100 (CET) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: Re: users-list Digest, Vol 29, Issue 12 > Message-ID: <895297092.253122.1421675893625.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset=utf-8 > > > Hi, > > MonetDB and Actian/VectorWise are two different systems that do not share > any code. > > MonetDB does not use any compression in the way Actian/VectorWise does. > > Best, > Stefan > > > ----- Original Message ----- > > Thank you Martin. > > So, did monetdb bought the codes from actian code base and included in > it's > > source code? > > Also kindly tell me if monetdb is following any other compression > techniques > > apart from pfor, pfor-delta and pdict. > > Thanks in advance. > > > > On Mon, Jan 19, 2015 at 4:30 PM, < users-list-request(a)monetdb.org > > wrote: > > > > > > Send users-list mailing list submissions to > > users-list(a)monetdb.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://www.monetdb.org/mailman/listinfo/users-list > > or, via email, send a message with subject or body 'help' to > > users-list-request(a)monetdb.org > > > > You can reach the person managing the list at > > users-list-owner(a)monetdb.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of users-list digest..." > > > > > > Today's Topics: > > > > 1. Re: Not utilising Mutithreads (Martin Kersten) > > 2. regarding source code (bhuvan Mitra) > > 3. Re: regarding source code (Martin Kersten) > > 4. information about compression/decompression techniques of > > monetdb (bhuvan Mitra) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Sun, 18 Jan 2015 12:20:15 +0100 > > From: Martin Kersten < martin(a)monetdb.org > > > To: users-list(a)monetdb.org > > Subject: Re: Not utilising Mutithreads > > Message-ID: < 54BB96EF.3030906(a)monetdb.org > > > Content-Type: text/plain; charset=windows-1252; format=flowed > > > > Hi > > > > Each MAL instruction is executed by a single thread. > > Some operators require the complete input to be available, > > such as in your case the group.subgroup(). > > > > martin > > > > On 18/01/15 01:27, shamsul hassan wrote: > > > Hi All, > > > > > > I have recently install monetdb "OCT 2014-SP1" release on my Oracle > > > Enterprise 6 Linux server. > > > > > > Server configuration - > > > > > > 8 CPU Cores > > > 64 GB RAM > > > > > > I have loaded a data set with 70 columns having 15 Millions Rows. > > > > > > I have started mserver5 as below -- > > > > > > mserver5 --debug=10 --set *gdk_nr_threads=8* > --dbpath=/data/monetdb/testdb > > > --set mapi_open=true --set mapi_port=54321 --set > > > merovingian_uri=mapi:monetdb://localhost:54321/testdb --set > > > mapi_usock=/data/monetdb/testdb/.mapi.sock --set > > > monet_vault_key=/data/monetdb/testdb/.vaultkey --set > > > sql_optimizer=default_pipe --set max_clients=64 > > > > > > Now I have ran a query as below -- > > > > > > select product_type,COUNT(DISTINCT sales_document),sum(ord_qty) FROM > > > bookings where ord_entry_date between '2014-04-01' and '2014-10-01' > group > > > by product_type; > > > > > > This query took around 90 Secs .. > > > So I started tomograph and checked plotted graph ... which shows the > > > parallelism usage as 39.9 % only and its utilizing only 1 thread with > > > maximum time went in the step - group.subgroupdone almost 98%. > > > > > > My Question is that even I have started the server with > gdk_nr_threads=8 , > > > then why it is just using single thread. > > > > > > More info -- > > > > > > mserver5 --version > > > > > > MonetDB 5 server v11.19.7 "Oct2014-SP1" (64-bit, 64-bit oids) > > > > > > Copyright (c) 1993-July 2008 CWI > > > > > > Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved > > > > > > Visit http://www.monetdb.org/ for further information > > > > > > Found 63.0GiB available memory, 8 available cpu cores > > > > > > Libraries: > > > > > > libpcre: 7.8 2008-09-05 (compiled with 7.8) > > > > > > openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips > 11 > > > Feb 2013) > > > > > > libxml2: 2.7.6 (compiled with 2.7.6) > > > > > > Compiled by: root@localhost (x86_64-unknown-linux-gnu) > > > > > > Compilation: gcc -O3 -fomit-frame-pointer -pipe -D_FORTIFY_SOURCE=2 > > > > > > Linking : /usr/bin/ld -m elf_x86_64 > > > > > > Any pointers ?? > > > > > > Thanks > > > > > > > > > > > > _______________________________________________ > > > users-list mailing list > > > users-list(a)monetdb.org > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Mon, 19 Jan 2015 12:28:05 +0530 > > From: bhuvan Mitra < bhuvanrjb(a)gmail.com > > > To: users-list(a)monetdb.org > > Subject: regarding source code > > Message-ID: > > < CAMmryP6eg5AAMY1x70KJZMdc3k7P8aqjnHuS5xN-fNSJ8mN_sQ(a)mail.gmail.com > > > Content-Type: text/plain; charset="utf-8" > > > > Hello, > > > > I am very interested to know and learn about implementations of > > compression/decompression techniques that monetdb is following. I have > seen > > the monetdb's light weight compression algorithms(pfor, pfor-delta, > pdict), > > cache involved decompression techniques and excited to know how actually > > are they implemented in form of codes. Kindly tell me in which > folder/file > > is the source code of this compression/decompression algorithms present. > I > > have downloaded the source tarball from the website and could not find > > these code snippets. Plese help me to identify the path. > > > > Regards, > > mg > >

2 1

Next release?
by g4＠novadsp.com 19 Jan '15

19 Jan '15

Hello - is there any timeframe for a next release? There are a couple of post Oct2014-SP1 bug-fixes which would be really good to have included. Many thanks.

2 1

Re: users-list Digest, Vol 29, Issue 12
by bhuvan Mitra 19 Jan '15

19 Jan '15

Thank you Martin. So, did monetdb bought the codes from actian code base and included in it's source code? Also kindly tell me if monetdb is following any other compression techniques apart from pfor, pfor-delta and pdict. Thanks in advance. On Mon, Jan 19, 2015 at 4:30 PM, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Not utilising Mutithreads (Martin Kersten) > 2. regarding source code (bhuvan Mitra) > 3. Re: regarding source code (Martin Kersten) > 4. information about compression/decompression techniques of > monetdb (bhuvan Mitra) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 18 Jan 2015 12:20:15 +0100 > From: Martin Kersten <martin(a)monetdb.org> > To: users-list(a)monetdb.org > Subject: Re: Not utilising Mutithreads > Message-ID: <54BB96EF.3030906(a)monetdb.org> > Content-Type: text/plain; charset=windows-1252; format=flowed > > Hi > > Each MAL instruction is executed by a single thread. > Some operators require the complete input to be available, > such as in your case the group.subgroup(). > > martin > > On 18/01/15 01:27, shamsul hassan wrote: > > Hi All, > > > > I have recently install monetdb "OCT 2014-SP1" release on my Oracle > Enterprise 6 Linux server. > > > > Server configuration - > > > > 8 CPU Cores > > 64 GB RAM > > > > I have loaded a data set with 70 columns having 15 Millions Rows. > > > > I have started mserver5 as below -- > > > > mserver5 --debug=10 --set *gdk_nr_threads=8* > --dbpath=/data/monetdb/testdb --set mapi_open=true --set mapi_port=54321 > --set merovingian_uri=mapi:monetdb://localhost:54321/testdb --set > mapi_usock=/data/monetdb/testdb/.mapi.sock --set > > monet_vault_key=/data/monetdb/testdb/.vaultkey --set > sql_optimizer=default_pipe --set max_clients=64 > > > > Now I have ran a query as below -- > > > > select product_type,COUNT(DISTINCT sales_document),sum(ord_qty) FROM > bookings where ord_entry_date between '2014-04-01' and '2014-10-01' group > by product_type; > > > > This query took around 90 Secs .. > > So I started tomograph and checked plotted graph ... which shows the > parallelism usage as 39.9 % only and its utilizing only 1 thread with > maximum time went in the step - group.subgroupdone almost 98%. > > > > My Question is that even I have started the server with gdk_nr_threads=8 > , then why it is just using single thread. > > > > More info -- > > > > mserver5 --version > > > > MonetDB 5 server v11.19.7 "Oct2014-SP1" (64-bit, 64-bit oids) > > > > Copyright (c) 1993-July 2008 CWI > > > > Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved > > > > Visit http://www.monetdb.org/ for further information > > > > Found 63.0GiB available memory, 8 available cpu cores > > > > Libraries: > > > > libpcre: 7.8 2008-09-05 (compiled with 7.8) > > > > openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL > 1.0.1e-fips 11 Feb 2013) > > > > libxml2: 2.7.6 (compiled with 2.7.6) > > > > Compiled by: root@localhost (x86_64-unknown-linux-gnu) > > > > Compilation: gcc -O3 -fomit-frame-pointer -pipe -D_FORTIFY_SOURCE=2 > > > > Linking : /usr/bin/ld -m elf_x86_64 > > > > Any pointers ?? > > > > Thanks > > > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > ------------------------------ > > Message: 2 > Date: Mon, 19 Jan 2015 12:28:05 +0530 > From: bhuvan Mitra <bhuvanrjb(a)gmail.com> > To: users-list(a)monetdb.org > Subject: regarding source code > Message-ID: > < > CAMmryP6eg5AAMY1x70KJZMdc3k7P8aqjnHuS5xN-fNSJ8mN_sQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I am very interested to know and learn about implementations of > compression/decompression techniques that monetdb is following. I have seen > the monetdb's light weight compression algorithms(pfor, pfor-delta, pdict), > cache involved decompression techniques and excited to know how actually > are they implemented in form of codes. Kindly tell me in which folder/file > is the source code of this compression/decompression algorithms present. I > have downloaded the source tarball from the website and could not find > these code snippets. Plese help me to identify the path. > > Regards, > mg >

2 1

information about compression/decompression techniques of monetdb
by bhuvan Mitra 19 Jan '15

19 Jan '15

I have been studying about various column-oriented databases and found monetdb to be very interesting as it is efficient and open source. Please tell me in which file/folder does source code of compression/decompression techniques written in monetdb. I am interested to learn the implementations of light weight compression algorithms and cache-decompression. I have source code of monetdb downloaded, just let me know the path of requested codes.

1 0

regarding source code
by bhuvan Mitra 19 Jan '15

19 Jan '15

Hello, I am very interested to know and learn about implementations of compression/decompression techniques that monetdb is following. I have seen the monetdb's light weight compression algorithms(pfor, pfor-delta, pdict), cache involved decompression techniques and excited to know how actually are they implemented in form of codes. Kindly tell me in which folder/file is the source code of this compression/decompression algorithms present. I have downloaded the source tarball from the website and could not find these code snippets. Plese help me to identify the path. Regards, mg

2 1

analyse performance slow query
by Pierre-Adrien Coustillas 16 Jan '15

16 Jan '15

Hi, I try to analyze the slowness of a query with EXPLAIN and stethoscope The query is : SELECT count (*) FROM ( SELECT id_unique, siteniveau1, siteniveau2, '0000-00-00', 0, 0, MAX( debutvisite ), 0, 0 FROM tmp __ at __ lm __visites_ agregees _id_unique GROUP BY id_unique,siteniveau1,siteniveau2 ) tbl ; the beginning of explain is ( there is 272 tuples ) +============================================================================================================================================================================================+ | function user.s2_5{ autoCommit = true }(A0: str ,A1: bte ,A2: bte ,A3: bte ,A4: bte ): void ; | | barrier X_429 := language . dataflow (); | | X_7 := sql . mvc (); | | X_55:bat[: oid ,: oid ] := sql . tid (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique",0,8); | | X_70:bat[: oid ,: int ] := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","id_unique",0,0,8); | | (X_78:bat[: oid ,: oid ] ,X_79:bat[: oid ,: int ] ) := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","id_unique",2,0,8); | | X_161 := sql .delta(X_70,X_78,X_79); | | X_169 := algebra . leftfetchjoin (X_55,X_161); | | X_96:bat[: oid ,: int ] := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau1",0,0,8); | | (X_111:bat[: oid ,: oid ] ,X_112:bat[: oid ,: int ] ) := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau1",2,0,8); | | X_184 := sql .delta(X_96,X_111,X_112); | | X_193 := algebra . leftfetchjoin (X_55,X_184); | | X_129:bat[: oid ,: int ] := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau2",0,0,8); | | (X_144:bat[: oid ,: oid ] ,X_145:bat[: oid ,: int ] ) := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau2",2,0,8); | | X_201 := sql .delta(X_129,X_144,X_145); | | X_209 := algebra . leftfetchjoin (X_55,X_201); | | (X_224,X_225,X_226) := group. subgroup (X_209); | | (X_248,X_249,X_250) := group. subgroup (X_193,X_224); | | (X_272,X_273,X_274) := group. subgroupdone (X_169,X_248); | | X_275 := algebra . leftfetchjoin (X_273,X_169); | | X_304 := algebra . leftfetchjoin (X_273,X_248); | | X_331 := algebra . leftfetchjoin (X_304,X_249); group. subgroupdone (X_169,X_248); set 10 minutes to run . Why ? how to understand this function ? Thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http ://1g6. biz Tél : 06 64 63 70 35

3 2

MSVCR80.dll Error Question
by Zola Javzmaa 16 Jan '15

16 Jan '15

Hello All, I am having trouble installing Monetdb on to my computer. Would anybody have any suggestions to fixing it? I'm on a Window 7, 64 bit operating system and I get the following error message when I try to load the server program: "Error message: This application has failed to start because MSVCR80.dll was not found. Re-installing the application might fix the problem." Thanks, Zola

3 2

rename database
by Pierre-Adrien Coustillas 15 Jan '15

15 Jan '15

Hi, Is it possible to rename a database ? I thought to do like this: monetdb stop database mv /var/lib/monetdb/database /var/lib/monetdb/new_database monetdb start new_database it's okay ? Thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

4 3

MonetDB Usage for spherical clustering
by Jiří Nádvorník 14 Jan '15

14 Jan '15

Dear MonetDB support, My name is Jiri Nadvornik and I am working for Czech astronomical institute (http://www.asu.cas.cz/en/about/about-the-institute) For more than a year I’m trying to generate identifiers for astronomical objects used when creating light curves for these objects. In general, I’m working with spherical coordinates and based on Euclidean parameters (and possibly other parameters, like magnitude) I am clustering these objects to assign multiple observations to one real object. A bottleneck when doing this in a relational database (like PostgreSQL) is not the finding of the neighbors as such (~ ms in a loop), but the inefficiency of working with disk IO when doing so. So after a little analysis of what could be done, I am quite interested into storing the data (mainly spherical coordinates) in an array DB, physically partition regions of the sky to multiple disks and then use the clustering on these partitions in parallel. My question no. 1 for you is whether MonetDB has some kind of support for spherical indexing (PgSphere, Q3C, HealPix, …) to ingest and correctly distribute the astronomical data. I need to control the way of how the data will be distributed to physical partitions. If question no.1 can be somehow answered as true, then question no 2. is: What is the easiest way to install MonetDB, ingest my data and test them? P.S.: Our processed dataset contains now cca 3e8 observations (entries for clustering) which resolve to about 6e6 real objects. We expect that number to double when we process all of our data. Thank you for your answer. Kind regards, Jiri Nadvornik Astronomical Institute of the Academy of Sciences of the Czech Republic nadvornik.ji(a)gmail.com

3 2

Bug in Query Plan??
by Vijay Krishna 09 Jan '15

09 Jan '15

Hi, I have been studying on multiple join performances in MonetDB. I am running the latest version of MonetDB Oct2014 - SP1 (MonetDB-11.19.3) on a CentOS machine with 128 GB RAM. All my tables have 100 to 300 columns and around 35 lakh rows. I have the following query with 5 joins and a few criteria. SELECT * FROM table1 INNER JOIN table2 on table1.col1=table2.col1 LEFT JOIN table3 ON table1.col2=table3.col1 LEFT JOIN table3 ON table1.col3=table3.col1 LEFT JOIN table4 ON table2.col2=table4.col1 LEFT JOIN table5 ON table2.col3=table5.col1 WHERE (((table2.col10 like '%a%' OR table5.col11 like '%s%') and ((table2.col12 like '%w%' *and table4.col13 like '%k'*) OR table2.col14 like '%h%') and (table1.col21 = 1) AND (table1.col22 IN (1,2,3))) AND (((table1.col23 >= 15842000000000000) AND (table1.col23 <= 15842999999999999)) OR ((table1.col23 >= 0) AND (table1.col23 <= 999999999999)))) ORDER BY 1 DESC LIMIT 10 OFFSET 0; This query has only 2000 matching rows and it returns the result in *2.9 seconds*. In the same query, if I remove one criteria alone (the bolded part), the result would still match only 2000 rows, but the result would come in *300 milliseconds* even. I used the PLAN statement and found that the query plan that the executor takes for the first query is a much complex path and *it performs 3 joins for all the 35 lakh rows* (i.e. before executing the criteria even) and hence the time has spiked to 3 seconds. May be the hot data of 35 lakh tuples would have not fit into the memory?? Not sure. But if I remove the highlighted criteria, the executor evaluates the criteria first and *it performs the joins for only 7000 rows* and hence it is faster 1000 times. Is this a bug? Or is there any way to optimise my query to match the executor to choose the right plan? Any help much appreciated. I could even demo this. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

1 0