February 2016 - users-list

Slow query - round 2
by Luciano Sasso 24 Feb '16

24 Feb '16

Hi, Do you think that many ( very many ) inserts / updates on this table , inserts that were aborted by network error could corrupt data and cause it? Remember , table have 100M rows. I have table with 30M rows, and run well work. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

File system level backup of running database
by Dennis Pallett 23 Feb '16

23 Feb '16

Hi all, Is it possible to create a file system level backup of a running MonetDB database? What are the downsides of doing this? For what it's worth, our databases are generally never updated outside of pre-fixed windows so only SELECT queries will be executed during any file system level backup. Best regards, Dennis Pallett

4 3

do these two commands create an error in the dev version of monetdb?
by Anthony Damico 23 Feb '16

23 Feb '16

they do work in the latest release but they crash monetdblite, making me wonder about the root culprit.. thanks all create table my_table (my_id clob, my_double double); SELECT COUNT( DISTINCT my_id ) AS unique_enrollees , QUANTILE( my_double , 0.25 ) AS some_quantile FROM my_table;

2 2

is there a downloadable nightly build exe installer for windows?
by Anthony Damico 22 Feb '16

22 Feb '16

or do i need to do something with these? http://dev.monetdb.org/hg/MonetDB/file/tip/NT/installer64 never done this before..curious how cumbersome it is i want to check whether this is a bug introduced within mserver or a bug specific to monetdblite https://www.monetdb.org/bugzilla/show_bug.cgi?id=3927 thanks!

2 2

Slow query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi Stefan About "prepare statements calls/traces ", in paralell select count(*), this service that insert in 10 and 10 minutes. But even without such active service , the time is the same, 90 minutes. My mistake, my server have 6 cores. Hi, (as a side note, I'm afraid I'm too "old fashioned" and do no longer understand "modern" MonetDB traces; at least, I have no idea, why there are so many prepare statements calls/traces in your trace of a simple select count(*) query ... ) Having said that, the trace suggests that your machine either has 6 cores, or MonetDB has decided to split it in 6 slices for processing, although you machine has only 4 cores? Also, the trace reveals that the vast majority of the time is spent in 6 (one per slice) sql.tid() calls: ....... -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, Martin Yes, before performance analysis, load is zero, the server is Linix console only. My RAM memory is low for this database size. what do you think ? Att, Hi, about slow query Yes, there is, but still left over resources. While the query (which takes 90 minutes) are running, to perform another in another table (9M rows), this latter returns usually in seconds. Hi Its all about concurrent processes while working with MonetDB. All those processes compete for RAM and disk bandwidth. Even running a browser already takes off a few GB from your RAM. At some point, you end up swapping intermediates to disk, which is a slow process (depending on your HDD). For performance analysis, you better shut down all the processes you don't need. Ideally, the CPU load = 0 before doing a performance analysis. > " > > A simple "select count(*) from table", run on 90 minutes. > > that is not normal. most likely there are a lot of other > processes competing for resources. > > > " -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, send now trace for query in link: https://drive.google.com/folderview?id=0B0L6U0kZpUaFNUU3ak9PNUFVZW8&usp=sha… Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow Query
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, send attached trace for query. Att, --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

3 2

Re: users-list Digest, Vol 42, Issue 17
by Luciano Sasso 21 Feb '16

21 Feb '16

Hi, about slow query Yes, there is, but still left over resources. While the query (which takes 90 minutes) are running, to perform another in another table (9M rows), this latter returns usually in seconds. " A simple "select count(*) from table", run on 90 minutes. that is not normal. most likely there are a lot of other processes competing for resources. " Em 21/02/2016 08:00, users-list-request(a)monetdb.org escreveu: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Slow Query (Martin Kersten) > 2. Re: Slow Query (Stefan Manegold) > 3. might be of interest (Anthony Damico) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 20 Feb 2016 21:34:06 +0100 > From: Martin Kersten <martin(a)monetdb.org> > To: users-list(a)monetdb.org > Subject: Re: Slow Query > Message-ID: <56C8CDBE.3060006(a)monetdb.org> > Content-Type: text/plain; charset=windows-1252; format=flowed > > On 20/02/16 19:39, Luciano Sasso wrote: >> Hi, >> >> I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, >> 4 of number type, others string type. >> >> A simple "select count(*) from table", run on 90 minutes. > that is not normal. most likely there are a lot of other > processes competing for resources. > >> Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. >> >> While monet run the query, command top, show 42G of virtual memory >> >> Its normal ?, any idea ? >> >> thanks. >> >> >> -- >> >> Luciano Sasso Vieira >> Data Scientist & Solutions Architect >> >> <http://www.gsgroup.com.br>luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 99706-9335 >> <http://www.gsgroup.com.br>www.gsgroup.com.br >> >> >> Este e-mail foi enviado por um computador sem v?rus e protegido pelo Avast. >> www.avast.com <https://www.avast.com/sig-email> >> >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list >> > > > ------------------------------ > > Message: 2 > Date: Sat, 20 Feb 2016 22:16:01 +0100 (CET) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: Re: Slow Query > Message-ID: <1956506938.11256849.1456002961125.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset="utf-8" > > Hi, > > this is indeed unexpected. > > To possibly better understand what's happening, > could you please run your query prefixed with keyword TRACE > and share the trace (profiling) result? > > Best, > Stefan > > ----- On Feb 20, 2016, at 7:39 PM, Luciano Sasso luciano(a)gsgroup.com.br wrote: > >> Hi, >> >> I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, >> 4 of number type, others string type. >> >> A simple "select count(*) from table", run on 90 minutes. >> >> Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. >> >> While monet run the query, command top, show 42G of virtual memory. >> >> Its normal ?, any idea ? >> >> thanks. >> >> >> -- >> >> >> Luciano Sasso Vieira >> Data Scientist & Solutions Architect >> >> luciano(a)gsgroup.com.br | tel: 17 3353-0833 | cel: 17 99706-9335 >> www.gsgroup.com.br >> >> >> Este e-mail foi enviado por um computador sem v?rus e protegido pelo Avast. >> www.avast.com >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/users-list -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

might be of interest
by Anthony Damico 21 Feb '16

21 Feb '16

http://stackoverflow.com/questions/32380663/improve-speed-of-queries-using-…

1 0