July 2020 - users-list

Many aggregated columns / using custom aggregation function for pivot
by Daniel Glöckner 10 Jul '20

10 Jul '20

Hi, today I checked out how to implement custom aggregation functions in Python. Works pretty neat and scales well when you have few columns which you want to aggregate, however I have many :) Imagine that the following aggregate functions has variants which perform a similar iteration but call a different function (not sum). The call to numpy.unique() and the iteration would have to be performed for each aggregated column which is bad in terms of performance. Background: I have records in a key / value like structure so there's a key column and a value column. I want to group by primary key and then create a column for each key. It's like pivot but without aggregation. Any ideas how to do this with SQL or custom functions in MonetDB? Kind regards, Daniel CREATE AGGREGATE python_aggregate(val INTEGER) RETURNS INTEGER LANGUAGE PYTHON { try: unique = numpy.unique(aggr_group) x = numpy.zeros(shape=(unique.size)) for i in range(0, unique.size): x[i] = numpy.sum(val[aggr_group==unique[i]]) except NameError: # aggr_group doesn't exist. no groups, aggregate on all data x = numpy.sum(val) return(x) };

1 0

Killing long running SQL queries
by Daniel Glöckner 09 Jul '20

09 Jul '20

Hi, sometimes things just go wrong ;) Is there a way to 1) list (long) running queries and 2) kill them? Kind regards, Daniel

2 1

Re: Re: Bit of documentation lacking (indexes, performance)
by gloeckner.daniel＠gmail.com 09 Jul '20

09 Jul '20

Hi, Thanks Martin Kersten, Alex Ossipov and Martin van Dinther for the valuable advice! It’s actually quite impressive to see how well MonetDB being a “knobless” system performs without any performance tuning. That’s what you do most of your day when using “big data” technologies from my experience 😉 but it’s of course not what you’re ultimately aiming to do… I think it could be considered as a unique selling point for MonetDB. Maybe the website should highlight this more and improve new user’s confidence level (indexes are not missing features but tuned automatically (how?)…., etc.). I’ll give PKs and FKs a shot to see how they affect performance. General advice, hire a working student and have her or him revamp the website 😃 There’s quite a bit of information but sometimes outdated and hard to navigate. My journey into MonetDB land continues, no blockers so far. Kind regards, Daniel

2 1

Bit of documentation lacking (indexes, performance)
by Daniel Glöckner 08 Jul '20

08 Jul '20

Hi, I'm currently in the process of evaluating MonetDB. Hats off for this great product! Very impressed by ease-of-use and performance so far. Looking for information what indexes are used by MonetDB I found a pretty empty page only: https://www.monetdb.org/Documentation/SQLReference/Indexes During our performance evaluation I was searching for common recipes to achieve good performance. Following page is unfortunately also under construction. https://www.monetdb.org/Documentation/SQLReference/PerformanceOptimization/… Is there any performance handbook available? I also did not find information regarding (performance) parameters for the DB server. Kind regards, Daniel

4 3

Huge merovingian.log after update
by Lynn Carol Johnson 07 Jul '20

07 Jul '20

HI all – A week or so ago we ran an upgrade to the lastest monetdb June-2020 patch. Our db was 2 patches old. While the log showed no specific problems, the upgrade effected our databases in that 2 of them would crash when they were accessed, and the third one (the largest one) I could access but could not see any tables. I ended up “destroying” all the dbs and recreating them fomr the saved binary files. Now, we are getting disk full messages (the db is only read – no one is adding to it). I look at the file sizes and there is a huge mervingian.log file: Is it a problem if I delete this file? What will it hurt? Should I “tail” it and save the end of it? Please advise. root@cbsudc01 dbfarmBL]# du -h --max-depth=1 1.3M ./testDB_lynn 52G ./maizeChrom10DB 955G ./maizeFullGenomeDB 1.3M ./testDB 6.9T . [lcj34@cbsudc01 dbfarmBL]$ l -ltr total 6320304952 -rw-rw---- 1 root buckler 339559112 Oct 9 2018 merovingian.log.old drwx------ 4 root root 121 Jun 26 13:54 testDB_lynn/ drwx------ 4 root root 121 Jul 1 17:23 maizeChrom10DB/ drwx------ 4 root root 121 Jul 1 17:23 testDB/ -rw-r--r-- 1 root root 6 Jul 1 17:23 merovingian.pid drwx------ 4 root root 4096 Jul 1 17:24 maizeFullGenomeDB/ -rw-r--r-- 1 root buckler 6471651893248 Jul 6 14:43 merovingian.log [lcj34@cbsudc01 dbfarmBL]$ Thanks - Lynn

3 5