- users-list - monetdb.org

Performance issue with count distinct
by AUBERT Simon 29 Mar '19

29 Mar '19

Hello all, Here a simple issue I have with performance : 1/ I create a table A and populate it with a bulk load 2/I create a table B, I populate the table with bulk load 3/I create table C with a CTAS statement. I analyze the statistics and set the primary key 4/I try a "count distinct" type query (on all the table) on the PK Field. It is pretty slow, with a high sollicitation of my hard drive for several dozens of second. the "select count" is however fast. So I have these following questions : 1/why despite statistics the count distinct query is slow? I thought the idea of statistics was to get some metrics about the table? 2/if the field I make a count distinct on is the PK (or the set of field is the PK), it's exactly the same than a count non null. So why the time are not the same? 3/if these issues are design issue or bugs, how can I ask to make it solved? For information : Monetdb 11.31.13 on Windows 10 Best regards, Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

2 1

Program contains errors.:(NONE).multiplex
by Niklas B 29 Mar '19

29 Mar '19

Hi, I'm upgrading from an old MonetDB (11.26.0 source snapshot) to the latest proper release. However, my Python aggregate functions appear to have stopped working (Program contains errors.:(NONE).multiplex): Running mserver5 with: /usr/local/bin/mserver5 --daemon=yes --set embedded_py=true --dbpath=/data/monetdb-farm/ --set gdk_nr_threads=1 --set max_clients=6 Reproduce with: CREATE FUNCTION test_udf("conum" int) RETURNS STRING LANGUAGE PYTHON3 {{ }}; Create table test("id" uuid, "date" DATE, "number" INT); select test_udf("number") from test; Gives: Program contains errors.:(NONE).multiplex I'm running it via Docker (stock Debian) with only MonetDB installed via apt-get. Not sure if this is connected: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378 <https://www.monetdb.org/bugzilla/show_bug.cgi?id=6378> Anyone having any similar issues? Tried compiling from source (April 2019 bransch) as well, no luck Regards, Niklas

3 2

sys.storage sorted attribute changes for primary key
by Jeremy Norris 29 Mar '19

29 Mar '19

Hello, I have a large table (~700 million rows, ~150 columns). Initially after import, the database is very fast on primary key lookups, however after a few thousand creates/updates/deletes on the database, "sorted" (as reported by sys.storage), flips from true to false. This results in major performance degradation. What are the conditions under which a database can lose it's sorted-ness on the primary key column? eg: sql>select schema, table, column, sorted from sys.storage where schema = 'test1' and table = 'cmdb' and column = 'pk_id'; +---------+-------+--------+--------+ | schema | table | column | sorted | +=========+=======+========+========+ | test1 | cmdb | pk_id | false | +---------+-------+--------+--------+ Thanks, -Jeremy Norris

2 1

Fwd: Question about partition bounds of updatable merge tables
by Yinjie Lin 06 Mar '19

06 Mar '19

Hi, I found that there is a new feature called updatable merge table in the new release (Updatable merge tables with partition predicates <https://www.monetdb.org/blog/updatable-merge-tables>). I think this makes merge table similar to *partitioned table* in other databases. But I found that the upper bound of range partitioning is an inclusive bound, while the upper bound of some databases like MySQL and PostgreSQL is an exclusive bound. So my question is: why is the upper bound declaration designed like this? Will it be possible if MonetDB be consistent with other databases? Besides, currently merge table doesn't support partition pruning, so all child tables will be scanned even if the filter condition contains the partitioning key. Is it possible if this feature will be added one day? Best Regards, Yinjie Lin

1 0

#!ERROR: GDKmove: cannot rename /mnt/g/monet/wal/sql/log to /mnt/g/monet/wal/sql/log.bak
by Michael Jansen 21 Feb '19

21 Feb '19

Hello, I have a crash after about 5 minutes when running in transaction replication mode. After the crash I cannot access the db. I am running: mserver5 --trace --dbpath=/data/monetdb/dbfarm/test_db/ --set gdk_keep_persisted_log_files=1000 --set gdk_logdir=/mnt/g/monet/wal --set sql_debug=4 I am using GlusterFS under /mnt/g to manage the replication. I am running on AWS Linux 2. and I have free space on in both the data and wal directories. (I assume this is enough space): Filesystem Size Used Avail Use% Mounted on 10.0.1.68:/gv0 8.0G 7.4G 669M 92% /mnt/g bash-4.2$ the db only has one test table: sql>create table t (n float, t text) ; operation successful sql>select * from t ; +--------------------------+------+ | n | t | +==========================+======+ | 1.1 | blah | | 1.1 | blah | | 1 | blah | +--------------------------+------+ 3 tuples any help is appreciated! -mj

1 0

Error when installing on Centos 7
by Trevor Andreas 25 Jan '19

25 Jan '19

Hello, I am getting an error trying to install MonetDB following the instructions here: https://dev.monetdb.org/downloads/epel/ The error I am getting is when running: yum install -y MonetDB-SQL-server5 MonetDB-client failure: repodata/repomd.xml from monetdb: [Errno 256] No more mirrors to try. https://dev.monetdb.org/downloads/epel/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found Looks like the repodata is missing for Centos 7? It looks to be there for Centos 6 https://dev.monetdb.org/downloads/epel/6/x86_64/repodata/ Thanks! Trevor

2 1

Invalid Credentials error
by AUBERT Simon 17 Jan '19

17 Jan '19

Good morning, I have used Monetdb for a few weeks now and this evening I cannot login after a full system restart (windows+monetdb). Environment : Windows 10 (well, it's my work computer, I don't have a lot of choice) Monetdb 5 server v11.31.11 I start monetdb by using C:\Program Files\MonetDB\MonetDB5\bin\mserver5.exe Then mclient on C:\Program Files\MonetDB\MonetDB5\mclient.bat (and usually it work just fine) I also tried with mclient.exe using the -x command to debug. Error message : C:\Program Files\MonetDB\MonetDB5\bin>mclient -X -d demo user(win32):monetdb password: MAPI = monetdb@localhost:50000 ACTION= mapi_reconnect ERROR = !InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' Of course, I use the default monetdb/monetdb credentials. I have no idea of what happens and moreover I have lost an entire schema (I see that if I connect through Tableau Desktop...). I have done an upgrade on the last version (11.31.13) and same result, impossible to connect. And same result. I spend the entire evening on that a Thank in advance. Simon AUBERT [cid:691B9F4089B9F34B8E0C22B39B7117EC@eurprd07.prod.outlook.com] Bourse Maritime - 1, place Lainé - 33 000 Bordeaux Mob : +33(0)6 66 28 52 04

2 3

table-returning function with subselect in input - no mitosis?
by Roberto Cornacchia 14 Jan '19

14 Jan '19

Hi, * Background: I have a custom string tokenizer that I invoke this way: SELECT * FROM tokenize( (SELECT id, s FROM my_strings) ); The output has of course a 1-to-N rows relation with the input. The function has a bulk implementation, so it takes M rows and it outputs M rows. * The question: No matter how large the input table my_strings is, mitosis does not seem to kick in. Is it not allowed in this case because it cannot assume that my custom function allows to concatenate partial results? Do you see any chance for me to enable mitosis with this scenario? Thanks, Roberto

1 1

Announcement: New Aug2018-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Jan '19

14 Jan '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP2 bugfix release (11.31.13) SQL Frontend * Disabled function sys.getcontent(url). Bug Fixes * 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered * 6645: optimizer treats all the functions with no or constant parameters as constant * 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s * 6651: Multi-column IN clause for subquery produces wrong results * 6653: CREATE TABLE accepts empty table/column name * 6654: Incorrect handling of 'TRUE' in compound select * 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message * 6660: GRANTing a ROLE is not idempotent * 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt. * 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489 * 6665: Creation of serial types does not accept negative numbers * 6666: COPY INTO from .. LOCKED doubles input data * 6668: The SAMPLE key word doesn't work in a subquery. * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value

1 0

Re: Performance degradation
by Jeremy Norris 12 Jan '19

12 Jan '19

Here's some additional observations on this puzzling issue: The slow table is "cmdb", the normal table is "cmdb2". Both have the same number of rows according to "SELECT count(*)" (73553310). However, sys.statistics shows something a bit different: sql>SELECT x.name, count(*) AS columns_with, z.count FROM sys.tables AS x, sys.columns AS y, sys.statistics AS z WHERE x.id = y.table_id AND y.id = z.column_id GROUP BY x.name, z.count; +-------+--------------+----------+ | name | columns_with | count | +=======+==============+==========+ | cmdb | 8 | 73553310 | | cmdb | 145 | 73553311 | | cmdb2 | 153 | 73553310 | +-------+--------------+----------+ 3 tuples sql> For the slow table, "cmdb", there are 145/153 columns that have an extra "row". This row count mismatch causes it to skip parallelization optimizations in "monetdb5/optimizer/opt_mitosis.c", line 212. A few related questions: 1) In the columnar model, is is valid for conceptual tuples to exist which don't have values for each column like this? (This seems strange as now there's a difference between an explicit NULL and "not existing"). How might this situation have happened? (Only transactional full-row inserts were applied to this DB). Can this be repaired? 2) Is the conditional in "monetdb5/optimizer/opt_mitosis.c", line 212 sufficiently robust? I may be misunderstanding the full context of this line, but it seems like we could still benefit from splitting/parallelization when r < rowcnt, no? Thanks, -Jeremy Norris

1 0