- developers-list - monetdb.org

Announcement: New Aug2018-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 17 Oct '18

17 Oct '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP1 bugfix release (11.31.11) Build Environment * On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object. Whole System * Some subtle dependencies between RPMs have been fixed. Bug Fixes * 6640: timestamp_to_str returning incorrectly adjusted results * 6641: race condition in SQL UDF with update * 6642: Hanging query * 6648: key property potentially wrong after type conversion * 6649: Projection inside within transaction gives wrong results

1 0

mvc_create_table new parameter
by Panagiotis Koutsourakis 03 Sep '18

03 Sep '18

Hi all, trying to merge the data vaults branch with the default branch I noticed we have a new parameter named 'properties' to the 'mvc_create_table' function. Is there somewhere a comment or even better some documentation about what it means and what the allowed values are? Thanks, Panos.

1 1

Announcement: New Aug2018 Feature release of MonetDB suite
by Sjoerd Mullender 31 Aug '18

31 Aug '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018 feature release (11.31.7) MonetDB5 Server * The lsst module was moved to a separate repository (https://dev.monetdb.org/hg/MonetDB-lsst/). Build Environment * Build the MonetDB-cfitsio RPM and libmonetdb5-server-cfitsio Debian/Ubuntu package. * On Windows, the separate MonetDB5-Geom installer has been incorporated into the main MonetDB5-SQL installer and is therefore no longer available as a separate download. Merovingian * Added a "logrotate" configuration file. See /etc/logrotate.d/monetdbd. * Changed the monetdb profilerstart command to be more robust. If the server or stethoscope crashed before, the pid file is still there, so the next time we try to start stethoscope, it will fail. Now the profilerstart command will check if a stethoscope process with the recorded pid is running. If not, we start stethoscope, assuming that something went wrong before. * Changed the monetdb stop command to try to stop stethoscope before stoping the server. The error conditions that can arrise from attempting to stop stethoscope are: + The database is not running. + The profilerlogpath is not set. + The profiler.pid file does not exist or is inaccessible. + The contents of the profiler.pid are not valid. + Shutdown of stethoscope did not succeed. + Removing the profiler.pid file failed. In all the cases, the attempt to stop the server can continue normally, so we actually ignore any errors that rise from the attempt to stop stethoscope. Client Package * ODBC: Implemented SQL_ATTR_QUERY_TIMEOUT parameter in SQLSetStmtAttr. * ODBC SQLGetInfo now returns a positive numeric value for InfoTypes: SQL_MAX_COLUMN_NAME_LEN, SQL_MAX_DRIVER_CONNECTIONS, SQL_MAX_IDENTIFIER_LEN, SQL_MAX_PROCEDURE_NAME_LEN, SQL_MAX_SCHEMA_NAME_LEN, SQL_MAX_TABLE_NAME_LEN and SQL_MAX_USER_NAME_LEN. * Added a '-f rowcount' option in mclient to repress printing the actual data of a resultset, but only print the number of returned tuples Stream Library * Added support for lz4 compressed files in the stream library MonetDB Common * Hash indexes are now persistent across server restarts. * The macros bunfastapp and tfastins and variants no longer set the dirty flag of the heap they write to. This now needs to be done separately (and preferably outside of the inner loop). * Removed batDirty flag from BAT record. Its function is completely superseded by batDirtydesc and the dirty flags on the various heaps. * Removed "masksize" argument of function BAThash. * A whole bunch of functions that took an int argument that was used as a Boolean (true/false) value now take a value of type bool. The functions BATkeyed, BATordered and BATordered_rev now return a bool instead of an int. * Removed the tdense property: it's function is completely replaced by whether or not tseqbase is equal to oid_nil. Testing Environment * Removed helper programs Mtimeout and MkillUsers: they have long been superseded by timeout handling by Mtest.py itself. SQL Frontend * Removed deprecated table producing system functions: sys.dependencies_columns_on_functions() sys.dependencies_columns_on_indexes() sys.dependencies_columns_on_keys() sys.dependencies_columns_on_triggers() sys.dependencies_columns_on_views() sys.dependencies_functions_on_functions() sys.dependencies_functions_on_triggers() sys.dependencies_keys_on_foreignkeys() sys.dependencies_owners_on_schemas() sys.dependencies_schemas_on_users() sys.dependencies_tables_on_foreignkeys() sys.dependencies_tables_on_functions() sys.dependencies_tables_on_indexes() sys.dependencies_tables_on_triggers() sys.dependencies_tables_on_views() sys.dependencies_views_on_functions() sys.dependencies_views_on_triggers() They are replaced by new system dependency_* views: sys.dependency_args_on_types sys.dependency_columns_on_functions sys.dependency_columns_on_indexes sys.dependency_columns_on_keys sys.dependency_columns_on_procedures sys.dependency_columns_on_triggers sys.dependency_columns_on_types sys.dependency_columns_on_views sys.dependency_functions_on_functions sys.dependency_functions_on_procedures sys.dependency_functions_on_triggers sys.dependency_functions_on_types sys.dependency_functions_on_views sys.dependency_keys_on_foreignkeys sys.dependency_owners_on_schemas sys.dependency_schemas_on_users sys.dependency_tables_on_foreignkeys sys.dependency_tables_on_functions sys.dependency_tables_on_indexes sys.dependency_tables_on_procedures sys.dependency_tables_on_triggers sys.dependency_tables_on_views sys.dependency_views_on_functions sys.dependency_views_on_procedures sys.dependency_views_on_views * Implemented group_concat(X,Y) aggregate function which also concatenates a column of strings X, but using a supplied string Y as the separator. This function is also a SQL extension. * Implemented group_concat(X) aggregate function which concatenates a column of strings using a comma as a separator. This function is not featured in the SQL standard. Bug Fixes * 4020: Importing timestamp with zone from copy into * 6506: Improper performance counters * 6556: Sqlitelogictest division by zero on COALESCE call * 6564: Changes to the Remote Table definition * 6575: Sqlitelogictest crash on groupby query with coalesce call * 6579: Sqlitelogic test infinite loop while compiling SQL query * 6586: Sqlitelogictest crash on complex aggregation query * 6593: Poor performance with like operator and escape clause * 6596: Multicolumn aggregation very slow after ANALYZE when persistent hashes are enabled * 6605: Sqlitelogictest set queries with wrong results * 6606: Misleading parameter name in generate_series function * 6610: Sqlitelogictest algebra.rangejoin undefined * 6611: Cannot compile with GCC 8.1 and --enable-debug=no * 6612: Implement BLOB handling in python UDFs * 6618: dependency column on sequence violated by DROP SEQUENCE * 6621: SELECT FROM REMOTE TABLE WHERE <> returns wrong results * 6624: "Cannot use non GROUP BY column in query results without an aggregate function" when using aggregate function in both HAVING and ORDER BY clauses. * 6625: OR in subselect causes the server to crash with segmentation fault * 6627: stddev_pop inconsistent behaviour * 6628: User cannot insert into own local temporary table * 6629: CREATE TABLE IF NOT EXISTS returns 42000! * 6630: Sqlitelogictest cast NULL to integer failing * 6632: Dataflow causes crash when THRnew fails * 6633: ILIKE clauses don't work on certain characters * 6635: monetdbd exits due to "Too many open files" error * 6637: Within a transaction, \d after an error causes mclient to exit * 6638: (sequences of) mkey.bulk_rotate_xor_hash() can generate NIL from non-NIL making multi-col joins return wrong results * 6639: COMMENT ON TABLE abc IS NULL invalidly sets the remark column to null where remark column is defined as NOT NULLable

1 0

New Aug2018 stable branch created
by Sjoerd Mullender 30 Jul '18

30 Jul '18

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Aug2018. So in each of your stable checkout directories you can do hg pull hg update -rAug2018 to update to this new branch. -- Sjoerd Mullender

1 0

Python UDFs with blob column
by Florestan De Moor 19 Jul '18

19 Jul '18

Hello, I've been recently working with Python UDFs in MonetDB and I've noticed that returning a blob column from an array of python objects is not allowed. A look at the code confirmed that this feature is not implemented yet: "FIXME: check for byte array/or pickle object to string". I have thus written an implementation of this feature. Could you please tell me how I can submit my code for you to review it and consider to apply it? Thanks. Best regards, Florestan

3 3

Insert/update value manipulation
by Schediwy, Thomas (itsc) 11 Jul '18

11 Jul '18

Hi there, I am looking for the function where I can manipulate the values from an insert or update statement. For now, I am using a Solaris 10 system. But I havent found the functions where the insert or update will get done. I have looked trogh the files stream.c, mapi.c and mclient.c. But everytime I tried to change the values in the variables nothing happend. Can somebody please help me? Thanks in advise Thomas

3 5

Announcement: New Mar2018-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 20 Jun '18

20 Jun '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Mar2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Mar 2018-SP1 bugfix release (11.29.7) Build Environment * On Linux and Unix, try not to link to libraries multiple times: that only causes the code (and worse, the variables in those libraries) to be included multiple times. On Windows, we do need to link to libraries multiple times (once for each DLL we create if the DLL references anything from the library) but that doesn't cause the linked library to be loaded multiple times. SQL Frontend * Corrected the definition of view: sys.ids. Bug Fixes * 6562: Sqlitelogictest crash on group by query with not in operator * 6565: Sqlitelogictest crash on complex select query with coalesce call * 6566: Sqlitelogictest unavailable calc.- MAL operations * 6568: Sqlitelogictest crash on complex case query * 6569: Sqlitelogictest select query with not between cause with wrong results * 6570: Sqlitelogictest select coalesce undefined calc * 6572: ordered index Error in optimizer garbageCollector * 6573: Sqlitelogictest crash on complex select query with case statement * 6574: server crashed could not find ordered index while creating a table * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6577: creating temp table kills performance of the original query * 6578: One two-tuple insert gives different results than two single inserts * 6581: Join condition errors. * 6583: Fixed size string concatenation with integer results in fixed size string of size 0 * 6584: SELECT FROM REMOTE TABLE WHERE IS NOT NULL produces wrong result * 6585: Nested Merge tables cause an infinite loop in rel_optimizer * 6587: Sqlitelogictest crash on complex case statement * 6589: Sqlitelogictest crash on complex on complex expression * 6594: Sqlitelogictest crash on complex case statement * 6595: Remote decimal division triggers assertion / returns wrong answer * 6598: Python 3.4 not supported (due to usage of Py_DecodeLocale) * 6600: Sqlitelogictest queries fail to execute * 6601: "where is null" clause on remote table causes problem with next query * 6602: Sqlitelogictest wrong results in IN query * 6603: Sqlitelogictest: Aggregation query with distinct clause produces duplicated rows

1 0

Vakantie
by Martin Kersten 22 Apr '18

22 Apr '18

wanneer was je op vakantie? Sent from my iPad > On 22 Apr 2018, at 11:54, Niels Nes <commits+niels=cwi.nl(a)monetdb.org> wrote: > > Changeset: 354f2d85e58b for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=354f2d85e58b > Added Files: > sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err > sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out > Modified Files: > sql/server/rel_select.c > Branch: Mar2018 > Log Message: > > fixes for bug 6589 > > > diffs (99 lines): > > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -2098,7 +2098,10 @@ rel_logical_value_exp(mvc *sql, sql_rel > > r = rel_value_exp(sql, &z, sval, f, ek); > if (l && r && IS_ANY(st->type->eclass)){ > - l = rel_check_type(sql, exp_subtype(r), l, type_equal); > + sql_exp *nl = rel_check_type(sql, exp_subtype(r), l, type_equal); > + if (nl != l && is_new) > + left = rel_project_exp(sql->sa, nl); > + l = nl; > if (l) > st = exp_subtype(l); > } > diff --git a/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err > @@ -0,0 +1,35 @@ > +stderr of test 'sqlitelogictest-complex-expression.Bug-6589` in directory 'sql/test/BugTracker-2018` itself: > + > + > +# 11:47:28 > > +# 11:47:28 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" "mapi_open=true" "--set" "mapi_port=31863" "--set" "mapi_usock=/var/tmp/mtest-18504/.s.monetdb.31863" "--set" "monet_prompt=" "--forcemito" "--dbpath=/home/niels/scratch/rc-old/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2018" "--set" "embedded_c=true" > +# 11:47:28 > > + > +# builtin opt gdk_dbpath = /home/niels/scratch/rc-old/Linux-x86_64/var/monetdb5/dbfarm/demo > +# builtin opt gdk_debug = 0 > +# builtin opt gdk_vmtrim = no > +# builtin opt monet_prompt = > > +# builtin opt monet_daemon = no > +# builtin opt mapi_port = 50000 > +# builtin opt mapi_open = false > +# builtin opt mapi_autosense = false > +# builtin opt sql_optimizer = default_pipe > +# builtin opt sql_debug = 0 > +# cmdline opt gdk_nr_threads = 0 > +# cmdline opt mapi_open = true > +# cmdline opt mapi_port = 31863 > +# cmdline opt mapi_usock = /var/tmp/mtest-18504/.s.monetdb.31863 > +# cmdline opt monet_prompt = > +# cmdline opt gdk_dbpath = /home/niels/scratch/rc-old/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2018 > +# cmdline opt embedded_c = true > +# cmdline opt gdk_debug = 553648138 > + > +# 11:47:28 > > +# 11:47:28 > "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" "--host=/var/tmp/mtest-18504" "--port=31863" > +# 11:47:28 > > + > + > +# 11:47:28 > > +# 11:47:28 > "Done." > +# 11:47:28 > > + > diff --git a/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out > @@ -0,0 +1,39 @@ > +stdout of test 'sqlitelogictest-complex-expression.Bug-6589` in directory 'sql/test/BugTracker-2018` itself: > + > + > +# 11:47:28 > > +# 11:47:28 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" "mapi_open=true" "--set" "mapi_port=31863" "--set" "mapi_usock=/var/tmp/mtest-18504/.s.monetdb.31863" "--set" "monet_prompt=" "--forcemito" "--dbpath=/home/niels/scratch/rc-old/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2018" "--set" "embedded_c=true" > +# 11:47:28 > > + > +# MonetDB 5 server v11.29.2 > +# This is an unreleased version > +# Serving database 'mTests_sql_test_BugTracker-2018', using 8 threads > +# Compiled for x86_64-unknown-linux-gnu/64bit > +# Found 15.387 GiB available main-memory. > +# Copyright (c) 1993 - July 2008 CWI. > +# Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved > +# Visit https://www.monetdb.org/ for further information > +# Listening for connection requests on mapi:monetdb://xps13:31863/ > +# Listening for UNIX domain connection requests on mapi:monetdb:///var/tmp/mtest-18504/.s.monetdb.31863 > +# MonetDB/GIS module loaded > +# MonetDB/SQL module loaded > + > +Ready. > + > +# 11:47:28 > > +# 11:47:28 > "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" "--host=/var/tmp/mtest-18504" "--port=31863" > +# 11:47:28 > > + > +#SELECT DISTINCT + - 82 * + 26 - 12 * + - 27 * + + CASE WHEN NOT NULL > +#IN ( - 69, ( + SUM ( DISTINCT CAST ( NULL AS INTEGER ) ) ) + + MAX ( - 6 ) + + - COUNT ( * ) / - - ( + 77 ) + 37, + 44, > +#- - 0 * + + 23, - - 86, 3, 12 ) THEN + 41 WHEN NOT - 21 / - 95 IS NOT NULL THEN 63 END AS col0; > +% .L42 # table_name > +% col0 # name > +% int # type > +% 1 # length > +[ NULL ] > + > +# 11:47:28 > > +# 11:47:28 > "Done." > +# 11:47:28 > > + > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 1

INSERT using multiple cores (inspired by COPY from CSV in LOCKED mode)
by Anton Kravchenko 12 Apr '18

12 Apr '18

Hi there, 1) It looks like INSERTing into tables is bounded by a single CPU. And it kills performance of a query that normally uses multiple CPUs... CREATE TEMP TABLE t1 AS (some 'select query' that uses multiple cores when run standalone) ON COMMIT PRESERVE ROWS; or INSERT into t1 (some 'select query' that uses multiple cores when run standalone); 2) Do you think you could implement INSERT with LOCKED option? (like it was done for COPY from CSV in LOCKED mode) Please let me know whether you think it's doable. Thank you, Anton

1 1

ordered index Error in optimizer garbageCollector
by Anton Kravchenko 06 Apr '18

06 Apr '18

Here is a reproducible example. create table t1(v0 int, v1 char(1)); insert into t1 values (1,'a'),(2,'b'),(3,'c'); create ordered index index_t1_v1 on t1(v1); create table t2(v1 char(1)); insert into t2 values ('a'); 1) gives "Error in optimizer garbageCollector" create temp table t3 as (select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1) on commit preserve rows; 2) works as expected create temp table t3 as (select t1.v0 from t1,t2 where t2.v1=t1.v1) on commit preserve rows; 3) works as expected select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1; 4) works as expected drop index index_t1_v1; create temp table t3 as (select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1) on commit preserve rows; I am using the latest MonetDB version: MonetDB 5 server v11.29.3 "Mar2018" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 31.3GiB available memory, 4 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.2k 26 Jan 2017 (compiled with ) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: mockbuild@ (x86_64-redhat-linux-gnu) Compilation: gcc -std=gnu99 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic Linking : /usr/bin/ld -m elf_x86_64 -Wl,-z,relro -Wl,-Bsymbolic-functions Thanks, Anton

1 0