- developers-list - monetdb.org

Announcement: New release candidate of MonetDB suite
by Sjoerd Mullender 14 Jun '16

14 Jun '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This is a heads up announcement. We at CWI and MonetDB have prepared a release candidate for the Jun2016 release. Usually we just foist a new release upon you, our unsuspecting users, but this time, because there are so many changes, we thought it better to give you a little warning. See http://dev.monetdb.org/downloads/testing/ for the release and the preliminary release notes [1]. Among the changes is a complete overhaul of the MonetDB-geom module. If you use that, make very sure you have your backups in order. Also note that since this new version depends on an up-to-date version of the geos library, there is no geom module available for older Linux releases (in particular, Debian 7 (wheezy), Ubuntu 12.04 LTS (Precise Pangolin) and Ubuntu 14.04 LTS (Trusty Tahr), and RedHat Enterprise Linux (CentOS and Scientific Linux) 6). The plan is to turn this release candidate into an actual release before the end of this month (June 2016). If you find a bug in this release candidate, please let us know on the bug tracker [2]. We will then determine whether to go ahead with the release (and hopefully fix the bug in the next bug-fix release) or postpone the release and fix the bug first. [1] https://dev.monetdb.org/downloads/testing/relnotes.html [2] http://bugs.monetdb.org (a.k.a. https://www.monetdb.org/bugzilla/) - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXX/EsAAoJEISMxT6LrWYg+AcIAIgIST/dUrL8fDjwp3/G41zc 4g9xWbSxIE/+nZLjlHM6Onr4sETceOVdDVuxUGoIbeu9+DZNQqEakUUrABrldvLx ie/kbNINtdntFvfF3W5Lyo/lWhel3IUZZhEXvsBT9JCSfg5kiLKYr/CPqw5QgOcU XgccIn7RAZYovfVoLo7jLxiMLeT9DJp74gy1OkjrIBwQ/GPVam/CzaO5Ungu0dRY cZiqoRhW1GEtUQHABq0O9J/gJcHfFbSlwDeQcnTouybO320gBRISYqgV8aTt0RdU /MVVkdOU3hLsShjAgWinRbQPNVwCLcZNAfxER9q4Z9tt15n/3E6WtPeKo2Bv030= =rnCK -----END PGP SIGNATURE-----

1 0

Announcement: New release candidate of MonetDB suite
by Sjoerd Mullender 14 Jun '16

14 Jun '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This is a heads up announcement. We at CWI and MonetDB have prepared a release candidate for the Jun2016 release. Usually we just foist a new release upon you, our unsuspecting users, but this time, because there are so many changes, we thought it better to give you a little warning. See http://dev.monetdb.org/downloads/testing/ for the release and the preliminary release notes [1]. Among the changes is a complete overhaul of the MonetDB-geom module. If you use that, make very sure you have your backups in order. Also note that since this new version depends on an up-to-date version of the geos library, there is no geom module available for older Linux releases (in particular, Debian 7 (wheezy), Ubuntu 12.04 LTS (Precise Pangolin) and Ubuntu 14.04 LTS (Trusty Tahr), and RedHat Enterprise Linux (CentOS and Scientific Linux) 6). The plan is to turn this release candidate into an actual release before the end of this month (June 2016). If you find a bug in this release candidate, please let us know on the bug tracker [2]. We will then determine whether to go ahead with the release (and hopefully fix the bug in the next bug-fix release) or postpone the release and fix the bug first. [1] https://dev.monetdb.org/downloads/testing/relnotes.html [2] http://bugs.monetdb.org (a.k.a. https://www.monetdb.org/bugzilla/) - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXX/EsAAoJEISMxT6LrWYg+AcIAIgIST/dUrL8fDjwp3/G41zc 4g9xWbSxIE/+nZLjlHM6Onr4sETceOVdDVuxUGoIbeu9+DZNQqEakUUrABrldvLx ie/kbNINtdntFvfF3W5Lyo/lWhel3IUZZhEXvsBT9JCSfg5kiLKYr/CPqw5QgOcU XgccIn7RAZYovfVoLo7jLxiMLeT9DJp74gy1OkjrIBwQ/GPVam/CzaO5Ungu0dRY cZiqoRhW1GEtUQHABq0O9J/gJcHfFbSlwDeQcnTouybO320gBRISYqgV8aTt0RdU /MVVkdOU3hLsShjAgWinRbQPNVwCLcZNAfxER9q4Z9tt15n/3E6WtPeKo2Bv030= =rnCK -----END PGP SIGNATURE-----

1 0

Updating for GCC 6
by Sahasranaman M S 24 May '16

24 May '16

Hi all, GCC 6.1 was relased late last month, and it was failing to build latest MonetDB, with the below 3 errors: stream.c:2116:24: error: logical ‘or’ of equal expressions [-Werror=logical-op] (errno == EAGAIN || errno == EWOULDBLOCK) && /* it was! */ ^~ stream.c:2142:24: error: logical ‘or’ of equal expressions [-Werror=logical-op] (errno == EAGAIN || errno == EWOULDBLOCK) ^~ mal_module.c: In function ‘getHelp’: mal_module.c:455:3: error: this ‘for’ clause does not guard... [-Werror=misleading-indentation] for(s= m->subscope[j]; s; s= s->peer) ^~~ mal_module.c:513:4: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘for’ m= m->sibling; ^ The first two appear to be because EAGAIN, EWOULDBLOCK have the same value on my machine (x86_64, Fedora 23 64bit). I don't know in what situation they would be different from each other, so I'm removing the check for EWOULDBLOCK and checking only for EAGAIN, as suggested by some on the Internet. The third error is due to bad indentation. If we reduce indentation of mal_module.c:513 by two levels, that solves it. It works for me with these changes, could someone confirm my solution to first two errors won't cause regressions, of suggest alternative solutions? Thanks, Sahas

2 6

Cache functions ?
by Brian Hood 16 May '16

16 May '16

Hi Guys, I've just noticed the cache functions in the mapi code what do these actually do and also look interesting ? mapi_export MapiMsg mapi_cache_limit(Mapi mid, int limit); mapi_export MapiMsg mapi_cache_shuffle(MapiHdl hdl, int percentage); mapi_export MapiMsg mapi_cache_freeup(MapiHdl hdl, int percentage); Regards, Brian Hood

1 0

Memory management - MonetDB on CentOS
by Vijay Krishna 07 May '16

07 May '16

Hi, We have been running tests on Memory management with the latest version of MonetDB on a CentOS installation. The machine has 1TB hard disk and 128 GB RAM space. The swap memory configured is 2GB. We were aware that MonetDB completely relies on the OS for Memory Management and is prone to OOM Killer crash and hence we wanted to simulate the crash. Hence we ran hundreds of threads executing various queries and noticed the following pattern of memory handling. 1. Initially, RAM was filled. 2. Upon RAM overflow, swap memory (2 GB) was used. 3. Upon swap overflow, the swap memory got extended and the 1TB memory that was free on the machine was used completely. 4. The OOM killer got invoked only when this 1TB memory got filled. 5. On consecutive executions, swap memory was started to be used even when RAM had 70GB of memory unused and free (not even used for cache files). 6. The /tmp partition was never used. Memory got overfilled only in the current partition. These observations led to the following questions about the memory management with CentOS. 1. Does MonetDB has no influence on the above behaviour and is it that only the OS is responsible for this? 2. Will the swap memory be expanded from the committed size by the OS if swap is running out of memory? In our case it expanded from 2GB to 1 TB. Our OS setting of overcommit is 0 (sysctl vm.overcommit_memory = 0) 3. When will the OOM Killer be invoked usually? Upon Swap fill or upon Hard disk space full? 4. Why was memory swapped even when there is 70GB free space in RAM? We tried reducing the swappiness to 10 (sysctl vm.swappiness=10), even then memory was swapped with 70 GB free space in RAM. 5. Does the OS writes intermediate results to /tmp directory? Or is it that MonetDB is bypassing this? 6. Is there any other OS configurations that we need to cross check regarding memory management? Any help much appreciated. Thanks in advance. Regards, Vijay.

1 0

Re: MonetDB: default - No need for IMPRINTS being a full-blown keyword.
by Lefteris 04 May '16

04 May '16

Sjoerd, there is a plan for imprints to become a keyword in SQL such that one may do the following: CREATE IMPRINTS INDEX idx_id ON table_id(column_id) just as ORDERED index. This is needed in many cases where for example Romulo wants to be able to create and drop imprints at will. On Wed, May 4, 2016 at 11:25 AM, Sjoerd Mullender <commits+sjoerd=acm.org(a)monetdb.org> wrote: > Changeset: 390e25b31196 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=390e25b31196 > Modified Files: > sql/server/sql_parser.y > Branch: default > Log Message: > > No need for IMPRINTS being a full-blown keyword. > > > diffs (11 lines): > > diff --git a/sql/server/sql_parser.y b/sql/server/sql_parser.y > --- a/sql/server/sql_parser.y > +++ b/sql/server/sql_parser.y > @@ -5089,6 +5089,7 @@ non_reserved_word: > | TIME { $$ = sa_strdup(SA, "time"); } > | TIMESTAMP { $$ = sa_strdup(SA, "timestamp"); } > | INTERVAL { $$ = sa_strdup(SA, "interval"); } > +| IMPRINTS { $$ = sa_strdup(SA, "imprints"); } > > | PREPARE { $$ = sa_strdup(SA, "prepare"); } > | EXECUTE { $$ = sa_strdup(SA, "execute"); } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Monetdb Python Issue
by Patel, Nikunj (CONT) 26 Apr '16

26 Apr '16

Hi, I have created a simple table (create table dept (dept_id integer, dept_nm varchar(25)) and trying to insert records from file using Python and getting errors. All execute statements gives me same error message: connection = monetdb.sql.connect(username="monetdb", password="monetdb", hostname="10.81.102.71", port=2234, database="demo") cursor = connection.cursor() cursor.execute("COPY INTO dept FROM 'C:\AWS\did.csv' USING DELIMITERS;") cursor.execute("COPY INTO dept FROM 'C:\AWS\did.csv' USING DELIMITERS") cursor.execute("COPY INTO dept FROM STDIN;\n44444444,'AL'\n55555555,'JAFFRI'") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\sql\cursors.py", line 175, in execute block = self.connection.execute(query) File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\sql\connections.py", line 138, in execute return self.command('s' + query + ';') File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\sql\connections.py", line 143, in command return self.mapi.cmd(command) File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\mapi.py", line 202, in cmd raise OperationalError(response[1:])monetdb.exceptions.OperationalError: 25005!current transaction is aborted (please ROLLBACK) Thanks, --------------------- Nik Patel ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

1 0

Announcement: New Jul2015-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 11 Apr '16

11 Apr '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP4 bugfix release MonetDB Common * Fixed a bug that caused various instances where old data returned or where crashes occurred. The problem was that internally data wasn't always marked dirty when it was being changed, causing later processing to not deal with the changed data correctly. Bug Fixes * 3905: MonetDB doesn't handle ANY/SOME/ALL operator correctly * 3929: R aggregate not recognized when using 3 or more parameters * 3965: Not possible to quote/escape single quote character in the name of the file to load. * 3968: Missing double use of column names

1 0

undefined reference to `pcre_config'
by John Krulick 05 Apr '16

05 Apr '16

Hi, I'm trying to build MonetDB from source and running into an error during configure. Running ./configure gives: ... ... checking for SHA256... yes checking for SHA384... yes checking for SHA512... yes checking for pcre... yes checking whether pcre comes with UTF-8 support... no configure: error: PCRE library not found but required for MonetDB5 At the bottom of the email is the error in the config.log. pcretest -C gives me: PCRE version 8.38 2015-11-23 Compiled with:q 8-bit support UTF-8 support Unicode properties support No just-in-time compiler support Newline sequence is LF \R matches all Unicode newlines Internal link size = 2 POSIX malloc threshold = 10 Parentheses nest limit = 250 Default match limit = 10000000 Default recursion depth limit = 10000000 Match recursion uses stack The issue, I think, is I have PCRE version 8.38 built in a non-standard location, /home/john/monetdb_prereq/pcre export pcre_LIBS=-L/home/john/monetdb_prereq/pcre/lib export pcre_CFLAGS=-I/home/john/monetdb_prereq/pcre/include Any idea what I'm missing? Thanks, John configure:20861: checking for SHA512 configure:20861: gcc -o conftest -g -O2 conftest.c -lssl -lcrypto -ldl -lz >&5 configure:20861: $? = 0 configure:20861: result: yes configure:20901: checking for pcre configure:20966: result: yes configure:20971: checking whether pcre comes with UTF-8 support configure:20992: gcc -o conftest -g -O2 -I/home/john/monetdb_prereq/pcre/include conftest.c -L/home/john/monetdb_prereq/pcre/lib >&5 /tmp/ccRjfKcQ.o: In function `main': /home/john/MonetDB-11.21.17/conftest.c:66: undefined reference to `pcre_config' collect2: ld returned 1 exit status configure:20992: $? = 1 configure: program exited with status 1 configure: failed program was: | /* confdefs.h */ | #define PACKAGE_NAME "MonetDB" | #define PACKAGE_TARNAME "MonetDB" | #define PACKAGE_VERSION "11.21.17" | #define PACKAGE_STRING "MonetDB 11.21.17" | #define PACKAGE_BUGREPORT "info(a)monetdb.org" | #define PACKAGE_URL "http://www.monetdb.org/" | #define PACKAGE "MonetDB" | #define VERSION "11.21.17" | #define HOST "x86_64-unknown-linux-gnu" | #define MONETDB_RELEASE "Jul2015-SP3" | #define HAVE_CONSOLE 1 | #define MONETDB5_PASSWDHASH "SHA512" | #define STDC_HEADERS 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define __EXTENSIONS__ 1 | #define _ALL_SOURCE 1 | #define _GNU_SOURCE 1 | #define _POSIX_PTHREAD_SEMANTICS 1 | #define _TANDEM_SOURCE 1 | #define SIZEOF_LONG 8 | #define HAVE_DLFCN_H 1 | #define LT_OBJDIR ".libs/" | #define NDEBUG 1 | #define DIFF "diff" | #define DIR_SEP '/' | #define DIR_SEP_STR "/" | #define PATH_SEP ':' | #define PATH_SEP_STR ":" | #define SO_PREFIX "lib" | #define SO_EXT ".so" | #define HAVE_LIBBZ2 1 | #define HAVE_MALLOPT 1 | #define HAVE_PTHREAD_H 1 | #define HAVE_SEMAPHORE_H 1 | #define HAVE_SCHED_H 1 | #define HAVE_PTHREAD_KILL 1 | #define HAVE_PTHREAD_SIGMASK 1 | #define HAVE_PTHREAD_SETSCHEDPRIO 1 | #define HAVE_LIBPTHREAD 1 | #define HAVE_LIBREADLINE 1 | #define HAVE_ICONV 1 | #define ICONV_CONST | #define HAVE_UUID 1 | #define HAVE_MD5 1 | #define HAVE_RIPEMD160 1 | #define HAVE_SHA1 1 | #define HAVE_SHA224 1 | #define HAVE_SHA256 1 | #define HAVE_SHA384 1 | #define HAVE_SHA512 1 | #define HAVE_OPENSSL 1 | /* end confdefs.h. */ | #include <pcre.h> | int | main () | { | int rc; pcre_config(PCRE_CONFIG_UTF8, &rc); exit(!rc); | ; | return 0; | } configure:20996: result: no configure:21021: error: PCRE library not found but required for MonetDB5

2 2

Announcement: New Jul2015-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Mar '16

25 Mar '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP3 bugfix release MonetDB5 Server * Fixed potential crash in MAL debugger when accessing BATs by index. Functionality dropped as it is also a security leak. Bug Fixes * 2972: SQL URL functionality contains errors * 3549: bulk string operations very slow * 3881: Server crashes on bulk load * 3890: Window function + group by in subselect, rel2bin_project: Assertion `0' failed * 3891: MonetDB crashes when executing SQL with window function * 3900: null handling in some sql statements is incorrect * 3906: Multi-column 1-N table-function with mitosis produces different column counts * 3908: LEFT JOIN with OR conditions triggers assertion * 3909: Incorrect column name in OR condition of LEFT JOIN crashes mserver * 3910: COPY INTO table (column1, column2) got wrong result * 3912: When table/column names conflicts, data ends in multiple tables! * 3917: Date difference returns month_interval instead of day_interval * 3918: MonetDB.R version 1.0.1 incorrectly constructs the batfile script * 3919: Table conflict when the table name and fields are identical * 3921: Creating a table from a complex query crashes mserver or triggers assertion * 3922: AVG( column ) returns NaN rather than Inf when column contains Inf * 3928: When killing a virtual machine, sql_logs/sql/log is empty * 3930: Wrong typecast on character columns in prepared statements when using Umlaute * 3932: CASE expressions with constants are not evaluated correctly * 3933: replace "exit" by "throw new Exception" * 3937: bad BAT properties with binary copy into and NULL values * 3938: Wrong error message on violating foreign key constraint * 3940: Date calculation and comparison produce wrong result * 3941: Wrong coercion priority * 3948: SQL: select * from sys.sys.table_name; is accepted but should return an error * 3951: extern table_funcs not visible from Windows DLL for extensions like vaults (crashes) * 3952: Stream table gives segfault * 3953: MIN/MAX of a UUID column produces wrong results * 3954: Consolidate table assertion error * 3955: (incorrect) MAL loop instead of manifold triggered by simple change in target list

1 0