- developers-list - monetdb.org

monetdb.org system maintenance
by Sjoerd Mullender 01 Mar '17

01 Mar '17

Tomorrow, March 2, we will be upgrading the hardware for the system on which all monetdb.org infrastructure runs. Expect some downtime. -- Sjoerd Mullender

1 0

Announcement: New Dec2016-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 26 Jan '17

26 Jan '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP1 bugfix release Bug Fixes * 4039: Slow mserver5 start after drop of tables (> 1 hour) * 4048: Segfault on vacuum with parallel updates * 6079: pushselect optimizer bug on MAL snippet * 6140: INNER JOIN gives the results of a CROSS JOIN * 6150: Query giving wrong results, extra records are appearing * 6175: The program can't start because python27.dll is missing from your computer. * 6178: AVG + GROUP BY returns NULL for some records that should have results * 6179: mergetable optimizer messes up sample * 6182: sys.shutdown triggers assertion in clients.c * 6184: Incorrect result set - Extra records in result set

1 0

python3 client bug+fix
by Bas Bloemsaat 23 Dec '16

23 Dec '16

Hi, Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3. The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply. Regards, Bas Bloemsaat --- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1

2 2

Announcement: New Dec2016 Feature release of MonetDB suite
by Panagiotis Koutsourakis 21 Dec '16

21 Dec '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016 feature release MonetDB5 Server * Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y. * Removed command bbp.getHeat(). * Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead. * Removed function BKCappend_reverse_val_wrap: it was unused. * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. * Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces Build Environment * New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB. * With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore. * Removed obsolete code associated with long gone static linking option. * Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture. . * The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github (https://github.com/gijzelaerr/pymonetdb) the other clients are maintained by CWI/MonetDB on our own server (https://dev.monetdb.org/hg/monetdb-java, https://dev.monetdb.org/hg/monetdb-perl, https://dev.monetdb.org/hg/monetdb-php) MonetDB Common * The tnokey values must now be 0 if it is not known whether all values in a column are distinct. * The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique. * Implemented conversion to str from any type (not just the internal types). * VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure. * BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8. * BATattach now copies the input file instead of "stealing" it. * Removed the lastused "timestamp" from the BBP. * Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function. * Removed unused functions BBPhot and BBPcold. * Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop. * Removed the macro BUNfirst. It can be replaced by 0. * Changed BATroles by removing the argument to set the name of the head column. * The head column is now completely gone. MonetDB is completely "headless". * The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase. * BATs now only have a single (logical) name. * The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column. * BATkey now works on the TAIL column instead of the HEAD column. * Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL. * Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID). * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. SQL * Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y. * The experimental recycler code is moved to the attic. * The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.) Bug Fixes * 3357: Implement setQueryTimeout() * 3445: Add support for database name to dotmonetdb file * 3597: SQL to MAL listing looses types * 3973: JDBC hangs * 3976: Performance enhancement to LIKE without wildcards * 4005: Correlated update causes incorrect null constraint violation * 4016: merge table only optimises for point query * 4040: sys.storage call can take a long time * 4047: Segfault when updating a dropped table * 4050: Database corruption when running low on inode * 4057: missing bulk operations between constant and bat * 4061: SIGSEGV in candscan_lng * 4066: Deadlocked monetdbd * 6068: Error message about incompatible BBP version should be clearer * 6069: query with union all silently crashes * 6070: setting negative session query timeout should not be possible/allowed * 6071: where clause with cast and floor fails to sigsegv * 6072: Bind to UPD delta column does not get/show type information in EXPLAIN * 6073: Missing type information for constants in MAL explain * 6074: SET ROLE command does not work * 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed. * 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed. * 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed. * 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed. * 6084: Merge table point to wrong columns if columns in partition tables are deleted * 6108: monetdb5-sql sysv init script does not wait for shutdown * 6114: segfault raised in the query rewriter due to a null pointer * 6115: Assertion hit in the codegen * 6116: Codegen does not support certain kind of selects on scalar subqueries * 6117: Assertion hit in the query rewriting stage during the push down phase * 6118: SIGSEGV in strPut due to shared heap * 6119: Assertion hit in the MAL optimiser on a complex query * 6120: QUANTILE() treats NULL as if it is zero * 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result * 6123: Assertion hit in the codegen #2 * 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result * 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression * 6126: batcalc.== can't handle void BATs * 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom

1 0

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka)
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded >joweria@ubuntu:~$ joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded But when i try to reach another database as shown below, the module is not loaded joweria@ubuntu:~$ mserver5 --dbpath=/home/joweria/Spartial/mydbfarm/Gisbase !FATAL: GDKlockHome: Database lock '.gdk_lock' denied joweria@ubuntu:~$ sudo su [sudo] password for joweria: root@ubuntu:/home/joweria# mserver5 --dbpath=/home/joweria/ Spartial/mydbfarm/Gisbase # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'Gisbase', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded > Also when i run with my user priviledges

1 0

Re: developers-list Digest, Vol 52, Issue 1
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded 2016-12-01 13:00 GMT+02:00 <developers-list-request(a)monetdb.org>: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) > 2. Re: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 1 Dec 2016 12:11:49 +0200 > From: Joweria Nakibuuka <joweria.nakibuuka(a)unitn.it> > To: developers-list(a)monetdb.org > Subject: RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > Message-ID: > <CANbNUqbw+N3=tb929XArQ8q+AiCG-o5DYNR9Ndk2POChHpdi1g@ > mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Admin, > I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with > 128bit integers. i recently installed MonetDB5 default branch from the > mercurial repository. > > I tried to rebuild and enable geom module but to no avail, it shows no > error, after compilation it shows that the module is enabled,However when i > run the mserver the monetDB/GIS module is not loaded as expected. I am > stuck and i don't seem to know how to proceed forward in order to run my > spatial queries > > Are there any setting i need to do explicitly to load this module? > > the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 > --enable-geom=yes --enable-pyintegration=yes --with-geos=yes > > and then after compilation under the LDFLAGS-enabled componets > geom is one of the enabled componet but when i run the mserver5 the module > is not loaded as expected. > > I rebuild monetDB again and for first time i run the mserver5 commans > and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is > loaded) but when i closed the session (\q) and ran the same command again > the module was not loaded. i tried exporting paths to no avail. what could > be the problem? please i need your kind help. > > Thank you > > Regards, > Joweria nakibuuka >

2 2

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

Dear Admin, I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with 128bit integers. i recently installed MonetDB5 default branch from the mercurial repository. I tried to rebuild and enable geom module but to no avail, it shows no error, after compilation it shows that the module is enabled,However when i run the mserver the monetDB/GIS module is not loaded as expected. I am stuck and i don't seem to know how to proceed forward in order to run my spatial queries Are there any setting i need to do explicitly to load this module? the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 --enable-geom=yes --enable-pyintegration=yes --with-geos=yes and then after compilation under the LDFLAGS-enabled componets geom is one of the enabled componet but when i run the mserver5 the module is not loaded as expected. I rebuild monetDB again and for first time i run the mserver5 commans and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is loaded) but when i closed the session (\q) and ran the same command again the module was not loaded. i tried exporting paths to no avail. what could be the problem? please i need your kind help. Thank you Regards, Joweria nakibuuka

2 1

MAL: How to interate through group values?
by Gatis Ozoliņš 30 Nov '16

30 Nov '16

I am trying to create my own MAL aggregate function. I have implemented main function and it works OK: function group_concat(b:bat[:oid, :str],s:bat[:oid, :str]):str; sep := ""; barrier (h, t) := iterator.new(s); sep := t; exit (h, t); value := nil:str; barrier (h, t) := iterator.new(b); barrier bnormal := value; value := value + sep; value := value + t; exit bnormal; barrier bnil := calc.isnil(value); value := t; exit bnil; redo (h, t) := iterator.next(b); exit (h, t); return value; end group_concat; Now I need to implemented subfunction with following definition: function subgroup_concat(b:bat[:oid,:str],s:bat[:oid,:str],g:bat[:oid,:oid],e:bat[:oid,:any_2],skip_nils:bit) :bat[:oid,:str]; I can iterate over all groups g, but I can't find correct MAL function to get all values (b,s) for this specific group. Which MAL function would return list of oid for specific group g entry ? Gatis

1 0

.gdk_lock and the GDK locking scheme
by Eyal Rozenberg 28 Nov '16

28 Nov '16

I used to think .gdk_lock was used by mserver the same way Linux daemons use lock files - the filesystem enforces serialization, so if you do: 1. if (lock file present) then locking failed 2. create the lock file 3. if (creation failed) then locking failed 4. locking successful that's a valid non-blocking locking procedure. However, it appears that: * .gdk_lock is not always empty, and can have text such as USR=12096163 PID=5844 TIME=Thu Nov 24 15:35:08 2016 @ LOGON USR=12096163 PID=5844 TIME=Thu Nov 24 15:42:26 2016 @ LOGOFF * If .gdk_lock is not always created on DB startup, nor deleted on shutdown. * The locking using .gdk_lock is actually an flock() on its 5th byte (offset 4) and never any other one. * The GDKunlockhome() file will cause a second file descriptor opening in addition to GDKlockfile So I'm a bit confused about the locking scheme. Help? Eyal

1 0

Questions about BATs and their names in BBP's
by Eyal Rozenberg 24 Nov '16

24 Nov '16

So, I'm listing the BATs in a BBP (of a TPCH database), and I'm noticing several strange things: 1. There's a h-u-g-e number of BATs. Actually I knew this to be the case already, since BBPs have so many files, but, it's still striking. 863 BATs for a database with 61 bona fide columns (in lineitem, customer etc.) 2. There are a lot of BATs with null storage. And I do mean a lot - 514 out of 863. What are these? why do they need to exist? And note this is a database in which nothing has occurred except one-time loading. 3. Most BATs have dummy generated names, e.g. tmp_547 - not, say, a concatenation of their table name and column name. Yet - some BATs _do_ have meaningful names: environment_key, sql_dsnapshots, stat_opt_calls and so on. Why not have meaningful names for all BATs then? and lastly, and perhaps most importantly: 5. What's the best/most appropriate/most quick-and-dirty way of obtaining table and column names for those BATs which have them? 6. If it's through the storage table - is there a way to determine which BATs correspond to its columns (table, column, location at the very least)?

2 1