November 2008

MonetDB Nov2008-SP2 release

Release “Nov2008-SP2” consists of the following updated components

IMPORTANT NOTE: Before upgrading to the latest “Nov2008-SP2” release from a release of MonetDB older than the Nov2008 release, please make a dump of your database, and rename/move/backup the database directories (‘dbfarm’, ‘sql_logs’, ‘xquery_logs’). After the upgrade, restore your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories._

Nov 2008 SP2 bug-fix release

This is the second bug fix release of the Nov2008 version. This version can be installed “on top of” the Nov2008 and Nov2008-SP1 releases. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

• MonetDB/XQuery: There was a bug when shredding documents into an existing updatable collection when the existing collection happened to end at a page boundary. (Bug [ 2141012 ] XQ: unclear error).
• General: Fix to configure to not complain when the raptor library isn’t found, unless it was specifically asked for.
• MonetDB/SQL: significantly speeded up loading of data when lines are extremely long (e.g. when loading BLOBs).
• XRPC: Since the change of the XRPC protocol (by June 2008), the XRPC server also sends response messages (with empty xrpc:response nodes for updating function calls. However, the XRPC serever failed to send response messages for calls to the Pathfinder document management functions, because they are handled differently than normal updating functions. This is now fixed.</xrpc:response>
• XRPC: The function XRPCHTTPConnection.sendReceive() did not check if HttpURLConnection.getResponseCode() returns -1 or not. If the return value is indeed -1, subsequent use of the stream returned by HttpURLConnection.getInputStream() or HttpURLConnection.getErrorStream() might cause a NullPointerException. This is now fixed. The return value and the stream returned by those two functions are now checked and a proper error message is given.

The following bug tracker items were fixed since the Nov2008-SP1 release:

• [ 1964658 ] Problem with names of reverse()’d BATs.
• [ 2141012 ] XQ: unclear error
• [ 2233581 ] MonetDB does not like ‘\0’ in its input
• [ 2388542 ] SQL bad blob insert causes crash
• [ 2405952 ] Merovingian takes too much CPU resources
• [ 2430201 ] Is distinct broken?
• [ 2457195 ] Broken query results in OOM, working one has no output
• [ 2460935 ] Pathfinder fails to compile with icc
• [ 2460953 ] make dist (and hence make rpm) fails for pathfinder
• [ 2466725 ] SubSelect with Outer Reference, result is wrong
• [ 2468513 ] SQL: copy into does not detect syntax errors
• [ 2471371 ] SQL: varchar bounds should be obeyed

Nov 2008 SP1 bug-fix release

This is the first bug fix release of the Nov2008 version. This version can be installed “on top of” the Nov2008 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

• Fixed formatted printing from MAL.
• mclient now exits with an error code if one of the queries in a non-interactive session failed. [2388047]
• Fixed a deadlock condition when multiple queries simultaneously update the same part of a document (XQuery).
• When writing to a file, mclient now flushes the output when the query is done instead of waiting until the file is closed. [2344251]
• Fixed a race condition in document management (XQuery). [2338542]

The following bug tracker items were fixed since the Nov2008 release:

• [ 1911209 ] XQ: MonetDB ‘hangs’ after requesting non-existing doc
• [ 1981762 ] PF/alg: “Mapping unique name to an original name failed.”
• [ 2049320 ] pf:collection doc count wrong (again?)
• [ 2075180 ] the unique column can’t insert more than 1 row with null
• [ 2075241 ] When the NULL is in a column,it shouldn’t add primary key
• [ 2103348 ] unique problem with nulls
• [ 2142522 ] XQ: Duplicates when not using intermediate result
• [ 2144639 ] XQ: fn:collection() in algebra version
• [ 2209871 ] most GEOM test fail after error handling changes
• [ 2224233 ] M4 test ID.914935 fails after warning changes
• [ 2233677 ] SQL Command kills DB
• [ 2276427 ] CMDmulti_merged_union: operation failed
• [ 2277120 ] XQ: following(-sibling)/preceding cause Mserver to crash
• [ 2305746 ] merovingian crashes when starting a vaultkeyless database
• [ 2338524 ] Deadlock on barrier
• [ 2344251 ] Output isn’t complete
• [ 2388047 ] Mclient exit code
• [ 2388424 ] ERROR: XML Generation: NULL BAT has a 24 head, but tail is N
• [ 2420768 ] col from another schema inside function body

Copyright & License

The copyright for the MonetDB suite was transferred to a newly formed company MonetDB B.V. One of the most important conditions for this transfer was that the software remains open source, and so the license under which this software is being distributed has not changed (apart from the copyright notice).

Windows Installers

The default installation location has been changed from ‘C:\Program Files\CWI’ to ‘C:\Program Files\MonetDB’.

MonetDB Common (GDK)

The kernel administration for BATs has been changed to prepare for future enhancements. All descriptors are now initialized and kept in memory, which reduces the number of files and related IO accessing the descriptors.

Memory management for large database has been improved, with the scavenging thread becoming more aggressive in moving dirty pages to disk when we reach 70% of the physical memory.

Code cleaning and simplification continues to provide ease of compilation on many platforms.

MonetDB Server 5

The MonetDB 5 kernel underwent a code clean up too, e.g. datatypes have been more strictly applied and all kernel errors are propagated to MAL exceptions.

The number of experimental modules have been developed. The ‘recycler’ provides support for automatic re-use of intermediate results. It improves query throughput for long sequences with partially overlapping queries.

The system supports intra-query parallelism using the multi-core systems capabilities using the ‘dataflow’ optimizer.

The ‘mergetable’ optimizer has been extended to perform symbolic reasoning over oid-reasoning. Together with the MAT and dataflow optimizers it provides a basis to improve multi-core performance.

Several bugs were fixed in Merovingian, the MonetDB Database Server daemon, including a file descriptor leak, which caused Merovingian to terminate after a certain amount of connections using the proxy method. Merovingian has been extended with neighbour discovery features, allowing to redirect to discovered neighbours. The ‘monetdb’ tool has been improved to give better feedback on starting and stopping databases. Most notably it now waits for completion of the start operation, and gives error feedback when starting failed for some reason.

SQL

This release comes with a few more optimizers enable by default. The most important one is the inter-query parallelism, which leads to better use of a multi-core platforms.

The dataloading code has been improved during our effort to load a 2.6TB astronomical database. Upon request we can provide a detailed report on this experiment.

A query log can be captured with the history.sql script. It provides a basis for post-session analysis of execution times.

Several bugs reported have been solved and functionality in many other areas has been improved.

Bug fixes

A detailed list of the 116 bug reports that have been closed since the previous release can be found in the MonetDB Bug Tracker at SourceForge: page 1/3, page 2/3, page 3/3. The detailed list of changes can be found in the source code CVS logs.

Known Issues in this Release

• Merovingian crashes when starting a vaultkeyless database. Actual workaround is not to try and start databases not created using ‘monetdb create’. Details: SF bug tracker #2305746
• Parallel loading is known to be buggy. Actual workaround is to start the server using ‘gdk_nr_threads=1’ set in your ‘monetdb5.conf’ file (or given via ‘–set gdk_nr_threads=1’ on your ‘mserver5’’s command line) during the bulkload using ‘COPY INTO FROM FILE’ with ‘mclient’. Details: (no known bug tracker reference)