November 2009

These release notes are replicated for historical purposes only. The release is not supported anymore.

Release “Nov2009-SP2” consists of the following updated components

NameVersionCVS moduleCVS branch
MonetDB Common1.34.4MonetDBNov2009
MonetDB Clients1.34.4clientsNov2009
MonetDB4 Server4.34.4MonetDB4Nov2009
MonetDB4/XQuery0.34.4pathfinderNov2009
MonetDB5 Server5.16.4MonetDB5Nov2009
MonetDB5/SQL2.34.4sqlNov2009
MonetDB5/SQL/GIS0.14.4geomNov2009
MonetDB Java1.34.4javaNov2009
MonetDB Testing1.34.4testingNov2009

NOTE: Before upgrading to the latest “Nov2009-SP2” release from a release of MonetDB older than the Nov2008 release, please make a dump of your database, and rename/move/backup the database directories (‘dbfarm’, ‘sql_logs’, ‘xquery_logs’). After the upgrade, restore your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the “recommended” instructions for SQL and/or XQuery.

Nov 2009 SP2 bug-fix release

This is the second bug fix release of the Nov2009 version. This version can be installed “on top of” the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

  • No need to require bash for running the *-config scripts. (SF bug #2914563)
  • MonetDB5 development package now contains all development include files. (SF RFE #2929558)
  • Added support for compiling on OpenBSD (contributed, i.e. not tested by us).
  • General code improvements, partly inspired by running Coverity Scan.
  • mclient -D will exit with a non-zero exit code if dumping failed. (SF bug #2925674)
  • The Python interface now uses the same socket options as the C version, giving a boost in performance. (SF bug #2925750)
  • Fixed a bug in the Python interface to deal with fields with embedded newline characters. (SF bug #2917219)
  • Added initial support to the Python client interface for specifying the encoding.
  • Fixed a bug in execute argument parsing in Python interface.
  • Added missing .sql files to the Windows installer.
  • In MonetDB/SQL, fixed bug in not(invalidtable). (SF bug #2927174)

The following bug tracker items were fixed since the Nov2009-SP1 release:

  • [ 2911008 ] Cannot start because lib_replication.dll was not found
  • [ 2914558 ] Modify MonetDB.m4 to support OpenBSD
  • [ 2917219 ] Python client doesn’t handle fields with newline characters
  • [ 2925674 ] mclient dump fails on UTF-8 char and gives no error
  • [ 2925750 ] setsockopt in Python client greatly improves performace
  • [ 2927174 ] SQL: not(invalidtable.bla) segfaults

Nov 2009 SP1 bug-fix release

This is a bug fix release of the Nov2009 version. This version can be installed “on top of” the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

  • The Windows installers for MonetDB/SQL in the Nov2009 release were incomplete. The missing files have been added.
  • Fixed performance issue with loading complex schemas in readonly mode in MonetDB/SQL.
  • Fixed performance issue with grouping in MonetDB/SQL when one of the columns is sorted.
  • Fixed problems with LIKE joins in MonetDB/SQL.
  • Load all files in a multi-file COPY INTO query.
  • Fix error reporting of unexpected input in XQuery queries to be Unicode aware.
  • Correctly handle circular dependencies in FD property check. (SF bug #2908615.)
  • Fixes throughout the code base to better cope with unexpected situations (such as memory shortage).

The following bug tracker items were fixed since the Nov2009 release:

  • [ 1681244 ] Mapi ports and patfhinder/pftijah
  • [ 1893448 ] PF: xrpcConvert2 fails on Windows
  • [ 2686008 ] SQL: multi-attribute GROUP BY may ignore sortedness
  • [ 2879008 ] SQL: WHERE A.x LIKE A.y misses results
  • [ 2879011 ] SQL: leftjoin undefined
  • [ 2884050 ] Broken link to VOC tutorial
  • [ 2902320 ] SQL: COPY multiple files
  • [ 2903111 ] Wrong error / no check for new database dir
  • [ 2908615 ] Not clause breaks result

Nov 2009 feature release

Fedora

In addition to Fedora 10 & 11, we now also provide RPMs for Fedora 12.

Python

Packaging of some Python support files which are only used for testing was changed. The MonetDB-python RPM has been removed, the files are now packaged in the MonetDB-testing RPM. Equally, the python-monetdb Debian/Ubuntu package has been removed.

Windows

On Windows, the bzip2 library is now also used. The server (e.g., using SQL’s ‘COPY INTO’) can read files that are compressed using gzip and bzip2.

MonetDB Common

MonetDB Server 5

  • The M5 code underwent code hardening to better deal with rare events that would materialize mostly on 32bit systems. Mostly minor changes, e.g. display of debugging output has been improved to be more informative.

SQL

  • There is a new implementation of the SQL ‘COPY INTO’ statement which uses multiple CPUs/cores when available. This version is a bit stricter when parsing the CSV file than the old version. E.g., when strings are quoted in the CSV file, the string delimiter must be specified in the ‘COPY INTO’ statement, and to let empty field be treated as ‘NULL’ values, the ‘COPY INTO’ statement must specify ‘NULL AS ‘’’.
  • The TRACE statement modifier has been changed to collect the events in BATs first and returning them after the result set. The support library has been adjusted accordingly. See the documentation: SQL runtime features.
  • Several experimental optimizer pipelines are researched and being worked upon. They are not (yet) meant as the default pipeline for production applications. See the documentation: SQL runtime features.
  • Changing usernames is now possible from SQL, including the monetdb adminstrator user. Feature Request #2830257. The SQL syntax is: ‘ALTER USER “monetdb” RENAME TO “newname”;’.

Merovingian

The following features were implemented in the merovingian and monetdb programs:

  • monetdb: Created databases are now always put under maintenance and as such they have to be explicitly released. This allows the DBA to initialise the new database without any client interference.
  • merovingian: Neighbour discovery now no longer announces databases under maintenance, such that they are not shown as being available to peers. This allows to pull a database out of the cluster easily.
  • monetdb: No operations can be performed with monetdb any more without a merovingian instance running.
  • merovingian: Implemented Feature Request #2830754. monetdb can now control a remote merovingian using the -h -p and -P global options to monetdb. The password used is stored at the merovingian site in the .merovingian_pass file in the controlled dbfarm. An example of how this can be used can be found in the Assets section under the VOC loader script.
  • merovingian: Remote controllable merovingians now announce their controlport to peers, which are marked with an asterisk in the output of monetdb discover.
  • monetdb: Implemented Feature Request #2855846. The global -q option to monetdb makes it suppress any status messages, causing only error messages to be printed when they occurr.
  • merovingian: The outside URI that Merovingian is available on is now announced to each mserver5 started by merovingian via the merovingian_uri GDK environment variable.
  • monetdb: For each database now the property optpipe can be set which allows per database control of the SQL optimiser pipeline. Valid values can be found in the monetdb5.conf configuration file under the sql_optimizer section.
  • monetdb: Fixed bug where the uptime of databases with an uptime longer than 7 days was reported incorrectly.

XQuery

  • Use of value indices on comparisions between a value and an XPath ending in ’text()’ or attribute access (e.g., ‘doc(x)//x/y[z/text() = “foo”]’ and ‘for $x in doc//x/y where $x/@z = “bar” return $x’); such queries do not need scanning data, but start at the index and navigate backwards from there.

NOTE: This required a change in the physical database storage. While the Nov2009 release of MonetDB/XQuery can still read older (pre-Nov2009) databases, it is recommended to re-shred all documents to benefit from the new indices and achieve optimal performance.

  • A session expression cache, where you can run multiple queries in the same session and within that session keep intermediate results present (somewhat like a database cursor). One use is quickly paging in a GUI through a large result, another is acceleration of series of subsequent queries that overlap (e.g. zoom in to a result set).

Client Packages

mclient
  • Result rendering has been changed to try harder to find an optimal rendering of results wider than the terminal allows. Unlike previous releases, mclient now emits results wider than the terminal width if it is impossible or unreasonable to fit the results, thereby aiming to avoid wrapping columns.
  • EXPLAIN, TRACE and PROFILE output are now correctly rendered.
  • Timer support for single statements was implemented, Bug #2872884.
  • Added support for .monetdb file with defaults for mclient. The DOTMONETDBFILE environment file can be used to point to a different location or file name. The file layout of a .monetdb file is a key=value file, where keys include user, password and language.
  • Data fed into mclient using a pipe is no longer read line by line, but per block, like files given as arguments are. This improves performance considerably, but changes the behaviour regarding error messages and understood commands. The internal \x commands of mclient are no longer respected when used with a pipe. To obtain the previous behaviour, add the -i option to mclient when using data fed from a pipe.
  • The default username (the name of the currently logged in user) is no longer used without asking. The -u option now always requires an argument, when this option is absent, mclient will ask for a username, unless a default was set in the .monetdb file. Note: mclient no longer logs in as monetdb/monetdb by default.
  • The start_with_help option was removed. –help now always gives the general help of mclient and exits afterwards. Instead, mclient now gives a welcome message with some diagnostic information and a hint how to get (language specific) help.
  • The \d command now describes all tables by reporting whether it is a TABLE or VIEW. Rendering mode of database object describe mode has been improved.
  • Autocommit status and changes therein are now reported by mclient. Empty responses are no longer reported as “0 tuples”, instead proper status messages are returned, giving e.g. information about a generated identity key.
  • Error messages no longer include the leading !.
Python & Ruby
  • The native implementations have been matured further.
PHP
  • The SWIG-based PHP interface (leveraging the C MAPI library) has been replaced by a native PHP implementation. For the time being, the old SWIG-based interface is still in the code base, but only the new native interface is activated and distributed in the installers.
JDBC
  • JDBC received a workaround for a longstanding problem when running the driver using Java 6.
  • A bug was fixed which caused Connection warning messages to disappear when a connection to Merovingian was made.

Bug fixes

The following bug tracker items were fixed since the previous Aug 2009 SP2 release:

The detailed list of changes can be found in the source code CVS logs.