August 2009

MonetDB Aug2009-SP2 release

Release “Aug2009-SP2” consists of the following updated components

NOTE:
Before upgrading to the latest “Aug2009-SP2” release from a release of MonetDB older than the Nov2008 release, please make a dump of your database, and rename/move/backup the database directories (‘dbfarm’, ‘sql_logs’, ‘xquery_logs’). After the upgrade, restore your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

Aug 2009 SP2 bug-fix release

This is the second bug fix release of the Aug2009 version. This version can be installed “on top of” the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

• Changes in the memory mapping code to less zealously allocate space.
• Changes to the use of the way the operating system is informed about how memory is used, resulting in a better fit between the predictions the operating system makes and the reality of what MonetDB actually does.
• Fix to exit code.
• Performance fix releated to anonymous virtual memory.
• Performance fix for when a single item is joined to a table. (MonetDB4/XQuery)
• When extracting fails, don’t leave a partial output file (Mx, SF bug 2840225).
• Added support for GCC version 4.4.* by disabling the strict-aiasing optimization in the compiler.
• Various improvements to the Ruby interface.
• Fix warning messages to disappear when a connection to merovingian is made.
• Fix physical plan for merge-adjacent-text-nodes. (MonetDB/XQuery)
• For some plans with aggregates, lax checking resulted in incorrect rewrites. (MonetDB/XQuery)
• Fix bug in order lists merging. In case the replaced column is sorted in descending order we need to reverse the order of all replacement order criteria. (MonetDB/XQuery)
• Fixes so that larger plans can be handled. (MonetDB/XQuery, SF bug 2860037)
• Fixes so that less stack space is used during processing, and so that running out of stack space is recognized in more places. (MonetDB/SQL)
• Fixed bug in IN handling. (MonetDB/SQL, SF bug 2869818)
• Fixed crash when attempting to create a function in read-only mode by disallowing this. (MonetDB/SQL, SF bug 2872421)
• Added partial support for compiling with bison++. Bison++ is not supported and can’t be used out of the box, but only minor tweaks are needed to use it. (SF bug 2872565)
• Fix warning messages to disappear when a connection to merovingian is made. (JDBC driver)

The following bug tracker items were fixed since the Aug2009-SP1 release:

• [ 2812963 ] JDBC connection hangs if result is too long
• [ 2839063 ] M5: property error in src/modules/mal/Tests/chopper03.mal
• [ 2840225 ] Mx creates output files even on errors
• [ 2847738 ] SuperBall build fails with PHP 5.3.0
• [ 2860037 ] PF: fatal error: aborted too deep recursion on linux
• [ 2860574 ] PF: fatal error: column not found / BATappend access denied
• [ 2865631 ] SQL: incref assertion
• [ 2869818 ] SQL: Program contains errors
• [ 2871426 ] SQL: mclient-lsql-D fails everywhere but on 64-bit Fedora
• [ 2872421 ] SQL: Crash when trying to create function to read only db
• [ 2872880 ] mclient shows prompt with “-s” option without “-i&qu ot; option
• [ 2872884 ] mclient -t -s… does not show “Timer” info
• [ 2876948 ] PF: error “We do not support the value NaN”
• [ 2879833 ] SQL: SIMILAR keyword does not exist

Aug 2009 SP1 bug-fix release

This is a bug fix release of the Aug2009 version. This version can be installed “on top of” the Aug2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is always a good idea to make backups.)

Some of the bugs that were fixed:

• On Windows, we switched to newer versions of the iconv and geos libraries (iconv 1.11.1 and geos 3.1.1, from iconv 1.11 and geos 3.0).
• On Windows, MonetDB4/XQuery was compiled with a newer version of the Intel compiler (version 11.1.046 from version 10.1.022).
• Better error handling if BATs are growing too large. (SF bug #2850513)
• The MAL parser was made more robust against syntax errors.
• A recursion depth check was added to the MAL interpreter. (SF bug #2852707)
• The PHP client is not compatible with PHP 5.3.0 or later. We now check for that. (Note that in a future release, the PHP client code will be replaced by a pure PHP implementation where this restriction does not apply.) (SF bug #284773)
• The SQL dump code (msqldump and mclient –dump) now specify all delimiters and the quote character on the generated COPY INTO query. This is needed to be compatible with the next feature release.
• The SQL dump code (msqldump and mclient –dump) now also knows about CLOBs (Character Large Objects) and properly quotes the values.
• The problem that occurred under certain circumstances that the JDBC driver would throw an SQLException when large result sets were retrieved is fixed. (SF bug #2812963)
• Fixed a bug in MonetDB/XQuery where the comparison between values of types strings and untypedAtomic was not done correctly. (SF bug #2853441)
• Fixed a bug in MonetDB/XQuery where the dead code elimination was a bit too eager. (SF bug #2852928)
• Fixed a bug in MonetDB/XQuery where a warning was generated that a pin count was below zero. (SF bug #2863458)
• Fixed a bug in PF/Tijah where OR queries were treated as AND queries. (SF bug #2836870)
• Fixed a bug in PF/Tijah where the nexi scanner would split strings on ‘_’.
• Fixed overflow checks in MonetDB/SQL. (SF bug #2853458)
• Fixes to get the embedded SQL example working again.
• Fixed startup of MonetDB/SQL server so that a client cannot connect before the server is ready. (SF bug #2855021)
• Fixed a bug in MonetDB/SQL where old and dropped view definitions were sometimes incorrectly reused. (SF bug #2863804)
• Fixed a bug in merovingian where the mero_port in the config file was not properly recognized. (SF bug #2844202)
• Fixed a bug in MonetDB/SQL where it was not possible to add month intervals to a timestamp. (SF bug #2854949)
• Fixed various bugs in the SQL engine. (SF bugs #2847106, #2862146)
• Use bigint for RESTART value in MonetDB/SQL. (SF bug #2846238)
• Fixed a bug in MonetDB/SQL with parsing hex values.
• Fixed a bug in MonetDB/SQL when attempting to make changes to the schema when the database is in read-only mode. (SF bug #2830238)
• Fixed a bug in MonetDB/SQL where adding a primary key constraint to a column containing a NULL value was allowed. (SF bug #2075241)
• Fixed a bug in MonetDB/SQL where updating values in one column could affect the values in another column. (SF bug #2864313)

The following bug tracker items were fixed since the Aug2009 release:

• [ 2808453 ] SQL: Trying a copy into in read only database segfault
• [ 2830238 ] SQL: Crash when trying to write on read-only DB
• [ 2831472 ] tijah:queryall crashes Mserver
• [ 2836870 ] PFtijah: Use of ‘or’ in phrase search yelds incorrect result
• [ 2840310 ] SQL: “Cannot access descriptor” on big endian
• [ 2844202 ] Merovingian: mero_port configuration option ignored
• [ 2846238 ] alter sequence restart on large integer
• [ 2847106 ] SQL: crash upon join with advanced between
• [ 2848525 ] mac10.6 and monetdb installation
• [ 2850341 ] SQL: bit_and returns invalid values
• [ 2852928 ] PF: updates broken using algebra front-end?
• [ 2853441 ] PF: distinct values on two sequences
• [ 2853458 ] SQL: cast exceeding maximum value, returns null
• [ 2854949 ] SQL: + interval ‘1’ year; Program contains errors
• [ 2855021 ] SQL: mserver accepts sql connections before scenario load
• [ 2855152 ] mac10.6 and monetdb installation
• [ 2862146 ] Two COUNT(*) with LIMIT doesn’t execute
• [ 2863458 ] PF/TIJAH: tokenize in a for-loop
• [ 2863804 ] SQL: old views incorrectly reused
• [ 2864313 ] SQL: Update changes all columns with same type and default

Aug 2009 feature release

Fedora We now provide RPMs for Fedora 10 & 11, but no longer for Fedora 9. However, a first check indicates that the Fedora 10 RPMs also install and work fine on Fedora 9.

MonetDB Common

• A bug-fix in the May2009 release had unnoticedly introduced a potential performance degradation, caused by non-optimal use of memory-mapped files, with workload that exceed physical memory, in particular with bulk-loading (“COPY INTO”) large tables in MonetDB/SQL. This performance problem has been fixed.
• Various problems related to sloppy type usage and type casting had surfaced with testing on big-endian hardware as well as on 64-bit systems when using 32-bit OIDs. These bugs have been fixed.
• The compilation process has been improved by splitting the “infamous” large (generated) “gdk_scanselect_defs.c” into multiple files.

MonetDB Server 5

• auto-sensing is no longer implicit based on the port number being 0, but can now be enabled and disabled via mapi_autosense in monetdb5.conf. When set to enabled, it will try to find a free port starting from the port given by mapi_port.

SQL

• fix crash with select * from ‘whatever’ = bla, select * where bla = “whatever”, Bug #2825974
• Ignore escapes for non-escape sequences, instead of inserting a literal backslash. Bug #2830900
• Decimals now are properly detected in PreparedStatements, allowing to use them. Bug #2831994
• Fixed hang after invalid statement + COMMIT outside auto-commit mode. Bug #2814622
• Fixed NUMERIC type detection. Bug #2818176
• No longer allow SELECT in value lists, causing confusing error messages. Bug #2807609
• Allow for NULL assignment and NULL as argument of a function. Bug #2812729
• Added missing power (^) operator. Bug #2812721
• DISTINCT was not properly recognised, causing incorrect count results. Bug #2826015
• When encountering ambigious columns, fail instead of crashing. Bug #2807336
• GROUP BY on a view with constants no longer yields in a crash. Bug #2837561
• PLAN output is fixed to no longer violate the protocol and to be tuples instead of comments.
• Optimisating pipelines can now be set in monetdb5.conf and changed without compiling SQL.
• sql_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.
• The ILIKE operator was added. This operator is a case-insensitive LIKE operator, as also found in PostgreSQL.

Merovingian

The following features were implemented in the merovingian and monetdb programs:

• merovingian: Implemented a configurable port for the neighbour discovery service. This also allows to entirely disable neighbour discovery by setting mero_discoveryport to 0. Feature request #2800964
• merovingian: Implemented full per-database property support. monetdb now can manage the following per-database properties: forward (whether to proxy or redirect), shared (if it will be available in the network and how) and nthreads (number of working threads the mserver process uses). Feature Request #1837591.
• merovingian: Increased the default mero_exittimeout from 7 to 60 seconds. This makes merovingian by default wait longer for mservers to exit before forcefully terminating them.
• merovingian: Implemented sharing of databases with tags. Using these tags, extra information is available on a remote merovingian to select a database during connect. For this, a client can now connect to a database using a glob expression, which is matched on the database plus tag. When multiple databases match such expression, a load-balancing technique is applied. See the merovingian man-page for details.
• monetdb: Implemented dynamic width of monetdb status output. If a database name is wider than 14 chars, the name column is made wider as long as the terminal window allows. If a column cannot be made any wider this way, it is abbreviated as before.
• monetdb: Upon ‘monetdb destroy’, avoid multiple questions by merging them into a single confirmation for the destroy command. This accidently also fixed a bug where a response such as ‘yes’ would abort the next database removal.
• monetdb: Databases created by monetdb now can only have names restricted to the pattern ‘[A-Za-z0-9-_]+’.
• monetdb: Allow setting the name of a database (a.k.a. renaming it).
• monetdb: Allow an expression to be used to match against the list of discovered or local databases as argument in the commands ‘discover’ and ‘status’.
• monetdb: ‘destroy’ now allows to remove a database under maintenance. This way an administrator can prevent users connecting to the database and hence starting it on demand, which in turn prevents removal of the database.

XQuery

• xquery_logdir can no longer be set and defaults to a location close to the actual data. This approach was chosen since the logs are useless without the data.

Client Packages

mclient

The rendering of the mclient utility for SQL has been improved to better utilise the available space on the screen. Main improvement is that columns are no longer wrapped if they contain only NULLs, even though enough space on the terminal is availble.

Python

• The python API now supports connections to Merovingian.
• Transaction support was added.
• Added compatibility with python3k.
• Full Unicode/UTF-8 support.
• autocommit, commit and revert functions were added.

Ruby

• Fixed bugs that prevented the correct working of activerecords’ migration; The activerecord connector now supports insertion, update and alter table operations; Type casting is working in activerecord; Added a rubygem and rakefile for activerecord-monetdb-adapter; Added a new usage example for activerecord to the README file; Added an example directory to the cvs tree; The driver now correctly works with Merovingian.
• Support for ruby 1.9.
• Both the standalone driver and activerecord now support transactions, nested transactions are simulated via savepoints in activerecord.
• Type conversion in the standalone driver is now performed after data has been retrieved and can be executed on single fields.

JDBC

• blockmode_blocksize option, and the -Xblksize option from JdbcClient were removed, as they are meaningless and inimplemented in the current JDBC driver.
• Full redirection support was added to support Merovingian’s tag-based connections for localisation and loadbalancing.