The Sep2022 documentation can be found here.
For more information on long term support releases visit MonetDB Solutions
(empty)
Introduced options wal_max_dropped, wal_max_file_age and wal_max_file_size that control the write-ahead log file rotation.
Fixed a (rare) race condition between copying a bat (COLcopy) and updates happening in parallel to that same bat. This may only be an actual problem with string bats, and then only in very particular circumstances.
Do a lot more error checking, mostly for allocation failures. More is still needed, though.
Improve performance of the ILIKE operator when the pattern contains only ASCII characters. In this case we do not need to treat any characters as UTF-8 and we can use much faster routines that perform byte comparisons.
Fixed a regression where after a while the write-ahead log files weren’t being rotated, meaning from some point onwards, the newest file just kept on growing.
When saving the SQL catalog during a low-level commit, we should only save the part of the catalog that corresponds to the part of the write-ahead log that has been processed. What we did was save more, which resulted in the catalog containing references to tables and columns whose disk presence is otherwise only in the write-ahead log.
A bug was fixed where the administration of which bats were in use was interpreted incorrectly during startup, causing problems later. One symptom that has been observed was failure to startup with a message that the catalog tables could not be loaded.
When memory is tight, it could happen a bat was backed up so that it could be saved. It was then possible that after commit the backup was not removed so that after a restart the old backup was used instead of the committed copy. This was fixed.
Fixed a number of data races (race conditions).
Fixed a reference counting problem when a BAT could nog be loaded, e.g. because of resource limitations.
Only check for virtual memory limits when creating or growing bats, not for general memory allocations. There is (still) too much code that doesn’t properly handle failing allocations, so we need to avoid those as much as possible. This has mostly an effect if there are virtual memory size restrictions imposed by cgroups (memory.swap.max in cgroups v2, memory.memsw.limit_in_bytes in cgroups v1).
The low-level commit turned out to always commit every persistent bat in the system. There is no need for that, it should only commit bats that were changed. This has now been fixed.
If the server is sent a SIGUSR1 signal, it prints out some useful information to the standard output. When run under monetdbd, this output will appear in the merovingian.log file.
There is now a new option –set tablet_threads=N to limit the number of threads used for a COPY INTO from CSV file query. This option can also be set for a specific database using the monetdb command using the ncopyintothreads property.
Added a missing interface function sys.timestamp_to_str with a TIMESTAMP (as opposed to TIMESTAMP WITH TIME ZONE) argument. The missing interface caused error messages being produced when the function was called with a TIMESTAMP argument, although it did give the correct result.
An upgrade that both creates a new .snapshot user and extends the sys.db_user_info table with (among others) a password column did these in such a way that the passord value for the new user was NULL. This is fixed by updating the password.
Warnings and informational messages are now sent to stdout instead of stderr, which means that monetdbd will now log them with the tag MSG instead of ERR.
Fixed parsing of the BBP.dir file when BAT ids grow larger than 2**24 (i.e. 100000000 in octal).
Fixed yet another occurrence of a missing .tailN file. This one could happen if a string bat was appended to in stages so that between appends the column was committed. If an append caused both a realloc of the tail heap because it was getting longer and a realloc because it got wider, the file might get removed before the GDK level commit happened.
When processing the WAL, if a to-be-destroyed object cannot be found, don’t stop, but keep processing the rest of the WAL.
A race condition was fixed where certain write-ahead log messages could get intermingled, resulting in a corrupted WAL file.
If opening of a file failed when it was supposed to get memory mapped, an incorrect value was returned to indicate the failure, causing crashes later on. This has been fixed.
A race condition was fixed that could result in a missing tail file for a string bat (i.e. a file with .tail1, .tail2, or .tail4 extension).
When saving a bat failed for some reason during a low-level commit, this was logged in the log file, but the error was then subsequently ignored, possibly leading to files that are too short or even missing.
The write-ahead log (WAL) is now rotated a bit more efficiently by doing multiple log files in one go (i.e. in one low-level transaction).
Fixed a regression introduced in the previous build: when a table is dropped in the same transaction where it was created, the data wasn’t removed from disk.
When creating a hot snapshot, allow other clients to proceed, even with updating queries.
Increased the size of a variable counting the number of changes made to the database (e.g. in case more than 2 billion rows are added to a table).
Improved cleanup after failures such as failed memory allocations.
An insert into a table from which a column was dropped in a parallel transaction was incorrectly not flagged as a transaction conflict.
Fix bug where boolean NULLs were not recognized by COPY BINARY INTO
7360: Aggregates returning string crash
7361: Table data is lost on DB restart, but only when a table has 2147483647 or more rows.
A crash in the ODBC driver was fixed when certain unsupported functions where used in a {fn …} escape.
Prepare of a query where the sum of the number of parameters (question marks in the query) and the number of output columns is larger than 100 could fail with an unexpected error. This has been fixed.
Fixed a race condition that could lead to a bat being added to the SQL catalog but nog being made persistent, causing a subsequent restart of the system to fail (and crash).
Fixed a race condition where a hash could have been created on a bat using the old bat count while in another thread the bat count got updated. This would make the hash be based on too small a size, causing failures later on.
When extending a bat failed, the capacity had been updated already and was therefore too large. This could then later cause a crash. This has been fixed by only updating the capacity if the extend succeeded.
Fixed a regression where when there are multiple concurrent transactions, the dependencies weren’t checked properly.
Added some error checking to prevent crashes. Errors would mainly occur under memory pressure.
Fixed cleanup after a failed allocation where the data being cleaned up was unitialized but still used as pointers to memory that also had to be freed.
Fixed a double cleanup after a failed allocation in COPY INTO. The double cleanup could cause a crash due to a race condition it enabled.
7343: GDKmmap requesting 0 virtual memory
7347: A bug where an exception occurs even though it is a query with normal syntax (Merge Table)
On Windows, use the wide-character interface of system calls when dealing with the environment (i.e. file names and getenv()).
Memory leaks have been fixed.
Improved maintenance of the estimated number of distinct values in BATs. The estimate helps in deciding which low-level algorithm to use.
Offset heaps (.tailN files) were growing too fast and unnecessarily under certain conditions. This has been fixed. Also, when such too large files are now loaded into the system, it is recognized they are too large and they are truncated to a more reasonable size.
Fixed a crash when the server runs out of client contexts (i.e. more concurrent clients than the server is configured to handle).
A race condition in the SHA hash code was fixed which resulted in occasional failed connection attempts when they occurred concurrently.
Fix a bug where the MAL optimizer would use the starttime of the previous query to determine whether a query timeout occurred.
7330: Creating temporary table fails after reconnect
7333: DLLs fail to load on Windows with accented characters in path
7336: Selecting from a literal-value table returns wrong values
7339: MonetDB corrupted state after SIGKILL
7342: column which datatype is double couldn’t group or aggregation in version 11.45.7
Improved SQLSetPos(). It now allows RowNumber to be 0 when Operation is SQL_POSITION.
Improved ODBC Error/Warning messages. They now include the name of the Data Source as required by the ODBC specification: [MonetDB][ODBC driver VERSION][data-source-name] data-source-supplied-text
Improved MonetDB ODBC Data Source Configuration dialog on MS Windows by hiding the typed in password text. It now shows dots for the characters. This fixes request https://github.com/MonetDB/MonetDB/issues/7314
Changed output of TABLE_CAT or PROCEDURE_CAT result columns as returned by ODBC functions: SQLTables(), SQLColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). They used to return the static database name but now they will return NULL as MonetDB does not support CATALOG objects or qualifiers.
Removed the possibility to retrieve or set the CURRENT_CATALOG via SQLGetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, …) and SQLSetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, …) as MonetDB does not support CATALOG objects (no SQL support for: CREATE CATALOG abc or SET CATALOG abc) and therefore there is no CURRENT_CATALOG.
Corrected ODBC functions SQLTablePrivileges() and SQLColumnPrivileges() for local temporary tables located in schema tmp. They did not return any rows when the temporary table had privileges set. Now they do return rows as expected.
Improved SQLProcedures() and SQLProcedureColumns(). They now list information also for all built-in system procedures and functions, not only those created via SQL. Also corrected the value of ORDINAL_POSITION for scalar function arguments. It would start at 2 instead of 1.
Extended output of SQLProcedures() and SQLProcedureColumns() resultsets with an extra column SPECIFIC_NAME. This column contains the name which uniquely identifies this procedure or function within its schema. As MonetDB supports overloaded procedures and functions, the combination of PROCEDURE_SCHEM and PROCEDURE_NAME is not enough to uniquely identify a procedure or function. This extra column allows you to correctly match the corresponding rows returned by SQLProcedureColumns() with the specific rows of SQLProcedures(). This extra column SPECIFIC_NAME is implemented similar to the JDBC DatabaseMetaData methods getProcedures() and getProcedureColumns().
For SQLForeignKeys() corrected the output of columns UPDATE_RULE and DELETE_RULE. These columns used to always return 3 (= SQL_NO_ACTION) but now they will report the action codes as specified in the FOREIGN KEY CONSTRAINT construction.
Fixed issue in function SQLProcedureColumns(). When the argument ColumnName was not empty it generated an invalid SQL query which failed with error: SELECT: no such column ‘c.name’. This has been resolved.
Corrected implementation of SQLGetInfo(hdbc, SQL_MAX_DRIVER_CONNECTIONS, …). It used to always return 64. Now it returns the value from the MonetDB server.
Changed the column names case of the result sets as returned by SQLTables(), SQLColumns(), SQLSpecialColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). The column names where all in lowercase but the ODBC specification defines them in uppercase, so changed them to uppercase.
Corrected SQLSpecialColumns(…, SQL_BEST_ROWID, …). Previously it only returned rows when the table had a primary key. Now it also returns rows when a table has a unique constraint.
Corrected SQLStatistics(…, SQL_INDEX_ALL, …). Previously it only returned rows when the table had a primary or unique key. Now it also returns rows for indices which are not based on a key.
Corrected SQLStatistics(…, SQL_ENSURE). It now returns CARDINALITY information for columns based on a primary/unique key. Previously it always returned NULL for the CARDINALITY result column.
Corrected ODBC functions SQLPrimaryKeys(), SQLSpecialColumns() and SQLStatistics() for local temporary tables located in schema tmp. They did not return any rows when the temp table had a primary or unique key or index. Now they do return rows as expected.
Fixed a bug in ORDER BY with both NULLS LAST and LIMIT when the ordering was on an interger or integer-like column and NULL values are present.
The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t).
The abort_on_error parameter of all GDK-level functions has been removed. Errors (e.g. overflow) now also results in an error.
Implemented BC/AD (and BCE/CE) suffixes when parsing dates.
The interface for using strimps has not changed (create an imprint index on a column of a read only table), but now construction happens at the beginning of the first query that uses the strimp and is performed in a multithreaded manner.
Implemented the use of strimps for NOT LIKE queries. The idea is to run the corresponding LIKE query using strimps and take the complement of the result. We keep around NULL values both during strimp filtering and during the pcre part of the LIKE query so that they get discarded automatically when we take the complement.
The function BBPkeepref now gets a BAT pointer as argument instead of a bat id.
Get rid of macro Tsize, use ->twidth instead.
Get rid of macro BUNlast, just use BATcount instead.
The BLOB type has been moved into the GDK layer.
When adding or subtracting months from a date or timestamp value, clamp the result to the calculated month instead of wrapping to the beginning of the next month. See bug 7227.
The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t).
The function sys.tracelog is now executable by anyone. The function (and view of the same name) returns the tracing information of a query that was prepended with the TRACE keyword.
Fixed a bug in COPY BINARY INTO where the input wasn’t checked thoroughly enough.
Extended the built-in profiler to emit non-MAL events related to query compilation, optimization, transactions and client connections. To minimize, simplify and optimize the process of generating and processing profiler output, only the events marking the end of an operation are emitted in most cases and the emitted json messages themselves are trimmed down to their essential fields. Furthermore, the MAL instruction profiler.openstream now requires an integer as a single argument, “0” for default behaviour or “4” to turn on the profiler in minimal mode, which causes it to only emit general events and excludes individual MAL instruction execution events from the profiler streams. The MAL instruction profiler.openstream with zero arguments is deprecated.
Extended system view sys.users with columns: schema_path, max_memory, max_workers, optimizer and default_role. Extended system table sys.db_user_info with columns: schema_path, max_memory, max_workers, optimizer, default_role and password. The password is encrypted. This table can only be queried when the user has been granted the select privilege from monetdb.
CREATE USER statement has been extended to take more optional arguments like MAX_MEMORY, MAX_WORKERS, OPTIMIZER, DEFAULT ROLE. ALTER USER statement has also been extended with DEFAULT ROLE.
A new function sys.url_extract_host(string, bool) which returns the host name from the given URL has been implemented. The bool argument, if true, causes the www. prefix to be removed.
The user authentication tables are now administered by the SQL layer instead of the MAL layer. This means that changing (adding, dropping, altering) user and role information is now transaction-safe.
Added the UNLOGGED TABLE feature.
Removed functions sys.index and sys.strings.
Added “loglevel” property for monetdbd logging (to merovingian.log). The loglevel can be set to: error or warning or information or debug. The loglevel property can be changed dynamically via command: monetdbd set loglevel=warning /path/to/dbfarm Default the loglevel is set to: information When loglevel is error, only errors are logged. When loglevel is warning, errors and warnings are logged. When loglevel is information, errors and warnings and information messages are logged. When loglevel is debug, all messages are logged.
There is now a loadmodules property that can be used to add –loadmodule arguments to the mserver5 command line. See the monetdb and mserver5 manual pages.
In certain cases (when an mserver5 process exits right after producing a message) the log message was logged over and over again, causing monetdbd to use 100% CPU. This has been fixed.
6982: Wrong constraint name in error message of constraint violation
7209: Configuration option for merovingian.log
7225: Invalid memory access when extending a BAT during appends
7227: Date calculations, bug or feature
7273: Concurrent reads and writes causes “BATproject2: does not match always” error
7282: call sys.dump_table_data(); fails
7285: C-UDFs: aggr_group.count has wrong value (number of input rows instead of number of groups).
7296: Implictly cast a timestamp string to DATE when appropriate
7297: Parsing partial dates behaves unpredictable
7306: ODBC Driver Assertion `stmt->Dbc->FirstStmt’ Failed
7314: ODBC Driver : please mask/hide password
7318: distinct in a subquery not working properly