February 2014 - users-list

Announcement: New Jan2014 Feature release of MonetDB suite
by Sjoerd Mullender 20 Feb '14

20 Feb '14

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jan2014 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jan 2014 feature release Build Environment * Created development packages for RPM based systems and Debian/Ubuntu containing the files needed to create extensions to the SQL front end. * Removed Mx, the literate programming tool. All code for the server is now pure C. * Created packages for RPM based systems and Debian/Ubunty containing the MonetDB interface to the GNU Scientific Library (gsl). * We no longer install the .la files in our Fedora/Debian/Ubuntu packages. Client Package * ODBC: Implemented {fn scalar()} and {interval ...} escapes. python2 * Changed defaults for connecting (defaults to unix socket now). * Unix sockets partially working for control protocol. * Add support for unix socket. python3 * Changed defaults for connecting (defaults to unix socket now). * Unix sockets partially working for control protocol. * Add support for unix socket. R * The R connector is now distributed in the source code packages. MonetDB Common * The join code has been completely rewritten. It now optionally uses candidate lists, like the select code that was introduced in the previous release. * A new indexing structure for range selections on unsorted data has been added. * The vmtrim thread is no longer started by default on 64 bit architectures. The vmtrim thread monitors memory usage and drops BATs from memory when memory gets tight. However, in the age of large address spaces and virtual memory, the kernel does a good enough job. And in addition to dropping BATs, the thread also destroyed indexing structures which would have to be recreated the next time they were needed. * Cleaned up some of the parameters dealing with memory usage. * If available on the system, we now use atomic instructions for latching. * Removed some unused fields in the atomDesc structure. This change requires a complete recompilation of the whole suite. * Replaced the mutex implementation for both GNU C and Visual Studio with a home-grown implementation that uses atomic instructions (__sync_*() in gcc, _Interlocked*() in VS). SQL * Added support for quantiles (generalization of median). Usage: SELECT quantile(column_name,0.25) FROM table_name; The value should be in the range 0..1. Bug Fixes * 3040: Wrong NULL behavior in EXCEPT and INTERSECT * 3092: ODBC client doesn't support scalar function escape * 3198: SIGSEGV insert_string_bat (b=0x7fffe419d0a0, n=0x7fffc4006010, append=0) at gdk_batop.c:196 * 3210: Unexpected concurrency conflict when inserting to 2 tables simultaneously and querying one of them * 3273: Add support to Python DBAPI package for timetz, inet and url types * 3285: no such table 'queryHistory' * 3298: GDKmmap messages and monetdb start db takes very long * 3354: Introduce query time-out * 3371: (i)like generates batloop instead of algebra.likesubselect * 3372: Large group by queries never complete - server at 100% cpu(all cores) until MonetDB stopped * 3383: Bad performance with DISTINCT GROUP BY * 3391: Bad performance with GROUP BY and FK with out aggregate function * 3393: "COPY .. INTO ..." - escape of string quotes * 3399: server crashed on simple (malformed) query * 3401: inconsistent/strange handling of invalid dates (e.g. 2013-02-29) in where clause * 3403: NOT NULL constraint can't be applied after deleting rows with null values * 3404: Assertion `h->storage == STORE_MMAP' failed. * 3408: nested concat query crashed server * 3411: (disguised) BETWEEN clause not recognised. Hence no rangejoin. * 3412: Boolean expressions in WHERE clause, result in incorrect resulsts * 3417: Nested Common Table Expressions Crash * 3418: Segmentation fault on a query from table expression * 3419: Database does not start after upgrade * 3420: Database does not start after upgrade * 3423: Group by alias with distinct count doesn't work * 3425: Temporal extraction glitches * 3427: Consistent use of current_timestamp and now() * 3428: Aggregation over two columns is broken * 3429: SAMPLE on JOIN result crashes server * 3430: Wrong temporary handling * 3431: SQLGetInfo returns incorrect value for SQL_FN_NUM_TRUNCATE * 3432: MonetDB SQL syntax incompatible with SQL-92 <delimited identifier> syntax * 3435: INDEX prevents JOIN from discovering matches * 3436: COPY INTO from file containing leading Byte Order Mark (BOM) causes corruption

1 0

slow performance with 10 joins and above
by imad hajj chahine 20 Feb '14

20 Feb '14

Hi, I have the following query that do a pivot for the data stored in different tables, the time of the query is acceptable as long as the number of table to join is up to 10, 10 to 15 the query take more than 90 seconds to return, above 15 it does not return. Is there something i am missing, or i have to slice my query in smaller joins and the join back the results? PS: the max number of records in each table is < 200k select tval1.value as countall,tval2.value as ana1,tval3.value as ana2,tval4.value as count5,tval5.value as min5,tval6.value as max5,tval7.value as sum5,tval8.value as avg5,tval9.value as count6,tval10.value as min6,tval11.value as max6,tval12.value as sum6,tval13.value as avg6,tval14.value as count7,tval15.value as min7,tval16.value as max7,tval17.value as sum7,tval18.value as avg7 from "RPME".t_entity_cache tec join "RPME".t_value_int_cache tval1 on tec.id_schema=tval1.id_schema and tec.id=tval1.id_entity and tval1.id_attribute=1 join "RPME".t_value_date_cache tval2 on tec.id_schema=tval2.id_schema and tec.id=tval2.id_entity and tval2.id_attribute=2 join "RPME".t_value_string_cache tval3 on tec.id_schema=tval3.id_schema and tec.id=tval3.id_entity and tval3.id_attribute=3 join "RPME".t_value_int_cache tval4 on tec.id_schema=tval4.id_schema and tec.id=tval4.id_entity and tval4.id_attribute=4 join "RPME".t_value_numeric_cache tval5 on tec.id_schema=tval5.id_schema and tec.id=tval5.id_entity and tval5.id_attribute=5 join "RPME".t_value_numeric_cache tval6 on tec.id_schema=tval6.id_schema and tec.id=tval6.id_entity and tval6.id_attribute=6 join "RPME".t_value_numeric_cache tval7 on tec.id_schema=tval7.id_schema and tec.id=tval7.id_entity and tval7.id_attribute=7 join "RPME".t_value_numeric_cache tval8 on tec.id_schema=tval8.id_schema and tec.id=tval8.id_entity and tval8.id_attribute=8 join "RPME".t_value_int_cache tval9 on tec.id_schema=tval9.id_schema and tec.id=tval9.id_entity and tval9.id_attribute=9 join "RPME".t_value_numeric_cache tval10 on tec.id_schema=tval10.id_schema and tec.id=tval10.id_entity and tval10.id_attribute=10 join "RPME".t_value_numeric_cache tval11 on tec.id_schema=tval11.id_schema and tec.id=tval11.id_entity and tval11.id_attribute=11 join "RPME".t_value_numeric_cache tval12 on tec.id_schema=tval12.id_schema and tec.id=tval12.id_entity and tval12.id_attribute=12 join "RPME".t_value_numeric_cache tval13 on tec.id_schema=tval13.id_schema and tec.id=tval13.id_entity and tval13.id_attribute=13 join "RPME".t_value_int_cache tval14 on tec.id_schema=tval14.id_schema and tec.id=tval14.id_entity and tval14.id_attribute=14 join "RPME".t_value_numeric_cache tval15 on tec.id_schema=tval15.id_schema and tec.id=tval15.id_entity and tval15.id_attribute=15 join "RPME".t_value_numeric_cache tval16 on tec.id_schema=tval16.id_schema and tec.id=tval16.id_entity and tval16.id_attribute=16 join "RPME".t_value_numeric_cache tval17 on tec.id_schema=tval17.id_schema and tec.id=tval17.id_entity and tval17.id_attribute=17 join "RPME".t_value_numeric_cache tval18 on tec.id_schema=tval18.id_schema and tec.id=tval18.id_entity and tval18.id_attribute=18 where tec.id_schema=3

3 3

what is the character limit for monetdb queries?
by Anthony Damico 17 Feb '14

17 Feb '14

i'm definitely hitting one around 1.7 million characters, but i'm wondering how much lower the actual limit is? thanks!

3 4

mapi_get_last_id in JDBC
by imad hajj chahine 16 Feb '14

16 Feb '14

Hi, How to get the last inserted id with JDBC, i see its supported in mapi library but i am unable to get it with JDBC interface using st.getGeneratedKeys(). Thanks

2 1

Choices of operating system
by Radovan Bičiště 12 Feb '14

12 Feb '14

Hello, We have monetdb deployed for our customers and due to decent experience we would like to solidify and standardize our enviroment. So far we've been using UBUNTU 12.04. I would like to ask for recommendation over operating system with good mix performance/support period. Internally we use Oracle Linux, but lack of prebuild packages pushed us to consider different distribution. We are looking at these options ordered by our priority: 1. Fedora, I have doubts about long term support. Is this what is used for Monet development? 2. FreeBSD, AFAIK it has different memory management and should scale better. Thank you, Radovan

2 1

database crash
by Radovan Bičiště 11 Feb '14

11 Feb '14

Hello, I hope anyone can help with database crash. In .gdk_lock is following: USR=1001 PID=6751 TIME=Mon Feb 10 16:04:17 2014 @ LOGON USR=1001 PID=6751 TIME=Mon Feb 10 16:04:18 2014 @ !FATAL: logger_new: there is a logger catalog, but no log file. Are you sure you are using the correct combination of database (--dbpath) and log directory (--set sql_logdir)? If you have done a recent update of the server, it may be that your logs are in an old location. You should then either use --set sql_logdir=<path to old log directory> or move the old log directory to the new location (/u01/ceosdata/database/monetdb/drmax/crm/zkdev1/sql_logs/sql/). USR=1001 PID=6751 TIME=Mon Feb 10 16:04:18 2014 @ LOGOFF Anyway to start the database without logs ? Thanks, Radovan

3 6

Announcement: (Hopefully) Final Release Candidate for Jan2014 Release
by Sjoerd Mullender 10 Feb '14

10 Feb '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The MonetDB team at CWI/MonetDB BV is pleased to announce a new release candidate for the upcoming Jan2014 feature release. Many things have changed in this release compared to the previous (Feb2013) release (that's why it took so long). For a smooth upgrade I would urge you to try out this release candidate and to provide feedback on our bug tracker (http://bugs.monetdb.org) A number of bugs in the previous release candidate have been fixed. We hope that this release candidate is good enough to be the new release, but we will give it a few more days. The release candidate can be found at http://dev.monetdb.org/downloads/testing. - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUviUtz7g04AjvIQpAQJy5wQArulRMgV7YWX1NoOJMoSSKINjEQLeneZC vj4NqMJ0igxYMRxqBLd7EcSfLKvgDR3ElmP7HhDkXbmCDk6POD4PjxPRmh66AEwY uYBhT3iIEe01PFzqjMhWm2HnDnfDOTzvn8Ai4n1/qCIVmW0p14kIYOkTRXWiNX5o tcGfy384cx8= =2un1 -----END PGP SIGNATURE-----

1 0

Some questions regarding MonetDB
by Alejandro Villamarin 07 Feb '14

07 Feb '14

Hi everyone, I've got some questions regarding MonetDB and scientifc data...more precisely, I'd like to know if MonetDB is suitable for ETL on datasets coming from NOAA in the following formats: GRIB2, NetCDF and HDF5, which all are multdimensional value grid files (to put it simple). I've already installed MonetDB and played with the tutorial, but so far I looks like a traditional RDBMS to me (yet)...any insights in this direction? Should I deep dive into the reference Manual? Another question would be...could I directly insert the data or do I have to do any parsing before? Does MonetDB provide analytical tools to do, for instance, correlation on that type of datasets? Thanks in advance, Alejandro

2 2

Fwd: Error when running Monet as cron jobs
by Nitin Gautam 07 Feb '14

07 Feb '14

Hi I am parsing my logs hourly as part of cron jobs and then uploading the data to Monet. However sometimes I do see some error messages in the logs *unsupported hash algorithms: gorithms: gorit* *monetdbd: internal error while starting mserver, please refer to the logs* Not sure what these error messages are, are they because of a bug or limitation of the Monet DB. Regards Nitin

4 9

mserver5 parameters from monetdbd
by Javier Picorel 06 Feb '14

06 Feb '14

Hello, I want to profile the data that is being read or created (e.g., tables, intermediate results) during the execution of a query. I would like to have more info than the tomograph or stethoscope show. I'm thinking about using the gdk_debug parameter (ALLOCMASK and HEAPMASK). However, I cannot specify the parameters of mserver5 when I run monetdbd. Running mserver5 manually doesn't seem to work. How can I specify the mserver5 parameter when starting the farm through monetdbd? Thanks! Regards, Javier

3 2