- developers-list - monetdb.org

Interaction of operators in the kernel.
by Xu,Wenjian 08 Jul '15

08 Jul '15

Hi, My objective is to investigate that, given an SPJ query, how are BATs *physically* processed (changed) among relational algebra operators (e.g., select, join). I do not care about how query is translated into execution plan in MAL, what I care is the interaction of operators in the kernel (aka., gdk) . My current plan is to add some debug information in those operators (e.g., functions in src/gdk_join.c, src/gdk_select.c) and re-compile the system. Then, I will pose an SPJ query (through console) and check the system log to see how those operators are invoked in sequence. But this method seems inefficient and I would like to ask if there are any alternatives, for example, is it possible to debug the system *step-by-step* given an input query (or its logical query plan). Thank you very much for any kind help. Best regards, Wenjian

2 1

Transaction-replication, monetdbd database properties and other changes on default
by Dimitar Nedev 23 Jun '15

23 Jun '15

Hello everyone, We've recently merged the transaction-replicaiton branch into the default. That means that you can try out the new, still experimental, transaction-replication if you build the MonetDB default branch. Since there is already a release branch for the next feature release (July2015), the transaction-replicaiton support will not be part of the next major release. In general, most changes affect a database instance if it runs as a slave, master instances should function just as before. So unless you flip the right switches (and it cannot happen accidentally*) you shouldn't notice any differences. Most code changes are in gdk_logger, where a few large functions (including logger_new) were split into smaller one to accommodate for the need of new read-only logger. That is, a logger that only reads transactions and persists them in the data store. In additional, you can now set any property to a database (mserver5) instance running under the MonetDB-Daemon (monetdbd). Previously only a few preset properties could be set via the monetdbd. The current preset properties will still function as before, both in terms of mapping monetdb-to-mserver5 names and allowed values. * I will provide more information how to turn on and try out the transaction-replication in a blog post on the website very soon. Best regards, Dimitar

1 0

New Jul2015 stable branch created
by Sjoerd Mullender 11 Jun '15

11 Jun '15

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2015. So in each of your stable checkout directories you can do hg pull hg update -rJul2015 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Oct2014-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 09 Jun '15

09 Jun '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP4 bugfix release Bug Fixes * 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. * 3707: var() possibly not working in debug builds * 3712: Concurrency issue on querying the SQL catalog * 3713: Long startup cost for simple session * 3715: Crash with two ALTER TABLE statements in a transaction * 3718: Adding and dropping a non existing tablename to/from a merge table is accepted without an error * 3719: Assertion failure in /MonetDB-11.19.11/gdk/gdk_bat.c:2841: BATassertHeadProps: Assertion `!b->H->key || cmp != 0' failed. * 3720: Incorrect results on joining with same table * 3723: Assertion failure in rel_bin.c:2548: rel2bin_groupby: Assertion `0' failed. * 3725: LEFT JOIN bug with CONST value * 3731: left shift for IP addresses not available to non-system users

1 0

Hashjoin performance with large vs small tables
by Roberto Cornacchia 12 May '15

12 May '15

Hi there, Triggered by a suspiciously slow join, I had a look at BATsubjoin, in particular at how it is decided which algorithm to use. I looked at a couple of cases with a join on strings. This is one (pseudo syntax): l.count ~16M l.T.sorted = 0 l.T.revsorted = 0 r.count = 1 r.T.sorted = 1 r.T.revsorted = 1 In this case, a hash table on r is created (because it's smaller). This join takes *430ms*. I forced swapping l and r, thus built the hash table on the larger bat, and then it takes *0.8ms*. Notice that though r.count = 1 is a bit of an extreme case, it does happen often in practice. And the case with a few tuples would not be very different. The question I wanted to ask is: is it always right to build the hash table on the small table? Perhaps some more heuristics can help? Like: if the smaller is very small, and the larger is not extremely large, then hash on the larger one? Roberto

2 12

Facing issues in implementing Embedded R in MonetDB-11.19.11
by Vishal Virmani 11 May '15

11 May '15

Hi I am implementing Embedded R in MonetDB version 11.19.11. My environment is VM with Ubuntu 14.04 Server. I have build MonetDB from source and R as well as from source. Running the command sudo mserver5 --version shows me the desired output. When i try to invoke a database using sudo monetdb start testdb1 command, where testdb1 is a database i successfully created, it repeatedly throws up an error...database 'testdb1' started up, but failed to open up a communication channel. Upon checking the associated merovingian log file i see further messages saying the object code to rapi.prelude missing. Any way to resolve this...? Regards Vishal Virmani @poisoncreed

1 0

Suggestion: Windows builds ...
by g4＠novadsp.com 11 May '15

11 May '15

Hello Might it be possible to include copies of the relevant header files for the support libraries (iconv through to zdll) in the GIT repo and/or tarballs? There is a considerable amount of pain involved in acquiring/building/installing matching versions of the lib/dll files on Windows (esp. on Windows!). Also libstream.dll does not advertise any version information which makes matching just that bit harder :) Thanks. Jerry.

1 0

Announcement: New Oct2014-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 07 May '15

07 May '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP3 bugfix release Build Environment * We now also create debug packages for Debian and Ubuntu. MonetDB Common * Replaced the rangejoin implementation with one that uses imprints if it can. Bug Fixes * 3466: UPDATE statements fails with "GDKerror: MT_mremap() failed" * 3602: Surprising overload resolution of generate_series * 3613: SQL data dictionary contains columns names which are also special keywords. This causes unexpected/unneeded SQL query errors * 3645: Network address operators such as << and <<= do not work * 3647: missing BAT for a column leads to crash in gtr_update_delta * 3648: memory corruption on unclean connection shutdown with local temporary tables * 3650: Naming of persistent BATs is fragile * 3653: PREPARE crashes mserver if unbound variable is function parameter * 3655: SQL WHERE -1 in (-1) issue? * 3656: error message after calling fitsload() * 3660: Incorrect Results for Comparison Operators on inet Datatype * 3661: Ship debug symbols for pre-built binaries * 3662: UPDATE row with row value constructor crashes monetdb server * 3663: Incorrect result ROW_NUMBER in subquery * 3664: SQLstatementIntern missing parameter when using jsonstore * 3665: inter-session starvation issue, particularly affects sys.queue * 3666: casting text column to inet truncating text column and resulting inet for first occurrence only * 3667: insert of negative value for oid column aborts mserver5 process with assertion failure * 3669: ALTER TABLE <tbl_nm> ADD CONSTRAINT <tbl_uc1> UNIQUE (col1, col2, col3) causes Assertion failure and abort * 3671: ODBC-Access on Windows 2012 does not work - E_FAIL * 3672: libbat_la-gdk_utils.o: relocation R_X86_64_PC32 against `MT_global_exit' can not be used when making a shared object * 3676: merovingian hangs trying to exit * 3677: Crash in BATgroup_internal (caused by 87379087770d?) * 3678: Ruby driver installation ignores prefix * 3680: Prepared statements fail on execution with message 'Symbol type not found' * 3684: Wrong query result set WHERE "IS NULL" or "NOT IN" clauses uses in combination with ORDER, LIMIT and OFFSET * 3687: 'bat.insert' undefined * 3688: Crash at exit (overrun THRerrorcount?) * 3689: No more connections accepted if a single client misbehaves * 3690: find_fk: Assertion `t && i' failed. * 3691: conversion of whitespaces string to double or float is accepted without an error during insert * 3693: algebra.join undefined (caused by non-existing variables in the plan) * 3696: Inconsistent behavior between dbl (SQL double) and flt (SQL real) data types and across platforms * 3697: mserver5[26946]: segfault at 0 ip 00007f3d0e1ab808 sp 00007f3cefbfcad0 error 4 in lib_sql.so[7f3d0e180000+16c000] * 3699: segfault again! (during last week I found 3 segfault bugs already) * 3703: INSERT INTO a MERGE TABLE crashes mserver5 * 3704: Unknown identifier from subquery * 3705: Assertion failure in rel_bin.c:2274: rel2bin_project: Assertion `0' failed. * 3706: Assertion failure in gdk_bat.c: BATassertHeadProps: Assertion `!b->H->sorted || cmp <= 0' failed. * 3709: "BATproject: does not match always" on abusive use of ALTER TABLE SET READ ONLY

1 0

C++ compiling Fixes
by Bastian Köcher 27 Apr '15

27 Apr '15

Hi, I want to write an optimizer for MonetDB in C++. So I included the necessary headers into my project, but I ran into some compile errors and warnings. I created a patch which solves some of these problems. Regards, Bastian

1 0

Re: MonetDB: default - remove jsonstore code we do not need for the ...
by Stefan Manegold 21 Apr '15

21 Apr '15

Arjen, are these left-overs occurrences of "jsonstore" in the default branch still used / required, or can they also be removed? ======== .../default/source/MonetDB $ hg locate | grep --color -i jsonstore sql/scripts/41_jsonstore.sql .../default/source/MonetDB $ hg locate -X \**/Tests/*.stable.* | xargs grep -l -i jsonstore MonetDB.spec NT/installer32/MonetDB5-SQL-Installer.vdproj NT/installer64/MonetDB5-SQL-Installer.vdproj NT/monetdb_config.h.in NT/rules.msc clients/Tests/All monetdb5/mal/mal.c sql/backends/monet5/sql_upgrades.c sql/scripts/41_jsonstore.sql sql/scripts/Makefile.ag testing/Mtest.py.in ======== Thanks! Stefan ----- On Apr 21, 2015, at 12:37 PM, Sjoerd Mullender commits(a)monetdb.org wrote: > Changeset: e6f13746b75b for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e6f13746b75b > Removed Files: > sql/backends/monet5/rest/73_jsonstore.mal > sql/backends/monet5/rest/Makefile.ag > sql/backends/monet5/rest/Tests/All > sql/backends/monet5/rest/Tests/jsonstore00.sql > sql/backends/monet5/rest/Tests/jsonstore00.stable.err > sql/backends/monet5/rest/Tests/jsonstore00.stable.out > sql/backends/monet5/rest/Tests/jsonstore01.sql > sql/backends/monet5/rest/Tests/jsonstore01.stable.err > sql/backends/monet5/rest/Tests/jsonstore01.stable.out > sql/backends/monet5/rest/jsonstore.mal > sql/backends/monet5/rest/rest_jsonstore.c > sql/backends/monet5/rest/rest_jsonstore.h > sql/backends/monet5/rest/rest_jsonstore_handle_get.c > sql/backends/monet5/rest/rest_jsonstore_handle_get.h > Modified Files: > configure.ag > monetdb5/mal/Makefile.ag > monetdb5/mal/mal_http_daemon.c > sql/backends/monet5/Makefile.ag > Branch: default > Log Message: > > remove jsonstore code we do not need for the next feature release > > > diffs (truncated from 2238 to 300 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -188,30 +188,13 @@ no-auto) > ;; > esac > > -dft_jsonstore=no > -AC_ARG_ENABLE(jsonstore, > - AS_HELP_STRING([--enable-jsonstore], > - [enable support for MonetDB/jsonstore (default=no)]), > - enable_jsonstore=$enableval, > - enable_jsonstore=$dft_jsonstore) > - > -dft_microhttpd=$enable_jsonstore > +dft_microhttpd=no > AC_ARG_ENABLE(microhttpd, > AS_HELP_STRING([--enable-microhttpd], > - [enable support for the builtin MircoHTTPD daemon (default: implied by > --enable-jsonstore)]), > + [enable support for the builtin MircoHTTPD daemon]), > enable_microhttpd=$enableval, > enable_microhttpd=$dft_microhttpd) > > -case "$enable_jsonstore-$enable_microhttpd" in > -yes-no) > - AC_MSG_ERROR([jsonstore require microhttpd]) > - ;; > -auto-no) > - enable_jsonstore=no > - disable_jsonstore="(jsonstore requires microhttpd)" > - ;; > -esac > - > dft_gsl=auto > AC_ARG_ENABLE(gsl, > AS_HELP_STRING([--enable-gsl], > @@ -2508,7 +2491,7 @@ if test x"$have_atomic_ops" = x"yes" ; t > fi > > dnl MicroHTTPD library > -dnl required for MonetDB5 SQL jsonstore, optional otherwise > +dnl optional > org_have_libmicrohttpd=auto > have_libmicrohttpd=$org_have_libmicrohttpd > if test "x$enable_microhttpd" != xno; then > @@ -2526,24 +2509,6 @@ if test x$enable_microhttpd != xno; then > AC_DEFINE(HAVE_MICROHTTPD, 1, [Define if the microhttpd server is to be > enabled]) > fi > > -org_have_liburiparser=auto > -have_liburiparser=$org_have_liburiparser > -if test "x$enable_jsonstore" != xno; then > - dnl Uriparser library > - dnl required for MonetDB5 SQL jsonstore, optional otherwise > - PKG_CHECK_MODULES([liburiparser], [liburiparser], > - [have_liburiparser=yes], > - [have_liburiparser=no; disable_jsonstore="(liburiparser not found)"; if test > "x$enable_jsonstore" = xyes; then AC_MSG_ERROR([LibUriParser library not found > but required for jsonstore]); fi]) > - > - if test "$have_libmicrohttpd-$have_liburiparser" = yes-yes; then > - AC_DEFINE(HAVE_JSONSTORE, 1, [Define that you want to build the jsonstore > library]) > - elif test "x$enable_jsonstore" = xyes; then > - AC_MSG_ERROR([cannot enable jsonstore > $why_have_libmicrohttpd$why_have_liburiparser]) > - else > - enable_jsonstore=no > - fi > -fi > - > dnl Check for gsl library for statistical functions > org_have_gsl=no > have_gsl=$org_have_gsl > @@ -2920,7 +2885,6 @@ AM_CONDITIONAL(HAVE_GDK, test "x$enable_ > AM_CONDITIONAL(HAVE_MONETDB5, test "x$enable_monetdb5" = xyes -o > "x$enable_monetdb5" = xauto) > AM_CONDITIONAL(HAVE_SQL, test "x$enable_sql" = xyes -o "x$enable_sql" = xauto) > AM_CONDITIONAL(HAVE_GEOM, test "x$enable_geom" = xyes -o "x$enable_geom" = > xauto) > -AM_CONDITIONAL(HAVE_JSONSTORE, test "x$enable_jsonstore" = xyes -o > "x$enable_jsonstore" = xauto) > AM_CONDITIONAL(HAVE_TESTING, test "x$enable_testing" = xyes -o > "x$enable_testing" = xauto) > AM_CONDITIONAL(HAVE_DEVELOPER, test "x$enable_developer" = xyes) > > @@ -3277,7 +3241,6 @@ for comp in \ > 'console ' \ > 'gsl ' \ > 'fits ' \ > - 'jsonstore ' \ > 'microhttpd ' \ > 'rintegration' \ > 'odbc ' \ > diff --git a/monetdb5/mal/Makefile.ag b/monetdb5/mal/Makefile.ag > --- a/monetdb5/mal/Makefile.ag > +++ b/monetdb5/mal/Makefile.ag > @@ -9,7 +9,6 @@ INCLUDES = ../../common/options \ > ../../common/utils \ > ../../clients/mapilib \ > ../../gdk \ > - #../../sql/backends/monet5/rest \ > $(READLINE_INCS) \ > $(libmicrohttpd_CFLAGS) > MTSAFE > diff --git a/monetdb5/mal/mal_http_daemon.c b/monetdb5/mal/mal_http_daemon.c > --- a/monetdb5/mal/mal_http_daemon.c > +++ b/monetdb5/mal/mal_http_daemon.c > @@ -12,262 +12,6 @@ > > #include "monetdb_config.h" > > -#ifdef HAVE_MICROHTTPD > -#include <sys/types.h> > -#include <sys/select.h> > -#include <sys/socket.h> > -#include <stdint.h> > -#include <stdarg.h> > -#include <stdio.h> > - > -#include "mal_client.h" > -#include "mal_http_daemon.h" > -#include "mal_private.h" > -#include <microhttpd.h> > - > -static MT_Id hdthread; > -static int volatile hdrunning; > -struct MHD_Daemon *http_daemon; > -#define PORT 8998 > -#define POSTBUFFERSIZE 512 > -#define MAXNAMESIZE 512 > -#define MAXANSWERSIZE 512 > - > -#define GET 0 > -#define POST 1 > - > -static > -http_request_handler http_handler; > - > -void register_http_handler(http_request_handler handler) { > - http_handler = handler; > -} > - > -struct connection_info_struct > -{ > - int connectiontype; > - char *answerstring; > - struct MHD_PostProcessor *postprocessor; > -}; > - > -static int > -send_page (struct MHD_Connection *connection, const char * url, > - const char * method, char *page, char * postdata) > -{ > - int ret; > - int rest; > - struct MHD_Response *response; > - > - rest = (*http_handler)(url, method, &page, postdata); > - (void)rest; > - response = > - MHD_create_response_from_buffer (strlen (page), > - (void *) page, > - MHD_RESPMEM_MUST_COPY); > - if (!response) > - return MHD_NO; > - > - ret = MHD_queue_response (connection, MHD_HTTP_OK, response); > - MHD_destroy_response (response); > - > - return ret; > -} > - > -static int > -iterate_post (void *coninfo_cls, enum MHD_ValueKind kind, const char *key, > - const char *filename, const char *content_type, > - const char *transfer_encoding, const char *data, uint64_t off, > - size_t size) > -{ > - struct connection_info_struct *con_info = coninfo_cls; > - char *answerstring; > - > - (void)key; > - (void)kind; > - (void)filename; > - (void)content_type; > - (void)transfer_encoding; > - (void)off; > - > - if (strcmp (key, "json") == 0) > - { > - if ((size > 0) && (size <= MAXNAMESIZE)) > - { > - answerstring = malloc (MAXANSWERSIZE); > - if (!answerstring) > - return MHD_NO; > - > - snprintf (answerstring, MAXANSWERSIZE, "%s", data); > - con_info->answerstring = answerstring; > - } else > - con_info->answerstring = NULL; > - > - return MHD_NO; > - } > - > - if (strcmp (key, "file") == 0) > - { > - if ((size > 0) && (size <= MAXNAMESIZE)) > - { > - answerstring = malloc (MAXANSWERSIZE); > - if (!answerstring) > - return MHD_NO; > - > - snprintf (answerstring, MAXANSWERSIZE, "%s", data); > - con_info->answerstring = answerstring; > - } else > - con_info->answerstring = NULL; > - > - return MHD_NO; > - } > - > - return MHD_YES; > -} > - > -static void > -request_completed (void *cls, struct MHD_Connection *connection, > - void **con_cls, enum MHD_RequestTerminationCode toe) > -{ > - struct connection_info_struct *con_info = *con_cls; > - > - (void)cls; > - (void)connection; > - (void)toe; > - > - if (con_info == NULL) > - return; > - > - if (con_info->connectiontype == POST) > - { > - MHD_destroy_post_processor (con_info->postprocessor); > - if (con_info->answerstring) > - free (con_info->answerstring); > - } > - > - free (con_info); > - *con_cls = NULL; > -} > - > -static int > -answer_to_connection (void *cls, struct MHD_Connection *connection, > - const char *url, const char *method, > - const char *version, const char *upload_data, > - size_t *upload_data_size, void **con_cls) > -{ > - char * page = NULL; > - struct connection_info_struct *con_info = NULL; > - int *done = cls; > - char *answerstring = NULL; > - char *errorpage = > - "<html><body>" > - "Failed to handle error in http request." > - "</body></html>"; > - > - (void)version; > - > - if (*con_cls == NULL) { > - con_info = malloc (sizeof (struct connection_info_struct)); > - if (con_info == NULL) > - return MHD_NO; > - con_info->answerstring = NULL; > - > - if (strcmp (method, "POST") == 0) { > - con_info->postprocessor = > - MHD_create_post_processor (connection, > - POSTBUFFERSIZE, > - iterate_post, > - (void *) con_info); > - > - if (con_info->postprocessor == NULL) { > - free (con_info); > - return MHD_NO; > - } > - con_info->connectiontype = POST; > - } else { > - con_info->connectiontype = GET; > - } > - *con_cls = (void *) con_info; > - return MHD_YES; > - } > - > - if (strcmp (method, "POST") == 0) { > - con_info = *con_cls; > - if (*upload_data_size != 0) { > - MHD_post_process (con_info->postprocessor, upload_data, > - *upload_data_size); > - *upload_data_size = 0; > - return MHD_YES; > - } else { > - if (con_info->answerstring != NULL) { > - return send_page(connection, url, method, page, > - con_info->answerstring); > - } > - } > - } > - > - if (strcmp (method, "PUT") == 0) { > - if (*upload_data_size != 0) { > - // TODO: check free answerstring > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1