- developers-list - monetdb.org

MonetDB in Fedora 21 and 22, and EPEL 7
by Jan Holčapek 10 Feb '15

10 Feb '15

Hello, I am aware that MonetDB is available as native Fedora 18-20 [1] and EPEL 6 [2] packages. However, I am missing native Fedora 21-22, as well as EPEL 7 packages. Would you find it beneficial if I try to push MonetDB directly to Fedora/EPEL packages, ie. it would have a record in Fedora Package Database [3] and be built on Fedora infrastructure? That said, I would become a mere package maintainer for the two distributions. Cheers Jan [1] https://www.monetdb.org/Documentation/UserGuide/Downloads/Fedora [2] https://www.monetdb.org/downloads/epel/ [3] https://admin.fedoraproject.org/pkgdb/

2 1

Adapting code Feb2013 -> Oct2014
by Roberto Cornacchia 09 Feb '15

09 Feb '15

Hi there, I have some custom code running on Feb2013 release. As I'm now trying to move to Oct2014, I need to update this custom code. MonetDB API has changed a bit in the meantime, for example if I just try to compile my code within Oct2014, I get error: implicit declaration of function 'ALGfetch' Indeed, looking at the previous commits, ALGfetch is one of the interfaces that have been removed. Question: Is there any documentation that can help me see more clearly what changed and how to adapt the code? Or is browsing the commit history my only resource? Thanks, Roberto

2 3

Announcement: New Oct2014-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 29 Jan '15

29 Jan '15

The MonetDB team at CWI/MonetDB BV is pleased to announce the Oct2014-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Oct 2014-SP2 bugfix release Client Package * Changes to the Perl interface, thanks to Stefan O'Rear: 1. removes "use sigtrap", because this has global effects and should not be used by modules, only by the application. 2. allows Perl 5.8.1+ Unicode strings to be passed to quote() and included in statements (UTF-8 encoded, as expected by Monet's str module) 3. quote and unquote now use the same quoting rules as the MonetDB server, allowing for all characters except NUL to be round-tripped 4. several character loops have been reimplemented in regex for much greater performance 5. micro-optimizations to the result fetch loop 6. block boundaries are preserved in piggyback data so that Xclose is not appended or prepended to a SQL command 7. diagnostic messages #foo before a result header are ignored, this is necessary to use recycler_pipe 8. fail quickly and loudly if we receive a continuation prompt (or any other response that starts with a non-ASCII character) 9. header lines must start with %, not merely contain %, fixing a bug when querying a table where string values contain % 10. after closing a large resultset, account for the fact that a reply will come and do not lose sync 11. allow a MAPI_TRACE environment variable to dump wire protocol frames to standard output 12. fixes maximum MAPI block size to match the server limit of 16k. previously would crash on blocks larger than 16k SQL * Fixed a typo in a column name of the sys.tablestoragemodel view (auxillary changed to auxiliary). Bug Fixes * 3467: Field aliases with '#' character excise field names in result set. * 3605: relational query without result * 3619: Missing dll on MonetDB Start * 3622: Type resolution error * 3624: insert of incomplete or invalid ip address values in inet column is silently accepted but the values are not stored (they become/show nil) * 3626: casting a type without alias results in program contains errors * 3628: mclient and ODBC driver report 'type mismatch' when stddev_pop used in a select which returns 0 rows * 3629: IF THEN ELSEIF always evaluates the first test as true * 3630: segv on rel_order_by_column_exp * 3632: running make clean twice gives an error in clients/ruby/adapter * 3633: Wrong result for HAVING with floating-point constant * 3640: Missing implementation of scalar function: sql_sub(<date>, <month interval>) * 3641: SQL lexer fails to detect string end if it the last character is U+FEFF ZERO WIDTH NO-BREAK SPACE * 3642: Combined WHERE conditions less-than plus equals-to produce incorrect results * 3643: Missing implementations of scalar function: sql_sub(<timetz>, arg2) * 3644: COPY INTO fails to import "inet" data type when value has prefix length in CIDR notation * 3646: ORDER BY clause does not produce proper results on 'inet' datatype * 3649: recycler crashes with concurrent transactions

1 0

Re: MonetDB: Oct2014 - Reimplemented rangejoin, now using imprints o...
by Roberto Cornacchia 28 Jan '15

28 Jan '15

Hi Sjoerd, This is very welcome! If I read well, this will use, in order of preference: 1. binary search (if l is sorted) 2. imprints if available 3. nested loop otherwise We use rangejoin extensively within Spinque and the previous one (just nested loop) has never an option. So far we have been using our own version which is perhaps naive but proved to be effective: - sort all inputs if not sorted already - perform a mergejon-like fast scan Though simple, I have not found so far a case where this strategy would not outperform nested loops by far. The cost of sorting was always far less than the cost of a full nested loop. I was wondering what your thoughts about this would be. Could this strategy replace the number 3 above? Does it make sense to keep the vanilla nested loop? Best, Roberto On 27 January 2015 at 14:09, Sjoerd Mullender <commits(a)monetdb.org> wrote: > Changeset: 5147add3bb38 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=5147add3bb38 > Modified Files: > gdk/ChangeLog.Oct2014 > gdk/gdk_join.c > gdk/gdk_private.h > gdk/gdk_select.c > Branch: Oct2014 > Log Message: > > Reimplemented rangejoin, now using imprints or binary search if possible. > >

2 2

Re: MonetDB: Oct2014 - Use imprints to calculate min/max of bat.
by Lefteris 22 Jan '15

22 Jan '15

Imprints code contains such a search for min/max, maybe we could expose it as a function and thus avoid having the same code twice (for the usual reasons of easy maintenance and bugfix) On Thu, Jan 22, 2015 at 2:15 PM, Sjoerd Mullender <commits(a)monetdb.org> wrote: > Changeset: 69a4f7d2fd58 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=69a4f7d2fd58 > Modified Files: > gdk/gdk_aggr.c > Branch: Oct2014 > Log Message: > > Use imprints to calculate min/max of bat. > But with limitations: no grouped min/max, and only on the complete bat > (i.e. if view, it's a view on the complete bat). > > > diffs (37 lines): > > diff --git a/gdk/gdk_aggr.c b/gdk/gdk_aggr.c > --- a/gdk/gdk_aggr.c > +++ b/gdk/gdk_aggr.c > @@ -2207,8 +2207,31 @@ BATminmax(BAT *b, void *aggr, > return NULL; > needdecref = 1; > } > - (void) (*minmax)(&pos, b, NULL, 1, 0, 0, 0, BATcount(b), > - NULL, NULL, BATcount(b), 1, 0); > + if (b->T->imprints && > + (VIEWtparent(b) == 0 || > + BATcount(b) == BATcount(BBPdescriptor(VIEWtparent(b))))) { > + pos = oid_nil; > + if (minmax == do_groupmin) { > + /* find first non-empty bin */ > + for (s = 0; s < b->T->imprints->bits; s++) { > + if (b->T->imprints->stats[s + 128]) { > + pos = b->T->imprints->stats[s] + > b->hseqbase; > + break; > + } > + } > + } else { > + /* find last non-empty bin */ > + for (s = b->T->imprints->bits - 1; s >= 0; s--) { > + if (b->T->imprints->stats[s + 128]) { > + pos = b->T->imprints->stats[s + > 64] + b->hseqbase; > + break; > + } > + } > + } > + } else { > + (void) (*minmax)(&pos, b, NULL, 1, 0, 0, 0, BATcount(b), > + NULL, NULL, BATcount(b), 1, 0); > + } > if (pos == oid_nil) { > res = ATOMnilptr(b->ttype); > } else { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 1

Perl driver improvements
by Stefan O'Rear 13 Jan '15

13 Jan '15

I have been integrating MonetDB into a Perl system for the last few months, and have made a few changes to the Perl driver to improve performance and reliability. I hereby release these changes under the same license as MonetDB itself. I have permission from my employer to do this. Is this to correct place to send and/or discuss patches? In order of patch hunks; more details will be provided for any part if requested: 1. removes "use sigtrap", because this has global effects and should not be used by modules, only by the application. 2. allows Perl 5.8.1+ Unicode strings to be passed to quote() and included in statements (UTF-8 encoded, as expected by Monet's str module) 3. quote and unquote now use the same quoting rules as the MonetDB server, allowing for all characters except NUL to be round-tripped 4. several character loops have been reimplemented in regex for much greater performance 5. micro-optimizations to the result fetch loop 6. block boundaries are preserved in piggyback data so that Xclose is not appended or prepended to a SQL command 7. diagnostic messages #foo before a result header are ignored, this is necessary to use recycler_pipe 8. fail quickly and loudly if we receive a continuation prompt (or any other response that starts with a non-ASCII character) 9. header lines must start with %, not merely contain %, fixing a bug when querying a table where string values contain % 10. after closing a large resultset, account for the fact that a reply will come and do not lose sync 11. allow a MAPI_TRACE environment variable to dump wire protocol frames to standard output 12. fixes maximum MAPI block size to match the server limit of 16k. previously would crash on blocks larger than 16k

2 1

Bug in Query Plan??
by Vijay Krishna 09 Jan '15

09 Jan '15

Hi, I have been studying on multiple join performances in MonetDB. I am running the latest version of MonetDB Oct2014 - SP1 (MonetDB-11.19.3) on a CentOS machine with 128 GB RAM. All my tables have 100 to 300 columns and around 35 lakh rows. I have the following query with 5 joins and a few criteria. SELECT * FROM table1 INNER JOIN table2 on table1.col1=table2.col1 LEFT JOIN table3 ON table1.col2=table3.col1 LEFT JOIN table3 ON table1.col3=table3.col1 LEFT JOIN table4 ON table2.col2=table4.col1 LEFT JOIN table5 ON table2.col3=table5.col1 WHERE (((table2.col10 like '%a%' OR table5.col11 like '%s%') and ((table2.col12 like '%w%' *and table4.col13 like '%k'*) OR table2.col14 like '%h%') and (table1.col21 = 1) AND (table1.col22 IN (1,2,3))) AND (((table1.col23 >= 15842000000000000) AND (table1.col23 <= 15842999999999999)) OR ((table1.col23 >= 0) AND (table1.col23 <= 999999999999)))) ORDER BY 1 DESC LIMIT 10 OFFSET 0; This query has only 2000 matching rows and it returns the result in *2.9 seconds*. In the same query, if I remove one criteria alone (the bolded part), the result would still match only 2000 rows, but the result would come in *300 milliseconds* even. I used the PLAN statement and found that the query plan that the executor takes for the first query is a much complex path and *it performs 3 joins for all the 35 lakh rows* (i.e. before executing the criteria even) and hence the time has spiked to 3 seconds. May be the hot data of 35 lakh tuples would have not fit into the memory?? Not sure. But if I remove the highlighted criteria, the executor evaluates the criteria first and *it performs the joins for only 7000 rows* and hence it is faster 1000 times. Is this a bug? Or is there any way to optimise my query to match the executor to choose the right plan? *I am looking on to install MonetDB in our production setup, but finding this as a showstopper. * Any help much appreciated. I could even demo this. Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305.

1 0

Some fun for Christmas
by Ying Zhang 25 Dec '14

25 Dec '14

Dear all, Just to add some fun for the Christmas, I have two new videos for you (starring Thibault, Hannes ;-)): /ufs/zhang/Blaeu_demo_screencast.mp4 /ufs/zhang/monetdb-r-screencast.mp4 They are not published yet. If you want to share them externally, please check with Hannes and/or Thibault. Feedback is highly appreciated. However, please keep in mind that these videos are primarily meant to be informative. They were in a hurry without prepared scripts :P Anyways, enjoy. Wish you a nice Christmas and New Year holiday. With kind regards, Jennie

1 0

Re: MonetDB: mosaic - Fix merge conflict.
by Martin Kersten 12 Dec '14

12 Dec '14

Thank you for taking the initiative. On 12/12/14 11:03, Sjoerd Mullender wrote: > Changeset: 3e1d5cbd649d for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3e1d5cbd649d > Modified Files: > sql/server/rel_optimizer.c > Branch: mosaic > Log Message: > > Fix merge conflict. > > > diffs (21 lines): > > diff --git a/sql/server/rel_optimizer.c b/sql/server/rel_optimizer.c > --- a/sql/server/rel_optimizer.c > +++ b/sql/server/rel_optimizer.c > @@ -6426,7 +6426,7 @@ rel_merge_table_rewrite(int *changes, mv > > /* do not include empty partitions */ > if ((nrel || nt->next) && > - pt && isTable(pt) && pt->readonly && !store_funcs.count_col(sql->session->tr, pt->columns.set->h->data, 1)){ > + pt && isTable(pt) && pt->access == TABLE_READONLY && !store_funcs.count_col(sql->session->tr, pt->columns.set->h->data, 1)){ > continue; > } > > @@ -6439,7 +6439,7 @@ rel_merge_table_rewrite(int *changes, mv > sql_exp *ne = m->data; > int i; > > - if (pt && isTable(pt) && pt->readonly && sel && (nrel || nt->next) && (i=find_col_exp(cols, e)) != -1) { > + if (pt && isTable(pt) && pt->access == TABLE_READONLY && sel && (nrel || nt->next) && (i=find_col_exp(cols, e)) != -1) { > /* check if incase of an expression if the part falls within the bounds else skip this (keep at least on part-table) */ > void *min, *max; > sql_column *col = NULL; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Win32 build question
by g4＠novadsp.com 29 Nov '14

29 Nov '14

Hello and thanks for MonetDB. Splendid stuff. Brilliant. I am trying to build a 32 bit version of 11.19.3 on Windows from the latest tarball. Root folder is R:\src\libraries\MonetDB-11.19.3. Building in R:\src\libraries\MonetDB-11.19.3\bin The build is failing at the point shown below. What have I missed here? The build folder is a sibling of /NT as indicated in the windowsbuild.rst ...? Many thanks for any pointers. Jerry. -------------------------snip-------------------------- copy ".\..\..\..\monetdb5\scheduler\run_isolate.mal" "run_isolate.mal" 1 file(s) copied. if not exist "tools" mkdir "tools" copy ".\..\..\monetdb5\tools\Makefile.msc" "tools\Makefile" 1 file(s) copied. cd "tools" && U:\vs2013\VC\BIN\nmake.exe /nologo DEBUG=1 "prefix=R:\src\ libraries\MonetDB-11.19.3\bin" "bits=32" all NMAKE : fatal error U1073: don't know how to make '..\..\gdk\libbat.lib' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: 'U:\vs2013\VC\BIN\nmake.exe' : return code '0x2' Stop. R:\src\libraries\MonetDB-11.19.3\bin> -------------------------snip-------------------------- Platform details: Windows 7x64 using VS2013 pro x86 command shell. I've got a very simple script which sets up the correct include/lib paths and then invokes nmake thus: SET LIBOPENSSL=U:\pkg\OpenSSL\Win32 SET LIBICONV=U:\pkg\monet\iconv-1.9.2.win32 SET LIBZLIB=U:\pkg\monet\zlib-1.2.5 SET LIBXML2=U:\pkg\monet\libxml2-2.7.8.win32 SET LIBPCRE=U:\pkg\monet\pcre-7.0-bin SET INCLUDE=%INCLUDE%;U:\pkg\monet\iconv-1.9.2.win32\include;U:\pkg\monet\pcre-7 .0-bin\include nmake /D -f ..\NT\makefile DEBUG=1 HAVE_OPENSSL=1 HAVE_SQL=1 HAVE_MONETDB5=1 all

1 0