September 2007 - developers-list

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/semantics heuristic.c
by Jan Rittinger 12 Sep '07

12 Sep '07

Hi Peter, I have two small comments to your changes: 1. Is there a reason for including monet_utils.h here? -- In my eyes it introduces an unnecessary dependency to MonetDB (as we want the SQL code generation to compile even without MonetDB). Index: milprint_summer.c =================================================================== RCS file: /cvsroot/monetdb/pathfinder/compiler/mil/milprint_summer.c,v retrieving revision 1.404 retrieving revision 1.405 diff -u -d -r1.404 -r1.405 --- milprint_summer.c 5 Sep 2007 17:06:54 -0000 1.404 +++ milprint_summer.c 11 Sep 2007 22:15:29 -0000 1.405 @@ -56,6 +56,7 @@ #define USE_DEPRECATED_ACCESS_TO_TYPE_SYSTEM 1 #include "pathfinder.h" +#include "monet_utils.h" #include <stdio.h> #include <assert.h> 2. I think you should use "#pf" (PFns_pf) as prefix for variables names instead of "pf" (PFns_lib). Namespace PFns_lib can be used in the query which allows a user to create pf:heur* variables that might conflict lateron (whereas PFns_pf can only be created inside pathfinder). --- NEW FILE: heuristic.c --- ... static PFpnode_t* var_(PFloc_t loc, int varnum) { PFpnode_t *r = p_leaf(p_varref, loc); if (r) { r->sem.qname_raw.prefix = "pf"; r->sem.qname_raw.loc = (char*) PFmalloc(8); if (r->sem.qname_raw.loc) snprintf(r->sem.qname_raw.loc, 8, "heur%03d", varnum); } return r; } On 09/12/2007 12:15 AM, Peter Boncz wrote with possible deletions: > Update of /cvsroot/monetdb/pathfinder/compiler/semantics > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv24884/compiler/semantics > > Modified Files: > Makefile.ag xquery_fo.c > Added Files: > heuristic.c > Log Message: > The infamous push-selections-down heuristic rewrite > =================================================== > > the good: equi comparisons on attributes, texts and elements in MXQ can now > be done in sublinear time thanks to indices. This involves the > "reversing"of multi-step XPath paths if required. > > the bad: it is a heuristic so no performance improvement assured, though much > of the performance pitfalls are mitigated by a loop-lifted run-time > decision whether to use the index plan or the original plan. > > the ugly: well, since I can't (or couldn't) stomach XQuery core plan, it is all > implemeted on the absssyn level. > > there are quite a few comments/explanations in compiler/semantics/heuristic.c > > runtime/pathfinder.mx > - major changes to the indexing scheme, supporting the pf:text/pf:attribute functions > - added run-time behavior to vx_lookup: for those iterations where index lookup is > not beneficial, it returns a bogus node (1@0, recognizable as a docnode) so > the original (non index-pushdown) plan is used for those iterations > > runtime/pf_support.mx > - new xquery hash function, with less collisions and the ability to preserve equality > semantics for any XQuery type (currently supported..). > - added a new dbl(byte1,...byte8) constructor that allows a IEEE double to be > constructed from its 8 binary bytes (per byte to avoid little/big endian issues) > without any precision loss. > - optimization for the ll_htordered_thetajoin in case we have constant join columns > > compiler/compile.c > compiler/include/Makefile.ag > compiler/semantics/Makefile.ag > - add heuristic.[ch] > - added heuristic index pushdown opt (currently only in mps, but alg/M4 should follow) > > compiler/semantics/xquery_fo.c > compiler/mil/milprint_summer.c > - rewrote pf:atttribute and pf:text to give approximate answers > - new pf:supernode function > - pass parsed decimals/doubles using the new byte-wise MIL dbl() constructor > (to avoid precision loss in the pf -> MonetDB transition) > - bugfix in empty ipik opt for the for-rule > > compiler/debug/abssynprint.c > - PFabssyn_stdout for debugging sanity > compiler/debug/prettyp.c > - margin=6 for better debug output readability > -- Jan Rittinger Database Systems Technische Universität München (Germany) http://www-db.in.tum.de/~rittinge/

2 1

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/mapiclient MapiClient.mx, , 1.86, 1.87
by Fabian Groffen 12 Sep '07

12 Sep '07

On 11-09-2007 15:51:28 +0000, Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/clients/src/mapiclient > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv2455/src/mapiclient > > Modified Files: > MapiClient.mx > Log Message: > Coalesced mclient's -o and -f options into a single -f (output format) > option. > > The -o option was used to specify output formatting for XQuery, > whereas -f was used to specify output formatting for the other > languages. Now the same option is used for both (similar) functions. > > This also frees up -o for a future, possible -o output file option. Is there a chance the two could be switched? -f traditionally is in use for [f]ile. JdbcClient also happens to have implemented the -f flag like that.

2 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_client.mx, , 1.157, 1.158
by Martin Kersten 11 Sep '07

11 Sep '07

Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15887 > > Modified Files: > mal_client.mx > Log Message: > dirty fix for block mode (from end of block find new line, shift all > characters up to new line add EOS and increment length of read data UGH > possible problem new line in string) Not dirty. They should be escaped to be acceptable in MAL. The new line terminates the instruction.

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/testing Makefile.ag, , 1.20, 1.21 Mtest.py.in, , 1.300, 1.301
by Stefan Manegold 11 Sep '07

11 Sep '07

On Tue, Sep 11, 2007 at 11:11:49AM +0000, Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/MonetDB/src/testing > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv17756/src/testing > > Modified Files: > Makefile.ag Mtest.py.in > Log Message: > Put Python files in their own Python module MonetDB. > In the RPMs, put Python files in the Python hierarchy. > Created a new RPM MonetDB-python which contains monet_options.py and other such files. > Added merovingian man page to RPM. > Fixed MonetDB5 spec file. > Removed Python ODBC interface: it was not directly related to MonetDB. These changes requires to set PYTHONPATH="MONETDB_PREFIX/`python -c 'import distutils.sysconfig; print distutils.sysconfig.get_python_lib(0,0,"")'`" before calling Mtest.py, unless MONETDB_PREFIX is /usr . Should we consider making Mtest.py search automatically for modules in the above path (as it IMHO did before), or shall we just document the above requirement? Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 3

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/src/sqlhelpers/xmlshred/include shred_helper.h, , 1.2, 1.3
by Stefan Manegold 11 Sep '07

11 Sep '07

On Tue, Sep 11, 2007 at 10:01:22AM +0000, Manuel Mayr wrote: > Update of /cvsroot/monetdb/pathfinder/src/sqlhelpers/xmlshred/include > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv17796/include > > Modified Files: > shred_helper.h > Log Message: > > > Adding definitions for SSZFMT and fixing some bugs > > > Index: shred_helper.h > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/src/sqlhelpers/xmlshred/include/shred_helper.h,v > retrieving revision 1.2 > retrieving revision 1.3 > diff -u -d -r1.2 -r1.3 > --- shred_helper.h 10 Sep 2007 10:09:37 -0000 1.2 > +++ shred_helper.h 11 Sep 2007 10:01:20 -0000 1.3 [...] > #if !(HAVE_STRING_H && HAVE_STRDUP) ^^ > +char * strndup (const char * s); ^^^ one 'n' too much? (or one 'N' missing??) Stefan > +#endif > + > +#if !(HAVE_STRING_H && HAVE_STRNDUP) > char * strndup (const char * s, size_t n); > #endif -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/tools merovingian.mx, , 1.46, 1.47
by Martin Kersten 11 Sep '07

11 Sep '07

Fabian wrote: > Update of /cvsroot/monetdb/MonetDB5/src/tools > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv3294 > > Modified Files: > merovingian.mx > Log Message: > Again, Merovingian is not the one to create directories, monetdb does > that. as reported. it doesn't

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/src/sqlhelpers/xmlshred xmlshred.c, , 1.2, 1.3
by Sjoerd Mullender 10 Sep '07

10 Sep '07

When printing a value of type size_t, use SZFMT. When printing a value of type ssize_t, use SSZFMT. The format needed to print these values varies from system to system, so that difference is hidden in a macro. Manuel Mayr wrote: > Update of /cvsroot/monetdb/pathfinder/src/sqlhelpers/xmlshred > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv13392 > > Modified Files: > xmlshred.c > Log Message: > > > Thx for noting. > > > Index: xmlshred.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/src/sqlhelpers/xmlshred/xmlshred.c,v > retrieving revision 1.2 > retrieving revision 1.3 > diff -u -d -r1.2 -r1.3 > --- xmlshred.c 9 Sep 2007 08:27:39 -0000 1.2 > +++ xmlshred.c 10 Sep 2007 06:55:38 -0000 1.3 > @@ -51,8 +51,7 @@ > > #define NAME_ID 0 > > -typedef long unsigned int nat; > -typedef int pre_t; > +typedef ssize_t nat; > > enum kind_t { > elem > @@ -138,8 +137,8 @@ > > typedef struct node_t node_t; > struct node_t { > - pre_t pre; > - pre_t apre; > + nat pre; > + nat apre; > nat post; > nat pre_stretched; > nat post_stretched; > @@ -155,7 +154,7 @@ > static node_t stack[STACK_MAX]; > static int level; > static int max_level; > -static pre_t pre; > +static nat pre; > static nat post; > static nat rank; > static nat att_id; > @@ -346,7 +345,7 @@ > attr_guide_node = insert_guide_node((char*)atts_back[0], > current_guide_node, attr); > > - fprintf (out_attr, "%lu, %i, \"%s\", \"%s\", %lu\n", > + fprintf (out_attr, "%i, %i, \"%s\", \"%s\", %i\n", > att_id++, pre, (char*)atts[0], (char*)atts[1], attr_guide_node->guide); > > atts += 2; > @@ -602,10 +601,10 @@ > case 'e': if(tuple.pre != -1) > fprintf (out, "%i", tuple.pre); > break; > - case 'o': fprintf (out, "%lu", tuple.post); break; > - case 'E': fprintf (out, "%lu", tuple.pre_stretched); break; > - case 'O': fprintf (out, "%lu", tuple.post_stretched); break; > - case 's': fprintf (out, "%lu", tuple.size); break; > + case 'o': fprintf (out, "%i", tuple.post); break; > + case 'E': fprintf (out, "%i", tuple.pre_stretched); break; > + case 'O': fprintf (out, "%i", tuple.post_stretched); break; > + case 's': fprintf (out, "%i", tuple.size); break; > case 'l': fprintf (out, "%u", tuple.level); break; > > case 'p': > @@ -616,7 +615,7 @@ > break; > case 'P': > if (tuple.parent) > - fprintf (out, "%lu",tuple.parent->pre_stretched); > + fprintf (out, "%i",tuple.parent->pre_stretched); > else > fprintf (out, "NULL"); > break; > @@ -653,7 +652,7 @@ > fprintf(out, "NULL"); > } > break; > - case 'g': fprintf (out, "%lu", tuple.guide); break; > + case 'g': fprintf (out, "%i", tuple.guide); break; > > default: putc (format[i], out); break; > } > @@ -966,7 +965,7 @@ > /* print the node self */ > fprintf ( > guide_out, > - "<node guide=\"%lu\" count=\"%lu\" kind=\"", > + "<node guide=\"%i\" count=\"%i\" kind=\"", > root->guide, > root->count); > print_kind (guide_out, root->kind); > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- Sjoerd Mullender

4 5

[Monetdb-developers] conf.bash still used?
by Stefan Manegold 10 Sep '07

10 Sep '07

Hi, I'm about to remove */conf/conf.bash, the script that "once" was provided to "conveniently" set your environment for building MonetDB, but --- thanks to improved configure setup and system installations --- should not be required any more. Please let me know a.s.a.p. in case there are any objections and/or anybody is still using this script. Thanks, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/tools merovingian.mx, , 1.46, 1.47
by Martin Kersten 10 Sep '07

10 Sep '07

That creates a deadlock. monetdb can not work without the mero and mero needs the dbfarm. Try testing your system without a dbfarm and var/log Fabian wrote: > Update of /cvsroot/monetdb/MonetDB5/src/tools > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv3294 > > Modified Files: > merovingian.mx > Log Message: > Again, Merovingian is not the one to create directories, monetdb does > that. > > > Index: merovingian.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/tools/merovingian.mx,v > retrieving revision 1.46 > retrieving revision 1.47 > diff -u -d -r1.46 -r1.47 > --- merovingian.mx 10 Sep 2007 07:44:09 -0000 1.46 > +++ merovingian.mx 10 Sep 2007 07:46:38 -0000 1.47 > @@ -1099,11 +1099,6 @@ > argp = dup(1); > topdp->fout = fdopen(argp, "w"); > } else { > - /* check the log directory */ > - if (access(msglog,F_OK) == -1 ){ > - GDKcreatedir(msglog); /* it doesn't necessarily create all */ > - fprintf(stderr, "log directory created '%s'\n", msglog); > - } > /* write to the given file */ > topdp->fout = fopen(msglog, "a"); > if (topdp->fout == NULL) { > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/testing difflib.c, , 1.46, 1.47
by Fabian Groffen 10 Sep '07

10 Sep '07

On 09-09-2007 11:36:26 +0000, Stefan Manegold wrote: > Update of /cvsroot/monetdb/MonetDB/src/testing > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv4669 > > Modified Files: > difflib.c > Log Message: > > no need to call "touch" via "system()" to create an empty file; > "fclose(fopen())" should do the job within pure C. Because you use the stream versions here, fclose(fopen()) can crash you tool if fopen() returns NULL, as fclose doesn't grok it. > Index: difflib.c > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/testing/difflib.c,v > retrieving revision 1.46 > retrieving revision 1.47 > diff -u -d -r1.46 -r1.47 > --- difflib.c 5 Sep 2007 17:05:58 -0000 1.46 > +++ difflib.c 9 Sep 2007 11:36:22 -0000 1.47 > @@ -922,7 +922,7 @@ > > if (!u_diff2lwc_diff(mindiff, LWC, u_diff_fn, lwc_diff_fn)) > /* { UNLINK(lwc_diff_fn); ERRHNDL(0,"u_diff2lwc_diff returns 0 in u_diff2html","",1); } */ > - SYSTEM(strconcat("touch ", lwc_diff_fn)); > + fclose(Wfopen(lwc_diff_fn)); Unless Wfopen does something fancy with the pointer, I suggest to check if it's non NULL.

2 1