February 2008 - developers-list

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/algebra/opt opt_algebra_cse.c, XQuery_0-22, 1.15.2.4, 1.15.2.5
by Manuel Mayr 19 Feb '08

19 Feb '08

No that's fine, I forgot to remove this one. Thx and best regards, Manuel On Feb 19, 2008, at 9:14 PM, Stefan Manegold wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/algebra/opt > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15698/ > compiler/algebra/opt > > Modified Files: > Tag: XQuery_0-22 > opt_algebra_cse.c > Log Message: > > making pathfinder compile with "--enable-strict", again: > removed unused variable. > > Manuel, > > please check, that I did not break anything ... > > > > > Index: opt_algebra_cse.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/algebra/opt/ > opt_algebra_cse.c,v > retrieving revision 1.15.2.4 > retrieving revision 1.15.2.5 > diff -u -d -r1.15.2.4 -r1.15.2.5 > --- opt_algebra_cse.c 19 Feb 2008 17:39:11 -0000 1.15.2.4 > +++ opt_algebra_cse.c 19 Feb 2008 20:14:42 -0000 1.15.2.5 > @@ -1964,7 +1964,6 @@ > PFla_op_t *seed = CSE(L(n)); > PFla_op_t *result = CSE(R(n)); > > - PFalg_proj_t *p = NULL; > PFarray_t *seed_actmap = create_actatt_map (); > PFarray_t *res_actmap = create_actatt_map (); > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins | Manuel Mayr, Dipl.- Inf . | | | | EMail: manuel .mayr @in.tum.de | | Web: http://www3.in.tum.de/cms/members/ mayrm | | Phone: +49 89 289 -17259 | | Fax: +49 89 289 -17263 | | | | Database Systems - Technische Universität München (Germany) |

1 0

[Monetdb-developers] New MonetDB Release
by Stefan Manegold 19 Feb '08

19 Feb '08

The MonetDB team at CWI is pleased to announce the new "Feb2008" release of the MonetDB suite of programs. This release marks the roll forward of the binary storage layout to a single column storage layout. This involved a major overhaul of the MonetDB Server code. IMPORTANT NOTE: The database format of this release is not compatible with that of any previous release. Before installing this release, make a dump of your database, and do a restore afterwards. Further changes include: SQL: The functional extensions to SQL are limited. The mostly concern minor bugs and some of the optimizers have been improved to better exploit re-occurring joins. The core cracker code has been cleaned up for better inspection and sharing in the community. XQuery: Next to the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS"), the development of the new improved and enhanced Algebra-based Pathfinder compiler ("ALG") is progressing quickly. While the original MPS version remains the default in this release of MonetDB/XQuery, a preview of the new ALG version is now available without recompilation. JDBC & XRPC: The Java code (JDBC driver & XRPC wrapper) has been moved from the clients and pathfinder source trees to a dedicated java source tree in CVS, which can simply be compiled using "ant". Apart from this, many bugs have been fixed, and many speed improvements have been made. See http://monetdb.cwi.nl/Development/Releases/Version4.22/ for more details. The new release consists of the following packages: MonetDB Common 1.22.0 MonetDB Clients 1.22.0 MonetDB4 Server 4.22.0 MonetDB XQuery 0.22.0 MonetDB5 Server 5.4.0 MonetDB SQL 2.22.0 monetdb-1.7-jdbc.jar There is of course also a new "Super Source Ball", a tarball containing the combined sources. All this is available on our SourceForge project page. See https://sourceforge.net/project/showfiles.php?group_id=56967 for the downloadable files. -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

[Monetdb-developers] PF: 334 tests fail with ALG that work fine with MPS
by Stefan Manegold 16 Feb '08

16 Feb '08

Hi, those of you involved it Pathfinder development but not following the Monetdb-bugs mailing list closely, might want to have a look at this one: [ 1894891 ] PF: 334 tests fail with ALG that work fine with MPS http://sourceforge.net/tracker/index.php?func=detail&aid=1894891&group_id=5… Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] Releasing: Please do not check-in to Stable release branches!
by Stefan Manegold 15 Feb '08

15 Feb '08

Dear all, we finally plan to produce a new release tomorrow (Wednesday) or Thursday. To ensure a smooth process, please refrain as of now from any check-ins to the Stable release branches until we're done and notify you accordingly. In case there are any (unexpected) urgend changes to be checked in, please check with Niels, Sjoerd and/or me, first! Thank you very much for your cooperation! Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/algebra builtins.c, , 1.79, 1.80
by Lefteris 14 Feb '08

14 Feb '08

Jan, is this the correct way to cope with empty input? The xquery function specs say that if input is ommited then it defaults to the context item (.) That also is expresed in xquery_fo.c with the .arity = 0 version of fn:name etc. If that is true, shouldn't be dealed at the core level? lefteris On Wed, Feb 13, 2008 at 9:54 PM, Jan Rittinger <tsheyar(a)users.sourceforge.net> wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/algebra > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv18492/algebra > > Modified Files: > builtins.c > Log Message: > -- Fix the translation of fn:name, fn:local-name, and fn:namespace-uri. > (All three have to cope with empty input.) > > > Index: builtins.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/algebra/builtins.c,v > retrieving revision 1.79 > retrieving revision 1.80 > diff -u -d -r1.79 -r1.80 > --- builtins.c 6 Feb 2008 16:36:28 -0000 1.79 > +++ builtins.c 13 Feb 2008 20:54:17 -0000 1.80 > @@ -2350,6 +2350,10 @@ > * is not even the right function to use there, because the constructors > * should do their job based on _statically-known namespace declarations_, > * not in-scope declarations. > + * > + * Furthermore a cast from string to xs:QName is only allowed at compile > + * time on constant strings. At runtime a cast to xs:QName is only allowed > + * for QNames inputs. > */ > struct PFla_pair_t > PFbui_fn_resolve_qname (const PFla_op_t *loop, bool ordering, > @@ -2416,22 +2420,34 @@ > struct PFla_pair_t > PFbui_fn_name (const PFla_op_t *loop, bool ordering, struct PFla_pair_t *args) > { > - (void) loop; (void) ordering; > + (void) ordering; > > - return (struct PFla_pair_t) { > - .rel = project ( > - fun_1to1 ( > - project (args[0].rel, > - proj (att_iter, att_iter), > - proj (att_pos, att_pos), > - proj (att_item, att_item)), > - alg_fun_fn_name, > - att_res, > - attlist(att_item)), > - proj (att_iter, att_iter), > - proj (att_pos, att_pos), > - proj (att_item, att_res)), > - .frag = PFla_empty_set() }; > + PFla_op_t *strings = project ( > + fun_1to1 ( > + project (args[0].rel, > + proj (att_iter, att_iter), > + proj (att_pos, att_pos), > + proj (att_item, att_item)), > + alg_fun_fn_name, > + att_res, > + attlist(att_item)), > + proj (att_iter, att_iter), > + proj (att_pos, att_pos), > + proj (att_item, att_res)); > + > + PFla_op_t *res = disjunion ( > + strings, > + attach ( > + attach ( > + difference ( > + loop, > + project ( > + strings, > + proj (att_iter, att_iter))), > + att_pos, lit_nat (1)), > + att_item, lit_str (""))); > + > + return (struct PFla_pair_t) { .rel = res, .frag = PFla_empty_set() }; > } > > /* ------------------- */ > @@ -2442,22 +2458,34 @@ > PFbui_fn_local_name (const PFla_op_t *loop, bool ordering, > struct PFla_pair_t *args) > { > - (void) loop; (void) ordering; > + (void) ordering; > > - return (struct PFla_pair_t) { > - .rel = project ( > - fun_1to1 ( > - project (args[0].rel, > - proj (att_iter, att_iter), > - proj (att_pos, att_pos), > - proj (att_item, att_item)), > - alg_fun_fn_local_name, > - att_res, > - attlist(att_item)), > - proj (att_iter, att_iter), > - proj (att_pos, att_pos), > - proj (att_item, att_res)), > - .frag = args[0].frag }; > + PFla_op_t *strings = project ( > + fun_1to1 ( > + project (args[0].rel, > + proj (att_iter, att_iter), > + proj (att_pos, att_pos), > + proj (att_item, att_item)), > + alg_fun_fn_local_name, > + att_res, > + attlist(att_item)), > + proj (att_iter, att_iter), > + proj (att_pos, att_pos), > + proj (att_item, att_res)); > + > + PFla_op_t *res = disjunion ( > + strings, > + attach ( > + attach ( > + difference ( > + loop, > + project ( > + strings, > + proj (att_iter, att_iter))), > + att_pos, lit_nat (1)), > + att_item, lit_str (""))); > + > + return (struct PFla_pair_t) { .rel = res, .frag = PFla_empty_set() }; > } > > /* --------------------- */ > @@ -2468,22 +2496,34 @@ > PFbui_fn_namespace_uri (const PFla_op_t *loop, bool ordering, > struct PFla_pair_t *args) > { > - (void) loop; (void) ordering; > + (void) ordering; > > - return (struct PFla_pair_t) { > - .rel = project ( > - fun_1to1 ( > - project (args[0].rel, > - proj (att_iter, att_iter), > - proj (att_pos, att_pos), > - proj (att_item, att_item)), > - alg_fun_fn_namespace_uri, > - att_res, > - attlist(att_item)), > - proj (att_iter, att_iter), > - proj (att_pos, att_pos), > - proj (att_item, att_res)), > - .frag = args[0].frag }; > + PFla_op_t *strings = project ( > + fun_1to1 ( > + project (args[0].rel, > + proj (att_iter, att_iter), > + proj (att_pos, att_pos), > + proj (att_item, att_item)), > + alg_fun_fn_namespace_uri, > + att_res, > + attlist(att_item)), > + proj (att_iter, att_iter), > + proj (att_pos, att_pos), > + proj (att_item, att_res)); > + > + PFla_op_t *res = disjunion ( > + strings, > + attach ( > + attach ( > + difference ( > + loop, > + project ( > + strings, > + proj (att_iter, att_iter))), > + att_pos, lit_nat (1)), > + att_item, lit_str (""))); > + > + return (struct PFla_pair_t) { .rel = res, .frag = PFla_empty_set() }; > } > > /* --------------- */ > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins >

3 3

[Monetdb-developers] python error during bootstrap
by Ying Zhang 14 Feb '08

14 Feb '08

Hi all, after cvs update all modules, rebuild monet from scratch (i.e. boostrap, confgure, make, make install). all other modules (i.e. buildtools, MonetDB, clients, java, MonetDB4, MonetDB5, sql) can be build without problem, however bootstrap pathfinder results in some python errors: [andorea:/ufs/zhang/monet/dev/pathfinder] $ ./bootstrap automake 1.10 is 1.7 or newer. Good. autoconf 2.61 is 2.57 or newer. Good. libtool 1.5.24 is 1.4 or newer. Good. Python 2.5.1 is 2.0.0 or newer. Good. /ufs/zhang/monet/dev/pathfinder/conf /ufs/zhang/monet/dev/pathfinder/compiler /ufs/zhang/monet/dev/pathfinder/compiler/utils /ufs/zhang/monet/dev/pathfinder/compiler/schema /ufs/zhang/monet/dev/pathfinder/compiler/debug /ufs/zhang/monet/dev/pathfinder/compiler/mem /ufs/zhang/monet/dev/pathfinder/compiler/parser /ufs/zhang/monet/dev/pathfinder/compiler/semantics /ufs/zhang/monet/dev/pathfinder/compiler/core /ufs/zhang/monet/dev/pathfinder/compiler/algebra /ufs/zhang/monet/dev/pathfinder/compiler/algebra/opt /ufs/zhang/monet/dev/pathfinder/compiler/algebra/prop /ufs/zhang/monet/dev/pathfinder/compiler/algebra/map /ufs/zhang/monet/dev/pathfinder/compiler/mil /ufs/zhang/monet/dev/pathfinder/compiler/include /ufs/zhang/monet/dev/pathfinder/compiler/sql /ufs/zhang/monet/dev/pathfinder/compiler/xmlimport /ufs/zhang/monet/dev/pathfinder/runtime ('error', '}', '=') Traceback (most recent call last): File "/export/scratch1/zhang/monet-install/dev/debug/bin/ autogen.py", line 163, in <module> (InstallList, DocList, OutList) = main(topdir, topdir, automake, []) File "/export/scratch1/zhang/monet-install/dev/debug/bin/ autogen.py", line 146, in main main(d, topdir, automake, incdirsmap, conditional + cond) File "/export/scratch1/zhang/monet-install/dev/debug/bin/ autogen.py", line 133, in main (InstallList, DocList, OutList) = am.output(p.curvar, cwd, topdir, automake, conditional) File "/export/scratch1/zhang/monet-install/dev/debug/lib/python2.5/ site-packages/autogen/am.py", line 1097, in output output_funcs[i](fd, i, v, am) File "/export/scratch1/zhang/monet-install/dev/debug/lib/python2.5/ site-packages/autogen/am.py", line 809, in am_libs if (libsmap.has_key("DIR")): AttributeError: 'list' object has no attribute 'has_key' Does someone has a idea what's happening here? I'm using the current branch. OS info: $ uname -a Linux andorea.ins.cwi.nl 2.6.23.15-137.fc8 #1 SMP Sun Feb 10 17:03:13 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Thanks, Jennie

1 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/tests/XRpc/Tests add_trusted.mil, XQuery_0-22, 1.2, 1.2.12.1
by Stefan Manegold 13 Feb '08

13 Feb '08

On Wed, Feb 13, 2008 at 02:50:03PM +0000, Ying Zhang wrote: > Update of /cvsroot/monetdb/pathfinder/tests/XRpc/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv23952/tests/XRpc/Tests > > Modified Files: > Tag: XQuery_0-22 > add_trusted.mil > Log Message: > [ monetdb-Bugs-1892582 ] PF: XRpc does not work on Windows > > Don't use hard coded directory name, '/' as directory separator will not > work on Windows. Thanks --- although I have NOT released the CVS check-in lock for the branches, yet ... Stefan > Index: add_trusted.mil > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/tests/XRpc/Tests/add_trusted.mil,v > retrieving revision 1.2 > retrieving revision 1.2.12.1 > diff -u -d -r1.2 -r1.2.12.1 > --- add_trusted.mil 22 Feb 2007 11:34:56 -0000 1.2 > +++ add_trusted.mil 13 Feb 2008 14:50:00 -0000 1.2.12.1 > @@ -1,7 +1,7 @@ > module("pathfinder"); > module("unix"); > > -var TSTSRCDIR := getenv("TSTSRCBASE") + "/tests/XRpc/Tests"; > +var TSTSRCDIR := getenv("TSTSRCDIR"); > var TSTTRGDIR := getenv("TSTTRGDIR"); > > add_xrpc_trusted(TSTSRCDIR); > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] monetdb/SQL development plan
by Marcin Zukowski 12 Feb '08

12 Feb '08

Hi Michael, Thank you for your interest in MonetDB. Since I'm not involved with the core MonetDB development, I forward this email to the monetdb-developers list. I'm sure the guys over there will be able to help you. All the best, Marcin Michael Yang wrote: > Hi Marcin, > > I saw that you're one of Peter's PhD students. I just got introduced > to MonetDB as a possible column-store solution for some time series I > work with. I'm in financial mathematics here at Princeton in the > Operations Research department. > > I am a heavy user of MS SQL Server and I use aggregate ranking > functions a lot - row_number(), rank(), etc. Could you give me a > rough estimate as to when MonetDB might add support for these > functions - LEAD(), LAG(), and X() over (partition by ... order by > ...). I see that it's in the development plan, but will it take 1 > year? 1 month? many years? > > Also, in the meantime is there a way you know to emulate these > aggregate functions using local variables in monetdb? > > thanks, > Michael Yang > > Princeton University Operations Research and Financial Engineering

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB4/src/modules/plain bat_arith.mx, MonetDB_4-22, 1.4.2.2, 1.4.2.3
by Stefan Manegold 11 Feb '08

11 Feb '08

my below fixes in MonetDB4/src/modules/plain/bat_arith.mx would need to be propagated / applied to MonetDB5/src/modules/kernel/batcalc.mx too, but a simple diff|patch does not work, and I haven't found time to do it by hand, yet, sorry ... Stefan On Sun, Feb 10, 2008 at 10:46:53PM +0000, Stefan Manegold wrote: > Update of /cvsroot/monetdb/MonetDB4/src/modules/plain > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv13856 > > Modified Files: > Tag: MonetDB_4-22 > bat_arith.mx > Log Message: > > fixed bug > [ 1890131 ] PF: property error: incorrectly marked sorted! > http://sourceforge.net/tracker/index.php?func=detail&aid=1890131&group_id=5… > > fixed sortedness propagation for "const [op] BAT" & "BAT [op] const", > in particular three cases: > - substracting a BAT from a constant > yields a result with the input's sort order reversed > - dividing a BAT by a negative constant > yields a result with the input's sort order reversed > - for modulo (%) (also on) float & double > we cannot predict any result order > > > Index: bat_arith.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB4/src/modules/plain/bat_arith.mx,v > retrieving revision 1.4.2.2 > retrieving revision 1.4.2.3 > diff -u -d -r1.4.2.2 -r1.4.2.3 > --- bat_arith.mx 8 Feb 2008 15:02:14 -0000 1.4.2.2 > +++ bat_arith.mx 10 Feb 2008 22:46:51 -0000 1.4.2.3 > @@ -57,42 +57,42 @@ > .END bat_arith; > > @= operations > -@:@1_operation(bte,+,add,,0,2)@ > -@:@1_operation(bte,-,sub,,0,2)@ > -@:@1_operation(bte,*,mul,1,0,3)@ > -@:@1_operation(bte,/,div,1,1,1)@ > +@:@1_operation(bte,+,add,,0,(1|2))@ > +@:@1_operation(bte,-,sub,,0,(1|8))@ > +@:@1_operation(bte,*,mul,1,0,(16|32))@ > +@:@1_operation(bte,/,div,1,1,16)@ > @:@1_operation(bte,%,mod,1,1,0)@ > -@:@1_operation(sht,+,add,,0,2)@ > -@:@1_operation(sht,-,sub,,0,2)@ > -@:@1_operation(sht,*,mul,1,0,3)@ > -@:@1_operation(sht,/,div,1,1,1)@ > +@:@1_operation(sht,+,add,,0,(1|2))@ > +@:@1_operation(sht,-,sub,,0,(1|8))@ > +@:@1_operation(sht,*,mul,1,0,(16|32))@ > +@:@1_operation(sht,/,div,1,1,16)@ > @:@1_operation(sht,%,mod,1,1,0)@ > -@:@1_operation(int,+,add,,0,2)@ > -@:@1_operation(int,-,sub,,0,2)@ > -@:@1_operation(int,*,mul,1,0,3)@ > -@:@1_operation(int,/,div,1,1,1)@ > +@:@1_operation(int,+,add,,0,(1|2))@ > +@:@1_operation(int,-,sub,,0,(1|8))@ > +@:@1_operation(int,*,mul,1,0,(16|32))@ > +@:@1_operation(int,/,div,1,1,16)@ > @:@1_operation(int,%,mod,1,1,0)@ > -@:@1_operation(wrd,+,add,,0,2)@ > -@:@1_operation(wrd,-,sub,,0,2)@ > -@:@1_operation(wrd,*,mul,1,0,3)@ > -@:@1_operation(wrd,/,div,1,1,1)@ > +@:@1_operation(wrd,+,add,,0,(1|2))@ > +@:@1_operation(wrd,-,sub,,0,(1|8))@ > +@:@1_operation(wrd,*,mul,1,0,(16|32))@ > +@:@1_operation(wrd,/,div,1,1,16)@ > @:@1_operation(wrd,%,mod,1,1,0)@ > -@:@1_operation(lng,+,add,,0,2)@ > -@:@1_operation(lng,-,sub,,0,2)@ > -@:@1_operation(lng,*,mul,1,0,3)@ > -@:@1_operation(lng,/,div,1,1,1)@ > +@:@1_operation(lng,+,add,,0,(1|2))@ > +@:@1_operation(lng,-,sub,,0,(1|8))@ > +@:@1_operation(lng,*,mul,1,0,(16|32))@ > +@:@1_operation(lng,/,div,1,1,16)@ > @:@1_operation(lng,%,mod,1,1,0)@ > -@:@1_operation(flt,+,add,,0,2)@ > -@:@1_operation(flt,-,sub,,0,2)@ > -@:@1_operation(flt,*,mul,1,0,3)@ > -@:@1_operation(flt,/,div,1,1,1)@ > -@:@1_operation(dbl,+,add,,0,2)@ > -@:@1_operation(dbl,-,sub,,0,2)@ > -@:@1_operation(dbl,*,mul,1,0,3)@ > -@:@1_operation(dbl,/,div,1,1,1)@ > +@:@1_operation(flt,+,add,,0,(1|2))@ > +@:@1_operation(flt,-,sub,,0,(1|8))@ > +@:@1_operation(flt,*,mul,1,0,(16|32))@ > +@:@1_operation(flt,/,div,1,1,16)@ > +@:@1_operation(dbl,+,add,,0,(1|2))@ > +@:@1_operation(dbl,-,sub,,0,(1|8))@ > +@:@1_operation(dbl,*,mul,1,0,(16|32))@ > +@:@1_operation(dbl,/,div,1,1,16)@ > @:@1_redefine_OP()@ > -@:@1_operation(flt,%,mod,1,1,1,%)@ > -@:@1_operation(dbl,%,mod,1,1,1,%)@ > +@:@1_operation(flt,%,mod,1,1,0,%)@ > +@:@1_operation(dbl,%,mod,1,1,0,%)@ > > @* Implementation > @c > @@ -108,9 +108,13 @@ > @3: add, sub, mul, div, mod basic arithmetic operator name > @4: not used > @5: 0 / 1 disable/enable division by zero check > - @6: 0 / 1 / 2 disable/enable sorting propagation > - 1 propagate on bat,const > - 2 propagate also on const,bat > + @6: (bit pattern) disable/enable sorting propagation > + 1 propagate as-is on bat,const > + 2 propagate as-is on const,bat > + 4 propagate reverted on bat,const (not required/used!) > + 8 propagate reverted on const,bat > + 16 propagate as-is / reverted on bat,const if const >=0 / <0 > + 32 propagate as-is / reverted on const,bat if const >=0 / <0 > */ > @= c_operation > int > @@ -201,7 +205,6 @@ > if (bn == NULL) > return GDK_FAIL; > BATseqbase(bn,b->hseqbase); > - bn->tsorted = 0; > bq = (@1*)Tloc(b,BUNlast(b)); > bp = (@1*)Tloc(b,BUNfirst(b)); > s = bnp = (@1*)Tloc(bn,BUNlast(bn)); > @@ -225,13 +228,19 @@ > BATsetcount(bn, bnp-s); > if (!bn->batDirty) bn->batDirty = TRUE; > > - if (@6 == 2) > + if (@6 & 2) { > bn->tsorted = BATtordered(b); > - if (@6 == 3) { > - if (*v > 0) > + } else > + if (@6 & 8) { > + bn->tsorted = REVERT_SORTED(BATtordered(b)); > + } else > + if (@6 & 32) { > + if (*v >= 0) > bn->tsorted = BATtordered(b); > else > bn->tsorted = REVERT_SORTED(BATtordered(b)); > + } else { > + bn->tsorted = 0; > } > > *ret = bn; > @@ -264,15 +273,20 @@ > } > if (!b->batDirty) b->batDirty = TRUE; > > - if (@6 == 2) > + if (@6 & 2) { > b->tsorted = BATtordered(b); > - if (@6 == 3) { > - if (*v > 0) > + } else > + if (@6 & 8) { > + b->tsorted = REVERT_SORTED(BATtordered(b)); > + } else > + if (@6 & 32) { > + if (*v >= 0) > b->tsorted = BATtordered(b); > else > b->tsorted = REVERT_SORTED(BATtordered(b)); > - } else > + } else { > b->tsorted = 0; > + } > > BBPfix(b->batCacheid); > *ret = b; > @@ -294,7 +308,6 @@ > if( bn == NULL) > return GDK_FAIL; > BATseqbase(bn,b->hseqbase); > - bn->tsorted = 0; > bq = (@1*)Tloc(b,BUNlast(b)); > bp = (@1*)Tloc(b,BUNfirst(b)); > s = bnp = (@1*)Tloc(bn,BUNlast(bn)); > @@ -317,14 +330,21 @@ > } > BATsetcount(bn, bnp-s); > if (!bn->batDirty) bn->batDirty = TRUE; > - if (@6 == 2) > + > + if (@6 & 1) { > bn->tsorted = BATtordered(b); > - if (@6 == 3) { > - if (*v > 0) > + } else > + if (@6 & 4) { > + bn->tsorted = REVERT_SORTED(BATtordered(b)); > + } else > + if (@6 & 16) { > + if (*v >= 0) > bn->tsorted = BATtordered(b); > else > bn->tsorted = REVERT_SORTED(BATtordered(b)); > - } > + } else { > + bn->tsorted = 0; > + } > > *ret = bn; > return GDK_SUCCEED; > @@ -356,15 +376,21 @@ > } > } > if (!b->batDirty) b->batDirty = TRUE; > - if (@6 == 2) > + > + if (@6 & 1) { > b->tsorted = BATtordered(b); > - if (@6 == 3) { > - if (*v > 0) > + } else > + if (@6 & 4) { > + b->tsorted = REVERT_SORTED(BATtordered(b)); > + } else > + if (@6 & 16) { > + if (*v >= 0) > b->tsorted = BATtordered(b); > else > b->tsorted = REVERT_SORTED(BATtordered(b)); > - } else > + } else { > b->tsorted = 0; > + } > > BBPfix(b->batCacheid); > *ret = b; -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins]MonetDB/src/gdkgdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
by Peter Boncz 11 Feb '08

11 Feb '08

Hi Stefan Thanks for your response. However, as I have said, but let me say it as clear as possble: I am *sure* there is a bug still unfound in the way re-allocations are done when mem_bigsize redirects it into virtual memory. For this reason, at this stage, with the goal of a stable release, it is just my recommendation *not* to re-enable it. In M5 nor in M5. Peter Dear all, On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote: > Hi Stefan, > > The pftijah bug was unrelated to vmalloc(), a relationship that I never > assumed. Indeed, that was (only) my mis-interpretation. Sorry. > The vmalloc() troubles occured, as you may recall, during the data > preparation of the sorted 100GB TPC-H. > > As it is entirely foreseeable that we at some point again will re-encounter an > OS with a malloc implementation that is prone to fragmentation, I'd keep the > vmalloc() code in for the moment (though in disabled state). Linux and Windows > do not need it. > > So, my advise is to for the monet let vmalloc() be disabled, until the issue > is properly investigated and a bugfix is found. Let's call it option (0). Ok. Just to sum up the current state. Please check, whether this is OK and consistent: a) Documentation in MonetDB4/conf/MonetDB.conf.in & MonetDB5/conf/monetdb5.conf.in says: " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 " b) Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says " @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. @item @code{gdk_vm_minsize}: column size above which memory mapped files are used always, e.g. @code{1749291171} " c) MonetDB/src/common/monet_options.mx sets " set[i].kind = opt_builtin; set[i].name = strdup("gdk_mem_bigsize"); set[i].value = strdup("262144"); " d) MonetDB/src/common/monet_options.py.in sets " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx gdk_mem_bigsize = '256K' gdk_vm_minsize = '128G' " e) GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets " if ((p = GDKgetenv("gdk_mem_bigsize"))) { /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ lng max_mem_bigsize = GDK_mem_maxsize/16; /* sanity check to avoid memory fragmentation */ GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10)); } " f) GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize g) HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* when using anonymous vm we malloc we need 64K chunks, also we * 20% extra malloc */ if (h->size > GDK_mem_bigsize) { h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; h->maxsize = (1 + (h->maxsize >> 16)) << 16; } " h) HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* extend a malloced heap, possibly switching over to file-mapped storage */ Heap bak = *h; int can_mmap = (h->filename && size >= GDK_mem_bigsize); int must_mmap = can_mmap && (size >= GDK_vm_minsize || (h->newstorage != STORE_MEM)); [...] if (can_mmap) { /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ if (size > h->maxsize) { h->maxsize = (size_t) ((double) size * BATMARGIN); } /* when using anonymous vm we malloc we need 64K chunks */ h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; [...] /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { char privext[PATHLENGTH], *of = h->filename; FILE *fp; h->filename = NULL; sprintf(privext, "%s.priv", ext); fp = GDKfilelocate(nme, "wb", privext); if (fp != NULL) { fclose(fp); /* a non-persistent heap: we create a .priv but *not* MMAP_PRIV !!! */ h->storage = STORE_MMAP; h->base = NULL; if (HEAPload(h, nme, privext, FALSE) >= 0) { memcpy(h->base, bak.base, bak.free); HEAPfree(&bak); return 0; } } GDKfree(of); } " i) MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize in MonetDB4/src/modules/plain/sys.mx j) both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however, only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize, while the MAL variants no longer do so: MonetDB4/src/modules/plain/sys.mx: " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } " MonetDB5/src/modules/kernel/status.mx " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); #endif GDK_mem_maxsize = sze; return GDK_SUCCEED; } int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); #endif GDK_vm_minsize = sze; return GDK_SUCCEED; } " If all this is considered OK and consistent, we're done. Otherwise, we need to do what ever is required to get the code and documentation consistent, preferably before the release. Stefan > Peter > > > > > > > > > > > > > > > > > Dear all, > > since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize > related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) > does not appear have any impact on this bug (just tested successfully both > with vmalloc() disabled and vmalloc() enabled), there are two ways to finish > this release (more or less) quickly: > > (1) finish the disabling of vmalloc() in all parts of the code as indicated > below (or better: remove all remains completely) --- this might need > some extra testing to ensure that it does not have any (significant) > impact on performance ... > > (2) for now (i.e., in the release branch) re-enable vmalloc(), and then > finish the removal of it (incl. extensive testing) in the development > trunk for th next release. > > Since I'm not completely aware of the impact of vmalloc(), I cannot really > make this decision --- the "save" back-up would IMHO be (2). > > Any other comments, opinions, suggestions, expertise? > > Thanks in advance! > > Stefan > > > On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote: > > Peter: Thanks for disabling mem_bigsize / vmalloc(). > > > > All: > > (a) > > Unfortunately, this does not (yet) solve > > [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. > > http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=5… > > > > But this appears to be more vm_minsize related than mem_bigsize related. > > > > > > (b) > > We should check all occurances of mem_bigsize and remove / "hide" those that > > have become irrelevant with the deactivation of mem_bigsize / vmalloc(): > > ======== > > $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize' > > -------- > > MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be > set/limited to > > MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize > (in bytes) will be mmaped anonymously > > MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 > > MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be > set/limited to > > MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize > (in bytes) will be mmaped anonymously > > MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 > > MonetDB/src/common/monet_options.mx:410: set[i].name = > strdup("gdk_mem_bigsize"); > > MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize & > gdk_vm_minsize will be set/limited to > > MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' > > MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename > && size >= GDK_mem_bigsize); > > MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size > after which we use VM rather than heap */ > > MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; > > MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous > virtual memory. The seamless > > MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_utils.mx:1529: if ((p = > GDKgetenv("gdk_mem_bigsize"))) { > > MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize = > GDK_mem_maxsize/16; > > MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t) > MIN(max_mem_bigsize, strtol(p, NULL, 10)); > > monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}: > minimum size for memory-mapped columns, e.g. @code{262144}. > > MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng = > get_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void = > set_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) > > MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) > > MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ > > MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); > > MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) > > MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); > > MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) > > MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); > > MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) > > MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; > > MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) > > MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ > > MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768, > sze); > > MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) > > MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); > > MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) > > MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); > > MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize = > MAX(GDK_mem_bigsize, sze); > > monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize", > "262144" > > ======== > > > > more tomorrow during MADAM ... > > > > Stefan > > > > > > On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote: > > > no consequences. > > > > > > > > > > > > > > > Peter, > > > > > > What about the consequences in: > > > build/MonetDB/src/gdk/gdk_heap.c > > > build/MonetDB5/src/modules/kernel/status.c > > > build/MonetDB/src/gdk/gdk.h > > > > > > Peter Boncz wrote: > > > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > > > > > > > Modified Files: > > > > Tag: MonetDB_1-22 > > > > gdk_utils.mx > > > > Log Message: > > > > disable use of vmalloc() > > > > > > > > > > > > > > > > Index: gdk_utils.mx > > > > =================================================================== > > > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > > > > retrieving revision 1.206 > > > > retrieving revision 1.206.2.1 > > > > diff -u -d -r1.206 -r1.206.2.1 > > > > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > > > > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > > > > @@ -1050,6 +1050,7 @@ > > > > #endif > > > > } > > > > size = (size + 7) & ~7; /* round up to a multiple of eight */ > > > > +#if 0 > > > > if (size > GDK_mem_bigsize) { > > > > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > > > > size_t newmax = MAX(*maxsize, newsize); > > > > @@ -1064,6 +1065,7 @@ > > > > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > > > > return (void *) s; > > > > } > > > > +#endif > > > > CHKMEM(size, 0); > > > > GDKmalloc_prefixsize(s, size); > > > > if (s == NULL) { > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > This SF.net email is sponsored by: Microsoft > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > _______________________________________________ > > > > Monetdb-checkins mailing list > > > > Monetdb-checkins(a)lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Microsoft > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > _______________________________________________ > > > Monetdb-developers mailing list > > > Monetdb-developers(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > > > > > > > > > > > > > > > > > -- > > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 2