On Thu, Mar 05, 2009 at 10:26:17AM +0100, Martin Kersten wrote:

Sjoerd Mullender wrote:

...

How is this not a new feature? Oops, yes, unless Stefan wants this in for McK

As I said, this was/is just an idea, has low(est) priority (for me), effects/benefits are (at least to me) yet unknown, is not crucial for McK --- hence, yes, it is a new feature that should be only in the development trunk --- I can later take care of revoming it from the release branch ... Stefan

...

Martin Kersten wrote:

...

Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv2998/optimizer

Modified Files: Tag: Feb2009 Makefile.ag opt_prelude.mx Added Files: Tag: Feb2009 opt_derivepath.mx Log Message: Added the framework for the commissioned derivepath optimizer.

U opt_prelude.mx Index: opt_prelude.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_prelude.mx,v retrieving revision 1.55.2.3 retrieving revision 1.55.2.4 diff -u -d -r1.55.2.3 -r1.55.2.4 --- opt_prelude.mx 1 Mar 2009 19:15:30 -0000 1.55.2.3 +++ opt_prelude.mx 5 Mar 2009 00:17:12 -0000 1.55.2.4 @@ -57,6 +57,7 @@ opt_export str deleteRef; opt_export str depositRef; opt_export str deriveRef; +opt_export str derivePathRef; opt_export str differenceRef; opt_export str divRef; opt_export str disconnectRef; @@ -216,6 +217,7 @@ str deleteRef; str depositRef; str deriveRef; +str derivePathRef; str differenceRef; str divRef; str disconnectRef; @@ -371,6 +373,7 @@ deleteRef = putName("delete",6); depositRef = putName("deposit",7); deriveRef = putName("derive",6); + derivePathRef = putName("derivePath",10); differenceRef= putName("difference",10); divRef = putName("/",1); disconnectRef= putName("disconnect",10);

U Makefile.ag Index: Makefile.ag =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/Makefile.ag,v retrieving revision 1.66 retrieving revision 1.66.2.1 diff -u -d -r1.66 -r1.66.2.1 --- Makefile.ag 18 Jan 2009 11:44:14 -0000 1.66 +++ Makefile.ag 5 Mar 2009 00:17:12 -0000 1.66.2.1 @@ -32,7 +32,7 @@ opt_singleton.mx opt_costModel.mx opt_reduce.mx opt_macro.mx \ opt_accumulators.mx opt_qep.mx opt_mergetable.mx \ opt_remoteQueries.mx opt_joinselect.mx opt_partitions.mx \ - opt_evaluate.mx opt_inline.mx opt_pushranges.mx \ + opt_evaluate.mx opt_inline.mx opt_pushranges.mx opt_derivepath.mx \ opt_accessmode.mx opt_joinpath.mx opt_heuristics.mx opt_remap.mx \ opt_statistics.mx opt_trace.mx opt_recycler.mx opt_dataflow.mx \ opt_mitosis.mx opt_octopus.mx opt_replicator.mx opt_history.mx @@ -53,7 +53,7 @@ opt_singleton.mx opt_costModel.mx opt_reduce.mx opt_macro.mx \ opt_accumulators.mx opt_qep.mx opt_mergetable.mx \ opt_remoteQueries.mx opt_joinselect.mx opt_partitions.mx \ - opt_evaluate.mx opt_inline.mx opt_pushranges.mx \ + opt_evaluate.mx opt_inline.mx opt_pushranges.mx opt_derivepath.mx \ opt_accessmode.mx opt_joinpath.mx opt_heuristics.mx opt_remap.mx \ opt_statistics.mx opt_trace.mx opt_recycler.mx opt_dataflow.mx \ opt_mitosis.mx opt_octopus.mx opt_replicator.mx opt_history.mx @@ -72,7 +72,7 @@ opt_singleton.mx opt_costModel.mx opt_reduce.mx opt_macro.mx \ opt_accumulators.mx opt_qep.mx opt_mergetable.mx \ opt_remoteQueries.mx opt_joinselect.mx opt_partitions.mx \ - opt_evaluate.mx opt_inline.mx opt_pushranges.mx \ + opt_evaluate.mx opt_inline.mx opt_pushranges.mx opt_derivepath.mx \ opt_accessmode.mx opt_joinpath.mx opt_heuristics.mx opt_remap.mx \ opt_statistics.mx opt_trace.mx opt_recycler.mx opt_dataflow.mx \ opt_mitosis.mx opt_octopus.mx opt_replicator.mx opt_history.mx

--- NEW FILE: opt_derivepath.mx --- @' The contents of this file are subject to the MonetDB Public License @' Version 1.1 (the "License"); you may not use this file except in @' compliance with the License. You may obtain a copy of the License at @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html @' @' Software distributed under the License is distributed on an "AS IS" @' basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the @' License for the specific language governing rights and limitations @' under the License. @' @' The Original Code is the MonetDB Database System. @' @' The Initial Developer of the Original Code is CWI. @' Portions created by CWI are Copyright (C) 1997-July 2008 CWI. @' Copyright August 2008-2009 MonetDB B.V. @' All Rights Reserved.

@f opt_derivepath @- Group derive paths The routine @sc{optimizer.derivepaths()} walks through the program looking for grouping operations and cascades them into a single multiple group paths. To illustrate, consider @verbatim (ext1,grp1) := group.new(b); (ext2,grp2) := group.derive(ext1,grp1, c); (ext3,grp3) := group.derive(ext2,grp2, d); @end verbatim The result becomes. @verbatim (ext3,grp3) := group.derivepath(b,c,d); @end verbatim @{ @mal pattern optimizer.derivePath():str address OPTderivePath; pattern optimizer.derivePath(mod:str, fcn:str):str address OPTderivePath comment "Join path constructor"; pattern group.derivePath(l:bat[:any,:any]...):bat[:any,:any] address ALGderivePath comment "internal routine to handle join paths. The type analysis is rather tricky."; @h #ifndef _OPT_DERIVEPATH_ #define _OPT_DERIVEPATH_ #include "opt_prelude.h" #include "opt_support.h" #include "mal_interpreter.h"

/* #define DEBUG_OPT_DERIVEPATH */ @c #include "mal_config.h" #include "opt_derivepath.h"

static int OPTderivePathImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p) { int i, actions=0; int *pc; str derivePathRef = putName("derivePath",10); InstrPtr q; InstrPtr *old; int limit,slimit;

(void) cntxt; (void) stk; if (varGetProp(mb, getArg(mb->stmt[0], 0), inlineProp) != NULL) return 0;

old= mb->stmt; limit= mb->stop; slimit= mb->ssize; newMalBlkStmt(mb,mb->ssize);

/* beware, new variables and instructions are introduced */ pc= (int*) GDKmalloc(sizeof(int)* mb->vtop * 2); /* to find last assignment */ memset((char*) pc, 0, sizeof(int)* mb->vtop *2);

for (i = 0; ifdout,"new derivePath instruction\n"); printInstruction(cntxt->fdout,mb, 0, q, LIST_MAL_ALL); #endif } } pushInstruction(mb,p); } for(; ifdout,"#opt_derivepath: %d statements glued\n",actions); return actions; } @include optimizerWrapper.mx @h @:exportOptimizer(derivePath)@ #endif opt_export str ALGderivePath(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); @c #include "opt_statistics.h" @:wrapOptimizer(derivePath,OPT_CHECK_ALL)@ @- The derive path optimizer takes a derivation sequence and attempts to minimize the intermediate result. The choice depends on a good estimate of intermediate results using properties. For the time being, we use a simplistic model, based on the assumption that most joins are foreign key joins anyway. @c str ALGderivePath(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) { (void) cntxt; (void) mb; (void) stk; (void) pci; throw(MAL,"group.derivePath","nyi"); } @}

------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

------------------------------------------------------------------------

------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H

------------------------------------------------------------------------

_______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |