Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

newer
Re: [Monetdb-developers]...

older
Re: [Monetdb-developers]...

Sjoerd Mullender

7 Feb 2008 7 Feb '08

6 p.m.

On 2008-02-07 18:45, Romulo Goncalves wrote:

...

Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested...

A better way to comment larger sections of code is to use #if 0/#endif. That nests.

...

Index: opt_joinpath.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- opt_joinpath.mx 7 Feb 2008 16:57:18 -0000 1.21 +++ opt_joinpath.mx 7 Feb 2008 17:44:59 -0000 1.22 @@ -231,22 +231,30 @@ The join path optimizer takes a join sequence and attempts to minimize the intermediate result. The choice depends on a good estimate of intermediate -results.SAMPLE_THRESHOLD_lOG +results. For the time being, we use a simplistic model, based on the assumption that most joins are foreign key joins anyway. + We use a sample based approach for sizeable tables. -The model is derived from the select statement. +The model is derived from the select statement. However, we did not succeed. +The code is now commented for future improvement. +Final conclusion from this exercise is: +The difference between the join input size and the join output size is not +the correct (or unique) metric which should be used to decide which order +should be used in the joinPath. + @c -#define SAMPLE_THRESHOLD_lOG 17 +/*#define SAMPLE_THRESHOLD_lOG 17*/ size_t ALGjoinCost(BAT *l, BAT *r) { - size_t lc, rc, lsize,rsize; - BAT *lsample, *rsample, *j; + size_t lc, rc; +/* size_t lsize,rsize; + BAT *lsample, *rsample, *j; */

lc = BATcount(l); rc = BATcount(r); - if( lc > 100000 || rc > 100000){ +/* if( lc > 100000 || rc > 100000){ lsize= MIN(lc/100, (1<ttype== TYPE_oid || r->htype== TYPE_oid ) return MIN(lc,rc); return lc * rc;

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- Sjoerd Mullender

Show replies by date

Romulo Goncalves

7 Feb 7 Feb

7 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

Sjoerd Mullender wrote:

...

On 2008-02-07 18:45, Romulo Goncalves wrote:

...
Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested...

A better way to comment larger sections of code is to use #if 0/#endif. #ifdef 0

#endif does not work: mizer_la-opt_joinpath.o /ufs/goncalve/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx:257:8: error: macro names must be identifiers make[5]: *** [lib_optimizer_la-opt_joinpath.lo] Error 1 make[5]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[4]: *** [all] Error 2 make[4]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build' make: *** [all] Error 2 [goncalve@amelia build]$ vim ~/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx Romulo

...

That nests.

...
Index: opt_joinpath.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- opt_joinpath.mx 7 Feb 2008 16:57:18 -0000 1.21 +++ opt_joinpath.mx 7 Feb 2008 17:44:59 -0000 1.22 @@ -231,22 +231,30 @@ The join path optimizer takes a join sequence and attempts to minimize the intermediate result. The choice depends on a good estimate of intermediate -results.SAMPLE_THRESHOLD_lOG +results. For the time being, we use a simplistic model, based on the assumption that most joins are foreign key joins anyway. + We use a sample based approach for sizeable tables. -The model is derived from the select statement. +The model is derived from the select statement. However, we did not succeed. +The code is now commented for future improvement. +Final conclusion from this exercise is: +The difference between the join input size and the join output size is not +the correct (or unique) metric which should be used to decide which order +should be used in the joinPath. + @c -#define SAMPLE_THRESHOLD_lOG 17 +/*#define SAMPLE_THRESHOLD_lOG 17*/ size_t ALGjoinCost(BAT *l, BAT *r) { - size_t lc, rc, lsize,rsize; - BAT *lsample, *rsample, *j; + size_t lc, rc; +/* size_t lsize,rsize; + BAT *lsample, *rsample, *j; */

lc = BATcount(l); rc = BATcount(r); - if( lc > 100000 || rc > 100000){ +/* if( lc > 100000 || rc > 100000){ lsize= MIN(lc/100, (1<ttype== TYPE_oid || r->htype== TYPE_oid ) return MIN(lc,rc); return lc * rc;

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

Stefan Manegold

7:04 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

On Thu, Feb 07, 2008 at 08:00:24PM +0100, Romulo Goncalves wrote:

...

Sjoerd Mullender wrote:

...
On 2008-02-07 18:45, Romulo Goncalves wrote:

...
Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested...

A better way to comment larger sections of code is to use #if 0/#endif. ^^^^^ #ifdef 0 ^^^!!!^^

no need not to say more, I gues ... Stefan

...

#endif

does not work: mizer_la-opt_joinpath.o /ufs/goncalve/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx:257:8: error: macro names must be identifiers make[5]: *** [lib_optimizer_la-opt_joinpath.lo] Error 1 make[5]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[4]: *** [all] Error 2 make[4]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build' make: *** [all] Error 2 [goncalve@amelia build]$ vim ~/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx

Romulo

...
That nests.

...
Index: opt_joinpath.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- opt_joinpath.mx 7 Feb 2008 16:57:18 -0000 1.21 +++ opt_joinpath.mx 7 Feb 2008 17:44:59 -0000 1.22 @@ -231,22 +231,30 @@ The join path optimizer takes a join sequence and attempts to minimize the intermediate result. The choice depends on a good estimate of intermediate -results.SAMPLE_THRESHOLD_lOG +results. For the time being, we use a simplistic model, based on the assumption that most joins are foreign key joins anyway. + We use a sample based approach for sizeable tables. -The model is derived from the select statement. +The model is derived from the select statement. However, we did not succeed. +The code is now commented for future improvement. +Final conclusion from this exercise is: +The difference between the join input size and the join output size is not +the correct (or unique) metric which should be used to decide which order +should be used in the joinPath. + @c -#define SAMPLE_THRESHOLD_lOG 17 +/*#define SAMPLE_THRESHOLD_lOG 17*/ size_t ALGjoinCost(BAT *l, BAT *r) { - size_t lc, rc, lsize,rsize; - BAT *lsample, *rsample, *j; + size_t lc, rc; +/* size_t lsize,rsize; + BAT *lsample, *rsample, *j; */

lc = BATcount(l); rc = BATcount(r); - if( lc > 100000 || rc > 100000){ +/* if( lc > 100000 || rc > 100000){ lsize= MIN(lc/100, (1<ttype== TYPE_oid || r->htype== TYPE_oid ) return MIN(lc,rc); return lc * rc;

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

Romulo Goncalves

7:04 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

Romulo Goncalves wrote:

...

Sjoerd Mullender wrote:

...
On 2008-02-07 18:45, Romulo Goncalves wrote:

...
Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested... A better way to comment larger sections of code is to use #if 0/#endif. #ifdef 0 Is it ok if I use #ifdef SAMPLE_THRESHOLD_lOG

#endif Because #define SAMPLE_THRESHOLD_lOG is commented? Or is there a better way to do it? Romulo

...

#endif

does not work: mizer_la-opt_joinpath.o /ufs/goncalve/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx:257:8: error: macro names must be identifiers make[5]: *** [lib_optimizer_la-opt_joinpath.lo] Error 1 make[5]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[4]: *** [all] Error 2 make[4]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build' make: *** [all] Error 2 [goncalve@amelia build]$ vim ~/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx

Romulo

...
That nests.

...
Index: opt_joinpath.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- opt_joinpath.mx 7 Feb 2008 16:57:18 -0000 1.21 +++ opt_joinpath.mx 7 Feb 2008 17:44:59 -0000 1.22 @@ -231,22 +231,30 @@ The join path optimizer takes a join sequence and attempts to minimize the intermediate result. The choice depends on a good estimate of intermediate -results.SAMPLE_THRESHOLD_lOG +results. For the time being, we use a simplistic model, based on the assumption that most joins are foreign key joins anyway. + We use a sample based approach for sizeable tables. -The model is derived from the select statement. +The model is derived from the select statement. However, we did not succeed. +The code is now commented for future improvement. +Final conclusion from this exercise is: +The difference between the join input size and the join output size is not +the correct (or unique) metric which should be used to decide which order +should be used in the joinPath. + @c -#define SAMPLE_THRESHOLD_lOG 17 +/*#define SAMPLE_THRESHOLD_lOG 17*/ size_t ALGjoinCost(BAT *l, BAT *r) { - size_t lc, rc, lsize,rsize; - BAT *lsample, *rsample, *j; + size_t lc, rc; +/* size_t lsize,rsize; + BAT *lsample, *rsample, *j; */

lc = BATcount(l); rc = BATcount(r); - if( lc > 100000 || rc > 100000){ +/* if( lc > 100000 || rc > 100000){ lsize= MIN(lc/100, (1<ttype== TYPE_oid || r->htype== TYPE_oid ) return MIN(lc,rc); return lc * rc;

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

Fabian Groffen

7:05 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

On 07-02-2008 20:04:42 +0100, Romulo Goncalves wrote:

...

Romulo Goncalves wrote:

...
Sjoerd Mullender wrote:

...
On 2008-02-07 18:45, Romulo Goncalves wrote:

...
Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested... A better way to comment larger sections of code is to use #if 0/#endif. #ifdef 0 Is it ok if I use #ifdef SAMPLE_THRESHOLD_lOG

#endif

Because

#define SAMPLE_THRESHOLD_lOG is commented?

Or is there a better way to do it?

Watch Sjoerd's suggestion closely: #if 0 :)

Romulo Goncalves

7:09 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

Fabian Groffen wrote:

...

On 07-02-2008 20:04:42 +0100, Romulo Goncalves wrote:

...
Romulo Goncalves wrote:

...
Sjoerd Mullender wrote:

...
On 2008-02-07 18:45, Romulo Goncalves wrote:

...
Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested... A better way to comment larger sections of code is to use #if 0/#endif. #ifdef 0 Is it ok if I use #ifdef SAMPLE_THRESHOLD_lOG

#endif

Because

#define SAMPLE_THRESHOLD_lOG is commented?

Or is there a better way to do it?

Watch Sjoerd's suggestion closely:

#if 0

:) Xiiii that was stupid one, time for a break....

Thanks... Romulo PS: Thanks to all that made me aware of such mistake :D

...

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

Sjoerd Mullender

7:05 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22

On 2008-02-07 20:00, Romulo Goncalves wrote:

...

Sjoerd Mullender wrote:

...
On 2008-02-07 18:45, Romulo Goncalves wrote:

...
Update of /cvsroot/monetdb/MonetDB5/src/optimizer In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937

Modified Files: opt_joinpath.mx Log Message: Revert Martin code because it does not work. The code will remain, but commented for future improvements as Martin suggested... A better way to comment larger sections of code is to use #if 0/#endif. #ifdef 0

#if 0 Read what I write! ;-)

...

#endif

does not work: mizer_la-opt_joinpath.o /ufs/goncalve/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx:257:8: error: macro names must be identifiers make[5]: *** [lib_optimizer_la-opt_joinpath.lo] Error 1 make[5]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[4]: *** [all] Error 2 make[4]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src/optimizer' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/export/scratch1/goncalve/MonetDB/debug/MonetDB5/build' make: *** [all] Error 2 [goncalve@amelia build]$ vim ~/MonetDB/MonetDB5/src/optimizer/opt_joinpath.mx

Romulo

...
That nests.

...
Index: opt_joinpath.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v retrieving revision 1.21 retrieving revision 1.22 diff -u -d -r1.21 -r1.22 --- opt_joinpath.mx 7 Feb 2008 16:57:18 -0000 1.21 +++ opt_joinpath.mx 7 Feb 2008 17:44:59 -0000 1.22 @@ -231,22 +231,30 @@ The join path optimizer takes a join sequence and attempts to minimize the intermediate result. The choice depends on a good estimate of intermediate -results.SAMPLE_THRESHOLD_lOG +results. For the time being, we use a simplistic model, based on the assumption that most joins are foreign key joins anyway. + We use a sample based approach for sizeable tables. -The model is derived from the select statement. +The model is derived from the select statement. However, we did not succeed. +The code is now commented for future improvement. +Final conclusion from this exercise is: +The difference between the join input size and the join output size is not +the correct (or unique) metric which should be used to decide which order +should be used in the joinPath. + @c -#define SAMPLE_THRESHOLD_lOG 17 +/*#define SAMPLE_THRESHOLD_lOG 17*/ size_t ALGjoinCost(BAT *l, BAT *r) { - size_t lc, rc, lsize,rsize; - BAT *lsample, *rsample, *j; + size_t lc, rc; +/* size_t lsize,rsize; + BAT *lsample, *rsample, *j; */

lc = BATcount(l); rc = BATcount(r); - if( lc > 100000 || rc > 100000){ +/* if( lc > 100000 || rc > 100000){ lsize= MIN(lc/100, (1<ttype== TYPE_oid || r->htype== TYPE_oid ) return MIN(lc,rc); return lc * rc;

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- Sjoerd Mullender

6169

Age (days ago)

6169

Last active (days ago)

List overview

Download

6 comments

4 participants

participants (4)

Fabian Groffen
Romulo Goncalves
Sjoerd Mullender
Stefan Manegold