March 2009 - developers-list

Re: [Monetdb-developers] [MonetDB-users] too deep recursion
by Jan Rittinger 10 Mar '09

10 Mar '09

On Mar 9, 2009, at 23:35, Martin Kersten wrote: > > Please run it also against the HEAD, because most of the problems > may have been resolved there. > > Jan Rittinger wrote: >> Hi Martin and others, >> I just tested what part the Pathfinder code generation plays and >> generated MIL code for the Aug2008 (0.24), the Nov2008, and the >> Feb2009 release branches. I ran all queries using the newest stable >> version (Feb2009) on Mac OS X. >> The observations are: >> * The problem with gdk_heap.mx, mmap, and Mac OS X still resides >> (all queries run in 10 seconds instead of 2 seconds)---Peter knows >> what I'm talking about. >> * Like Nils reported the queries are getting slower. >> * The main performance decrease in my scenario is the document >> loading. >> * The problem does not stem from Pathfinder's MIL code generation. >> For more details see the attached file... >> ------------------------------------------------------------------------ >> BTW: For todays' head version the results are even worse... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Attached you find a bundle with the queries and the test results. (I'm using a Macbook Pro 2.6GHz Intel Core 2 Duo with 4GB of RAM running Mac OS X 10.5.6.) >> >> Jan >> On Mar 9, 2009, at 18:08, Martin Kersten wrote: >>> For all interested. Indeed there are performance differences >>> between the various releases. Some can be traced back to >>> functional enhancements, others are a result from internal >>> administrative activities. >>> >>> Recent experiments with the TPC-H scale-factor 2 on Feb 2009 >>> branch show a performance degradation compared to Aug 2008, >>> as reported on the website. >>> >>> It appears that some low-level actions related to allocation >>> of BATs and their management in memory-scarce situations are >>> debet to this situation. >>> >>> Solutions are integrated with the HEAD, and may (depending >>> on our resources) be back propagated into a bugfix release >>> of the Feb 2009 version. >>> >>> Nils Grimsmo wrote: >>>> On Wed, Mar 04, 2009 at 11:08:40PM +0100, Jan Rittinger wrote: >>>>> Hi Nils, >>>>> >>>>> I just ran your queries with the latest (not yet announced) >>>>> Feb2009 >>>>> release (http://monetdb.cwi.nl/downloads/sources/Feb2009/) and >>>>> received an answer in 1.5 (Q1) and 2.5 (Q2) seconds. If you >>>>> still have >>>>> problems with the new version, then please let us know. >>>> >>>> Thank you for your answer, Jan. Feb2009 is indeed faster than >>>> Nov2008, >>>> but on my computer it is still slower than Aug2008. I also see >>>> some >>>> strange and unfavorable performance characteristics on subsequent >>>> queries >>>> for Nov2008 and Feb2009 (see below). >>>> >>>> >>>> Aug2008: >>>> # MonetDB Server v4.24.0 >>>> # based on GDK v1.24.0 >>>> # PF/Tijah module v0.5.0 loaded. http://dbappl.cs.utwente.nl/ >>>> pftijah >>>> # MonetDB/XQuery module v0.24.0 loaded (default back-end is >>>> 'algebra') >>>> >>>> Nov2008-SP2: >>>> # MonetDB Server v4.26.4 >>>> # based on GDK v1.26.4 >>>> # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/ >>>> pftijah >>>> # MonetDB/XQuery module v0.26.4 loaded (default back-end is >>>> 'algebra') >>>> >>>> Feb2009: >>>> # MonetDB Server v4.28.0 >>>> # Based on GDK v1.28.0 >>>> # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/ >>>> pftijah >>>> # MonetDB/XQuery module v0.28.0 loaded (default back-end is >>>> 'algebra') >>>> >>>> >>>> I run the queries multiple times in different scenarios. >>>> >>>> A - Have just indexed the document, first run. >>>> B - Second run (subsequent have similar timing). >>>> C - Restart the server (Mserver), then first run. >>>> D - Second run (subsequent have similar timing). >>>> >>>> >>>> Query Q0: >>>> Aug2008 Nov2008 Feb2009 >>>> A 1101 3687 1760 >>>> B 1031 4510 3015 >>>> C 1350 5216 3390 >>>> D 1035 12620 9533 >>>> >>>> >>>> Query Q1: >>>> Aug2008 Nov2008 Feb2009 >>>> A 2161 15119 3013 >>>> B 2099 19292 4072 >>>> C 2526 18523 4567 >>>> D 2117 42555 10602 >>>> >>>> >>>> This seems very strange to me. The timings make sense for >>>> Aug2008, where >>>> the query is slightly slower right after restarting the server >>>> (C). For >>>> Nov2008 and Feb2009, the second (and subsequent) runs are slower >>>> than the >>>> first. How can this be? It can make sense for the first run after >>>> restarting the server (C) to be slower (reading stuff from disk >>>> etc.), but >>>> why is the second (D) terribly slower? If I just keep running >>>> the query, >>>> the timings are similar to D. >>>> >>>> Note: If I start mixing Q0 and Q1 after step D, they are both as >>>> slow as >>>> in step D. >>>> >>>> >>>> I hope this feedback is helpful. Is there something strange with >>>> my >>>> setup, or is this a "bug"? (My timings in step (A) seem similar >>>> to Jan's >>>> timings). >>>> >>>> >>>> If I want to compare MonetDB/XQuery to other implementations in a >>>> scientific paper, I typically want to warm up the system, then >>>> run the >>>> query multiple times to get an average timing. It is kind of >>>> inconvenient >>>> not to be able to close down Mserver between experiments... >>>> >>>> >>>>> P.S.: The E-Mail subject seems slightly off topic here :) >>>> >>>> Yes, thought I'd avoid touching the mouse to copy the email >>>> address. Cut >>>> away In-Reply-To:, but forgot to change Subject:... >>>> >>>> >>>> Thank you for your assistance! >>>> >>>> >>>> Klem fra Nils >>>> >>>>> On Mar 4, 2009, at 16:30, Nils Grimsmo wrote: >>>>> >>>>>> Hi, I just upgraded from the Augst to the Noveber super-ball, >>>>>> and the >>>>>> performance has worsened badly. >>>>>> >>>>>> Example queries on dblp.xml (441 MB): >>>>>> >>>>>> Q0: count(/dblp//author[text()="Michael Stonebraker"]) >>>>>> Q1: count(/dblp/*/author[text()="Michael Stonebraker"]) >>>>>> >>>>>> Query time in milliseconds: >>>>>> >>>>>> August November >>>>>> Q0 1100 4867 >>>>>> Q1 3993 17999 >>>>>> >>>>>> I have compiled with --enable-optimise both times. I query with: >>>>>> >>>>>> mclient --language=xquery --algebra --time < $QUERYFILE >>>>>> >>>>>> Is this performance degradation expected? If so, why? >>>>>> >>>>>> >>>>>> BTW: Is there any way of finding how much disk space a >>>>>> collection >>>>>> uses? >>>>>> >>>>>> >>>>>> Thank you for contributing free software! >>>>>> >>>>>> >>>>>> Klem fra Nils -- Jan Rittinger Lehrstuhl Datenbanken und Informationssysteme Wilhelm-Schickard-Institut für Informatik Eberhard-Karls-Universität Tübingen http://www-db.informatik.uni-tuebingen.de/team/rittinger

1 0

Re: [Monetdb-developers] [MonetDB-users] too deep recursion
by Jan Rittinger 09 Mar '09

09 Mar '09

Hi Martin and others, I just tested what part the Pathfinder code generation plays and generated MIL code for the Aug2008 (0.24), the Nov2008, and the Feb2009 release branches. I ran all queries using the newest stable version (Feb2009) on Mac OS X. The observations are: * The problem with gdk_heap.mx, mmap, and Mac OS X still resides (all queries run in 10 seconds instead of 2 seconds)---Peter knows what I'm talking about. * Like Nils reported the queries are getting slower. * The main performance decrease in my scenario is the document loading. * The problem does not stem from Pathfinder's MIL code generation. For more details see the attached file... BTW: For todays' head version the results are even worse... Jan On Mar 9, 2009, at 18:08, Martin Kersten wrote: > For all interested. Indeed there are performance differences > between the various releases. Some can be traced back to > functional enhancements, others are a result from internal > administrative activities. > > Recent experiments with the TPC-H scale-factor 2 on Feb 2009 > branch show a performance degradation compared to Aug 2008, > as reported on the website. > > It appears that some low-level actions related to allocation > of BATs and their management in memory-scarce situations are > debet to this situation. > > Solutions are integrated with the HEAD, and may (depending > on our resources) be back propagated into a bugfix release > of the Feb 2009 version. > > Nils Grimsmo wrote: >> On Wed, Mar 04, 2009 at 11:08:40PM +0100, Jan Rittinger wrote: >>> Hi Nils, >>> >>> I just ran your queries with the latest (not yet announced) Feb2009 >>> release (http://monetdb.cwi.nl/downloads/sources/Feb2009/) and >>> received an answer in 1.5 (Q1) and 2.5 (Q2) seconds. If you still >>> have >>> problems with the new version, then please let us know. >> >> Thank you for your answer, Jan. Feb2009 is indeed faster than >> Nov2008, >> but on my computer it is still slower than Aug2008. I also see some >> strange and unfavorable performance characteristics on subsequent >> queries >> for Nov2008 and Feb2009 (see below). >> >> >> Aug2008: >> # MonetDB Server v4.24.0 >> # based on GDK v1.24.0 >> # PF/Tijah module v0.5.0 loaded. http://dbappl.cs.utwente.nl/pftijah >> # MonetDB/XQuery module v0.24.0 loaded (default back-end is >> 'algebra') >> >> Nov2008-SP2: >> # MonetDB Server v4.26.4 >> # based on GDK v1.26.4 >> # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah >> # MonetDB/XQuery module v0.26.4 loaded (default back-end is >> 'algebra') >> >> Feb2009: >> # MonetDB Server v4.28.0 >> # Based on GDK v1.28.0 >> # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah >> # MonetDB/XQuery module v0.28.0 loaded (default back-end is >> 'algebra') >> >> >> I run the queries multiple times in different scenarios. >> >> A - Have just indexed the document, first run. >> B - Second run (subsequent have similar timing). >> C - Restart the server (Mserver), then first run. >> D - Second run (subsequent have similar timing). >> >> >> Query Q0: >> Aug2008 Nov2008 Feb2009 >> A 1101 3687 1760 >> B 1031 4510 3015 >> C 1350 5216 3390 >> D 1035 12620 9533 >> >> >> Query Q1: >> Aug2008 Nov2008 Feb2009 >> A 2161 15119 3013 >> B 2099 19292 4072 >> C 2526 18523 4567 >> D 2117 42555 10602 >> >> >> This seems very strange to me. The timings make sense for Aug2008, >> where >> the query is slightly slower right after restarting the server >> (C). For >> Nov2008 and Feb2009, the second (and subsequent) runs are slower >> than the >> first. How can this be? It can make sense for the first run after >> restarting the server (C) to be slower (reading stuff from disk >> etc.), but >> why is the second (D) terribly slower? If I just keep running the >> query, >> the timings are similar to D. >> >> Note: If I start mixing Q0 and Q1 after step D, they are both as >> slow as >> in step D. >> >> >> I hope this feedback is helpful. Is there something strange with my >> setup, or is this a "bug"? (My timings in step (A) seem similar to >> Jan's >> timings). >> >> >> If I want to compare MonetDB/XQuery to other implementations in a >> scientific paper, I typically want to warm up the system, then run >> the >> query multiple times to get an average timing. It is kind of >> inconvenient >> not to be able to close down Mserver between experiments... >> >> >>> P.S.: The E-Mail subject seems slightly off topic here :) >> >> Yes, thought I'd avoid touching the mouse to copy the email >> address. Cut >> away In-Reply-To:, but forgot to change Subject:... >> >> >> Thank you for your assistance! >> >> >> Klem fra Nils >> >>> On Mar 4, 2009, at 16:30, Nils Grimsmo wrote: >>> >>>> Hi, I just upgraded from the Augst to the Noveber super-ball, and >>>> the >>>> performance has worsened badly. >>>> >>>> Example queries on dblp.xml (441 MB): >>>> >>>> Q0: count(/dblp//author[text()="Michael Stonebraker"]) >>>> Q1: count(/dblp/*/author[text()="Michael Stonebraker"]) >>>> >>>> Query time in milliseconds: >>>> >>>> August November >>>> Q0 1100 4867 >>>> Q1 3993 17999 >>>> >>>> I have compiled with --enable-optimise both times. I query with: >>>> >>>> mclient --language=xquery --algebra --time < $QUERYFILE >>>> >>>> Is this performance degradation expected? If so, why? >>>> >>>> >>>> BTW: Is there any way of finding how much disk space a collection >>>> uses? >>>> >>>> >>>> Thank you for contributing free software! >>>> >>>> >>>> Klem fra Nils -- Jan Rittinger Lehrstuhl Datenbanken und Informationssysteme Wilhelm-Schickard-Institut für Informatik Eberhard-Karls-Universität Tübingen http://www-db.informatik.uni-tuebingen.de/team/rittinger

1 0

[Monetdb-developers] Possible bug in MonetDB and JDBC driver in sequence management
by Cimballi 09 Mar '09

09 Mar '09

Hi, I'm a recent user of MonetDB. I'm using it to store data from XML files from "http://oasissta.caiso.com/". I parse the XML files and I insert the data into MonetDB using the JDBC driver (1.9). I make a batch insert of each xml data element. The ID column of my table is a sequence I created like this : CREATE TABLE "oasis_report_data" ( "oasis_report_data_id" INT NOT NULL, "oasis_report_header_id" INT NOT NULL, "data_item" VARCHAR(255) NOT NULL, "resource_name" VARCHAR(255) NOT NULL, "opr_date" DATE NOT NULL, "interval_num" INT NOT NULL, "value" DOUBLE NOT NULL, CONSTRAINT "pk_oasis_report_data" PRIMARY KEY ("oasis_report_data_id"), CONSTRAINT "fk_oasis_report_data_oasis_report_header" FOREIGN KEY ("oasis_report_header_id") REFERENCES "oasis_report_header" ("oasis_report_header_id") ); CREATE SEQUENCE seq_oasis_report_data_id AS INT; And I'm using iBatis to insert the data like that : <insert id="insertOasisReportData" parameterClass="oasisReportData"> <selectKey keyProperty="id" resultClass="int" type="pre"> SELECT NEXT VALUE FOR seq_oasis_report_data_id </selectKey> INSERT INTO oasis_report_data (oasis_report_data_id, oasis_report_header_id, data_item, interval_num, opr_date, resource_name, value) VALUES (#id#, #header.id#, #dataItem#, #intervalNum#, #oprDate#, #resourceName#, #value#) </insert> Here is the bug I found : When the file is small (up to 2 Mo, around 15 000 items), no problem. If the file is bigger (20 Mo, around 100 000 items), then at a moment the select next value return null. Here is the java.sql log : 2009-03-02 21:14:30,324 | DEBUG | java.sql.Connection | debug | {conn-170420} Connection 2009-03-02 21:14:30,324 | DEBUG | java.sql.Connection | debug | {conn-170420} Preparing Statement: SELECT NEXT VALUE FOR seq_oasis_report_data_id 2009-03-02 21:14:30,334 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170421} Executing Statement: SELECT NEXT VALUE FOR seq_oasis_report_data_id 2009-03-02 21:14:30,334 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170421} Parameters: [] 2009-03-02 21:14:30,334 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170421} Types: [] 2009-03-02 21:14:30,344 | DEBUG | java.sql.ResultSet | debug | {rset-170422} ResultSet 2009-03-02 21:14:30,344 | DEBUG | java.sql.ResultSet | debug | {rset-170422} Header: [next_value_for_sys] 2009-03-02 21:14:30,344 | DEBUG | java.sql.ResultSet | debug | {rset-170422} Result: [16901] 2009-03-02 21:14:30,344 | DEBUG | java.sql.Connection | debug | {conn-170420} Preparing Statement: INSERT INTO oasis_report_data (oasis_report_data_id, oasis_report_header_id, data_item, interval_num, opr_date, resource_name, value) VALUES (?, ?, ?, ?, ?, ?, ?) 2009-03-02 21:14:30,354 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170423} Executing Statement: INSERT INTO oasis_report_data (oasis_report_data_id, oasis_report_header_id, data_item, interval_num, opr_date, resource_name, value) VALUES (?, ?, ?, ?, ?, ?, ?) 2009-03-02 21:14:30,354 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170423} Parameters: [16901, 705, LMP_PRC, 5, 2009-01-26 00:00:00.0, AVOCADO_6_N002, 31.80941] 2009-03-02 21:14:30,354 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170423} Types: [java.lang.Integer, java.lang.Integer, java.lang.String, java.lang.Integer, java.sql.Timestamp, java.lang.String, java.lang.Double] 2009-03-02 21:14:30,364 | DEBUG | java.sql.Connection | debug | {conn-170424} Connection 2009-03-02 21:14:30,364 | DEBUG | java.sql.Connection | debug | {conn-170424} Preparing Statement: SELECT NEXT VALUE FOR seq_oasis_report_data_id 2009-03-02 21:14:30,364 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170425} Executing Statement: SELECT NEXT VALUE FOR seq_oasis_report_data_id 2009-03-02 21:14:30,364 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170425} Parameters: [] 2009-03-02 21:14:30,364 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170425} Types: [] 2009-03-02 21:14:34,500 | DEBUG | java.sql.Connection | debug | {conn-170424} Preparing Statement: INSERT INTO oasis_report_data (oasis_report_data_id, oasis_report_header_id, data_item, interval_num, opr_date, resource_name, value) VALUES (?, ?, ?, ?, ?, ?, ?) 2009-03-02 21:14:34,510 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170426} Executing Statement: INSERT INTO oasis_report_data (oasis_report_data_id, oasis_report_header_id, data_item, interval_num, opr_date, resource_name, value) VALUES (?, ?, ?, ?, ?, ?, ?) 2009-03-02 21:14:34,510 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170426} Parameters: [null, 705, LMP_PRC, 6, 2009-01-26 00:00:00.0, AVOCADO_6_N002, 35.59336] 2009-03-02 21:14:34,510 | DEBUG | java.sql.PreparedStatement | debug | {pstm-170426} Types: [null, java.lang.Integer, java.lang.String, java.lang.Integer, java.sql.Timestamp, java.lang.String, java.lang.Double] You can see that in the second case there are two lines missing : 2009-03-02 21:14:30,344 | DEBUG | java.sql.ResultSet | debug | {rset-170422} Header: [next_value_for_sys] 2009-03-02 21:14:30,344 | DEBUG | java.sql.ResultSet | debug | {rset-170422} Result: [16901] And what strange is that each time it appears at a different moment (I mean on a different set of data, using the same file). Maybe on nig stress the sequence generator of MonetDB bugs or maybe the JDBC driver. HTH, Cimballi

2 1

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5 sql.mx, , 1.321, 1.322 sql_scenario.mx, , 1.337, 1.338
by Fabian Groffen 07 Mar '09

07 Mar '09

On 07-03-2009 21:41:21 +0000, Stefan de Konink wrote: > This code adds the last_id variable to MonetDB5/SQL to implement 'in SQL' > access to the last updated sequence, in a similar fashion as the @@IDENTITY > found in MSSQL. For future compatibility a procedure like get_last_row_id() > found in other databases, this function would have to return the variable > found. The variable is updated when m->last_id is updated. > > The code was added to have last_id support within the ODBC interface; > required for phpBB compatibility. Weird, ODBC already knows it, and more reliable, via the response of Q_UPDATE. I think it was already implemented in ODBC even? It's available in mapilib, using mapi_get_last_id.

2 4

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_derivepath.mx, Feb2009, NONE, 1.1.2.1 Makefile.ag, Feb2009, 1.66, 1.66.2.1 opt_prelude.mx, Feb2009, 1.55.2.3, 1.55.2.4
by Sjoerd Mullender 05 Mar '09

05 Mar '09

How is this not a new feature? Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv2998/optimizer > > Modified Files: > Tag: Feb2009 > Makefile.ag opt_prelude.mx > Added Files: > Tag: Feb2009 > opt_derivepath.mx > Log Message: > Added the framework for the commissioned derivepath optimizer. > > > U opt_prelude.mx > Index: opt_prelude.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_prelude.mx,v > retrieving revision 1.55.2.3 > retrieving revision 1.55.2.4 > diff -u -d -r1.55.2.3 -r1.55.2.4 > --- opt_prelude.mx 1 Mar 2009 19:15:30 -0000 1.55.2.3 > +++ opt_prelude.mx 5 Mar 2009 00:17:12 -0000 1.55.2.4 > @@ -57,6 +57,7 @@ > opt_export str deleteRef; > opt_export str depositRef; > opt_export str deriveRef; > +opt_export str derivePathRef; > opt_export str differenceRef; > opt_export str divRef; > opt_export str disconnectRef; > @@ -216,6 +217,7 @@ > str deleteRef; > str depositRef; > str deriveRef; > +str derivePathRef; > str differenceRef; > str divRef; > str disconnectRef; > @@ -371,6 +373,7 @@ > deleteRef = putName("delete",6); > depositRef = putName("deposit",7); > deriveRef = putName("derive",6); > + derivePathRef = putName("derivePath",10); > differenceRef= putName("difference",10); > divRef = putName("/",1); > disconnectRef= putName("disconnect",10); > > U Makefile.ag > Index: Makefile.ag > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/Makefile.ag,v > retrieving revision 1.66 > retrieving revision 1.66.2.1 > diff -u -d -r1.66 -r1.66.2.1 > --- Makefile.ag 18 Jan 2009 11:44:14 -0000 1.66 > +++ Makefile.ag 5 Mar 2009 00:17:12 -0000 1.66.2.1 > @@ -32,7 +32,7 @@ > opt_singleton.mx opt_costModel.mx opt_reduce.mx opt_macro.mx \ > opt_accumulators.mx opt_qep.mx opt_mergetable.mx \ > opt_remoteQueries.mx opt_joinselect.mx opt_partitions.mx \ > - opt_evaluate.mx opt_inline.mx opt_pushranges.mx \ > + opt_evaluate.mx opt_inline.mx opt_pushranges.mx opt_derivepath.mx \ > opt_accessmode.mx opt_joinpath.mx opt_heuristics.mx opt_remap.mx \ > opt_statistics.mx opt_trace.mx opt_recycler.mx opt_dataflow.mx \ > opt_mitosis.mx opt_octopus.mx opt_replicator.mx opt_history.mx > @@ -53,7 +53,7 @@ > opt_singleton.mx opt_costModel.mx opt_reduce.mx opt_macro.mx \ > opt_accumulators.mx opt_qep.mx opt_mergetable.mx \ > opt_remoteQueries.mx opt_joinselect.mx opt_partitions.mx \ > - opt_evaluate.mx opt_inline.mx opt_pushranges.mx \ > + opt_evaluate.mx opt_inline.mx opt_pushranges.mx opt_derivepath.mx \ > opt_accessmode.mx opt_joinpath.mx opt_heuristics.mx opt_remap.mx \ > opt_statistics.mx opt_trace.mx opt_recycler.mx opt_dataflow.mx \ > opt_mitosis.mx opt_octopus.mx opt_replicator.mx opt_history.mx > @@ -72,7 +72,7 @@ > opt_singleton.mx opt_costModel.mx opt_reduce.mx opt_macro.mx \ > opt_accumulators.mx opt_qep.mx opt_mergetable.mx \ > opt_remoteQueries.mx opt_joinselect.mx opt_partitions.mx \ > - opt_evaluate.mx opt_inline.mx opt_pushranges.mx \ > + opt_evaluate.mx opt_inline.mx opt_pushranges.mx opt_derivepath.mx \ > opt_accessmode.mx opt_joinpath.mx opt_heuristics.mx opt_remap.mx \ > opt_statistics.mx opt_trace.mx opt_recycler.mx opt_dataflow.mx \ > opt_mitosis.mx opt_octopus.mx opt_replicator.mx opt_history.mx > > --- NEW FILE: opt_derivepath.mx --- > @' The contents of this file are subject to the MonetDB Public License > @' Version 1.1 (the "License"); you may not use this file except in > @' compliance with the License. You may obtain a copy of the License at > @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html > @' > @' Software distributed under the License is distributed on an "AS IS" > @' basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > @' License for the specific language governing rights and limitations > @' under the License. > @' > @' The Original Code is the MonetDB Database System. > @' > @' The Initial Developer of the Original Code is CWI. > @' Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > @' Copyright August 2008-2009 MonetDB B.V. > @' All Rights Reserved. > > @f opt_derivepath > @- Group derive paths > The routine @sc{optimizer.derivepaths()} > walks through the program looking for grouping operations > and cascades them into a single multiple group paths. > To illustrate, consider > @verbatim > (ext1,grp1) := group.new(b); > (ext2,grp2) := group.derive(ext1,grp1, c); > (ext3,grp3) := group.derive(ext2,grp2, d); > @end verbatim > The result becomes. > @verbatim > (ext3,grp3) := group.derivepath(b,c,d); > @end verbatim > @{ > @mal > pattern optimizer.derivePath():str > address OPTderivePath; > pattern optimizer.derivePath(mod:str, fcn:str):str > address OPTderivePath > comment "Join path constructor"; > pattern group.derivePath(l:bat[:any,:any]...):bat[:any,:any] > address ALGderivePath > comment "internal routine to handle join paths. > The type analysis is rather tricky."; > @h > #ifndef _OPT_DERIVEPATH_ > #define _OPT_DERIVEPATH_ > #include "opt_prelude.h" > #include "opt_support.h" > #include "mal_interpreter.h" > > /* #define DEBUG_OPT_DERIVEPATH */ > @c > #include "mal_config.h" > #include "opt_derivepath.h" > > static int > OPTderivePathImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p) > { > int i, actions=0; > int *pc; > str derivePathRef = putName("derivePath",10); > InstrPtr q; > InstrPtr *old; > int limit,slimit; > > (void) cntxt; > (void) stk; > if (varGetProp(mb, getArg(mb->stmt[0], 0), inlineProp) != NULL) > return 0; > > old= mb->stmt; > limit= mb->stop; > slimit= mb->ssize; > newMalBlkStmt(mb,mb->ssize); > > /* beware, new variables and instructions are introduced */ > pc= (int*) GDKmalloc(sizeof(int)* mb->vtop * 2); /* to find last assignment */ > memset((char*) pc, 0, sizeof(int)* mb->vtop *2); > > for (i = 0; i<limit; i++){ > p= old[i]; > if( getModuleId(p)== groupRef && getFunctionId(p)== newRef ){ > pc[getArg(p,0)] = i; > pc[getArg(p,1)] = i; > } > if( getModuleId(p)== groupRef && getFunctionId(p)== deriveRef ){ > @- > Try to expand its argument list with what we have found so far. > This creates a series of join paths, many of which will be removed during deadcode elimination. > @c > if (pc[getArg(p,2)]== pc[getArg(p,3)]){ > q= copyInstruction(getInstrPtr(mb,pc[getArg(p,2)])); > q= pushArgument(mb,q, getArg(p,5)); > pc[getArg(p,0)] = i; > pc[getArg(p,1)] = i; > setFunctionId(q,derivePathRef); > p= q; > actions++; > #ifdef DEBUG_OPT_DERIVEPATH > stream_printf(cntxt->fdout,"new derivePath instruction\n"); > printInstruction(cntxt->fdout,mb, 0, q, LIST_MAL_ALL); > #endif > } > } > pushInstruction(mb,p); > } > for(; i<slimit; i++) > if(old[i]) > freeInstruction(old[i]); > GDKfree(old); > GDKfree(pc); > OPTDEBUG > if ( actions) > stream_printf(cntxt->fdout,"#opt_derivepath: %d statements glued\n",actions); > return actions; > } > @include optimizerWrapper.mx > @h > @:exportOptimizer(derivePath)@ > #endif > opt_export str ALGderivePath(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > @c > #include "opt_statistics.h" > @:wrapOptimizer(derivePath,OPT_CHECK_ALL)@ > @- > The derive path optimizer takes a derivation sequence and > attempts to minimize the intermediate result. > The choice depends on a good estimate of intermediate > results using properties. > For the time being, we use a simplistic model, based > on the assumption that most joins are foreign key joins anyway. > @c > str > ALGderivePath(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > { > (void) cntxt; > (void) mb; > (void) stk; > (void) pci; > throw(MAL,"group.derivePath","nyi"); > } > @} > > > ------------------------------------------------------------------------------ > Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA > -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise > -Strategies to boost innovation and cut costs with open source participation > -Receive a $600 discount off the registration fee with the source code: SFAD > http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

3 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_debugger.mx, Feb2009, 1.256.2.2, 1.256.2.3
by Stefan Manegold 04 Mar '09

04 Mar '09

[Pre-nighlty testing compilation check service] Fails to compile with icc & gcc with optimization enabled: .../MonetDB5/src/mal/mal_debugger.mx(1501): error #592: variable "p" is used before its value is set mdbCommand(cntxt, mb, stk, p, pc); ^ Stefam On Wed, Mar 04, 2009 at 03:54:07PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv10776 > > Modified Files: > Tag: Feb2009 > mal_debugger.mx > Log Message: > The debugger command 'bbp trace <pattern>' has been added to > keep a watch over the lref status of bats in the bbp. > > > U mal_debugger.mx > Index: mal_debugger.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_debugger.mx,v > retrieving revision 1.256.2.2 > retrieving revision 1.256.2.3 > diff -u -d -r1.256.2.2 -r1.256.2.3 > --- mal_debugger.mx 12 Feb 2009 06:34:16 -0000 1.256.2.2 > +++ mal_debugger.mx 4 Mar 2009 15:54:04 -0000 1.256.2.3 > @@ -489,6 +489,7 @@ > mal_export void printBatInfo(stream *f, VarPtr n, ValPtr v); > mal_export void printBatProperties(stream *f, VarPtr n, ValPtr v, str props); > mal_export void printTraceCall(stream *out, MalBlkPtr mb, MalStkPtr stk, int pc, int flags); > +mal_export char BBPTraceCall(Client cntxt, MalBlkPtr mb, MalStkPtr stk, int pc); > mal_export str call2str(MalBlkPtr mb, MalStkPtr stk, int pc, int flags); > > mal_export str runMALDebugger(Client cntxt, Symbol s); > @@ -742,6 +743,63 @@ > stream_printf(out, "#%s\n", msg); > GDKfree(msg); > } > +static void > +mdbBacktrace(Client cntxt, MalStkPtr stk, int pci){ > + for (; stk != NULL; stk = stk->up) { > + printCall(cntxt, stk->blk, stk, pci); > + if (stk->up) > + pci = stk->up->pcup; > + } > +} > +@- > +Sometimes we may want to trace the changes applied to the > +BAT buffer pool and report them together with the MAL > +instruction where it happened. > +@c > +static int BBPtraceEnabled = 0; > +static str BBPtracePattern = NULL; > +char > +BBPTraceCall(Client cntxt, MalBlkPtr mb, MalStkPtr stk, int pc) > +{ > + static int *BBPmirror = NULL; > + static int bbpsize; > + char lbuf[1024], pbuf[1024]; > + int i, action; > + (void) mb; > + > + if ( BBPtraceEnabled == 0) > + return 0; > + if (BBPmirror == NULL ){ > + bbpsize= BBPsize; > + BBPmirror = (int*) GDKzalloc(sizeof(int) * BBPsize); > + } > + if( BBPsize > bbpsize) { > + int *old = BBPmirror; > + BBPmirror = (int*) GDKzalloc(sizeof(int) * BBPsize); > + memcpy((char*)BBPmirror, (char*)old, sizeof(int) * bbpsize); > + bbpsize= BBPsize; > + GDKfree(old); > + } > + /* no growing BBP yet */ > + action = 0; > + for ( i=0; i<bbpsize; i++){ > + /* what happened to this BAT */ > + if ( BBPmirror[i] != BBP_lrefs(i)){ > + BBPlogical(i,lbuf); > + if ( BBPtracePattern && > + strncmp(lbuf,BBPtracePattern,strlen(BBPtracePattern)) ) > + continue; > + BBPphysical(i,pbuf); > + stream_printf(cntxt->fdout, "#BBP state change of %s %s entry %d refs %d:\n", > + lbuf,pbuf,i, BBP_lrefs(i)); > + action = 1; > + } > + BBPmirror[i] = BBP_lrefs(i); > + } > + if ( action ) > + mdbBacktrace(cntxt, stk, pc); > + return 's'; > +} > > static void > printBATproperties(stream *f, BAT *b){ > @@ -756,7 +814,7 @@ > if (b->T->parentid) > stream_printf(f, "tail view on %s ", BBPname(b->T->parentid)); > } > -@+ MAL parser > +@+ MAL debugger parser > The debugger structure is inherited from GDB. > The routine mdbCommand is called with p=0 after finishing a mal- function call > and before continuing at the next level of invocation. > @@ -802,14 +860,6 @@ > return getMalBlkHistory(mb,-1); > } > > -static void > -mdbBacktrace(Client cntxt, MalStkPtr stk, int pci){ > - for (; stk != NULL; stk = stk->up) { > - printCall(cntxt, stk->blk, stk, pci); > - if (stk->up) > - pci = stk->up->pcup; > - } > -} > > void > mdbCommand(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p, int pc) > @@ -1089,6 +1139,16 @@ > int i,limit; > > skipWord(cntxt, b); > + /* bbp change tracing enabling */ > + if ( strncmp(b,"trace",5) == 0){ > + skipWord(cntxt, b); > + BBPtraceEnabled = !BBPtraceEnabled; > + if ( BBPtraceEnabled && *b ) > + BBPtracePattern = GDKstrdup(b); > + stream_printf(out,"#bbp trace enabled %d\n", BBPtraceEnabled); > + stk->cmd = 'c'; > + break; > + } > i = BBPindex(b); > if( i) > limit=i+1; > @@ -1437,6 +1497,8 @@ > return; > } > > + if ( BBPTraceCall(cntxt,mb,stk,pc)) > + mdbCommand(cntxt, mb, stk, p, pc); > switch (stk->cmd) { > case 'c': > p = getInstrPtr(mb, pc); > > > ------------------------------------------------------------------------------ > Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA > -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise > -Strategies to boost innovation and cut costs with open source participation > -Receive a $600 discount off the registration fee with the source code: SFAD > http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-bugs] [ monetdb-Bugs-2654300 ] problem with views reusing aliases ?
by Romulo Goncalves 02 Mar '09

02 Mar '09

SourceForge.net wrote: > Bugs item #2654300, was opened at 2009-03-02 12:41 > Message generated for change (Tracker Item Submitted) made by Item Submitter > You can respond by visiting: > https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2654300&group_… > > Please note that this message will contain a full copy of the comment thread, > including the initial issue submission, for this request, > not just the latest update. > Category: SQL/Core > Group: SQL "stable" > Status: Open > Resolution: None > Priority: 5 > Private: No > Submitted By: Roberto Cornacchia (cornuz) > Assigned to: Niels Nes (nielsnes) > Summary: problem with views reusing aliases ? > > Initial Comment: > [ on Feb2009 branch ] > > create table x (n int, s varchar(10)); > insert into x values (1, 'one'); > insert into x values (2, 'two'); > insert into x values (3, 'three'); > > -- swap the two attributes > create view x1 as > select s as a1, n as a2 from x; > > -- swap them again > create view x2 as > select a2 as a1, a1 as a2 from x1; > > -- select on the original 's' attribute fails > select * from x2 where a2='two'; > > This is the output: > > $ mclient -lsql -dtest02 -G < bug2.sql The algebra version is the default one in the current as well as in the stable branch... So how the -G option is still valid? Romulo > [ 1 ] > [ 1 ] > [ 1 ] > MAPI = monetdb@localhost:50000 > QUERY = select * from x2 where a2='two'; > ERROR = !TypeException:user.s0_1[4]:'algebra.uselect' undefined in: _9:any := algebra.uselect(_2:bat[:oid,:str], _8:int) > !TypeException:user.s0_1[6]:'algebra.kdifference' undefined in: _12:any := algebra.kdifference(_9:any, _10:bat[:oid,:str]) > !TypeException:user.s0_1[7]:'algebra.uselect' undefined in: _13:any := algebra.uselect(_10:bat[:oid,:str], _8:int) > !TypeException:user.s0_1[8]:'algebra.kunion' undefined in: _14:any := algebra.kunion(_12:any, _13:any) > !TypeException:user.s0_1[10]:'algebra.uselect' undefined in: _17:any := algebra.uselect(_15:bat[:oid,:str], _8:int) > !TypeException:user.s0_1[11]:'algebra.kunion' undefined in: _18:any := algebra.kunion(_14:any, _17:any) > !TypeException:user.s0_1[14]:'algebra.kdifference' undefined in: _21:any := algebra.kdifference(_18:any, _20:bat[:oid,:oid]) > !TypeException:user.s0_1[16]:'algebra.markT' undefined in: _24:any := algebra.markT(_21:any, _22:oid) > !TypeException:user.s0_1[17]:'bat.reverse' undefined in: _25:any := bat.reverse(_24:any) > !TypeException:user.s0_1[24]:'algebra.join' undefined in: _33:any := algebra.join(_25:any, _32:bat[:oid,:int]) > !TypeException:user.s0_1[28]:'algebra.join' undefined in: _37:any := algebra.join(_25:any, _36:bat[:oid,:str]) > !SQLException:int:conversion of string 'two' failed > > > ---------------------------------------------------------------------- > > You can respond by visiting: > https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2654300&group_… > > ------------------------------------------------------------------------------ > Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA > -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise > -Strategies to boost innovation and cut costs with open source participation > -Receive a $600 discount off the registration fee with the source code: SFAD > http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > Monetdb-bugs mailing list > Monetdb-bugs(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-bugs

1 0