- developers-list - monetdb.org

[Monetdb-developers] BAT-leak in gdk_logger.mx ?
by Stefan Manegold 04 Jun '08

04 Jun '08

Niels, Sjoerd, Peter, after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&group_id=5… I found the --- or at least one possible --- cause for this bug. In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents. Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()). However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown. The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm. Thank you very much in advance! Stefan ======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) { + BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE); } ======== -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 4

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/BugDay_2005-10-06_2.9.3/Tests All, SQL_2-24, 1.10, 1.10.4.1 having.SF-922614.stable.out, SQL_2-24, 1.3, 1.3.14.1
by Stefan Manegold 03 Jun '08

03 Jun '08

On Tue, Jun 03, 2008 at 07:03:54AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/sql/src/test/BugDay_2005-10-06_2.9.3/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv28722 > > Modified Files: > Tag: SQL_2-24 > All having.SF-922614.stable.out > Log Message: > approved new output > > > U All > Index: All > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugDay_2005-10-06_2.9.3/Tests/All,v > retrieving revision 1.10 > retrieving revision 1.10.4.1 > diff -u -d -r1.10 -r1.10.4.1 > --- All 11 Dec 2007 22:19:06 -0000 1.10 > +++ All 3 Jun 2008 07:03:52 -0000 1.10.4.1 > @@ -1,7 +1,7 @@ > jdbc_crashes_on_getTables.SF_944316 > variable_arg_substring.SF_942981 > UNION_JOIN_vs_plain_set_UNION-2.SF-920585 > -huge_expression_and_column_name.SF-921173 > +#huge_expression_and_column_name.SF-921173 will not be fixed in the MonetDB 5.6.0, SQL 2.24 release > CrashMe_SQL_server_crash-2.SF-921673 > CrashMe_SQL_server_crash-1.SF-921673 > inserts_with_defaults.SF-921977 I just prevented the propagation of this change ;-) Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/snodgrass/Tests All, SQL_2-24, 1.4, 1.4.10.1
by Stefan Manegold 03 Jun '08

03 Jun '08

On Tue, Jun 03, 2008 at 07:13:37AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/sql/src/test/snodgrass/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv559/src/test/snodgrass/Tests > > Modified Files: > Tag: SQL_2-24 > All > Log Message: > disabled test (will not be fixed in this release) > > THIS SHOULD NOT BE PROPAGATED!! propagation has been prevented. Stefan > U All > Index: All > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/snodgrass/Tests/All,v > retrieving revision 1.4 > retrieving revision 1.4.10.1 > diff -u -d -r1.4 -r1.4.10.1 > --- All 25 Jan 2007 01:28:36 -0000 1.4 > +++ All 3 Jun 2008 07:13:35 -0000 1.4.10.1 > @@ -1 +1 @@ > -cast_select > +#cast_select will not be fixed in the MonetDB 5.6.0, SQL 2.24 release > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_recycler.mx, MonetDB_5-6, 1.23.2.1, 1.23.2.2
by Romulo Goncalves 02 Jun '08

02 Jun '08

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv24665 > > Modified Files: > Tag: MonetDB_5-6 > opt_recycler.mx > Log Message: > Since we may introduce new instructions and variables, > the recycled[] object may have been too small. The recycler is feature and it is used in the current branch. It is not tested in the stable and it might stop to be compiled in the stable. All the implementation related to the recycler should be checked in at the current one and not in the stable one. This avoids problems in the stable and saves time in the recycler development (I do not need to wait for the propagations to have an actualized code)... I am going to revert this change, and do it in the current Note: Martin is aware of this... :) Regards, Romulo > > > U opt_recycler.mx > Index: opt_recycler.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_recycler.mx,v > retrieving revision 1.23.2.1 > retrieving revision 1.23.2.2 > diff -u -d -r1.23.2.1 -r1.23.2.2 > --- opt_recycler.mx 31 May 2008 16:29:54 -0000 1.23.2.1 > +++ opt_recycler.mx 2 Jun 2008 20:26:08 -0000 1.23.2.2 > @@ -107,7 +107,8 @@ > > (void) cntxt; > (void) stk; > - recycled= GDKzalloc(sizeof(char)*mb->vtop); > + /* watch out, instructions may introduce new variables */ > + recycled= GDKzalloc(sizeof(char)*mb->vtop*2); > setLifespan(mb,span); > limit= mb->stop; > old = mb->stmt; > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_recycle.mx, MonetDB_5-6, 1.75.2.2, 1.75.2.3
by Romulo Goncalves 02 Jun '08

02 Jun '08

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv28477 > > Modified Files: > Tag: MonetDB_5-6 > mal_recycle.mx > Log Message: > Protect the garbage collector against removing NULL bats. The recycler is feature and it is used in the current branch. It is not tested in the stable and it might stop to be compiled in the stable. All the implementation related to the recycler should be checked in at the current one and not in the stable one. This avoids problems in the stable and saves time in the recycler development (I do not need to wait for the propagations to have an actualized code)... I am going to revert this change, and do it in the current Note: Martin is aware of this... :) Regards, Romulo > > > U mal_recycle.mx > Index: mal_recycle.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_recycle.mx,v > retrieving revision 1.75.2.2 > retrieving revision 1.75.2.3 > diff -u -d -r1.75.2.2 -r1.75.2.3 > --- mal_recycle.mx 31 May 2008 20:06:48 -0000 1.75.2.2 > +++ mal_recycle.mx 2 Jun 2008 17:55:04 -0000 1.75.2.3 > @@ -273,7 +273,8 @@ > for(j=0; j< q->argc; j++){ > v= &getVarConstant(mb,getArg(q,j)); > if(getArgType(mb,q,j)==TYPE_bat || isaBatType(getArgType(mb, q,j)) ){ > - BBPdecref(ABS(v->val.bval), TRUE); > + if( v->val.bval || !BBP_lrefs(v->val.bval)) > + BBPdecref(ABS(v->val.bval), TRUE); > v->val.bval= 0; > } > if( v->vtype == TYPE_str) { > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/BugTracker/Tests authorization.SF-1430616-drop_user.sql, SQL_2-24, NONE, 1.1.2.1 authorization.SF-1430616.SQL.bat, SQL_2-24, 1.3, 1.3.4.1 authorization.SF-1430616.SQL.sh, SQL_2-24, 1.2, 1.2.4.1
by Stefan Manegold 02 Jun '08

02 Jun '08

On Mon, Jun 02, 2008 at 03:17:02PM +0000, Romulo Goncalves wrote: > Update of /cvsroot/monetdb/sql/src/test/BugTracker/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv8577 > > Modified Files: > Tag: SQL_2-24 > authorization.SF-1430616.SQL.bat > authorization.SF-1430616.SQL.sh > Added Files: > Tag: SQL_2-24 > authorization.SF-1430616-drop_user.sql > Log Message: > If you create the user (voc) do not forget to drop it. > > This was the reason to break some following tests which were trying to create user voc... That's indeed the better solution than my renaming user "voc" to "voc2" in the later test. Thanks! --- I was simply "too lazy" to check all the subsequent test that output some catalog statistics, and hence, might need to have their output approved, too. Did you check that? Stefan > Thanks Stefan for dectecting the problem... > > > --- NEW FILE: authorization.SF-1430616-drop_user.sql --- > START TRANSACTION; > > DROP USER "voc"; > DROP SCHEMA "voc"; > > commit; > > U authorization.SF-1430616.SQL.sh > Index: authorization.SF-1430616.SQL.sh > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugTracker/Tests/authorization.SF-1430616.SQL.sh,v > retrieving revision 1.2 > retrieving revision 1.2.4.1 > diff -u -d -r1.2 -r1.2.4.1 > --- authorization.SF-1430616.SQL.sh 29 Oct 2007 20:11:51 -0000 1.2 > +++ authorization.SF-1430616.SQL.sh 2 Jun 2008 15:16:59 -0000 1.2.4.1 > @@ -17,4 +17,11 @@ > > Mlog -x "$MTIMEOUT mjclient -h $HOST -p $MAPIPORT -d ${TSTDB} -q" > > +cat << EOF > .monetdb > +user=monetdb > +password=monetdb > +EOF > + > +Mlog -x "$MTIMEOUT mjclient -h $HOST -p $MAPIPORT -d ${TSTDB} -f \"$RELSRCDIR/authorization.SF-1430616-drop_user.sql\"" > + > rm -f .monetdb > > U authorization.SF-1430616.SQL.bat > Index: authorization.SF-1430616.SQL.bat > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugTracker/Tests/authorization.SF-1430616.SQL.bat,v > retrieving revision 1.3 > retrieving revision 1.3.4.1 > diff -u -d -r1.3 -r1.3.4.1 > --- authorization.SF-1430616.SQL.bat 29 Oct 2007 20:11:51 -0000 1.3 > +++ authorization.SF-1430616.SQL.bat 2 Jun 2008 15:16:58 -0000 1.3.4.1 > @@ -15,4 +15,9 @@ > > call mjclient -h %HOST% -p %MAPIPORT% -d %TSTDB% -q > > +echo user=monetdb> .monetdb > +echo password=monetdb>> .monetdb > + > +call mjclient -h %HOST% -p %MAPIPORT% -d %TSTDB% -f "%RELSRCDIR%\authorization.SF-1430616-drop_user.sql" > + > @del .monetdb > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/algebra/opt opt_join_pd.c, XQuery_0-24, 1.46.2.1, 1.46.2.2
by Stefan Manegold 02 Jun '08

02 Jun '08

Jan, Sjoerd, I'll take care of these. Sjoerd, please do not propagate until I'm done. Stefan On Mon, Jun 02, 2008 at 08:44:32AM +0000, Jan Rittinger wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/algebra/opt > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv1643/algebra/opt > > Modified Files: > Tag: XQuery_0-24 > opt_join_pd.c > Log Message: > -- circumvent bugs #1981491 and #1908822 for the release. > > This 'hack' does not fix the real problem (that we have to get rid > of bit-encoded column names), but just disables some optimizations > in case we might get into trouble. Thus we can apply less rewrites > and get 'suboptimal' plans. > > THIS IS A RELEASE ONLY CHECKIN --- PLEASE DO NOT PROPAGATE! > > > U opt_join_pd.c > Index: opt_join_pd.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/algebra/opt/opt_join_pd.c,v > retrieving revision 1.46.2.1 > retrieving revision 1.46.2.2 > diff -u -d -r1.46.2.1 -r1.46.2.2 > --- opt_join_pd.c 28 May 2008 11:37:25 -0000 1.46.2.1 > +++ opt_join_pd.c 2 Jun 2008 08:44:30 -0000 1.46.2.2 > @@ -580,6 +580,11 @@ > } break; > > case la_project: > + /* disable rewrite in case we might get wider relations > + than we can map back to the bit-encoded (ori) column > + names */ > + if (PFarray_last (rproj) + L(lp)->schema.count < 30) > + > /* Arbitrary projections are pushed through the join operator > by: > 1.) merging the projection lists of the projection > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 1

Re: [Monetdb-developers] [Monetdb-sql-checkins] geom/src/monetdb5 geom.mx, Geom_0-4, 1.13.4.1, 1.13.4.2
by Sjoerd Mullender 30 May '08

30 May '08

Stefan Manegold wrote: > Update of /cvsroot/monetdb/geom/src/monetdb5 > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv30443/src/monetdb5 > > Modified Files: > Tag: Geom_0-4 > geom.mx > Log Message: > > attempt to fix compilation problem > " > geom/src/monetdb5/geom.mx:791: error: initialization discards qualifiers from pointer target type > " > with gcc 4.3.0 on Gentoo > > > U geom.mx > Index: geom.mx > =================================================================== > RCS file: /cvsroot/monetdb/geom/src/monetdb5/geom.mx,v > retrieving revision 1.13.4.1 > retrieving revision 1.13.4.2 > diff -u -d -r1.13.4.1 -r1.13.4.2 > --- geom.mx 28 May 2008 09:43:48 -0000 1.13.4.1 > +++ geom.mx 30 May 2008 04:58:29 -0000 1.13.4.2 > @@ -788,7 +788,7 @@ > GEOSGeom geosGeometry = wkb2geos(*geom); > > if (geosGeometry) { > - GEOSCoordSeq gcs = GEOSGeom_getCoordSeq(geosGeometry); > + GEOSCoordSeq gcs = (GEOSCoordSeq)GEOSGeom_getCoordSeq(geosGeometry); > > /* we could also check if geom is a > LineString, LinearRing or Point */ > > It seems to me a better fix is to keep the qualifier and to declare gcs as const. -- Sjoerd Mullender

1 0

[Monetdb-developers] HOST_NAME_MAX op mac os 10.5
by Ying Zhang 29 May '08

29 May '08

Dear all, I use HOST_NAME_MAX in my code. Compiling the code on a CWI machine does not have any problem, however, compilation on Mac OS 10.5 complains that HOST_NAME_MAX is not defined. fgrep on mac 10.5 says: $fgrep -r HOST_NAME_MAX /usr/include/* /usr/include/limits.h:#define _POSIX_HOST_NAME_MAX 255 /usr/include/unistd.h:#define _SC_HOST_NAME_MAX 72 How (and where) can I add proper check to define HOST_NAME_MAX for darwin 9? Thanks! Jennie

3 3

Re: [Monetdb-developers] Error in compilation of PF-ROX
by Peter Boncz 28 May '08

28 May '08

Hi Stefan, Riham PF_ROX has the new indexing code, that Lefteris has tried to describe in his recent paper. This code was only checked for correct operation from MIL in the code generated by the pf-rox java program of Riham. The correct operation of the new indexing code when activated automatically from mps was not tested yet. It may well be the source of trouble. i will pick this up once I have some time again (..). Peter Dear all, I checked all (modified) files in the PF_ROX branch by hand, and all but runtime/pathfinder.mx & runtime/pf_support.mx (this means in particular the compiler/mil/milprint_summer.c I fixed the other day) are OK, i.e., they contain all (intended) changes applied in the PF_ROX branch as well as all chnages that were made on the development trunk since the creation of the PF_ROX branch (and hence properly propagated). For runtime/pathfinder.mx & runtime/pf_support.mx, I am not yet sure, whether everything is OK. There have been numerous "interleaved" (and partly "redundant" or "conflicting") changes both genuine on the PF_ROX branch and propagated from the development trunk, and the (very tedious) analysis requires more time than I could spent today. More to come "soon", I hope ... Unfortunately, while only 50 (100) tests fail in the new stable branch (and in the development trunk), more than 500 tests seem to fail in the PF_ROX branch --- also this requires "some" more ananlysis ... Stefan

1 0