- developers-list - monetdb.org

[Monetdb-developers] MAPI cache bug: explained
by Stefan de Konink 05 Jun '08

05 Jun '08

Hi, Already two days I am very annoyed with the MAPI interface, having 'random' fallout in mapi_fetch_field. With high confidence of trust in my own code I decided to walk the road of debugging the Mapiclient library and I'm reporting here. Please point your editors at clients/src/mapilib/Mapi.mx Around line 4800 you will find the function mapi_fetch_row. This is a public function and gets borked randomly but there seems to be a correlation between the amount of results fetched, lets move on. I was able to find the source of the problem in the while loop, where mapi_fetch_line is accessed. This function is the first place this function can return 0. This seems to be consistent with my logs. So what is this mapi_fetch_line all about? Around line 4260, you can find it. The idea of this function would be not to return NULL. After some debugging it seems that the function on the 'bug' occasion always returns 0. Why? If we explore the if statement something interesting can be found: mapi_fetch_line bypass: 709 + 91 < 800 = 0 result->cache.first + result->cache.tuplecount < result->row_count That seems to make sense, but it also prevents the mapi_fetch_line function to be processed. So what is not doing its job? The logical answer would be the function above mapi_fetch_line_internal. In this cute function there is an interesting check: if (mid->active != hdl || hdl->needmore) I was a bit distracted here, but when I explicitly have set a check on both constraints it seems the mid->active != hdl is the failure here. And I guessed it would be possible that mid->active (or actually hdl->active) was in fact equal to NULL. When I read about the MapiHdl active; MapiHdl active; /* set when not all rows have been received */ Isn't it strange that needmore is actually 0, but we would still require an active handle? I thought this would suggest the replacement of || with &&, but even with the assert in read_into_cache removed, this comes out with even MORE failed data. I hope I have outlined the problem in sufficient details so that one of the core developers can look where possible failures of hdl->mid->active takes place. If someone wants to see the live failures he is welcome to ask a private login. Yours Sincerely, Stefan de Konink

2 1

[Monetdb-developers] BAT-leak in gdk_logger.mx ?
by Stefan Manegold 04 Jun '08

04 Jun '08

Niels, Sjoerd, Peter, after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&group_id=5… I found the --- or at least one possible --- cause for this bug. In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents. Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()). However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown. The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm. Thank you very much in advance! Stefan ======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) { + BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE); } ======== -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 4

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/BugDay_2005-10-06_2.9.3/Tests All, SQL_2-24, 1.10, 1.10.4.1 having.SF-922614.stable.out, SQL_2-24, 1.3, 1.3.14.1
by Stefan Manegold 03 Jun '08

03 Jun '08

On Tue, Jun 03, 2008 at 07:03:54AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/sql/src/test/BugDay_2005-10-06_2.9.3/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv28722 > > Modified Files: > Tag: SQL_2-24 > All having.SF-922614.stable.out > Log Message: > approved new output > > > U All > Index: All > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugDay_2005-10-06_2.9.3/Tests/All,v > retrieving revision 1.10 > retrieving revision 1.10.4.1 > diff -u -d -r1.10 -r1.10.4.1 > --- All 11 Dec 2007 22:19:06 -0000 1.10 > +++ All 3 Jun 2008 07:03:52 -0000 1.10.4.1 > @@ -1,7 +1,7 @@ > jdbc_crashes_on_getTables.SF_944316 > variable_arg_substring.SF_942981 > UNION_JOIN_vs_plain_set_UNION-2.SF-920585 > -huge_expression_and_column_name.SF-921173 > +#huge_expression_and_column_name.SF-921173 will not be fixed in the MonetDB 5.6.0, SQL 2.24 release > CrashMe_SQL_server_crash-2.SF-921673 > CrashMe_SQL_server_crash-1.SF-921673 > inserts_with_defaults.SF-921977 I just prevented the propagation of this change ;-) Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/snodgrass/Tests All, SQL_2-24, 1.4, 1.4.10.1
by Stefan Manegold 03 Jun '08

03 Jun '08

On Tue, Jun 03, 2008 at 07:13:37AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/sql/src/test/snodgrass/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv559/src/test/snodgrass/Tests > > Modified Files: > Tag: SQL_2-24 > All > Log Message: > disabled test (will not be fixed in this release) > > THIS SHOULD NOT BE PROPAGATED!! propagation has been prevented. Stefan > U All > Index: All > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/snodgrass/Tests/All,v > retrieving revision 1.4 > retrieving revision 1.4.10.1 > diff -u -d -r1.4 -r1.4.10.1 > --- All 25 Jan 2007 01:28:36 -0000 1.4 > +++ All 3 Jun 2008 07:13:35 -0000 1.4.10.1 > @@ -1 +1 @@ > -cast_select > +#cast_select will not be fixed in the MonetDB 5.6.0, SQL 2.24 release > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_recycler.mx, MonetDB_5-6, 1.23.2.1, 1.23.2.2
by Romulo Goncalves 02 Jun '08

02 Jun '08

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv24665 > > Modified Files: > Tag: MonetDB_5-6 > opt_recycler.mx > Log Message: > Since we may introduce new instructions and variables, > the recycled[] object may have been too small. The recycler is feature and it is used in the current branch. It is not tested in the stable and it might stop to be compiled in the stable. All the implementation related to the recycler should be checked in at the current one and not in the stable one. This avoids problems in the stable and saves time in the recycler development (I do not need to wait for the propagations to have an actualized code)... I am going to revert this change, and do it in the current Note: Martin is aware of this... :) Regards, Romulo > > > U opt_recycler.mx > Index: opt_recycler.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_recycler.mx,v > retrieving revision 1.23.2.1 > retrieving revision 1.23.2.2 > diff -u -d -r1.23.2.1 -r1.23.2.2 > --- opt_recycler.mx 31 May 2008 16:29:54 -0000 1.23.2.1 > +++ opt_recycler.mx 2 Jun 2008 20:26:08 -0000 1.23.2.2 > @@ -107,7 +107,8 @@ > > (void) cntxt; > (void) stk; > - recycled= GDKzalloc(sizeof(char)*mb->vtop); > + /* watch out, instructions may introduce new variables */ > + recycled= GDKzalloc(sizeof(char)*mb->vtop*2); > setLifespan(mb,span); > limit= mb->stop; > old = mb->stmt; > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_recycle.mx, MonetDB_5-6, 1.75.2.2, 1.75.2.3
by Romulo Goncalves 02 Jun '08

02 Jun '08

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv28477 > > Modified Files: > Tag: MonetDB_5-6 > mal_recycle.mx > Log Message: > Protect the garbage collector against removing NULL bats. The recycler is feature and it is used in the current branch. It is not tested in the stable and it might stop to be compiled in the stable. All the implementation related to the recycler should be checked in at the current one and not in the stable one. This avoids problems in the stable and saves time in the recycler development (I do not need to wait for the propagations to have an actualized code)... I am going to revert this change, and do it in the current Note: Martin is aware of this... :) Regards, Romulo > > > U mal_recycle.mx > Index: mal_recycle.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_recycle.mx,v > retrieving revision 1.75.2.2 > retrieving revision 1.75.2.3 > diff -u -d -r1.75.2.2 -r1.75.2.3 > --- mal_recycle.mx 31 May 2008 20:06:48 -0000 1.75.2.2 > +++ mal_recycle.mx 2 Jun 2008 17:55:04 -0000 1.75.2.3 > @@ -273,7 +273,8 @@ > for(j=0; j< q->argc; j++){ > v= &getVarConstant(mb,getArg(q,j)); > if(getArgType(mb,q,j)==TYPE_bat || isaBatType(getArgType(mb, q,j)) ){ > - BBPdecref(ABS(v->val.bval), TRUE); > + if( v->val.bval || !BBP_lrefs(v->val.bval)) > + BBPdecref(ABS(v->val.bval), TRUE); > v->val.bval= 0; > } > if( v->vtype == TYPE_str) { > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/BugTracker/Tests authorization.SF-1430616-drop_user.sql, SQL_2-24, NONE, 1.1.2.1 authorization.SF-1430616.SQL.bat, SQL_2-24, 1.3, 1.3.4.1 authorization.SF-1430616.SQL.sh, SQL_2-24, 1.2, 1.2.4.1
by Stefan Manegold 02 Jun '08

02 Jun '08

On Mon, Jun 02, 2008 at 03:17:02PM +0000, Romulo Goncalves wrote: > Update of /cvsroot/monetdb/sql/src/test/BugTracker/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv8577 > > Modified Files: > Tag: SQL_2-24 > authorization.SF-1430616.SQL.bat > authorization.SF-1430616.SQL.sh > Added Files: > Tag: SQL_2-24 > authorization.SF-1430616-drop_user.sql > Log Message: > If you create the user (voc) do not forget to drop it. > > This was the reason to break some following tests which were trying to create user voc... That's indeed the better solution than my renaming user "voc" to "voc2" in the later test. Thanks! --- I was simply "too lazy" to check all the subsequent test that output some catalog statistics, and hence, might need to have their output approved, too. Did you check that? Stefan > Thanks Stefan for dectecting the problem... > > > --- NEW FILE: authorization.SF-1430616-drop_user.sql --- > START TRANSACTION; > > DROP USER "voc"; > DROP SCHEMA "voc"; > > commit; > > U authorization.SF-1430616.SQL.sh > Index: authorization.SF-1430616.SQL.sh > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugTracker/Tests/authorization.SF-1430616.SQL.sh,v > retrieving revision 1.2 > retrieving revision 1.2.4.1 > diff -u -d -r1.2 -r1.2.4.1 > --- authorization.SF-1430616.SQL.sh 29 Oct 2007 20:11:51 -0000 1.2 > +++ authorization.SF-1430616.SQL.sh 2 Jun 2008 15:16:59 -0000 1.2.4.1 > @@ -17,4 +17,11 @@ > > Mlog -x "$MTIMEOUT mjclient -h $HOST -p $MAPIPORT -d ${TSTDB} -q" > > +cat << EOF > .monetdb > +user=monetdb > +password=monetdb > +EOF > + > +Mlog -x "$MTIMEOUT mjclient -h $HOST -p $MAPIPORT -d ${TSTDB} -f \"$RELSRCDIR/authorization.SF-1430616-drop_user.sql\"" > + > rm -f .monetdb > > U authorization.SF-1430616.SQL.bat > Index: authorization.SF-1430616.SQL.bat > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugTracker/Tests/authorization.SF-1430616.SQL.bat,v > retrieving revision 1.3 > retrieving revision 1.3.4.1 > diff -u -d -r1.3 -r1.3.4.1 > --- authorization.SF-1430616.SQL.bat 29 Oct 2007 20:11:51 -0000 1.3 > +++ authorization.SF-1430616.SQL.bat 2 Jun 2008 15:16:58 -0000 1.3.4.1 > @@ -15,4 +15,9 @@ > > call mjclient -h %HOST% -p %MAPIPORT% -d %TSTDB% -q > > +echo user=monetdb> .monetdb > +echo password=monetdb>> .monetdb > + > +call mjclient -h %HOST% -p %MAPIPORT% -d %TSTDB% -f "%RELSRCDIR%\authorization.SF-1430616-drop_user.sql" > + > @del .monetdb > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/algebra/opt opt_join_pd.c, XQuery_0-24, 1.46.2.1, 1.46.2.2
by Stefan Manegold 02 Jun '08

02 Jun '08

Jan, Sjoerd, I'll take care of these. Sjoerd, please do not propagate until I'm done. Stefan On Mon, Jun 02, 2008 at 08:44:32AM +0000, Jan Rittinger wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/algebra/opt > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv1643/algebra/opt > > Modified Files: > Tag: XQuery_0-24 > opt_join_pd.c > Log Message: > -- circumvent bugs #1981491 and #1908822 for the release. > > This 'hack' does not fix the real problem (that we have to get rid > of bit-encoded column names), but just disables some optimizations > in case we might get into trouble. Thus we can apply less rewrites > and get 'suboptimal' plans. > > THIS IS A RELEASE ONLY CHECKIN --- PLEASE DO NOT PROPAGATE! > > > U opt_join_pd.c > Index: opt_join_pd.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/algebra/opt/opt_join_pd.c,v > retrieving revision 1.46.2.1 > retrieving revision 1.46.2.2 > diff -u -d -r1.46.2.1 -r1.46.2.2 > --- opt_join_pd.c 28 May 2008 11:37:25 -0000 1.46.2.1 > +++ opt_join_pd.c 2 Jun 2008 08:44:30 -0000 1.46.2.2 > @@ -580,6 +580,11 @@ > } break; > > case la_project: > + /* disable rewrite in case we might get wider relations > + than we can map back to the bit-encoded (ori) column > + names */ > + if (PFarray_last (rproj) + L(lp)->schema.count < 30) > + > /* Arbitrary projections are pushed through the join operator > by: > 1.) merging the projection lists of the projection > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 1

Re: [Monetdb-developers] [Monetdb-sql-checkins] geom/src/monetdb5 geom.mx, Geom_0-4, 1.13.4.1, 1.13.4.2
by Sjoerd Mullender 30 May '08

30 May '08

Stefan Manegold wrote: > Update of /cvsroot/monetdb/geom/src/monetdb5 > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv30443/src/monetdb5 > > Modified Files: > Tag: Geom_0-4 > geom.mx > Log Message: > > attempt to fix compilation problem > " > geom/src/monetdb5/geom.mx:791: error: initialization discards qualifiers from pointer target type > " > with gcc 4.3.0 on Gentoo > > > U geom.mx > Index: geom.mx > =================================================================== > RCS file: /cvsroot/monetdb/geom/src/monetdb5/geom.mx,v > retrieving revision 1.13.4.1 > retrieving revision 1.13.4.2 > diff -u -d -r1.13.4.1 -r1.13.4.2 > --- geom.mx 28 May 2008 09:43:48 -0000 1.13.4.1 > +++ geom.mx 30 May 2008 04:58:29 -0000 1.13.4.2 > @@ -788,7 +788,7 @@ > GEOSGeom geosGeometry = wkb2geos(*geom); > > if (geosGeometry) { > - GEOSCoordSeq gcs = GEOSGeom_getCoordSeq(geosGeometry); > + GEOSCoordSeq gcs = (GEOSCoordSeq)GEOSGeom_getCoordSeq(geosGeometry); > > /* we could also check if geom is a > LineString, LinearRing or Point */ > > It seems to me a better fix is to keep the qualifier and to declare gcs as const. -- Sjoerd Mullender

1 0

[Monetdb-developers] HOST_NAME_MAX op mac os 10.5
by Ying Zhang 29 May '08

29 May '08

Dear all, I use HOST_NAME_MAX in my code. Compiling the code on a CWI machine does not have any problem, however, compilation on Mac OS 10.5 complains that HOST_NAME_MAX is not defined. fgrep on mac 10.5 says: $fgrep -r HOST_NAME_MAX /usr/include/* /usr/include/limits.h:#define _POSIX_HOST_NAME_MAX 255 /usr/include/unistd.h:#define _SC_HOST_NAME_MAX 72 How (and where) can I add proper check to define HOST_NAME_MAX for darwin 9? Thanks! Jennie

3 3