Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder configure.ag, 1.91, 1.92
Hi Jan, As I have indicated, use of the Boehm garbage collector inside the MonetDB process is not really an option. If algebra plans can only be compiled with Boehm, than it will become very difficult to use pathfinder inside MonetDB/XQuery. Did you try the option with memory sandboxes that at certain critical stages can be cleared? Peter -----Original Message----- From: monetdb-pf-checkins-bounces@lists.sourceforge.net [mailto:monetdb-pf-checkins-bounces@lists.sourceforge.net]On Behalf Of Jan Rittinger Sent: 22 January 2007 12:26 To: monetdb-pf-checkins@lists.sourceforge.net Subject: [Monetdb-pf-checkins] pathfinder configure.ag,1.91,1.92 Update of /cvsroot/monetdb/pathfinder In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv7979 Modified Files: configure.ag Log Message: -- Re-integrated Hans Boehms garbage collection library. * This was necessary because more complex queries did run out of memory. (Before we did allocate memory and only freed it at the end. Snapshot based freeing -- meaning after a full tree copy was made -- did help but was a lot slower than using the garbage collection library.) * The old memory allocation (without garbage collection) is still in use whenever pathfinder is used in embedded mode (e.g. if the xquery() command in MonetDB is used). * If the garbage collection library is not detected during configure we avoid to generate an error and use the old memory allocation instead. * The drawback of this change is that the compilation of simple queries is now lot slower (e.g. XMark Q6 takes 400ms to compile). Index: configure.ag =================================================================== RCS file: /cvsroot/monetdb/pathfinder/configure.ag,v retrieving revision 1.91 retrieving revision 1.92 diff -u -d -r1.91 -r1.92 --- configure.ag 21 Jan 2007 16:01:38 -0000 1.91 +++ configure.ag 22 Jan 2007 11:25:59 -0000 1.92 @@ -334,6 +334,70 @@ AC_SUBST(LIBXML2_LDADD, $LIBXML2_LDADD) AM_CONDITIONAL(HAVE_LIBXML2, test x"$have_libxml2" != xno) +dnl See if we have the Hans Boehm garbage collector available +dnl +dnl First allow the user to specify a location on the command line. +dnl Further down see if we find the library and abort if not. +have_gc="auto" +AC_ARG_WITH(gc, + [ --with-gc=DIR Boehm garbage collector library is installed in DIR], + [have_gc="$withval"], [have_gc="auto"]) + +if test "x$have_gc" != xauto; then + LIBGC_CFLAGS="-I$have_gc/include" + LIBGC_LIBS="-L$have_gc/lib" +fi + +if test "x$have_gc" = xauto; then + LIBGC_CFLAGS="" + LIBGC_LIBS="" +fi + +save_CPPFLAGS="$CPPFLAGS" +save_LDFLAGS="$LDFLAGS" +save_LIBS="$LIBS" +CPPFLAGS="$CPPFLAGS $LIBGC_CFLAGS" +LDFLAGS="$LDFLAGS $LIBGC_LIBS" +LIBS="" + +dnl If we have the dl and/or the pthread library, use it +dnl (dl and/or pthread are required for the garbage collection +dnl library on some systems.) +LIBGC_LDADD="" +AC_CHECK_LIB(dl, dlopen, [LIBGC_LDADD="$LIBGC_LDADD -ldl"]) +AC_CHECK_LIB(pthread, pthread_create, [LIBGC_LDADD="$LIBGC_LDADD -lpthread"]) + +AC_CHECK_HEADER([gc.h], + [ dnl See if we can link in the gc library. + dnl If we have found the dl and/or pthread library as well, also use it. + dnl If the gc library was not found, clear LIBGC_LDADD. + AC_CHECK_LIB(gc, GC_malloc, + [LIBGC_LDADD="$LIBGC_LDADD -lgc"], + [LIBGC_LDADD=""], [$LIBGC_LDADD])], + [ LIBGC_LDADD=""]) + + +LIBS="$save_LIBS" +LDFLAGS="$save_LDFLAGS" +CPPFLAGS="$save_CPPFLAGS" + +if test "x$LIBGC_LDADD" = "x"; then + AC_MSG_WARN([Unable to find the Hans Boehm garbage collector. + Please go get it from + http://www.hpl.hp.com/personal/Hans_Boehm/gc/ + (Compilation will use memory allocation only strategy instead.)]) + AC_DEFINE(HAVE_GC, 0, [Define if you use garbage collection]) +fi +if test "x$LIBGC_LDADD" != "x"; then + AC_DEFINE(HAVE_GC, 1, [Define if you use garbage collection]) +fi + + +dnl Propagate the variable to the Makefiles +AC_SUBST(LIBGC_CFLAGS, $LIBGC_CFLAGS) +AC_SUBST(LIBGC_LIBS, $LIBGC_LIBS) +AC_SUBST(LIBGC_LDADD, $LIBGC_LDADD) + # have_pftijah=no # AM_CONDITIONAL(HAVE_PFTIJAH, test "x$have_pftijah" == xyes) ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Monetdb-pf-checkins mailing list Monetdb-pf-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
On 01/22/2007 01:43 PM, Peter Boncz wrote with possible deletions:
Hi Jan,
As I have indicated, use of the Boehm garbage collector inside the MonetDB process is not really an option.
If algebra plans can only be compiled with Boehm, than it will become very difficult to use pathfinder inside MonetDB/XQuery.
As I mentioned the embedded variant uses the old memory allocation routine. It just will run out of memory for more complex queries. (But applying less optimization phases will certainly avoid many memory allocation problems).
Did you try the option with memory sandboxes that at certain critical stages can be cleared?
Yes, of course. In the check-in message I mentioned that the sandbox idea works but is way slower than the Boehm GC (7min vs 3min -- and yes I know it takes some time to compile the query :)). Furthermore it requires a lot of changes in various places to ensure that we address the correct sandbox and to make sure that we do not throw away too much/less. This was one of the main reasons why we decided to use the garbage collector instead. Jan
Peter
-----Original Message----- From: monetdb-pf-checkins-bounces@lists.sourceforge.net [mailto:monetdb-pf-checkins-bounces@lists.sourceforge.net]On Behalf Of Jan Rittinger Sent: 22 January 2007 12:26 To: monetdb-pf-checkins@lists.sourceforge.net Subject: [Monetdb-pf-checkins] pathfinder configure.ag,1.91,1.92
Update of /cvsroot/monetdb/pathfinder In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv7979
Modified Files: configure.ag Log Message: -- Re-integrated Hans Boehms garbage collection library.
* This was necessary because more complex queries did run out of memory. (Before we did allocate memory and only freed it at the end. Snapshot based freeing -- meaning after a full tree copy was made -- did help but was a lot slower than using the garbage collection library.)
* The old memory allocation (without garbage collection) is still in use whenever pathfinder is used in embedded mode (e.g. if the xquery() command in MonetDB is used).
* If the garbage collection library is not detected during configure we avoid to generate an error and use the old memory allocation instead.
* The drawback of this change is that the compilation of simple queries is now lot slower (e.g. XMark Q6 takes 400ms to compile).
Index: configure.ag =================================================================== RCS file: /cvsroot/monetdb/pathfinder/configure.ag,v retrieving revision 1.91 retrieving revision 1.92 diff -u -d -r1.91 -r1.92 --- configure.ag 21 Jan 2007 16:01:38 -0000 1.91 +++ configure.ag 22 Jan 2007 11:25:59 -0000 1.92 @@ -334,6 +334,70 @@ AC_SUBST(LIBXML2_LDADD, $LIBXML2_LDADD) AM_CONDITIONAL(HAVE_LIBXML2, test x"$have_libxml2" != xno)
+dnl See if we have the Hans Boehm garbage collector available +dnl +dnl First allow the user to specify a location on the command line. +dnl Further down see if we find the library and abort if not. +have_gc="auto" +AC_ARG_WITH(gc, + [ --with-gc=DIR Boehm garbage collector library is installed in DIR], + [have_gc="$withval"], [have_gc="auto"]) + +if test "x$have_gc" != xauto; then + LIBGC_CFLAGS="-I$have_gc/include" + LIBGC_LIBS="-L$have_gc/lib" +fi + +if test "x$have_gc" = xauto; then + LIBGC_CFLAGS="" + LIBGC_LIBS="" +fi + +save_CPPFLAGS="$CPPFLAGS" +save_LDFLAGS="$LDFLAGS" +save_LIBS="$LIBS" +CPPFLAGS="$CPPFLAGS $LIBGC_CFLAGS" +LDFLAGS="$LDFLAGS $LIBGC_LIBS" +LIBS="" + +dnl If we have the dl and/or the pthread library, use it +dnl (dl and/or pthread are required for the garbage collection +dnl library on some systems.) +LIBGC_LDADD="" +AC_CHECK_LIB(dl, dlopen, [LIBGC_LDADD="$LIBGC_LDADD -ldl"]) +AC_CHECK_LIB(pthread, pthread_create, [LIBGC_LDADD="$LIBGC_LDADD -lpthread"]) + +AC_CHECK_HEADER([gc.h], + [ dnl See if we can link in the gc library. + dnl If we have found the dl and/or pthread library as well, also use it. + dnl If the gc library was not found, clear LIBGC_LDADD. + AC_CHECK_LIB(gc, GC_malloc, + [LIBGC_LDADD="$LIBGC_LDADD -lgc"], + [LIBGC_LDADD=""], [$LIBGC_LDADD])], + [ LIBGC_LDADD=""]) + + +LIBS="$save_LIBS" +LDFLAGS="$save_LDFLAGS" +CPPFLAGS="$save_CPPFLAGS" + +if test "x$LIBGC_LDADD" = "x"; then + AC_MSG_WARN([Unable to find the Hans Boehm garbage collector. + Please go get it from + http://www.hpl.hp.com/personal/Hans_Boehm/gc/ + (Compilation will use memory allocation only strategy instead.)]) + AC_DEFINE(HAVE_GC, 0, [Define if you use garbage collection]) +fi +if test "x$LIBGC_LDADD" != "x"; then + AC_DEFINE(HAVE_GC, 1, [Define if you use garbage collection]) +fi + + +dnl Propagate the variable to the Makefiles +AC_SUBST(LIBGC_CFLAGS, $LIBGC_CFLAGS) +AC_SUBST(LIBGC_LIBS, $LIBGC_LIBS) +AC_SUBST(LIBGC_LDADD, $LIBGC_LDADD) + # have_pftijah=no # AM_CONDITIONAL(HAVE_PFTIJAH, test "x$have_pftijah" == xyes)
------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Monetdb-pf-checkins mailing list Monetdb-pf-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- Jan Rittinger Database Systems Technische Universität München (Germany) http://www-db.in.tum.de/~rittinge/
Hi Jan, Thanks for the clarifications. However, from what you tell me know, I do wonder whether it would not be a better idea to go for the sandbox approach. For me, it is not an issue of performance, but of dependability. Given this choice, maybe we should exctually decouple the pathfinder compiler altogether from the runtime, and fork off a process. That also resolves the MT safeness issue. Peter -----Original Message----- From: Jan Rittinger [mailto:rittinge@in.tum.de] Sent: 22 January 2007 14:04 To: Peter Boncz Cc: monetdb-developers@lists.sourceforge.net Subject: Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder configure.ag, 1.91, 1.92 On 01/22/2007 01:43 PM, Peter Boncz wrote with possible deletions:
Hi Jan,
As I have indicated, use of the Boehm garbage collector inside the MonetDB process is not really an option.
If algebra plans can only be compiled with Boehm, than it will become very difficult to use pathfinder inside MonetDB/XQuery.
As I mentioned the embedded variant uses the old memory allocation routine. It just will run out of memory for more complex queries. (But applying less optimization phases will certainly avoid many memory allocation problems).
Did you try the option with memory sandboxes that at certain critical
stages
can be cleared?
Yes, of course. In the check-in message I mentioned that the sandbox idea works but is way slower than the Boehm GC (7min vs 3min -- and yes I know it takes some time to compile the query :)). Furthermore it requires a lot of changes in various places to ensure that we address the correct sandbox and to make sure that we do not throw away too much/less. This was one of the main reasons why we decided to use the garbage collector instead. Jan
Peter
-----Original Message----- From: monetdb-pf-checkins-bounces@lists.sourceforge.net [mailto:monetdb-pf-checkins-bounces@lists.sourceforge.net]On Behalf Of Jan Rittinger Sent: 22 January 2007 12:26 To: monetdb-pf-checkins@lists.sourceforge.net Subject: [Monetdb-pf-checkins] pathfinder configure.ag,1.91,1.92
Update of /cvsroot/monetdb/pathfinder In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv7979
Modified Files: configure.ag Log Message: -- Re-integrated Hans Boehms garbage collection library.
* This was necessary because more complex queries did run out of
memory.
(Before we did allocate memory and only freed it at the end. Snapshot based freeing -- meaning after a full tree copy was made -- did help but was a lot slower than using the garbage collection library.)
* The old memory allocation (without garbage collection) is still in
use
whenever pathfinder is used in embedded mode (e.g. if the xquery() command in MonetDB is used).
* If the garbage collection library is not detected during configure we avoid to generate an error and use the old memory allocation instead.
* The drawback of this change is that the compilation of simple queries is now lot slower (e.g. XMark Q6 takes 400ms to compile).
Index: configure.ag =================================================================== RCS file: /cvsroot/monetdb/pathfinder/configure.ag,v retrieving revision 1.91 retrieving revision 1.92 diff -u -d -r1.91 -r1.92 --- configure.ag 21 Jan 2007 16:01:38 -0000 1.91 +++ configure.ag 22 Jan 2007 11:25:59 -0000 1.92 @@ -334,6 +334,70 @@ AC_SUBST(LIBXML2_LDADD, $LIBXML2_LDADD) AM_CONDITIONAL(HAVE_LIBXML2, test x"$have_libxml2" != xno)
+dnl See if we have the Hans Boehm garbage collector available +dnl +dnl First allow the user to specify a location on the command line. +dnl Further down see if we find the library and abort if not. +have_gc="auto" +AC_ARG_WITH(gc, + [ --with-gc=DIR Boehm garbage collector library is
installed
in DIR], + [have_gc="$withval"], [have_gc="auto"]) + +if test "x$have_gc" != xauto; then + LIBGC_CFLAGS="-I$have_gc/include" + LIBGC_LIBS="-L$have_gc/lib" +fi + +if test "x$have_gc" = xauto; then + LIBGC_CFLAGS="" + LIBGC_LIBS="" +fi + +save_CPPFLAGS="$CPPFLAGS" +save_LDFLAGS="$LDFLAGS" +save_LIBS="$LIBS" +CPPFLAGS="$CPPFLAGS $LIBGC_CFLAGS" +LDFLAGS="$LDFLAGS $LIBGC_LIBS" +LIBS="" + +dnl If we have the dl and/or the pthread library, use it +dnl (dl and/or pthread are required for the garbage collection +dnl library on some systems.) +LIBGC_LDADD="" +AC_CHECK_LIB(dl, dlopen, [LIBGC_LDADD="$LIBGC_LDADD -ldl"]) +AC_CHECK_LIB(pthread, pthread_create, [LIBGC_LDADD="$LIBGC_LDADD -lpthread"]) + +AC_CHECK_HEADER([gc.h], + [ dnl See if we can link in the gc library. + dnl If we have found the dl and/or pthread library as well, also use it. + dnl If the gc library was not found, clear LIBGC_LDADD. + AC_CHECK_LIB(gc, GC_malloc, + [LIBGC_LDADD="$LIBGC_LDADD -lgc"], + [LIBGC_LDADD=""], [$LIBGC_LDADD])], + [ LIBGC_LDADD=""]) + + +LIBS="$save_LIBS" +LDFLAGS="$save_LDFLAGS" +CPPFLAGS="$save_CPPFLAGS" + +if test "x$LIBGC_LDADD" = "x"; then + AC_MSG_WARN([Unable to find the Hans Boehm garbage collector. + Please go get it from + http://www.hpl.hp.com/personal/Hans_Boehm/gc/ + (Compilation will use memory allocation only strategy instead.)]) + AC_DEFINE(HAVE_GC, 0, [Define if you use garbage collection]) +fi +if test "x$LIBGC_LDADD" != "x"; then + AC_DEFINE(HAVE_GC, 1, [Define if you use garbage collection]) +fi + + +dnl Propagate the variable to the Makefiles +AC_SUBST(LIBGC_CFLAGS, $LIBGC_CFLAGS) +AC_SUBST(LIBGC_LIBS, $LIBGC_LIBS) +AC_SUBST(LIBGC_LDADD, $LIBGC_LDADD) + # have_pftijah=no # AM_CONDITIONAL(HAVE_PFTIJAH, test "x$have_pftijah" == xyes)
------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Monetdb-pf-checkins mailing list Monetdb-pf-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- Jan Rittinger Database Systems Technische Universität München (Germany) http://www-db.in.tum.de/~rittinge/
participants (2)
-
Jan Rittinger -
Peter Boncz