- developers-list - monetdb.org

Re: [Monetdb-developers] [Monetdb-checkins]MonetDB/src/gdkgdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
by Peter Boncz 11 Feb '08

11 Feb '08

Hi Stefan Thanks for your response. However, as I have said, but let me say it as clear as possble: I am *sure* there is a bug still unfound in the way re-allocations are done when mem_bigsize redirects it into virtual memory. For this reason, at this stage, with the goal of a stable release, it is just my recommendation *not* to re-enable it. In M5 nor in M5. Peter Dear all, On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote: > Hi Stefan, > > The pftijah bug was unrelated to vmalloc(), a relationship that I never > assumed. Indeed, that was (only) my mis-interpretation. Sorry. > The vmalloc() troubles occured, as you may recall, during the data > preparation of the sorted 100GB TPC-H. > > As it is entirely foreseeable that we at some point again will re-encounter an > OS with a malloc implementation that is prone to fragmentation, I'd keep the > vmalloc() code in for the moment (though in disabled state). Linux and Windows > do not need it. > > So, my advise is to for the monet let vmalloc() be disabled, until the issue > is properly investigated and a bugfix is found. Let's call it option (0). Ok. Just to sum up the current state. Please check, whether this is OK and consistent: a) Documentation in MonetDB4/conf/MonetDB.conf.in & MonetDB5/conf/monetdb5.conf.in says: " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 " b) Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says " @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. @item @code{gdk_vm_minsize}: column size above which memory mapped files are used always, e.g. @code{1749291171} " c) MonetDB/src/common/monet_options.mx sets " set[i].kind = opt_builtin; set[i].name = strdup("gdk_mem_bigsize"); set[i].value = strdup("262144"); " d) MonetDB/src/common/monet_options.py.in sets " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx gdk_mem_bigsize = '256K' gdk_vm_minsize = '128G' " e) GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets " if ((p = GDKgetenv("gdk_mem_bigsize"))) { /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ lng max_mem_bigsize = GDK_mem_maxsize/16; /* sanity check to avoid memory fragmentation */ GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10)); } " f) GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize g) HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* when using anonymous vm we malloc we need 64K chunks, also we * 20% extra malloc */ if (h->size > GDK_mem_bigsize) { h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; h->maxsize = (1 + (h->maxsize >> 16)) << 16; } " h) HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* extend a malloced heap, possibly switching over to file-mapped storage */ Heap bak = *h; int can_mmap = (h->filename && size >= GDK_mem_bigsize); int must_mmap = can_mmap && (size >= GDK_vm_minsize || (h->newstorage != STORE_MEM)); [...] if (can_mmap) { /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ if (size > h->maxsize) { h->maxsize = (size_t) ((double) size * BATMARGIN); } /* when using anonymous vm we malloc we need 64K chunks */ h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; [...] /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { char privext[PATHLENGTH], *of = h->filename; FILE *fp; h->filename = NULL; sprintf(privext, "%s.priv", ext); fp = GDKfilelocate(nme, "wb", privext); if (fp != NULL) { fclose(fp); /* a non-persistent heap: we create a .priv but *not* MMAP_PRIV !!! */ h->storage = STORE_MMAP; h->base = NULL; if (HEAPload(h, nme, privext, FALSE) >= 0) { memcpy(h->base, bak.base, bak.free); HEAPfree(&bak); return 0; } } GDKfree(of); } " i) MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize in MonetDB4/src/modules/plain/sys.mx j) both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however, only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize, while the MAL variants no longer do so: MonetDB4/src/modules/plain/sys.mx: " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } " MonetDB5/src/modules/kernel/status.mx " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); #endif GDK_mem_maxsize = sze; return GDK_SUCCEED; } int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); #endif GDK_vm_minsize = sze; return GDK_SUCCEED; } " If all this is considered OK and consistent, we're done. Otherwise, we need to do what ever is required to get the code and documentation consistent, preferably before the release. Stefan > Peter > > > > > > > > > > > > > > > > > Dear all, > > since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize > related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) > does not appear have any impact on this bug (just tested successfully both > with vmalloc() disabled and vmalloc() enabled), there are two ways to finish > this release (more or less) quickly: > > (1) finish the disabling of vmalloc() in all parts of the code as indicated > below (or better: remove all remains completely) --- this might need > some extra testing to ensure that it does not have any (significant) > impact on performance ... > > (2) for now (i.e., in the release branch) re-enable vmalloc(), and then > finish the removal of it (incl. extensive testing) in the development > trunk for th next release. > > Since I'm not completely aware of the impact of vmalloc(), I cannot really > make this decision --- the "save" back-up would IMHO be (2). > > Any other comments, opinions, suggestions, expertise? > > Thanks in advance! > > Stefan > > > On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote: > > Peter: Thanks for disabling mem_bigsize / vmalloc(). > > > > All: > > (a) > > Unfortunately, this does not (yet) solve > > [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. > > http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=5… > > > > But this appears to be more vm_minsize related than mem_bigsize related. > > > > > > (b) > > We should check all occurances of mem_bigsize and remove / "hide" those that > > have become irrelevant with the deactivation of mem_bigsize / vmalloc(): > > ======== > > $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize' > > -------- > > MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be > set/limited to > > MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize > (in bytes) will be mmaped anonymously > > MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 > > MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be > set/limited to > > MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize > (in bytes) will be mmaped anonymously > > MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 > > MonetDB/src/common/monet_options.mx:410: set[i].name = > strdup("gdk_mem_bigsize"); > > MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize & > gdk_vm_minsize will be set/limited to > > MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' > > MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename > && size >= GDK_mem_bigsize); > > MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size > after which we use VM rather than heap */ > > MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; > > MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous > virtual memory. The seamless > > MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_utils.mx:1529: if ((p = > GDKgetenv("gdk_mem_bigsize"))) { > > MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize = > GDK_mem_maxsize/16; > > MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t) > MIN(max_mem_bigsize, strtol(p, NULL, 10)); > > monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}: > minimum size for memory-mapped columns, e.g. @code{262144}. > > MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng = > get_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void = > set_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) > > MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) > > MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ > > MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); > > MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) > > MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); > > MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) > > MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); > > MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) > > MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; > > MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) > > MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ > > MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768, > sze); > > MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) > > MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); > > MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) > > MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); > > MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize = > MAX(GDK_mem_bigsize, sze); > > monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize", > "262144" > > ======== > > > > more tomorrow during MADAM ... > > > > Stefan > > > > > > On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote: > > > no consequences. > > > > > > > > > > > > > > > Peter, > > > > > > What about the consequences in: > > > build/MonetDB/src/gdk/gdk_heap.c > > > build/MonetDB5/src/modules/kernel/status.c > > > build/MonetDB/src/gdk/gdk.h > > > > > > Peter Boncz wrote: > > > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > > > > > > > Modified Files: > > > > Tag: MonetDB_1-22 > > > > gdk_utils.mx > > > > Log Message: > > > > disable use of vmalloc() > > > > > > > > > > > > > > > > Index: gdk_utils.mx > > > > =================================================================== > > > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > > > > retrieving revision 1.206 > > > > retrieving revision 1.206.2.1 > > > > diff -u -d -r1.206 -r1.206.2.1 > > > > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > > > > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > > > > @@ -1050,6 +1050,7 @@ > > > > #endif > > > > } > > > > size = (size + 7) & ~7; /* round up to a multiple of eight */ > > > > +#if 0 > > > > if (size > GDK_mem_bigsize) { > > > > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > > > > size_t newmax = MAX(*maxsize, newsize); > > > > @@ -1064,6 +1065,7 @@ > > > > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > > > > return (void *) s; > > > > } > > > > +#endif > > > > CHKMEM(size, 0); > > > > GDKmalloc_prefixsize(s, size); > > > > if (s == NULL) { > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > This SF.net email is sponsored by: Microsoft > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > _______________________________________________ > > > > Monetdb-checkins mailing list > > > > Monetdb-checkins(a)lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Microsoft > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > _______________________________________________ > > > Monetdb-developers mailing list > > > Monetdb-developers(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > > > > > > > > > > > > > > > > > -- > > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 2

[Monetdb-developers] mem_bigsize
by Peter Boncz 11 Feb '08

11 Feb '08

Hi Stefan, Thanks for so much Grundlichkeit. It is appreciated. It seems that the comment in the XQuery documentation describing mem_bigsize is not correct. Given the current disabled status of mem_bigsize I will remove it. The behavior to limit mem_bigsize to the values of both mem_maxsize and vm_minsize I did not recall; but it probably makes sense. The rest is.. indeed correct. The repitition of configuration values in the python and monet_options has always bothered me, but I recall there was some reason why it cannot be avoided (though not what it was). However, I have spent the entire night&day, re-thinking the mmap-heap situation; and there are still some problems. I think it can be fixed once and for all, but it requires abandoning the X.priv approach to STORE_PRIV copy-on-write heaps in favor of always mapping the X file and writing into X.new files (which take preference over X files if encountered). But that also affects the BBP, the commit protocol. Peter Dear all, On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote: > Hi Stefan, > > The pftijah bug was unrelated to vmalloc(), a relationship that I never > assumed. Indeed, that was (only) my mis-interpretation. Sorry. > The vmalloc() troubles occured, as you may recall, during the data > preparation of the sorted 100GB TPC-H. > > As it is entirely foreseeable that we at some point again will re-encounter an > OS with a malloc implementation that is prone to fragmentation, I'd keep the > vmalloc() code in for the moment (though in disabled state). Linux and Windows > do not need it. > > So, my advise is to for the monet let vmalloc() be disabled, until the issue > is properly investigated and a bugfix is found. Let's call it option (0). Ok. Just to sum up the current state. Please check, whether this is OK and consistent: a) Documentation in MonetDB4/conf/MonetDB.conf.in & MonetDB5/conf/monetdb5.conf.in says: " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously #gdk_mem_bigsize=262144 # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped; #gdk_vm_minsize=137438953472 " b) Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says " @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. @item @code{gdk_vm_minsize}: column size above which memory mapped files are used always, e.g. @code{1749291171} " c) MonetDB/src/common/monet_options.mx sets " set[i].kind = opt_builtin; set[i].name = strdup("gdk_mem_bigsize"); set[i].value = strdup("262144"); " d) MonetDB/src/common/monet_options.py.in sets " # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to # 1/2 of the physically available amount of main-memory # during start-up in src/tools/Mserver.mx gdk_mem_bigsize = '256K' gdk_vm_minsize = '128G' " e) GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets " if ((p = GDKgetenv("gdk_mem_bigsize"))) { /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ lng max_mem_bigsize = GDK_mem_maxsize/16; /* sanity check to avoid memory fragmentation */ GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10)); } " f) GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize g) HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* when using anonymous vm we malloc we need 64K chunks, also we * 20% extra malloc */ if (h->size > GDK_mem_bigsize) { h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; h->maxsize = (1 + (h->maxsize >> 16)) << 16; } " h) HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows: " /* extend a malloced heap, possibly switching over to file-mapped storage */ Heap bak = *h; int can_mmap = (h->filename && size >= GDK_mem_bigsize); int must_mmap = can_mmap && (size >= GDK_vm_minsize || (h->newstorage != STORE_MEM)); [...] if (can_mmap) { /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ if (size > h->maxsize) { h->maxsize = (size_t) ((double) size * BATMARGIN); } /* when using anonymous vm we malloc we need 64K chunks */ h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; [...] /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { char privext[PATHLENGTH], *of = h->filename; FILE *fp; h->filename = NULL; sprintf(privext, "%s.priv", ext); fp = GDKfilelocate(nme, "wb", privext); if (fp != NULL) { fclose(fp); /* a non-persistent heap: we create a .priv but *not* MMAP_PRIV !!! */ h->storage = STORE_MMAP; h->base = NULL; if (HEAPload(h, nme, privext, FALSE) >= 0) { memcpy(h->base, bak.base, bak.free); HEAPfree(&bak); return 0; } } GDKfree(of); } " i) MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize in MonetDB4/src/modules/plain/sys.mx j) both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however, only the MIL variants (still) related mem_maxsize & vm_minsize with mem_bigsize, while the MAL variants no longer do so: MonetDB4/src/modules/plain/sys.mx: " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); return GDK_SUCCEED; } " MonetDB5/src/modules/kernel/status.mx " int set_mem_maxsize(lng *num) { @:num2sze(mem_maxsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); #endif GDK_mem_maxsize = sze; return GDK_SUCCEED; } int set_vm_minsize(lng *num) { @:num2sze(vm_minsize)@ #if 0 if (sze < GDK_mem_bigsize) set_mem_bigsize(num); GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); #endif GDK_vm_minsize = sze; return GDK_SUCCEED; } " If all this is considered OK and consistent, we're done. Otherwise, we need to do what ever is required to get the code and documentation consistent, preferably before the release. Stefan > Peter > > > > > > > > > > > > > > > > > Dear all, > > since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize > related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) > does not appear have any impact on this bug (just tested successfully both > with vmalloc() disabled and vmalloc() enabled), there are two ways to finish > this release (more or less) quickly: > > (1) finish the disabling of vmalloc() in all parts of the code as indicated > below (or better: remove all remains completely) --- this might need > some extra testing to ensure that it does not have any (significant) > impact on performance ... > > (2) for now (i.e., in the release branch) re-enable vmalloc(), and then > finish the removal of it (incl. extensive testing) in the development > trunk for th next release. > > Since I'm not completely aware of the impact of vmalloc(), I cannot really > make this decision --- the "save" back-up would IMHO be (2). > > Any other comments, opinions, suggestions, expertise? > > Thanks in advance! > > Stefan > > > On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote: > > Peter: Thanks for disabling mem_bigsize / vmalloc(). > > > > All: > > (a) > > Unfortunately, this does not (yet) solve > > [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. > > http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=5… > > > > But this appears to be more vm_minsize related than mem_bigsize related. > > > > > > (b) > > We should check all occurances of mem_bigsize and remove / "hide" those that > > have become irrelevant with the deactivation of mem_bigsize / vmalloc(): > > ======== > > $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize' > > -------- > > MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be > set/limited to > > MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize > (in bytes) will be mmaped anonymously > > MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 > > MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be > set/limited to > > MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize > (in bytes) will be mmaped anonymously > > MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 > > MonetDB/src/common/monet_options.mx:410: set[i].name = > strdup("gdk_mem_bigsize"); > > MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize & > gdk_vm_minsize will be set/limited to > > MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' > > MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename > && size >= GDK_mem_bigsize); > > MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size > after which we use VM rather than heap */ > > MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; > > MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous > virtual memory. The seamless > > MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { > > MonetDB/src/gdk/gdk_utils.mx:1529: if ((p = > GDKgetenv("gdk_mem_bigsize"))) { > > MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize = > GDK_mem_maxsize/16; > > MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t) > MIN(max_mem_bigsize, strtol(p, NULL, 10)); > > monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}: > minimum size for memory-mapped columns, e.g. @code{262144}. > > MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng = > get_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void = > set_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) > > MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; > > MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) > > MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ > > MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); > > MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) > > MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); > > MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) > > MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); > > MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) > > MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; > > MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) > > MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ > > MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768, > sze); > > MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) > > MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); > > MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize = > MAX(GDK_mem_bigsize, sze); > > MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) > > MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); > > MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize = > MAX(GDK_mem_bigsize, sze); > > monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize", > "262144" > > ======== > > > > more tomorrow during MADAM ... > > > > Stefan > > > > > > On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote: > > > no consequences. > > > > > > > > > > > > > > > Peter, > > > > > > What about the consequences in: > > > build/MonetDB/src/gdk/gdk_heap.c > > > build/MonetDB5/src/modules/kernel/status.c > > > build/MonetDB/src/gdk/gdk.h > > > > > > Peter Boncz wrote: > > > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > > > > > > > Modified Files: > > > > Tag: MonetDB_1-22 > > > > gdk_utils.mx > > > > Log Message: > > > > disable use of vmalloc() > > > > > > > > > > > > > > > > Index: gdk_utils.mx > > > > =================================================================== > > > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > > > > retrieving revision 1.206 > > > > retrieving revision 1.206.2.1 > > > > diff -u -d -r1.206 -r1.206.2.1 > > > > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > > > > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > > > > @@ -1050,6 +1050,7 @@ > > > > #endif > > > > } > > > > size = (size + 7) & ~7; /* round up to a multiple of eight */ > > > > +#if 0 > > > > if (size > GDK_mem_bigsize) { > > > > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > > > > size_t newmax = MAX(*maxsize, newsize); > > > > @@ -1064,6 +1065,7 @@ > > > > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > > > > return (void *) s; > > > > } > > > > +#endif > > > > CHKMEM(size, 0); > > > > GDKmalloc_prefixsize(s, size); > > > > if (s == NULL) { > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > This SF.net email is sponsored by: Microsoft > > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > > _______________________________________________ > > > > Monetdb-checkins mailing list > > > > Monetdb-checkins(a)lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Microsoft > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > _______________________________________________ > > > Monetdb-developers mailing list > > > Monetdb-developers(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > > > > > > > > > > > > > > > > > -- > > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdkgdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
by Peter Boncz 11 Feb '08

11 Feb '08

Hi Stefan, The pftijah bug was unrelated to vmalloc(), a relationship that I never assumed. The vmalloc() troubles occured, as you may recall, during the data preparation of the sorted 100GB TPC-H. As it is entirely foreseeable that we at some point again will re-encounter an OS with a malloc implementation that is prone to fragmentation, I'd keep the vmalloc() code in for the moment (though in disabled state). Linux and Windows do not need it. So, my advise is to for the monet let vmalloc() be disabled, until the issue is properly investigated and a bugfix is found. Let's call it option (0). Peter Dear all, since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related) does not appear have any impact on this bug (just tested successfully both with vmalloc() disabled and vmalloc() enabled), there are two ways to finish this release (more or less) quickly: (1) finish the disabling of vmalloc() in all parts of the code as indicated below (or better: remove all remains completely) --- this might need some extra testing to ensure that it does not have any (significant) impact on performance ... (2) for now (i.e., in the release branch) re-enable vmalloc(), and then finish the removal of it (incl. extensive testing) in the development trunk for th next release. Since I'm not completely aware of the impact of vmalloc(), I cannot really make this decision --- the "save" back-up would IMHO be (2). Any other comments, opinions, suggestions, expertise? Thanks in advance! Stefan On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote: > Peter: Thanks for disabling mem_bigsize / vmalloc(). > > All: > (a) > Unfortunately, this does not (yet) solve > [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed. > http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=5… > > But this appears to be more vm_minsize related than mem_bigsize related. > > > (b) > We should check all occurances of mem_bigsize and remove / "hide" those that > have become irrelevant with the deactivation of mem_bigsize / vmalloc(): > ======== > $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize' > -------- > MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be set/limited to > MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously > MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144 > MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be set/limited to > MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped anonymously > MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144 > MonetDB/src/common/monet_options.mx:410: set[i].name = strdup("gdk_mem_bigsize"); > MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to > MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K' > MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) { > MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename && size >= GDK_mem_bigsize); > MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size after which we use VM rather than heap */ > MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20; > MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous virtual memory. The seamless > MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) { > MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) { > MonetDB/src/gdk/gdk_utils.mx:1529: if ((p = GDKgetenv("gdk_mem_bigsize"))) { > MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize = GDK_mem_maxsize/16; > MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10)); > monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g. @code{262144}. > MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng = get_mem_bigsize; > MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void = set_mem_bigsize; > MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num) > MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize; > MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num) > MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@ > MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze); > MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize) > MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num); > MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); > MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize) > MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num); > MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); > MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num) > MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize; > MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num) > MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@ > MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768, sze); > MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize) > MonetDB5/src/modules/kernel/status.mx:440: set_mem_bigsize(num); > MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze); > MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize) > MonetDB5/src/modules/kernel/status.mx:467: set_mem_bigsize(num); > MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize = MAX(GDK_mem_bigsize, sze); > monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize", "262144" > ======== > > more tomorrow during MADAM ... > > Stefan > > > On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote: > > no consequences. > > > > > > > > > > Peter, > > > > What about the consequences in: > > build/MonetDB/src/gdk/gdk_heap.c > > build/MonetDB5/src/modules/kernel/status.c > > build/MonetDB/src/gdk/gdk.h > > > > Peter Boncz wrote: > > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > > > > > Modified Files: > > > Tag: MonetDB_1-22 > > > gdk_utils.mx > > > Log Message: > > > disable use of vmalloc() > > > > > > > > > > > > Index: gdk_utils.mx > > > =================================================================== > > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > > > retrieving revision 1.206 > > > retrieving revision 1.206.2.1 > > > diff -u -d -r1.206 -r1.206.2.1 > > > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > > > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > > > @@ -1050,6 +1050,7 @@ > > > #endif > > > } > > > size = (size + 7) & ~7; /* round up to a multiple of eight */ > > > +#if 0 > > > if (size > GDK_mem_bigsize) { > > > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > > > size_t newmax = MAX(*maxsize, newsize); > > > @@ -1064,6 +1065,7 @@ > > > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > > > return (void *) s; > > > } > > > +#endif > > > CHKMEM(size, 0); > > > GDKmalloc_prefixsize(s, size); > > > if (s == NULL) { > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Microsoft > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > _______________________________________________ > > > Monetdb-checkins mailing list > > > Monetdb-checkins(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > Monetdb-developers mailing list > > Monetdb-developers(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > > > > > > > > > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
by Peter Boncz 11 Feb '08

11 Feb '08

no consequences. Peter, What about the consequences in: build/MonetDB/src/gdk/gdk_heap.c build/MonetDB5/src/modules/kernel/status.c build/MonetDB/src/gdk/gdk.h Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > Modified Files: > Tag: MonetDB_1-22 > gdk_utils.mx > Log Message: > disable use of vmalloc() > > > > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.206 > retrieving revision 1.206.2.1 > diff -u -d -r1.206 -r1.206.2.1 > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > @@ -1050,6 +1050,7 @@ > #endif > } > size = (size + 7) & ~7; /* round up to a multiple of eight */ > +#if 0 > if (size > GDK_mem_bigsize) { > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > size_t newmax = MAX(*maxsize, newsize); > @@ -1064,6 +1065,7 @@ > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > return (void *) s; > } > +#endif > CHKMEM(size, 0); > GDKmalloc_prefixsize(s, size); > if (s == NULL) { > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

2 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
by Martin Kersten 10 Feb '08

10 Feb '08

Peter, What about the consequences in: build/MonetDB/src/gdk/gdk_heap.c build/MonetDB5/src/modules/kernel/status.c build/MonetDB/src/gdk/gdk.h Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > Modified Files: > Tag: MonetDB_1-22 > gdk_utils.mx > Log Message: > disable use of vmalloc() > > > > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.206 > retrieving revision 1.206.2.1 > diff -u -d -r1.206 -r1.206.2.1 > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > @@ -1050,6 +1050,7 @@ > #endif > } > size = (size + 7) & ~7; /* round up to a multiple of eight */ > +#if 0 > if (size > GDK_mem_bigsize) { > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > size_t newmax = MAX(*maxsize, newsize); > @@ -1064,6 +1065,7 @@ > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > return (void *) s; > } > +#endif > CHKMEM(size, 0); > GDKmalloc_prefixsize(s, size); > if (s == NULL) { > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_utils.mx, MonetDB_1-22, 1.206, 1.206.2.1
by Martin Kersten 10 Feb '08

10 Feb '08

Shouldn't then also monetdb.conf be adjusted Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651 > > Modified Files: > Tag: MonetDB_1-22 > gdk_utils.mx > Log Message: > disable use of vmalloc() > > > > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.206 > retrieving revision 1.206.2.1 > diff -u -d -r1.206 -r1.206.2.1 > --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206 > +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1 > @@ -1050,6 +1050,7 @@ > #endif > } > size = (size + 7) & ~7; /* round up to a multiple of eight */ > +#if 0 > if (size > GDK_mem_bigsize) { > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > size_t newmax = MAX(*maxsize, newsize); > @@ -1064,6 +1065,7 @@ > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > return (void *) s; > } > +#endif > CHKMEM(size, 0); > GDKmalloc_prefixsize(s, size); > if (s == NULL) { > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_replicator.mx, , 1.10, 1.11
by Stefan Manegold 09 Feb '08

09 Feb '08

On Sat, Feb 09, 2008 at 03:49:10PM +0000, Milena Ivanova wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv450 > > Modified Files: > opt_replicator.mx > Log Message: > Fix make warning. Why didn't it show with my flags set to > CONFIGURE_OPTIONS="--enable-debug --disable-optimize --enable-strict"? `man gcc` says ======== [...] -Wuninitialized Warn if an automatic variable is used without first being initialized or if a variable may be clobbered by a "setjmp" call. These warnings are possible only in optimizing compilation, because they require data flow information that is computed only when optimizing. If you don’t specify -O, you simply won’t get these warnings. [...] ======== hence, these warnings (=> error) only occur with optimization enabled. Stefan > > Index: opt_replicator.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_replicator.mx,v > retrieving revision 1.10 > retrieving revision 1.11 > diff -u -d -r1.10 -r1.11 > --- opt_replicator.mx 8 Feb 2008 16:44:33 -0000 1.10 > +++ opt_replicator.mx 9 Feb 2008 15:49:08 -0000 1.11 > @@ -242,7 +242,7 @@ > Replica r; > int (*cmp) (ptr, ptr); > ptr nilptr; > - int i, minr; > + int i, minr = -1; > size_t minc = (size_t) GDK_lng_max; > bit leq, lsub, heq, hsub; > int rs[replnr], ri=0; /* subset replicas */ > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] buildtools/doc windowsbuild.rst, MonetDB_1-22, 1.10, 1.10.2.1
by Niels Nes 08 Feb '08

08 Feb '08

On Fri, Feb 08, 2008 at 10:12:44AM +0000, Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/buildtools/doc > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv11533 > > Modified Files: > Tag: MonetDB_1-22 > windowsbuild.rst > Log Message: > Updated for ICC 1.10. > Added CC and CXX variables for ICC. Small request. Could the line cscript configure.js compiler=msvc prefix=C:\libxml2-2.6.30.win64 include=C:\iconv-1.11.win64\include;C:\zlib-1.2.3.win64\include lib=C:\iconv-1.11.win64\lib;C:\zlib-1.2.3.win64\lib iconv=yes zlib=yes somehow be split correctly (ie for windows and rst correct that is) over say 3 lines? Niels > > > Index: windowsbuild.rst > =================================================================== > RCS file: /cvsroot/monetdb/buildtools/doc/windowsbuild.rst,v > retrieving revision 1.10 > retrieving revision 1.10.2.1 > diff -u -d -r1.10 -r1.10.2.1 > --- windowsbuild.rst 15 Jan 2008 15:29:27 -0000 1.10 > +++ windowsbuild.rst 8 Feb 2008 10:12:42 -0000 1.10.2.1 > @@ -114,7 +114,8 @@ > > - Microsoft Visual Studio .NET 2003 (also known as Microsoft Visual Studio 7); > - Microsoft Visual Studio 2005 (also known as Microsoft Visual Studio 8); > -- Intel(R) C++ Compiler 9.1 (which actually needs one of the above). > +- Intel(R) C++ Compiler 9.1 (which actually needs one of the above); > +- Intel(R) C++ Compiler 10.1 (which also needs one of the Microsoft compilers). > > Note that the pathfinder component can currently not be compiled with > any of the Microsoft compilers. It can be compiled with the Intel > @@ -124,7 +125,7 @@ > Compiler gcc under Cygwin__. Using that it (probably still) is possible > to build a version that runs using the Cygwin DLLs, but also a version > that uses the MinGW__ (Minimalist GNU for Windows) package. This is > -not further described here. > +not supported and not further described here. > > __ http://www.cygwin.com/ > __ http://www.mingw.org/ > @@ -534,9 +535,18 @@ > > call "%ProgramFiles%\Microsoft Visual Studio 8\Common7\Tools\vsvars32.bat" > > -- Intel(R) C++ Compiler 9.1:: > +- Intel(R) C++ Compiler 10.1.013:: > > - call "%ProgramFiles%\Intel\Compiler\C++\9.1\IA32\Bin\iclvars.bat" > + call "C:%ProgramFiles%\Intel\Compiler\C++\10.1.013\IA32\Bin\iclvars.bat" > + > +When using the Intel compiler, you also need to set the ``CC`` and > +``CXX`` variables:: > + > + set CC=icl -Qstd=c99 -GR- -Qsafeseh- > + set CXX=icl -Qstd=c99 -GR- -Qsafeseh- > + > +(These are the values for the 10.1 version, for 9.1 replace > +``-Qstd=c99`` with ``-Qc99``.) > > Internal Variables > ~~~~~~~~~~~~~~~~~~ > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_joinpath.mx, , 1.21, 1.22
by Sjoerd Mullender 07 Feb '08

07 Feb '08

On 2008-02-07 18:45, Romulo Goncalves wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv22937 > > Modified Files: > opt_joinpath.mx > Log Message: > Revert Martin code because it does not work. The code will remain, but commented > for future improvements as Martin suggested... A better way to comment larger sections of code is to use #if 0/#endif. That nests. > > > > Index: opt_joinpath.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_joinpath.mx,v > retrieving revision 1.21 > retrieving revision 1.22 > diff -u -d -r1.21 -r1.22 > --- opt_joinpath.mx 7 Feb 2008 16:57:18 -0000 1.21 > +++ opt_joinpath.mx 7 Feb 2008 17:44:59 -0000 1.22 > @@ -231,22 +231,30 @@ > The join path optimizer takes a join sequence and > attempts to minimize the intermediate result. > The choice depends on a good estimate of intermediate > -results.SAMPLE_THRESHOLD_lOG > +results. > For the time being, we use a simplistic model, based > on the assumption that most joins are foreign key joins anyway. > + > We use a sample based approach for sizeable tables. > -The model is derived from the select statement. > +The model is derived from the select statement. However, we did not succeed. > +The code is now commented for future improvement. > +Final conclusion from this exercise is: > +The difference between the join input size and the join output size is not > +the correct (or unique) metric which should be used to decide which order > +should be used in the joinPath. > + > @c > -#define SAMPLE_THRESHOLD_lOG 17 > +/*#define SAMPLE_THRESHOLD_lOG 17*/ > size_t > ALGjoinCost(BAT *l, BAT *r) > { > - size_t lc, rc, lsize,rsize; > - BAT *lsample, *rsample, *j; > + size_t lc, rc; > +/* size_t lsize,rsize; > + BAT *lsample, *rsample, *j; */ > > lc = BATcount(l); > rc = BATcount(r); > - if( lc > 100000 || rc > 100000){ > +/* if( lc > 100000 || rc > 100000){ > lsize= MIN(lc/100, (1<<SAMPLE_THRESHOLD_lOG)/3); > lsample= BATsample(l,lsize); > BBPreclaim(lsample); > @@ -257,7 +265,7 @@ > lsize= BATcount(j); > BBPreclaim(j); > return lsize; > - } > + }*/ > if( l->ttype== TYPE_oid || r->htype== TYPE_oid ) > return MIN(lc,rc); > return lc * rc; > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

4 6

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal bpm.mx, , 1.123, 1.124
by Stefan Manegold 07 Feb '08

07 Feb '08

Martin I guess this should have gone into the MonetDB_5-4 branch, right? Please sync with Niels before back-porting/fixing! Stefan On Thu, Feb 07, 2008 at 08:18:17AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv8081 > > Modified Files: > bpm.mx > Log Message: > typos > > > Index: bpm.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/bpm.mx,v > retrieving revision 1.123 > retrieving revision 1.124 > diff -u -d -r1.123 -r1.124 > --- bpm.mx 11 Jan 2008 10:41:37 -0000 1.123 > +++ bpm.mx 7 Feb 2008 08:18:12 -0000 1.124 > @@ -183,13 +183,13 @@ > @example > bpm.open(); > Ralias:= bpm.take("myversion",:bat[:oid,:int]); # get the partition alias > - R0{hlow=nil:oid,hhigh=nil:oid,tlow=nil:int,thigh=100}:= bpm.take(Ralias, 0); > + R0@{hlow=nil:oid,hhigh=nil:oid,tlow=nil:int,thigh=100@}:= bpm.take(Ralias, 0); > T0:=algebra.select(R0,0,100); > > - R1{hlow=nil:oid,hhigh=nil:oid,tlow=101,thigh=200}:= bpm.take(Ralias, 1); > + R1@{hlow=nil:oid,hhigh=nil:oid,tlow=101,thigh=200@}:= bpm.take(Ralias, 1); > T1:= algebra.select(R1,0,100); > > - R2{hlow=nil:oid,hhigh=nil:oid,tlow=201,thigh=nil:int}:= bpm.take(Ralias, 2); > + R2@{hlow=nil:oid,hhigh=nil:oid,tlow=201,thigh=nil:int@}:= bpm.take(Ralias, 2); > R:= mat.new(T0,T1,T2); > T2:=algebra.select(R2,0,100); > @end example > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0