Sjoerd, this checkin seems to fix the following tests, while not harming any others (on my 64-bit Fedora 10 desktop): MonetDB4/src/gdk/malloc_mmap_priv.milS MonetDB4/src/modules/plain/bat.py sql/src/test/BugTracker-2009/union_leak.SF-2540169.SQL.sh Stefan On Thu, Aug 13, 2009 at 02:41:08PM +0000, Sjoerd Mullender wrote:

Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv31891

Modified Files: Tag: Aug2009 gdk_heap.mx Log Message: Experimental checkin.

Memory maps come in two flavors: normal and private (a.k.a. copy-on-write). The former are more efficient when saving since less data needs to be copied, but they can be a pain when dealing with pre-existing heaps, since then a backup needs to be made before starting to update the data.

When switching from malloced to memory mapped allocation in a heap, we now see whether the heap file already existed, and if so, we use a private memory map (as before), but if the file did not exist, we now use a normal memory map. We don't need to make a backup first, because there is no pre-existing data to backup, so this should work.

The result of this change is that loading data into fresh bats should be quicker. Since the new heaps are now "normal" memory maps, less copying takes place when the data is saved. A quick test suggests that this is indeed the case: the loading part of the 15M Periscope test took longer than the timeout (1800 seconds) allowed without this change, but was finished in about 1000 seconds with the change.

Tonight's testing will hopefully show whether this idea does indeed work.

U gdk_heap.mx Index: gdk_heap.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.125 retrieving revision 1.125.2.1 diff -u -d -r1.125 -r1.125.2.1 --- gdk_heap.mx 25 Jul 2009 21:51:40 -0000 1.125 +++ gdk_heap.mx 13 Aug 2009 14:41:05 -0000 1.125.2.1 @@ -206,13 +206,26 @@ } /* too big: convert it to a disk-based temporary heap */ if (can_mmap) { - FILE *fp = GDKfilelocate(nme, "wb", ext); + FILE *fp; char *of = h->filename; + int existing = 0; + + /* if the heap file already exists, we want to + switch to STORE_PRIV (copy-on-write memory + mapped files), but if the heap file doesn't + exist yet, the BAT is new and we can use + STORE_MMAP */ + fp = GDKfilelocate(nme, "rb", ext); + if (fp != NULL) { + existing = 1; + fclose(fp); + } h->filename = NULL; + fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp); if (h->storage == STORE_MEM) { - int newmode = (h->newstorage != STORE_MMAP)?h->newstorage:STORE_PRIV; + int newmode = h->newstorage == STORE_MMAP && existing ? STORE_PRIV : h->newstorage; /* make sure we realy MMAP */ if (must_mmap && h->newstorage == STORE_MEM) newmode = STORE_MMAP;

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |