Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_storage.mx, Feb2010, 1.149.2.5, 1.149.2.6
On Sun, Jan 31, 2010 at 04:43:29PM +0000, Martin Kersten wrote:
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28035
Modified Files: Tag: Feb2010 gdk_storage.mx Log Message: A pity that the first code was not properly checked, because it did not even tried to force the preload by memory touching properly.
As per request (though admittedly belated ...): Wouldn't it be sufficient (and faster and less memory consuming) to preload only that part of the heaps that is indeed used (i.e., holds useful/relevant data) instead of preloading the whole (possibly consciously over-allocated) heaps? I.e., replacing ...heap->size by ...heap->free (as it was in the original code, only that the original code incorrecly assumed ...heap->free to be an absolute address instead of a relative byte offset ...) Stefan
Index: gdk_storage.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_storage.mx,v retrieving revision 1.149.2.5 retrieving revision 1.149.2.6 diff -u -d -r1.149.2.5 -r1.149.2.6 --- gdk_storage.mx 31 Jan 2010 15:50:19 -0000 1.149.2.5 +++ gdk_storage.mx 31 Jan 2010 16:43:27 -0000 1.149.2.6 @@ -703,28 +703,32 @@ @c size_t BATpreload_(BAT *b) { - size_t dummy = 0, *i; + size_t dummy = 0, *i, *limit; size_t step = MT_pagesize()/sizeof(size_t); size_t pages = (size_t) (0.8 * MT_npages());
- if ( b->T->vheap && b->T->vheap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->vheap\n", BATgetId(b)); - for ( i = (size_t *) b->T->vheap->base; i < (size_t *) b->T->vheap->free && pages > 0; i+= step, pages--) + if ( b->T->vheap && b->T->vheap->base && pages){ + IODEBUG THRprintf(GDKout,"#BATpreloaded(%s): T->vheap\n", BATgetId(b)); + limit = (size_t *) (b->T->vheap->base + b->T->vheap->size); + for ( i = (size_t *) b->T->vheap->base; i < limit && pages > 0; i+= step, pages--) dummy += *i; } - if ( b->H->vheap && b->H->vheap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b)); - for ( i = (size_t *) b->H->vheap->base; i < (size_t *) b->H->vheap->free && pages > 0 ; i+= step, pages--) + if ( b->T->heap.base && pages ){ + IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->heap\n", BATgetId(b)); + limit = (size_t *) (b->T->heap.base + b->T->heap.size); + for ( i = (size_t *) b->T->heap.base; i < limit && pages > 0; i+= step, pages--) dummy += *i; } - if ( b->hheap && b->hheap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): hheap\n", BATgetId(b)); - for ( i = (size_t *) b->hheap->base; i < (size_t *)b->hheap->free && pages > 0 ; i+= step, pages--) + if ( b->H->heap.base && pages ){ + IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->heap\n", BATgetId(b)); + limit = (size_t *) (b->H->heap.base + b->H->heap.size); + for ( i = (size_t *) b->H->heap.base; i < limit && pages > 0 ; i+= step, pages--) dummy += *i; } - if ( b->theap && b->theap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): theap\n", BATgetId(b)); - for ( i = (size_t *) b->theap->base; i < (size_t *) b->theap->free && pages > 0; i+= step, pages--) + if ( b->H->vheap && b->H->vheap->base && pages ){ + IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b)); + limit = (size_t *) (b->H->vheap->base + b->H->vheap->size); + for ( i = (size_t *) b->H->vheap->base; i < limit && pages > 0 ; i+= step, pages--) dummy += *i; } return dummy;
------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |
On Sun, Jan 31, 2010 at 04:43:29PM +0000, Martin Kersten wrote:
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28035
Modified Files: Tag: Feb2010 gdk_storage.mx Log Message: A pity that the first code was not properly checked, because it did not even tried to force the preload by memory touching properly.
As per request (though admittedly belated ...): nothing is too late and indeed this was the previous code. It could be that the BATs you touch will be filled, then the size is the option.
Wouldn't it be sufficient (and faster and less memory consuming) to preload The total amount of memory consumed is controlled by the dataflow admission
Stefan Manegold wrote: policy. See the respective check-ins last two weeks. Monitoring the run of Q10 shows that with the current setting we stay properly within the memory bounds most of the time.
only that part of the heaps that is indeed used (i.e., holds useful/relevant data) instead of preloading the whole (possibly consciously over-allocated) heaps?
I.e., replacing ...heap->size by ...heap->free
(as it was in the original code, only that the original code incorrecly assumed ...heap->free to be an absolute address instead of a relative byte offset ...) original code incorrectly used b->hheap and b->theap, which also need to be preloaded upon need. Martin
Stefan
Index: gdk_storage.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_storage.mx,v retrieving revision 1.149.2.5 retrieving revision 1.149.2.6 diff -u -d -r1.149.2.5 -r1.149.2.6 --- gdk_storage.mx 31 Jan 2010 15:50:19 -0000 1.149.2.5 +++ gdk_storage.mx 31 Jan 2010 16:43:27 -0000 1.149.2.6 @@ -703,28 +703,32 @@ @c size_t BATpreload_(BAT *b) { - size_t dummy = 0, *i; + size_t dummy = 0, *i, *limit; size_t step = MT_pagesize()/sizeof(size_t); size_t pages = (size_t) (0.8 * MT_npages());
- if ( b->T->vheap && b->T->vheap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->vheap\n", BATgetId(b)); - for ( i = (size_t *) b->T->vheap->base; i < (size_t *) b->T->vheap->free && pages > 0; i+= step, pages--) + if ( b->T->vheap && b->T->vheap->base && pages){ + IODEBUG THRprintf(GDKout,"#BATpreloaded(%s): T->vheap\n", BATgetId(b)); + limit = (size_t *) (b->T->vheap->base + b->T->vheap->size); + for ( i = (size_t *) b->T->vheap->base; i < limit && pages > 0; i+= step, pages--) dummy += *i; } - if ( b->H->vheap && b->H->vheap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b)); - for ( i = (size_t *) b->H->vheap->base; i < (size_t *) b->H->vheap->free && pages > 0 ; i+= step, pages--) + if ( b->T->heap.base && pages ){ + IODEBUG THRprintf(GDKout,"#BATpreload(%s): T->heap\n", BATgetId(b)); + limit = (size_t *) (b->T->heap.base + b->T->heap.size); + for ( i = (size_t *) b->T->heap.base; i < limit && pages > 0; i+= step, pages--) dummy += *i; } - if ( b->hheap && b->hheap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): hheap\n", BATgetId(b)); - for ( i = (size_t *) b->hheap->base; i < (size_t *)b->hheap->free && pages > 0 ; i+= step, pages--) + if ( b->H->heap.base && pages ){ + IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->heap\n", BATgetId(b)); + limit = (size_t *) (b->H->heap.base + b->H->heap.size); + for ( i = (size_t *) b->H->heap.base; i < limit && pages > 0 ; i+= step, pages--) dummy += *i; } - if ( b->theap && b->theap->base ){ - IODEBUG THRprintf(GDKout,"#BATpreload(%s): theap\n", BATgetId(b)); - for ( i = (size_t *) b->theap->base; i < (size_t *) b->theap->free && pages > 0; i+= step, pages--) + if ( b->H->vheap && b->H->vheap->base && pages ){ + IODEBUG THRprintf(GDKout,"#BATpreload(%s): H->vheap\n", BATgetId(b)); + limit = (size_t *) (b->H->vheap->base + b->H->vheap->size); + for ( i = (size_t *) b->H->vheap->base; i < limit && pages > 0 ; i+= step, pages--) dummy += *i; } return dummy;
------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
participants (2)
-
Martin Kersten -
Stefan Manegold