On Sat, Feb 06, 2010 at 10:17:54AM +0000, Martin Kersten wrote:
Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv685
Modified Files: Tag: Feb2010 batcast.mx batstr.mx Log Message: All string BATs should be preloaded to avoid random access to disk.
Preloading their string heap(s) is indeed a good idea (unless they exceeds memory size, anyway) due to their inherently random access. However, preloading sequentially traversed BUN heaps might just create an unnecessarily huge memory footprint, in particular with large BATs. Hence, we might want to consider extending BATpreload to not uncoditionally preload all heaps of a BAT (up to four right now, since we ignore hash tables), but rather allow selective preload of those heaps that are (a) accessed at all, and (b) access randomly (or not accessed sequentially). I'll look into this later ... Stefan
Index: batstr.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batstr.mx,v retrieving revision 1.68 retrieving revision 1.68.2.1 diff -u -d -r1.68 -r1.68.2.1 --- batstr.mx 7 Jan 2010 15:24:26 -0000 1.68 +++ batstr.mx 6 Feb 2010 10:17:49 -0000 1.68.2.1 @@ -239,6 +239,7 @@
bi = bat_iterator(b);
+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi,p); x = (@3) BUNtail(bi,p); @@ -275,6 +276,7 @@
bi = bat_iterator(b);
+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi,p); str y = (str)str_nil, *yp = &y; @@ -322,6 +324,7 @@
if (strcmp(*cst, str_nil) == 0) isnil = 1; + BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p);
@@ -369,6 +372,8 @@ lefti = bat_iterator(left); righti = bat_iterator(right);
+ BATpreload(left); + BATpreload(right); BATloop(left, p, q) { bit y = TRUE; ptr h = BUNhead(lefti, p); @@ -411,6 +416,8 @@ lefti = bat_iterator(left); righti = bat_iterator(right);
+ BATpreload(left); + BATpreload(right); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); ptr tl = BUNtail(lefti,p); @@ -444,6 +451,7 @@
lefti = bat_iterator(left);
+ BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); ptr tl = BUNtail(lefti,p); @@ -476,6 +484,8 @@ lefti = bat_iterator(left); righti = bat_iterator(right);
+ BATpreload(left); + BATpreload(right); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); str tl = (str) BUNtail(lefti,p); @@ -509,6 +519,7 @@
lefti = bat_iterator(left);
+ BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); str tl = (str) BUNtail(lefti,p); @@ -539,6 +550,7 @@
righti = bat_iterator(right);
+ BATpreload(right); BATloop(right, p, q) { ptr h = BUNhead(righti,p); str tr = (str) BUNtail(righti,p); @@ -579,6 +591,7 @@
bi = bat_iterator(b);
+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p); str y = (str)str_nil, *yp = &y; @@ -622,6 +635,7 @@
bi = bat_iterator(b);
+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p); ptr t = BUNtail(bi, p); @@ -666,6 +680,7 @@ bn->tsorted = b->tsorted;
bi = bat_iterator(b); + BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p); str t = (str) BUNtail(bi, p); @@ -720,6 +735,7 @@ lefti = bat_iterator(left); starti = bat_iterator(start); lengthi = bat_iterator(length); + BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); str tl = (str) BUNtail(lefti,p); @@ -764,6 +780,7 @@ bn->tsorted=0;
li = bat_iterator(left); + BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(li,p); str tl = (str) BUNtail(li,p);
Index: batcast.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batcast.mx,v retrieving revision 1.31 retrieving revision 1.31.2.1 diff -u -d -r1.31 -r1.31.2.1 --- batcast.mx 7 Jan 2010 15:24:26 -0000 1.31 +++ batcast.mx 6 Feb 2010 10:17:47 -0000 1.31.2.1 @@ -253,6 +253,7 @@ @:voidresultBAT(TYPE_@1,"batcalc.@1")@ bi = bat_iterator(b);
+ BATpreload(b); BATloop(b, p, q) { ptr v = NULL; int len = 0;
------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |