Stefan Manegold wrote:

On Sat, Feb 06, 2010 at 10:17:54AM +0000, Martin Kersten wrote:

...

Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv685

Modified Files: Tag: Feb2010 batcast.mx batstr.mx Log Message: All string BATs should be preloaded to avoid random access to disk.

Preloading their string heap(s) is indeed a good idea (unless they exceeds memory size, anyway) due to their inherently random access. Again, the pre-loading is always limited to the amount of main memory available!

However, preloading sequentially traversed BUN heaps might just create an unnecessarily huge memory footprint, in particular with large BATs. Again, it preloads items that are needed anyway and the admission policy 'ensures' that all operands together fit in memory unless unavoidable. The negative effect is a memory probe extra per page as part of the loop.

The preloading gives preference to the vheaps, ensuring random probing there does not trigger io.

Hence, we might want to consider extending BATpreload to not uncoditionally preload all heaps of a BAT (up to four right now, since we ignore hash tables), but rather allow selective preload of those heaps that are (a) accessed at all, and (b) access randomly (or not accessed sequentially).

Yes, a refinement is possible. For now it marks the places where it should be considered. I am sure there are more places. Martin

I'll look into this later ...

Stefan

...

Index: batstr.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batstr.mx,v retrieving revision 1.68 retrieving revision 1.68.2.1 diff -u -d -r1.68 -r1.68.2.1 --- batstr.mx 7 Jan 2010 15:24:26 -0000 1.68 +++ batstr.mx 6 Feb 2010 10:17:49 -0000 1.68.2.1 @@ -239,6 +239,7 @@

bi = bat_iterator(b);

+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi,p); x = (@3) BUNtail(bi,p); @@ -275,6 +276,7 @@

bi = bat_iterator(b);

+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi,p); str y = (str)str_nil, *yp = &y; @@ -322,6 +324,7 @@

if (strcmp(*cst, str_nil) == 0) isnil = 1; + BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p);

@@ -369,6 +372,8 @@ lefti = bat_iterator(left); righti = bat_iterator(right);

+ BATpreload(left); + BATpreload(right); BATloop(left, p, q) { bit y = TRUE; ptr h = BUNhead(lefti, p); @@ -411,6 +416,8 @@ lefti = bat_iterator(left); righti = bat_iterator(right);

+ BATpreload(left); + BATpreload(right); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); ptr tl = BUNtail(lefti,p); @@ -444,6 +451,7 @@

lefti = bat_iterator(left);

+ BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); ptr tl = BUNtail(lefti,p); @@ -476,6 +484,8 @@ lefti = bat_iterator(left); righti = bat_iterator(right);

+ BATpreload(left); + BATpreload(right); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); str tl = (str) BUNtail(lefti,p); @@ -509,6 +519,7 @@

lefti = bat_iterator(left);

+ BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); str tl = (str) BUNtail(lefti,p); @@ -539,6 +550,7 @@

righti = bat_iterator(right);

+ BATpreload(right); BATloop(right, p, q) { ptr h = BUNhead(righti,p); str tr = (str) BUNtail(righti,p); @@ -579,6 +591,7 @@

bi = bat_iterator(b);

+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p); str y = (str)str_nil, *yp = &y; @@ -622,6 +635,7 @@

bi = bat_iterator(b);

+ BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p); ptr t = BUNtail(bi, p); @@ -666,6 +680,7 @@ bn->tsorted = b->tsorted;

bi = bat_iterator(b); + BATpreload(b); BATloop(b, p, q) { ptr h = BUNhead(bi, p); str t = (str) BUNtail(bi, p); @@ -720,6 +735,7 @@ lefti = bat_iterator(left); starti = bat_iterator(start); lengthi = bat_iterator(length); + BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(lefti,p); str tl = (str) BUNtail(lefti,p); @@ -764,6 +780,7 @@ bn->tsorted=0;

li = bat_iterator(left); + BATpreload(left); BATloop(left, p, q) { ptr h = BUNhead(li,p); str tl = (str) BUNtail(li,p);

Index: batcast.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batcast.mx,v retrieving revision 1.31 retrieving revision 1.31.2.1 diff -u -d -r1.31 -r1.31.2.1 --- batcast.mx 7 Jan 2010 15:24:26 -0000 1.31 +++ batcast.mx 6 Feb 2010 10:17:47 -0000 1.31.2.1 @@ -253,6 +253,7 @@ @:voidresultBAT(TYPE_@1,"batcalc.@1")@ bi = bat_iterator(b);

+ BATpreload(b); BATloop(b, p, q) { ptr v = NULL; int len = 0;

------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins