- developers-list - monetdb.org

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_posix.mx, Feb2010, 1.176.2.14, 1.176.2.15
by Stefan Manegold 15 Feb '10

15 Feb '10

On Sun, Feb 14, 2010 at 09:23:10PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28118 > > Modified Files: > Tag: Feb2010 > gdk_posix.mx > Log Message: > Move the load advice outside the lock. FYI: This checkin also changes the semantics of MT_mmap_inform() such that, now, (1) with preload > 0, posix_madvise(base, len, advise) is also called on areas that are not maintained in MT_mmap_tab[] (2) with preload < 0, posix_madvise(base, len, BUF_NORMAL) is never called on areas that are not maintained in MT_mmap_tab[] Was this intended? Stefan > > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.176.2.14 > retrieving revision 1.176.2.15 > diff -u -d -r1.176.2.14 -r1.176.2.15 > --- gdk_posix.mx 14 Feb 2010 13:17:50 -0000 1.176.2.14 > +++ gdk_posix.mx 14 Feb 2010 21:22:29 -0000 1.176.2.15 > @@ -884,7 +884,7 @@ > void > MT_mmap_inform(void *base, size_t len, int preload, int advise, int writable) > { > - int i; > + int i, unload =0; > > (void) pthread_mutex_lock(&MT_mmap_lock); > i = MT_mmap_idx(base, len); > @@ -892,12 +892,13 @@ > if (writable) MT_mmap_tab[i].writable = (writable > 0); > MT_mmap_tab[i].random += preload * (advise == BUF_WILLNEED); /* done as a counter to keep track of multiple threads */ > MT_mmap_tab[i].usecnt += preload; /* active thread count */ > - if (preload > 0) > - (void) posix_madvise(base, len, advise); > - else if (MT_mmap_tab[i].usecnt == 0) > - (void) posix_madvise(base, len, BUF_NORMAL); > + unload = MT_mmap_tab[i].usecnt == 0; > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > + if (preload > 0) > + (void) posix_madvise(base, len, advise); > + else if (unload) > + (void) posix_madvise(base, len, BUF_NORMAL); > > } > > > > ------------------------------------------------------------------------------ > SOLARIS 10 is the OS for Data Centers - provides features such as DTrace, > Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW > http://p.sf.net/sfu/solaris-dev2dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel group.mx, Feb2010, 1.146.2.14, 1.146.2.15
by Martin Kersten 14 Feb '10

14 Feb '10

Stefan Manegold wrote: > when BATpreloading the sequentially accessed outer BAT, > make sure we also BATpreload the (often) randomly > accessed "inner" BAT We probably also have to inject advice for the target variable(s), because it is likely to be used in the future a BUF_WILLNEED might be appropriate to avoid early eviction. This advice should be given when we create them, e.g. in BATnew?

1 0

[Monetdb-developers] cracking pipeline
by Isidor Zeuner 13 Feb '10

13 Feb '10

Dear MonetDB developers, I'd like to write some words of gratitude for your work on the cracking code. I have read your paper describing the approach, which looks promising. And current code already seems to be somehow stable (I could get it to segfault using integer equality, and got wrong results using integer comparisons, but both were easy to work around .. nothing serious compared with the MALexception whack-a-mole I experienced with mitosis_pipe). Is there a roadmap somewhere for this part of the code, so I could see what to expect, or where to test/patch/contribute most appropriately? Best regards, Isidor Zeuner

1 0

[Monetdb-developers] working with the SQL engine's MAL explanations
by Isidor Zeuner 13 Feb '10

13 Feb '10

Dear MonetDB developers, when using the EXPLAIN keyword within "mclient --language=sql", I get output like: % .explain # table_name % mal # name % clob # type % 0 # length function user.s0_2{autoCommit=true}():void; .. .. Most notably, apart from the lines starting with %, the syntax resembles the code "mclient --language=mal" would expect. So, a naive approach to use this observation would be to filter out the % lines and feed the output together with some boilerplate code into "mclient --language=mal" (example using a bash unix shell): echo "EXPLAIN SELECT 1;"| mclient --language=sql| ( echo "sql.init();" grep -v ^% echo "user.s0_2();" )| mclient --language=mal For very simple SQL statements, this works and creates the expected output. However, as soon as tables come into play, I'm getting errors because of uninitialized variables. So there seem to be at least semantical differences between EXPLAIN's MAL output and the MAL "mclient --language=mal" expects. Is this difference documented somewhere? Is there a better way for ad-hoc translation from executable SQL to executable MAL? Best regards, Isidor Zeuner

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, Feb2010, 1.301.2.9, 1.301.2.10 gdk_bat.mx, Feb2010, 1.236.2.4, 1.236.2.5 gdk_batop.mx, Feb2010, 1.180.2.15, 1.180.2.16 gdk_bbp.mx, Feb2010, 1.269.2.5, 1.269.2.6 gdk_posix.mx, Feb2010, 1.176.2.11, 1.176.2.12 gdk_relop.mx, Feb2010, 1.179.2.13, 1.179.2.14 gdk_storage.mx, Feb2010, 1.149.2.21, 1.149.2.22 gdk_utils.mx, Feb2010, 1.262.2.10, 1.262.2.11
by Stefan Manegold 12 Feb '10

12 Feb '10

Peter, > +/* a thread informs it is goin to (preload==1) or stops (preload==-1) using a range of memory */ > void > -MT_mmap_pin(void *base, size_t len) > +MT_mmap_inform(void *base, size_t len, int preload, int advise, int writable) > { > int i; > > (void) pthread_mutex_lock(&MT_mmap_lock); > i = MT_mmap_idx(base, len); > if (i >= 0) { > - MT_mmap_tab[i].pincnt++; > + if (writable) MT_mmap_tab[i].writable = (writable > 0); > + MT_mmap_tab[i].random += (advise == BUF_WILLNEED); /* done as a counter to keep track of multiple threads */ ^^^^^^^^^^^^^^^^^^^^^^^^ shouldn't this rather be something like MT_mmap_tab[i].random += preload * (advise == BUF_WILLNEED); ? Otherwise (i.e., in the current case), both BATpreload_() & _BATpreload() only increment MT_mmap_tab[i].random, but never decrement it, right? Stefan > + MT_mmap_tab[i].usecnt += preload; /* active thread count */ > + if (preload > 0) (void) posix_madvise(base, len, advise); > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > > } -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/mapiclient MapiClient.mx, , 1.194, 1.195
by Stefan de Konink 12 Feb '10

12 Feb '10

Op 12-02-10 18:01, Fabian Groffen schreef: > On 12-02-2010 16:37:28 +0000, Stefan de Konink wrote: >> Update of /cvsroot/monetdb/clients/src/mapiclient >> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv30000 >> >> Modified Files: >> MapiClient.mx >> Log Message: >> Minor issue with save_history = false. > > you forgot a changelog entry for this feature. I didn't know there was a changelog ;) But it looks that that is branch related? And I just checked it into head? Stefan

2 1

[Monetdb-developers] @= atomtostr memory usage
by Stefan de Konink 11 Feb '10

11 Feb '10

Hi, While creating some new code and seeing unexpected, but interesting, behavior I stumbled upon the @= atomtostr code, some simple questions regarding this. It seems that the memory requested in @= atommem is rather pessimistic. It seems that for a regular integer 24 characters are allocated. Now since this integer seems to go up to 2^31 (10 chars) + sign + \0 (2 chars), why is the double amount needed if a per type approach is available in MAL? The actual toStr code finishes with a strlen, now this is nice behavior, but if we now know what the length of the string actually is, why not shoot a GDKrealloc on it, and make the *len the actual size? Regarding to memory management and memory function calling, the code isn't very efficient. Effectively all the functions seem to introduce their own buffers. If we look from a perspective of for example strConcat, the value is copied several times from serveral position. We could overcome this bahavior if we would return a maximum length for a type, the concat function would add the length of the second string alloc this buffer, forwards the pointer to the typical atomToStr call, and instead of allocing a new string, using the prealloced buffer. (...and yes, doing a GDKrealloc after that function wouldn't be good.) I took the liberty to come up with a patch for the second issue; Since snprintf will return the amount of bytes written, and since we have established that there is no numeric type overflowing the string space, we can ommit the strlen, use the amount of bytes + 1 from snprintf, realloc, and return the actual amount of bytes representing the string. Catches; 1) Old implementation resulted bestcase in an off by one return vs len 2) Is there any code dependent on the *len or that length will include the \0 value? If there are no objections, I'll check in that patch later today in head. >From benchmarking point of view, it might be worthwhile to test with and without the realloc. Stefan

2 2

[Monetdb-developers] confused by Timer/QueryHistory/CallHistory
by Mo Zhou 08 Feb '10

08 Feb '10

Hi, Sorry for bothering you again... I am confused by the time shown in Timer/queryHistory/callHistory. In queryHistory, there are two time: parse time and optimization time. In callHistory, there are two important time: xtime and rtime. These times should not overlap each other. but when I sum up these times, the number is different from the one returned by Timer. I wonder which time should I use for performance test. Thank you very much. -- Thanks, Mo

1 1

[Monetdb-developers] \f none not working
by Mo Zhou 08 Feb '10

08 Feb '10

Hi, I found "\f none" not working in XQuery. here is my commands: >\f none >\f Current formatter: none >for $t in doc("...")//Product return <pid> {pf:nid($t)} <XQueryResult>.....</pid><pid>1711511</pid></XQueryResult> >\f >Current formatter: none The results should not be printed according to the manual. "none: Finally, the none mode suppresses any output. This mode is mostly useful for performance measurements, if one wants to exclude the result serialization time." Could you please let me know if there are any solutions? Thanks! -- Thanks, Mo

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel batcast.mx, Feb2010, 1.31, 1.31.2.1 batstr.mx, Feb2010, 1.68, 1.68.2.1
by Stefan Manegold 06 Feb '10

06 Feb '10

On Sat, Feb 06, 2010 at 10:17:54AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv685 > > Modified Files: > Tag: Feb2010 > batcast.mx batstr.mx > Log Message: > All string BATs should be preloaded to avoid random access to disk. Preloading their string heap(s) is indeed a good idea (unless they exceeds memory size, anyway) due to their inherently random access. However, preloading sequentially traversed BUN heaps might just create an unnecessarily huge memory footprint, in particular with large BATs. Hence, we might want to consider extending BATpreload to not uncoditionally preload all heaps of a BAT (up to four right now, since we ignore hash tables), but rather allow selective preload of those heaps that are (a) accessed at all, and (b) access randomly (or not accessed sequentially). I'll look into this later ... Stefan > > Index: batstr.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batstr.mx,v > retrieving revision 1.68 > retrieving revision 1.68.2.1 > diff -u -d -r1.68 -r1.68.2.1 > --- batstr.mx 7 Jan 2010 15:24:26 -0000 1.68 > +++ batstr.mx 6 Feb 2010 10:17:49 -0000 1.68.2.1 > @@ -239,6 +239,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi,p); > x = (@3) BUNtail(bi,p); > @@ -275,6 +276,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi,p); > str y = (str)str_nil, *yp = &y; > @@ -322,6 +324,7 @@ > > if (strcmp(*cst, str_nil) == 0) > isnil = 1; > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > > @@ -369,6 +372,8 @@ > lefti = bat_iterator(left); > righti = bat_iterator(right); > > + BATpreload(left); > + BATpreload(right); > BATloop(left, p, q) { > bit y = TRUE; > ptr h = BUNhead(lefti, p); > @@ -411,6 +416,8 @@ > lefti = bat_iterator(left); > righti = bat_iterator(right); > > + BATpreload(left); > + BATpreload(right); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > ptr tl = BUNtail(lefti,p); > @@ -444,6 +451,7 @@ > > lefti = bat_iterator(left); > > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > ptr tl = BUNtail(lefti,p); > @@ -476,6 +484,8 @@ > lefti = bat_iterator(left); > righti = bat_iterator(right); > > + BATpreload(left); > + BATpreload(right); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > str tl = (str) BUNtail(lefti,p); > @@ -509,6 +519,7 @@ > > lefti = bat_iterator(left); > > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > str tl = (str) BUNtail(lefti,p); > @@ -539,6 +550,7 @@ > > righti = bat_iterator(right); > > + BATpreload(right); > BATloop(right, p, q) { > ptr h = BUNhead(righti,p); > str tr = (str) BUNtail(righti,p); > @@ -579,6 +591,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > str y = (str)str_nil, *yp = &y; > @@ -622,6 +635,7 @@ > > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > ptr t = BUNtail(bi, p); > @@ -666,6 +680,7 @@ > bn->tsorted = b->tsorted; > > bi = bat_iterator(b); > + BATpreload(b); > BATloop(b, p, q) { > ptr h = BUNhead(bi, p); > str t = (str) BUNtail(bi, p); > @@ -720,6 +735,7 @@ > lefti = bat_iterator(left); > starti = bat_iterator(start); > lengthi = bat_iterator(length); > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(lefti,p); > str tl = (str) BUNtail(lefti,p); > @@ -764,6 +780,7 @@ > bn->tsorted=0; > > li = bat_iterator(left); > + BATpreload(left); > BATloop(left, p, q) { > ptr h = BUNhead(li,p); > str tl = (str) BUNtail(li,p); > > Index: batcast.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/batcast.mx,v > retrieving revision 1.31 > retrieving revision 1.31.2.1 > diff -u -d -r1.31 -r1.31.2.1 > --- batcast.mx 7 Jan 2010 15:24:26 -0000 1.31 > +++ batcast.mx 6 Feb 2010 10:17:47 -0000 1.31.2.1 > @@ -253,6 +253,7 @@ > @:voidresultBAT(TYPE_@1,"batcalc.@1")@ > bi = bat_iterator(b); > > + BATpreload(b); > BATloop(b, p, q) { > ptr v = NULL; > int len = 0; > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1