February 2010 - developers-list

[Monetdb-developers] encoding of strings returned by Python client
by Mark Bucciarelli 28 Jun '10

28 Jun '10

Any chance the Python client could be modified to return strings in Unicode instead of UTF-8? Seems like a better fit for Python; for example, that's how Python's sqlite3 module behaves. An alternative approach is to expose a set_encoding method on the connection that let's you specify the encoding of all text fields returned. If backward compatibility is an issue, the latter idea seems better even though it adds a knob. Any comments? (Is the maintainer of the Python adapter active even on this list? I didn't see many Python-specific posts in the archives.) Thanks, m

3 12

[Monetdb-developers] pf:new-doc
by Tinco Andringa 26 Feb '10

26 Feb '10

Hello MonetDB developers, In my quest to develop an object oriented interface to monetdb/xquery on ruby using datamapper, I've stumbled upon the following problem. It is at this moment impossible to add a document to monetdb that does not have a source outside monetdb. The only function to add documents to monetdb seems to be pf:add-doc, which takes an url parameter with a location of an xml document it should import. To fully abstract the database from the ruby application, it would be nice if the developers could add a new (empty) document to the database. Would it be a nice idea to have the following function: pf:new-doc("document_name.xml"[, "collection_name.xml"[,"<some><initial>content</initial></some>"]]) It would make users of my api a little happier, and perhaps MonetDB/XQuery as a whole, a bit more normal ;) Kind regards, Tinco Andringa

1 1

[Monetdb-developers] SQL concurrency/serialization
by Isidor Zeuner 25 Feb '10

25 Feb '10

Dear MonetDB developers, on concurrently running writing transactions, I am getting a rather narrowly descriptive error message: "!SQLException:sql.trans:COMMIT: failed". My first attempts to deal with this issue suggest that it happens exactly in those cases where the concurrent writes create some sort of conflict. I am aware that MonetDB does not focus on concurrency, and of the suggestion to serialize multiple writers. However, I'd like to know what the current status of concurrency is. To what extent are SQL primitives implemented which allow to let the server take care of serializing writes, for example through explicit table locking or transaction isolation? Best regards, Isidor Zeuner

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_mapreduce.mx, , 1.13, 1.14
by Martin Kersten 24 Feb '10

24 Feb '10

Fabian wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv10474 > > Modified Files: > opt_mapreduce.mx > Log Message: > One of the optimisers 'inline,remap,evaluate,costModel,coercions,emptySet,aliases' replaces a nil statement in such a way that it ends up as argument 'nil' witout type, causing execution to fail. Push barrier up, such that we don't get undefined var errors because the reduce phase was performed in the dataflow block too. You can freeze the type of variables using setVarUDFtype. As a line of defense, at the end of an optimizer all non-UDFtypes are thrown away and a fresh type resolve step is taken.

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, Feb2010, 1.13, 1.13.2.1
by Sjoerd Mullender 23 Feb '10

23 Feb '10

Are you sure *all* these changes are for disabling tokenizer? Including the property changes? None of this will now be propagated, so if any of this should land in the HEAD, you need to do it manually (but then please identically). On 2010-02-22 16:43, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv32552/src/modules/mal/rdf > > Modified Files: > Tag: Feb2010 > rdf_shredder.mx > Log Message: > disable the tokenizer.mx from m5 > > > Index: rdf_shredder.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v > retrieving revision 1.13 > retrieving revision 1.13.2.1 > diff -u -d -r1.13 -r1.13.2.1 > --- rdf_shredder.mx 8 Jan 2010 12:08:58 -0000 1.13 > +++ rdf_shredder.mx 22 Feb 2010 15:43:41 -0000 1.13.2.1 > @@ -27,7 +27,9 @@ > #include "url.h" > #include "rdf.h" > #include "raptor.h" > +#if 0 > #include "../tokenizer.h" > +#endif > > typedef struct graphBATdef { > graphBATType batType; /* BAT type */ > @@ -36,6 +38,8 @@ > int tailType; /* type of right column */ > } graphBATdef; > > +static BUN batsz = 10000000; > + > /* this list should be kept alligned with the graphBATType enum */ > #if STORE == TRIPLE_STORE > static graphBATdef graphdef[N_GRAPH_BAT] = { > @@ -132,7 +136,8 @@ > @:rdf_BUNappend_unq(@1, @2)@ > #endif > > -@= rdf_BUNappend_unq > + > +@= rdf_BUNappend_unq_1 > bun = BUNfnd(BATmirror(@1),(ptr)@2); > if (bun == BUN_NONE) { > if (BATcount(@1) > 4 * @1->T->hash->mask) { > @@ -148,6 +153,20 @@ > bun = (@1)->hseqbase + bun; > } > > +@= rdf_BUNappend_unq > +bun = BUNfnd(BATmirror(@1),(ptr)@2); > +if (bun == BUN_NONE) { > + if (BATcount(@1) > 4 * @1->T->hash->mask) { > + HASHdestroy(@1); > + BAThash(BATmirror(@1), 2*BATcount(@1)); > + } > + bun = (BUN) @1->batCount; > + @1 = BUNappend(@1, (ptr)@2, TRUE); > + if (@1 == NULL) { > + @:raptor_exception(pdata, "could not append in@1")@ > + } > +} > + > @= rdf_tknzr_insert > { > str t = @1; > @@ -226,12 +245,9 @@ > > /* disable all properties */ > b->tsorted = FALSE; > - b->T->nosorted = 0; > b->tdense = FALSE; > - b->T->nodense = 0; > b->tkey = FALSE; > - b->T->nokey[0] = 0; > - b->T->nokey[1] = 1; > + b->hdense = TRUE; > > return b; > } > @@ -264,7 +280,7 @@ > pdata->graph[i] = create_BAT ( > graphdef[i].headType, > graphdef[i].tailType, > - BATTINY); /* DOTO: estimate size */ > + batsz); /* DOTO: estimate size */ > if (pdata->graph[i] == NULL) { > return NULL; > } > @@ -274,7 +290,7 @@ > pdata->graph[MAP_LEX] = create_BAT ( > graphdef[MAP_LEX].headType, > graphdef[MAP_LEX].tailType, > - BATTINY); /* DOTO: estimate size */ > + batsz); /* DOTO: estimate size */ > if (pdata->graph[MAP_LEX] == NULL) { > return NULL; > } > @@ -346,10 +362,10 @@ > static str > post_processing (parserData *pdata) > { > -#if STORE == TRIPLE_STORE > BUN cnt; > BAT *map_oid = NULL, *S = NULL, *P = NULL, *O = NULL, *ctref= NULL; > BAT **graph = pdata->graph; > +#ifdef _TKNZR_H > BATiter bi, mi; > BUN p, d, r; > oid *bt; > @@ -379,7 +395,31 @@ > P = graph[P_sort]; > O = graph[O_sort]; > cnt = BATcount(S); > +#else > + /* order MAP_LEX */ > + BATorder(BATmirror(graph[MAP_LEX])); > + map_oid = BATmark(graph[MAP_LEX], 0); /* BATmark will create a copy */ > + BATorder(map_oid); > + BATsetaccess(map_oid, BAT_READ); /* force BAtmark not to copy bat */ > + map_oid = BATmirror(BATmark(BATmirror(map_oid), 0)); > + BATsetaccess(graph[MAP_LEX], BAT_READ); /* force BATmark not to copy bat */ > + graph[MAP_LEX] = BATmirror(BATmark(BATmirror(graph[MAP_LEX]), 0)); > + > + /* convert old oids of S_sort, P_sort, O_sort to new ones */ > + cnt = BATcount(graph[S_sort]); > + S = BATleftfetchjoin(graph[S_sort], map_oid, cnt); > + if (S == NULL) goto bailout; > + BBPreclaim(graph[S_sort]); > + P = BATleftfetchjoin(graph[P_sort], map_oid, cnt); > + if (P == NULL) goto bailout; > + BBPreclaim(graph[P_sort]); > + O = BATleftfetchjoin(graph[O_sort], map_oid, cnt); > + if (O == NULL) goto bailout; > + BBPreclaim(graph[O_sort]); > + BBPreclaim(map_oid); > +#endif > > +#if STORE == TRIPLE_STORE > /* order SPO/SOP */ > graph[S_sort] = BATmirror(BATsort(BATmirror(S))); /* sort on S */ > @:order(graph[S_sort],P,O,PO)@ > @@ -408,6 +448,14 @@ > > return MAL_SUCCEED; > > +#elif STORE == MLA_STORE > + graph[S_sort] = S; > + graph[P_sort] = P; > + graph[O_sort] = O; > + > + return MAL_SUCCEED; > +#endif > + > bailout: > if (map_oid != NULL) BBPreclaim(map_oid); > if (ctref != NULL) BBPreclaim(ctref); > @@ -415,11 +463,6 @@ > if (P != NULL) BBPreclaim(P); > if (O != NULL) BBPreclaim(O); > return NULL; > - > -#elif STORE == MLA_STORE > - (void) pdata; > - return MAL_SUCCEED; > -#endif > } > > @= clean_raptor > @@ -455,15 +498,21 @@ > (void) graphname; > > /* init tokenizer */ > +#ifdef _TKNZR_H > if (TKNZRopen (NULL, schema) != MAL_SUCCEED) { > throw(RDF, "rdf.rdfShred", > "could not open the tokenizer\n"); > } > +#else > + (void) schema; > +#endif > > /* Init pdata */ > pdata = parserData_create(*location); > if (pdata == NULL) { > +#ifdef _TKNZR_H > TKNZRclose(&iret); > +#endif > @:clean@ > throw(RDF, "rdf.rdfShred", > "could not allocate enough memory for pdata\n"); > @@ -473,7 +522,9 @@ > raptor_init(); > pdata->rparser = rparser = raptor_new_parser("guess"); > if (rparser == NULL) { > +#ifdef _TKNZR_H > TKNZRclose(&iret); > +#endif > raptor_finish(); > @:clean@ > throw(RDF, "rdf.rdfShred", "could not create raptor parser object\n"); > @@ -484,7 +535,9 @@ > /* Parse URI or local file. */ > ret = URLisaURL(&isURI, location); > if (ret != MAL_SUCCEED) { > +#ifdef _TKNZR_H > TKNZRclose(&iret); > +#endif > @:clean@ > return ret; > } else if (isURI) { > @@ -496,6 +549,9 @@ > iret = raptor_parse_file(rparser, uri, NULL); > } > @:clean_raptor@ > +#ifdef _TKNZR_H > + TKNZRclose(&iret); > +#endif > > graph = pdata->graph; > assert (pdata->tcount == BATcount(graph[S_sort]) && > @@ -504,23 +560,18 @@ > > /* error check */ > if (iret) { > - TKNZRclose(&iret); > @:clean@ > throw(RDF, "rdf.rdfShred", "parsing failed\n"); > } > if (pdata->exception) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "%s\n", pdata->exceptionMsg); > } else if (pdata->fatal) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "last fatal error was:\n%s\n", > pdata->fatalMsg); > } else if (pdata->error) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "last error was:\n%s\n", > pdata->errorMsg); > } else if (pdata->warning) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "last warning was:\n%s\n", > pdata->warningMsg); > } > @@ -528,7 +579,6 @@ > /* post processing step */ > ret = post_processing(pdata); > if (ret != MAL_SUCCEED) { > - TKNZRclose(&iret); > @:clean@ > throw(RDF, "rdf.rdfShred", "could not post-proccess data"); > } > @@ -536,7 +586,6 @@ > /* prepare return bat of bats */ > retbat = BATnew(TYPE_void, TYPE_bat, N_GRAPH_BAT); > if (retbat == NULL) { > - TKNZRclose(&iret); > @:clean@ > throw(RDF, "rdf.rdfShred", > "could not allocate enough memory for return bat"); > @@ -548,7 +597,6 @@ > } > > GDKfree(pdata); > - TKNZRclose(&iret); > BBPkeepref(*retval = retbat->batCacheid); > > return MAL_SUCCEED; > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, Feb2010, 1.13.2.1, 1.13.2.2
by Sjoerd Mullender 22 Feb '10

22 Feb '10

Propagation prevented. On 2010-02-22 17:24, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv8115/rdf > > Modified Files: > Tag: Feb2010 > rdf_shredder.mx > Log Message: > remove any files and code related to the tokenizer from the stable branch. These changes should not be propagated to the head > > > Index: rdf_shredder.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v > retrieving revision 1.13.2.1 > retrieving revision 1.13.2.2 > diff -u -d -r1.13.2.1 -r1.13.2.2 > --- rdf_shredder.mx 22 Feb 2010 15:43:41 -0000 1.13.2.1 > +++ rdf_shredder.mx 22 Feb 2010 16:24:23 -0000 1.13.2.2 > @@ -27,9 +27,6 @@ > #include "url.h" > #include "rdf.h" > #include "raptor.h" > -#if 0 > -#include "../tokenizer.h" > -#endif > > typedef struct graphBATdef { > graphBATType batType; /* BAT type */ > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, Feb2010, 1.13, 1.13.2.1
by Lefteris 22 Feb '10

22 Feb '10

the tokenizer has been removed from the Makefile.ag of the stable branch, and this #include can be deleted all the way. I have said many times that none of this code is ready for stable branches and yes they should have never been propagate to the stable. We haven't announced any MonetDB/RDF module official so it should not be used by any user. For me the #if 0 is not a problem On Mon, Feb 22, 2010 at 4:46 PM, Fabian Groffen <Fabian.Groffen(a)cwi.nl> wrote: > On 22-02-2010 15:43:44 +0000, Lefteris Sidirourgos wrote: >> Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf >> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv32552/src/modules/mal/rdf >> >> Modified Files: >> Tag: Feb2010 >> rdf_shredder.mx >> Log Message: >> disable the tokenizer.mx from m5 > > Dunno if this was/is discussed, but isn't it a much cleaner solution to > just remove it from the code base in this branch, and keep it as is in > the HEAD (e.g. request to prevent propagation?). The 'if 0's and > commented out code look a bit messy. > > >> Index: rdf_shredder.mx >> =================================================================== >> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v >> retrieving revision 1.13 >> retrieving revision 1.13.2.1 >> diff -u -d -r1.13 -r1.13.2.1 >> --- rdf_shredder.mx 8 Jan 2010 12:08:58 -0000 1.13 >> +++ rdf_shredder.mx 22 Feb 2010 15:43:41 -0000 1.13.2.1 >> @@ -27,7 +27,9 @@ >> #include "url.h" >> #include "rdf.h" >> #include "raptor.h" >> +#if 0 >> #include "../tokenizer.h" >> +#endif > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel calc.mx, Feb2010, 1.96, 1.96.2.1
by Stefan Manegold 21 Feb '10

21 Feb '10

On Sun, Feb 21, 2010 at 08:41:08AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv5630 > > Modified Files: > Tag: Feb2010 > calc.mx > Log Message: > The semantics assumed overflow to produce nil-values, not an exception. Is this in line with SQL semantic (requirements)? AFAIK, SQL standard requires to report under-/overflows during casts as execption/errors. Moreover, our SQL compiler/front-end cannot (easily) destinguish between NULL (NIL) values that have been in the input and NULL (NIL) values that are added to the result of a cast to indicate under-/overflows. This might have been the reason why this execption was added by Niels ... Just my €0.02 (or less) Stefan > > Index: calc.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/calc.mx,v > retrieving revision 1.96 > retrieving revision 1.96.2.1 > diff -u -d -r1.96 -r1.96.2.1 > --- calc.mx 10 Jan 2010 15:30:59 -0000 1.96 > +++ calc.mx 21 Feb 2010 08:41:02 -0000 1.96.2.1 > @@ -1273,8 +1273,10 @@ > { > if (*a == @2_nil @3) { > *res= @1_nil; > +/* obey semantics above, don't check for information loss > if (*a != @2_nil) > throw(MAL, "calc.@1", ARGUMENT_TOO_LARGE); > +*/ > } else { > *res= @4; > } > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer/Tests All, Feb2010, 1.79, 1.79.2.1
by Stefan Manegold 21 Feb '10

21 Feb '10

On Sun, Feb 21, 2010 at 08:41:49AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer/Tests > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv5697 > > Modified Files: > Tag: Feb2010 > All > Log Message: > Datacyclotron testing is not for this release. prevented propagations > > Index: All > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/Tests/All,v > retrieving revision 1.79 > retrieving revision 1.79.2.1 > diff -u -d -r1.79 -r1.79.2.1 > --- All 10 Jan 2010 19:14:21 -0000 1.79 > +++ All 21 Feb 2010 08:41:47 -0000 1.79.2.1 > @@ -39,7 +39,7 @@ > reorder00 > reorder01 > > -cycloon00 > +#cycloon00 > #mat00 > #mat01 > #mat02 > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_posix.mx, Feb2010, 1.176.2.24, 1.176.2.25 gdk_storage.mx, Feb2010, 1.149.2.36, 1.149.2.37
by Stefan Manegold 19 Feb '10

19 Feb '10

On Fri, Feb 19, 2010 at 03:40:10PM +0000, Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv10221 > > Modified Files: > Tag: Feb2010 > gdk_posix.mx gdk_storage.mx > Log Message: > - fix newly introduced bug (BUF_SEQUENTIAL passed to posix_madvise iso MMAP_SEQUENTIAL) > most of this was done by Stefan, but the intent of the code was when all users > are gone, to give uniform (sequential) advise to the whole heap > - make sure the length of advise is page aligned (not strictly needed for > Linux, but maybe of other Un*ces) > - in case of shared vheaps (as done by leftfetchjoin into a string bat), assume that > access to the string heap will be random. shouldn't len & ~MT_pagesize() then rather be len & ~(MT_pagesize()-1) (assuming pages size is usually a power of 2) ? Stefan > > > Index: gdk_storage.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_storage.mx,v > retrieving revision 1.149.2.36 > retrieving revision 1.149.2.37 > diff -u -d -r1.149.2.36 -r1.149.2.37 > --- gdk_storage.mx 19 Feb 2010 13:13:38 -0000 1.149.2.36 > +++ gdk_storage.mx 19 Feb 2010 15:40:07 -0000 1.149.2.37 > @@ -707,17 +707,19 @@ > * Peter Feb2010: I tried to do prefetches further apart, to trigger multiple readahead > * units in parallel, but it does improve performance visibly > */ > -static size_t access_heap(str id, str hp, Heap *h, char* base, size_t sz, int touch, int preload, int advise) { > +static size_t access_heap(str id, str hp, Heap *h, char* base, size_t sz, int touch, int preload, int adv) { > size_t v0 = 0, v1 = 0, v2 = 0, v3 = 0, v4 = 0, v5 =0, v6 = 0, v7 = 0, page = MT_pagesize(); > + str advise = (adv==MMAP_WILLNEED)?"WILLNEED":(adv==MMAP_SEQUENTIAL)?"SEQUENTIAL":(adv==MMAP_RANDOM)?"RANDOM":(adv==MMAP_NORMAL)?"NORMAL":NULL; > int t = GDKms(); > + assert(advise); > if (h->storage != STORE_MEM && h->size > MT_MMAP_TILE) { > - MT_mmap_inform(h->base, h->size, preload, advise, 0); > + MT_mmap_inform(h->base, h->size, preload, adv, 0); > if (preload > 0) { > - void* alignedbase = (void*) (((size_t) base) & ~(page-1)); > - size_t alignedsz = (sz + (page-1)) & ~(page-1); > - int ret = posix_madvise(alignedbase, sz, advise); > - if (ret) THRprintf(GDKerr, "#MT_mmap_inform: posix_madvise(file=%s, base="PTRFMT", len="SZFMT"MB, advice=%d) = %d\n", > - h->filename, PTRFMTCAST alignedbase, alignedsz >> 20, advise, errno); > + size_t alignskip = (page - (((size_t) base) & (page-1))) & (page-1); > + size_t alignedsz = (size_t) (((sz < alignskip)?0:((size_t) (sz-alignskip))) & ~(page-1)); > + int ret = posix_madvise(base + alignskip, alignedsz, adv); > + if (ret) THRprintf(GDKerr, "#MT_mmap_inform: posix_madvise(file=%s, base="PTRFMT", len="SZFMT"MB, advice=%s) = %d\n", > + h->filename, PTRFMTCAST (base + alignskip), alignedsz >> 20, advise, errno); > } > } > if (touch && preload > 0) { > @@ -731,8 +733,7 @@ > } > for (hi += 7*page; lo <= hi; lo +=page) v0 += *lo; > } > - IODEBUG THRprintf(GDKout,"#BATpreload(%s->%s,preload=%d,sz=%dMB,%s) = %dms \n", id, hp, preload, (int) (sz>>20), > - (advise==MMAP_WILLNEED)?"WILLNEED":(advise==MMAP_SEQUENTIAL)?"SEQUENTIAL":"UNKNOWN", GDKms()-t); > + IODEBUG THRprintf(GDKout,"#BATpreload(%s->%s,preload=%d,sz=%dMB,%s) = %dms \n", id, hp, preload, (int) (sz>>20), advise, GDKms()-t); > return v0+v1+v2+v3+v4+v5+v6+v7; > } > > @@ -743,7 +744,6 @@ > str id = BATgetId(b); > BATiter bi = bat_iterator(b); > > - assert(advise==MMAP_NORMAL||advise==MMAP_RANDOM||advise==MMAP_SEQUENTIAL||advise==MMAP_WILLNEED||advise==MMAP_DONTNEED); > if (BATcount(b) == 0) return 0; > > /* HASH indices (inherent random access). handle first as they *will* be access randomly (one can always hope for locality on the other heaps) */ > @@ -760,30 +760,46 @@ > gdk_unset_lock(GDKhashLock(ABS(b->batCacheid) & BBP_BATMASK), "BATaccess"); > } > > - /* we only touch stuff that is going to be read randomly (WILLNEED). Note varheaps are sequential wrt to the references, or small */ > - if ( what&USE_HEAD) { > + /* vheaps next, as shared vheaps are not seq-correlated needing WILLNEED (use prefetch budget for this first) */ > + if ( what&USE_HEAD ) { > + if (b->H->vheap && b->H->vheap->base) { > + char *lo = BUNhead(bi, BUNfirst(b)), *hi = BUNhead(bi, BUNlast(b)-1); > + int heap_advise = advise; > + if (b->H->vheap->copied) { /* shared string heaps are not (likely) to be sequentially correlated */ > + lo = b->H->vheap->base; hi = lo + b->H->vheap->free; > + heap_advise = MADV_WILLNEED; > + } > + budget -= sz = ((hi-lo) > budget)?budget:(hi-lo); > + v += access_heap(id, "hheap", b->H->vheap, lo, sz, (advise == BUF_WILLNEED), preload, heap_advise); > + } > + } > + if ( what&USE_TAIL ) { > + if (b->T->vheap && b->T->vheap->base) { > + char *lo = BUNtail(bi, BUNfirst(b)), *hi = BUNtail(bi, BUNlast(b)-1); > + int heap_advise = advise; > + if (b->T->vheap->copied) { /* shared string heaps are not (likely) to be sequentially correlated */ > + lo = b->T->vheap->base; hi = lo + b->T->vheap->free; > + heap_advise = MADV_WILLNEED; > + } > + budget -= sz = ((hi-lo) > budget)?budget:(hi-lo); > + v += access_heap(id, "theap", b->T->vheap, lo, sz, (advise == BUF_WILLNEED), preload, heap_advise); > + } > + } > + > + /* BUN heaps are last in line for prefetch budget */ > + if ( what&USE_HEAD ) { > if (b->H->heap.base) { > char *lo = BUNhloc(bi, BUNfirst(b)), *hi = BUNhloc(bi, BUNlast(b)-1); > budget -= sz = ((hi-lo) > budget)?budget:(hi-lo); > v += access_heap(id, "hbuns", &b->H->heap, lo, sz, (advise == MMAP_WILLNEED), preload, advise); > } > - if (b->H->vheap && b->H->vheap->base) { > - char *lo = BUNhead(bi, BUNfirst(b)), *hi = BUNhead(bi, BUNlast(b)-1); > - budget -= sz = ((hi-lo) > budget)?budget:(hi-lo); > - v += access_heap(id, "hheap", b->H->vheap, lo, sz, (advise == MMAP_WILLNEED), preload, advise); > - } > } > - if ( what&USE_TAIL) { > + if ( what&USE_TAIL ) { > if (b->T->heap.base) { > char *lo = BUNtloc(bi, BUNfirst(b)), *hi = BUNtloc(bi, BUNlast(b)-1); > budget -= sz = ((hi-lo) > budget)?budget:(hi-lo); > v += access_heap(id, "tbuns", &b->T->heap, lo, sz, (advise == MMAP_WILLNEED), preload, advise); > } > - if (b->T->vheap && b->T->vheap->base) { > - char *lo = BUNtail(bi, BUNfirst(b)), *hi = BUNtail(bi, BUNlast(b)-1); > - budget -= sz = ((hi-lo) > budget)?budget:(hi-lo); > - v += access_heap(id, "theap", b->T->vheap, lo, sz, (advise == MMAP_WILLNEED), preload, advise); > - } > } > return v; > } > > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.176.2.24 > retrieving revision 1.176.2.25 > diff -u -d -r1.176.2.24 -r1.176.2.25 > --- gdk_posix.mx 19 Feb 2010 13:13:38 -0000 1.176.2.24 > +++ gdk_posix.mx 19 Feb 2010 15:40:07 -0000 1.176.2.25 > @@ -675,7 +675,7 @@ > #ifdef HAVE_POSIX_FADVISE > if (!do_not_use_posix_fadvise && MT_mmap_tab[victim].fd >= 0) { > /* tell the OS quite clearly that you want to drop this */ > - ret = posix_fadvise(MT_mmap_tab[victim].fd, 0LL, MT_mmap_tab[victim].len, POSIX_FADV_DONTNEED); > + ret = posix_fadvise(MT_mmap_tab[victim].fd, 0LL, MT_mmap_tab[victim].len & ~MT_pagesize(), POSIX_FADV_DONTNEED); > #ifdef MMAP_DEBUG > stream_printf(GDKerr, "#MT_mmap_del: posix_fadvise(%s,fd=%d,%uMB,POSIX_FADV_DONTNEED) = %d\n", MT_mmap_tab[victim].path, MT_mmap_tab[victim].fd, (unsigned int) (MT_mmap_tab[victim].len >> 20), ret); > #endif > @@ -709,7 +709,7 @@ > i = MT_mmap_idx(base, len); > if (i >= 0) { > if (MT_mmap_tab[i].fd >= 0) { > - ret = posix_fadvise(MT_mmap_tab[i].fd, 0, len, advice); > + ret = posix_fadvise(MT_mmap_tab[i].fd, 0, len & ~MT_pagesize(), advice); > #ifdef MMAP_DEBUG > stream_printf(GDKerr, "#MT_fadvise: posix_fadvise(%s,fd=%d,%uMB,%d) = %d\n", MT_mmap_tab[i].path, MT_mmap_tab[i].fd, (unsigned int) (len >> 20), advice, ret); > #endif > @@ -733,7 +733,7 @@ > { > size_t len = MIN((size_t) MT_MMAP_TILE, MT_mmap_tab[i].len - off); > /* tell Linux to please stop caching this stuff */ > - int ret = posix_madvise(MT_mmap_tab[i].base + off, len, POSIX_MADV_DONTNEED); > + int ret = posix_madvise(MT_mmap_tab[i].base + off, len & ~MT_pagesize(), POSIX_MADV_DONTNEED); > > if (err) { > stream_printf(err, "#MT_mmap_unload_tile: posix_madvise(%s,off=%uMB,%uMB,fd=%d,POSIX_MADV_DONTNEED) = %d\n", > @@ -743,7 +743,7 @@ > #ifdef HAVE_POSIX_FADVISE > if (!do_not_use_posix_fadvise) { > /* tell the OS quite clearly that you want to drop this */ > - ret = posix_fadvise(MT_mmap_tab[i].fd, off, len, POSIX_FADV_DONTNEED); > + ret = posix_fadvise(MT_mmap_tab[i].fd, off, len & ~MT_pagesize(), POSIX_FADV_DONTNEED); > if (err) { > stream_printf(err, "#MT_mmap_unload_tile: posix_fadvise(%s,off=%uMB,%uMB,fd=%d,POSIX_MADV_DONTNEED) = %d\n", > MT_mmap_tab[i].path, (unsigned int) (off >> 20), > @@ -908,10 +908,9 @@ > MT_mmap_tab[i].random += preload * (advise == MMAP_WILLNEED); /* done as a counter to keep track of multiple threads */ > MT_mmap_tab[i].usecnt += preload; /* active thread count */ > unload = MT_mmap_tab[i].usecnt == 0; > + if (unload) ret = posix_madvise(MT_mmap_tab[i].base, MT_mmap_tab[i].len & ~MT_pagesize(), MMAP_SEQUENTIAL); > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > - if (unload) > - ret = posix_madvise(base, len, MMAP_SEQUENTIAL); > if (ret) { > stream_printf(GDKerr, "#MT_mmap_inform: posix_madvise(file=%s, fd=%d, base="PTRFMT", len="SZFMT"MB, advice=%d) = %d\n", > (i >= 0 ? MT_mmap_tab[i].path : ""), (i >= 0 ? MT_mmap_tab[i].fd : -1), > @@ -1156,7 +1155,7 @@ > > if (ret != (void *) -1L) { > if (hdl->mode & MMAP_ADVISE) { > - (void) MT_madvise(ret, len, hdl->mode & MMAP_ADVISE); > + (void) MT_madvise(ret, len & ~MT_pagesize(), hdl->mode & MMAP_ADVISE); > } > hdl->fixed = (void *) ((char *) ret + len); > } > @@ -1199,7 +1198,7 @@ > int > MT_madvise(void *p, size_t len, int advise) > { > - int ret = posix_madvise(p, len, advise); > + int ret = posix_madvise(p, len & ~MT_pagesize(), advise); > > #ifdef MMAP_DEBUG > stream_printf(GDKerr, "#posix_madvise(" PTRFMT "," SZFMT ",%d) = %d\n", PTRFMTCAST p, len, advise, ret); > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0