April 2009 - developers-list

[Monetdb-developers] XQ: unaligned access
by Wouter Alink 17 Apr '09

17 Apr '09

Hello, I currently get many of the following messages on the Mserver console, while shredding a collection of XML documents with yesterdays stable nightly build (using 32-bit oids on a 64-bit machine): Mserver(31123): unaligned access to 0x20000000de29726f, ip=0x200000000133c101 Mserver(31123): unaligned access to 0x20000000de1f746e, ip=0x200000000133c101 Mserver(31123): unaligned access to 0x20000000de1f746e, ip=0x200000000133c0f0 Mserver(31123): unaligned access to 0x200000013013696d, ip=0x200000000133c101 Mserver(31123): unaligned access to 0x200000013013696d, ip=0x200000000133c0f0 Mserver(31123): unaligned access to 0x20000000de29726f, ip=0x200000000133c101 A new one appears every few seconds. There seems to be only 4 or 5 different addresses accessed for which this message appears. Is this a bug, a feature, or debug info? Can it be safely ignored? Cheers, Wouter

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_octopus.mx, , 1.29, 1.30
by Lefteris 14 Apr '09

14 Apr '09

Is it only my copy because I messed something up or there is a problem with compiling octapus functions in sql.mx? cc1: warnings being treated as errors /home/lsidir/develop//current/sql/src/backends/monet5/sql.mx: In function ‘SQLnewAquarium’: /home/lsidir/develop//current/sql/src/backends/monet5/sql.mx:4091: error: implicit declaration of function ‘OCTOPUSworker’ /home/lsidir/develop//current/sql/src/backends/monet5/sql.mx:4091: error: return makes pointer from integer without a cast /home/lsidir/develop//current/sql/src/backends/monet5/sql.mx: In function ‘SQLdropAquarium’: /home/lsidir/develop//current/sql/src/backends/monet5/sql.mx:4097: error: implicit declaration of function ‘OCTOPUSdrop’ /home/lsidir/develop//current/sql/src/backends/monet5/sql.mx:4097: error: return makes pointer from integer without a cast make[7]: *** [lib_sql_la-sql.lo] Error 1 make[7]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql/src/backends/monet5' make[6]: *** [all] Error 2 make[6]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql/src/backends/monet5' make[5]: *** [all-recursive] Error 1 make[5]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql/src/backends' make[4]: *** [all] Error 2 make[4]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql/src/backends' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/export/scratch0/lsidir/monet/build.current/sql' make: *** [all] Error 2 On Tue, Apr 14, 2009 at 12:59 PM, Martin Kersten <mlkersten(a)users.sourceforge.net> wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv7426 > > Modified Files: > opt_octopus.mx > Log Message: > Silence the compilers on unused variables. > > > U opt_octopus.mx > Index: opt_octopus.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_octopus.mx,v > retrieving revision 1.29 > retrieving revision 1.30 > diff -u -d -r1.29 -r1.30 > --- opt_octopus.mx 14 Apr 2009 07:13:43 -0000 1.29 > +++ opt_octopus.mx 14 Apr 2009 10:59:52 -0000 1.30 > @@ -178,7 +178,7 @@ > OPTtentacle(Client cntxt, MalBlkPtr mb, InstrPtr pci, int start, int idx, int v2){ > MalBlkPtr tm= NULL; > InstrPtr p, q,anchor=NULL; > - int varid= pci->argv[idx],i,j,top=0, fnd, version; > + int varid= pci->argv[idx],i,j,top=0, fnd; > char buf[BUFSIZ]; > InstrPtr *list = (InstrPtr*) GDKzalloc(sizeof(InstrPtr) * mb->ssize); > char *needed= (char*) GDKzalloc(mb->vtop); > @@ -219,7 +219,6 @@ > setArgType(mb,q,0,getVarType(mb,varid)); > q= pushStr(mb,q,"tbd"); > q= pushStr(mb,q,name); > - version = newVariable(mb,GDKstrdup("version"),TYPE_int); > q= pushArgument(mb,q,v2); > needed[varid]= 1; > anchor = q; > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.279, 1.280 gdk_atoms.mx, , 1.161, 1.162 gdk_bat.mx, , 1.214, 1.215 gdk_batop.mx, , 1.170, 1.171 gdk_bbp.mx, , 1.252, 1.253 gdk_heap.mx, , 1.117, 1.118 gdk_qsort.mx, , 1.34, 1.35 gdk_ssort.mx, , 1.15, 1.16 gdk_utils.mx, , 1.246, 1.247
by Sjoerd Mullender 14 Apr '09

14 Apr '09

Is this a backward compatible change? In other words, can databases created before this change be read by the new version? I really want backward compatibility, so if it isn't, some kind of conversion would be needed. Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv5363/src/gdk > > Modified Files: > gdk.mx gdk_atoms.mx gdk_bat.mx gdk_batop.mx gdk_bbp.mx > gdk_heap.mx gdk_qsort.mx gdk_ssort.mx gdk_utils.mx > Log Message: > support for 32GB string heaps in 64bits/oid32 mode > (implies bat format change but ONLY for 64bits/oid32) > > src/gdk/gdk.mx > - introduce GDK_VARSHIFT, var_t (str indices) are to be shifted that > amount of bits to get to the real offset (padding is 8, in case > of 64-bits and oid32 -- otherwise it is 0 => no change) > - clean up usage of var_t in HEAP_* interface > > src/gdk/gdk_atoms.mx > - str heaps with difficult double lim distrubution do not maintain > a linked list (direct open hashing only) > - allow internal string heap hash table pointers to be unsigned shorts > instead of var_t (only switched on for 64bits/oid32) > - double-elim string heaps scaled back to 64KB (from 256-512KB) > - support for GDK_VARSHIFT > > src/gdk/gdk_bat.mx > - bugfix in 64bits/oid32 offset calculation (precision loss averted) > > src/gdk/gdk_batop.mx > src/gdk/gdk_bbp.mx > src/gdk/gdk_qsort.mx > src/gdk/gdk_ssort.mx > - support for GDK_VARSHIFT > > src/gdk/gdk_heap.mx > - HEAPmargin allocates large VM area after a block alloc in 64-bits > (this to stimulate in-place HEAPextend without memcpy) > - clean up use of var_t/size_t in HEAP_* functions, and support for GDK_VARSHIFT > > src/gdk/gdk_utils.mx > - increase VM size for 64-bits systems to 4TB > > > > U gdk.mx > Index: gdk.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk.mx,v > retrieving revision 1.279 > retrieving revision 1.280 > diff -u -d -r1.279 -r1.280 > --- gdk.mx 3 Apr 2009 09:14:51 -0000 1.279 > +++ gdk.mx 13 Apr 2009 23:36:24 -0000 1.280 > @@ -1068,9 +1068,9 @@ > @item void > HEAP_destroy (Heap* h) > @item var_t > -HEAP_malloc (Heap* heap, var_t nbytes) > +HEAP_malloc (Heap* heap, size_t nbytes) > @item void > -HEAP_free (Heap *heap, size_t block) > +HEAP_free (Heap *heap, var_t block) > @item int > HEAP_private (Heap* h) > @item void > @@ -1111,7 +1111,7 @@ > int (*sizefcn) (ptr) /* BATatoms[].atomLen function */ > ); > > -gdk_export var_t HEAP_malloc(Heap *heap, var_t nbytes); > +gdk_export var_t HEAP_malloc(Heap *heap, size_t nbytes); > gdk_export void HEAP_free(Heap *heap, var_t block); > gdk_export var_t HEAP_private(Heap *h); > gdk_export void HEAP_checkformat(Heap *h); > @@ -1350,12 +1350,37 @@ > #define Hloc(b,p) ((b)->H->heap.base+((p)<<(b)->H->shift)) > #define Tloc(b,p) ((b)->T->heap.base+((p)<<(b)->T->shift)) > > +#if SIZEOF_VAR_T < SIZEOF_VOID_P > +/* NEW 11/4/2009: when compiled with 32-bits oids/var_t on 64-bits systems, align heap strings > + * on 8 byte boundaries always (wasting 4 padding bytes on avg). Note that in heaps where > + * duplicate elimination is succesful, such padding occurs anyway (as an aside, a better > + * implementation with two-bytes pointers in the string heap hash table, could reduce that > + * padding to avg 1 byte wasted -- see TODO below). > + * > + * This 8 byte alignment allows the offset in the fixed part of the BAT string column to be > + * interpreted as an index, which should be multiplied by 8 to get the position (VARSHIFT). The > + * overall effect is that 32GB heaps can be addressed even when oids are limited to 4Gtuples. > + * > + * In the future, we might extend this such that the string alignment is set in the BAT header > + * (columns with long strings take more storage space, but could tolerate more padding). > + * It would mostly work, only the sort routine and strPut/strLocate (which do not see the BAT > + * header) extra parameters would be needed in their APIs. > + */ > +typedef unsigned short stridx_t; > +#define GDK_VARSHIFT 3 > +#define GDK_VARALIGN (1<<GDK_VARSHIFT) > +#else > +typedef var_t stridx_t; /* TODO: should also be unsigned short, but kept at var_t not to break BAT images */ > +#define GDK_VARSHIFT 0 > +#define GDK_VARALIGN sizeof(stridx_t) > +#endif > + > #define BUNhloc(bi,p) Hloc((bi).b,p) > #define BUNtloc(bi,p) Tloc((bi).b,p) > #define BUNhpos(bi,p) (Hpos(&(bi),p)) > #define BUNtpos(bi,p) (Tpos(&(bi),p)) > -#define BUNhvar(bi,p) ((bi).b->htype?(Hbase((bi).b)+*((var_t*)BUNhloc(bi,p))):BUNhpos(bi,p)) > -#define BUNtvar(bi,p) ((bi).b->ttype?(Tbase((bi).b)+*((var_t*)BUNtloc(bi,p))):BUNtpos(bi,p)) > +#define BUNhvar(bi,p) ((bi).b->htype?(Hbase((bi).b)+((*(var_t*)BUNhloc(bi,p))<<GDK_VARSHIFT)):BUNhpos(bi,p)) > +#define BUNtvar(bi,p) ((bi).b->ttype?(Tbase((bi).b)+((*(var_t*)BUNtloc(bi,p))<<GDK_VARSHIFT)):BUNtpos(bi,p)) > #define BUNhead(bi,p) ((bi).b->hvarsized?BUNhvar(bi,p):BUNhloc(bi,p)) > #define BUNtail(bi,p) ((bi).b->tvarsized?BUNtvar(bi,p):BUNtloc(bi,p)) > > > U gdk_atoms.mx > Index: gdk_atoms.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_atoms.mx,v > retrieving revision 1.161 > retrieving revision 1.162 > diff -u -d -r1.161 -r1.162 > --- gdk_atoms.mx 7 Apr 2009 11:53:22 -0000 1.161 > +++ gdk_atoms.mx 13 Apr 2009 23:36:24 -0000 1.162 > @@ -175,7 +175,6 @@ > gdk_export int strLen(const char *s); > gdk_export int strCmp(str l, str r); > gdk_export int strNil(str s); > -gdk_export void strHeapConvert(Heap *h, int directon); > gdk_export int strElimDoubles(Heap *h); > gdk_export var_t strLocate(Heap *h, str v); > gdk_export int strCmpNoNil(unsigned char *l, unsigned char *r); > @@ -457,7 +456,7 @@ > 0, 0, > (var_t (*)(Heap *, var_t *, ptr)) strPut, 0, > (int (*)(ptr)) strLen, strHeap, > - (void (*)(Heap *, int)) strHeapConvert, 0}, > + (void (*)(Heap *, int)) 0, 0}, > }; > int GDKatomcnt = TYPE_str + 1; > > @@ -931,24 +930,25 @@ > } \ > } while (0) > > -#define GDK_STRHASHTABLE (1<<10) > +/* string heaps: > + * - strings are 8 byte aligned > + * - start with a 1024 bucket hash table > + * - heaps < 64KB are fully duplicate eliminated with this hash tables > + * - heaps >= 64KB are opportunistically (imperfect) duplicate eliminated > + * as only the last 128KB chunk is considered and there is no linked list > + * - buckets and next pointers are unsigned short "indices" > + * - indices should be multiplied by 8 and takes from ELIMBASE to get an offset > + * > + * Note that a 64KB chunk of the heap contains at most 8K 8-byte aligned > + * strings. The 1K bucket list means that in worst load, the list length is 8 (OK). > + */ > +#define GDK_STRHASHTABLE (1<<10) /* 1024 */ > #define GDK_STRHASHMASK (GDK_STRHASHTABLE-1) > -#define GDK_STRHASHSIZE (GDK_STRHASHTABLE * sizeof(var_t)) > -#define GDK_STRHASHCREDIT(h) (((var_t*) (h)->base)[GDK_STRHASHTABLE]) > +#define GDK_STRHASHSIZE (GDK_STRHASHTABLE * sizeof(stridx_t)) > +#define GDK_ELIMPOWER 16 /* 64KB is the threshold */ > #define GDK_ELIMDOUBLES(h) ((h)->free < GDK_ELIMLIMIT) > -#define GDK_ELIMLIMIT (1<<GDK_ELIMPOWER) > +#define GDK_ELIMLIMIT (1<<GDK_ELIMPOWER) /* equivalently: ELIMBASE == 0 */ > #define GDK_ELIMBASE(x) (((x) >> GDK_ELIMPOWER) << GDK_ELIMPOWER) > -#if SIZEOF_SIZE_T == SIZEOF_INT > -#define GDK_ELIMPOWER 16 /* makes for a max 64KB hash table > - * ie 256 string bytes per hash bucket > - * ~ 4 strings of UP4(8<=len<=11)=12 + 4 bytes > - */ > -#else > -#define GDK_ELIMPOWER 17 /* makes for a max 128KB hash table > - * ie 512 string bytes per hash bucket > - * ~ 5 strings of UP8(8<=len<=15)=16 + 8 bytes > - */ > -#endif > > @- Atomic ADT functions > @c > @@ -1767,46 +1767,34 @@ > Because in many typical situations lots of double string values occur > in tables, the string insertion provides automatic double elimination. > To do this, a GDK_STRHASHTABLE(=1024) bucket hashtable is hidden in the first > -4096 (8192 on 64-bit architectures) bytes of the string heap, consisting of integer offsets of the first > -string hashing to that bucket in the heap. Furthermore, the first 4(8) bytes > -before each string in the heap is an integer offset to the next string hashing > -to the same number. > +4096 bytes of the string heap, consisting an offset to the first string > +hashing to that bucket in the heap. > +These offsets are made small (stridx_t is an unsigned short) by exploiting > +the fact that the double elimination chunks are (now) 64KB, hence a short > +suffices. > > -However, in many other situations the cardinality of string columns is large, > +In many other situations the cardinality of string columns is large, > or the string values might even be unique. In those cases, our fixed-size hash > table will start to overflow quickly. Therefore, after the hash table is full > (this is measured very simplistically by looking whether the string heap exceeds a > -heap size = GDK_ELIMLIMIT -- done this way to keep compatibility with old bat images) > -we flush the hash table. If one views the string heaps as consecutive chunks > -of size GDK_ELIMLIMIT bytes, then all strings within one chunk are double-eliminated. > -There is a macro GDK_ELIMBASE(offset) that computes the base of the chunk in which > -a certain byte-offset falls. > -@- > -This is a departure from our previous policy of not looking at the hash tables at > -all after overflow occurred. The advantage of the new approach is that if we have > -a value distribution that is skewed (ie some values are very frequent), these > -values will always be double eliminated, saving a considerable amount of space. > -Disadvantage of the approach is that we always have to reserve space for the next > -pointer (4(8) byte integer offset) that is stored right in front of the string (and > -consequently have to keep all string chunks and offsets aligned to 4(8)). All this > -translates into some wasted space. However, if there are that many different strings > -that the hash table overflows, the strings must be relatively long and the relative > -storage overhead should be low. > +heap size = GDK_ELIMLIMIT = 64KB) we flush the hash table. Even more, from that moment > +on, we do not use a linked list, but a lossy hash table that just contains > +the last position for each bucket. Basically, after exceeding GDK_ELIMLIMIT, > +we get a probabilistic/opportunistic duplicate elimination mechanism, > +that only looks at the last GDK_ELIMLIMIT chunk in the heap, in a lossy way. > + > +When comparing with the previous string implementation, the biggest difference > +is that on 64-nbits bt with 32-bits oids, strings are always 8-byte aligned > +and var_t numbers are multiplied by 8 to get the true offset. The goal to do > +this is to allow 32-bits var_t on 64-bits systems to address 32GB (using string > +alignment=8). For large database, the cost of padding (4 bytes avg) is offset > +by the savings in var_t (allowing to go from 64- to 32-bits). Nothing lost there, > +and 32-bits var_t also pay in smaller OIDs and smaller hash tables, reducing memory > +pressure. For small duplicate eliminated heaps, the short indices > +used in the hash table have now allowed more buckets (2K instead of 1K) > +and average 2 bytes overhead for the next pointers instead of 6-12. Therefore > +small heaps are now more compact than before. > @- > -Notice that this mechanism enables to keep a certain linear storage property > -in the string heaps. This is important if we want to take a BATslice on a BAT > -by simply loading or @strong{mmap()}ping slices of the BAT files on disk into memory. > -This is relevant in order to process a very large BAT iteratively by taking slices > -in order to reduce memory consumption. Notice that if there are few different string > -values, the hash table has not overflowed, and the string heap size will be small > -(i.e. < GDK_ELIMLIMIT), so in those cases it is not a problem to load the entire string heap. > -If the hash table @strong{has} overflowed, we want to be able to only map a slice of the > -string heap as well. Now, given that the first string in the BAT-slice is called F1 > -and its heap offset is O1 and the last string in the slice is F2 and its > -offset is O2, then the slice we should take from the string heap is: > -@example > -GDK_ELIMBASE(F1) .. MAX(GDK_ELIMBASE(F2)+GDK_ELIMLIMIT), O2+strlen(F2)) > -@end example > The routine strElimDoubles() can be used to check whether all > strings are still being double-eliminated in the original hash-table. > Only then we know that unequal offset-integers in the BUN array means > @@ -1877,16 +1865,12 @@ > strHeap(Heap *d, size_t cap) > { > size_t size; > - var_t *h, *e; > > cap = MAX(cap, BATTINY); > - size = (GDK_STRHASHTABLE + 1) * sizeof(var_t) + MIN(GDK_ELIMLIMIT, cap * 12); > + size = GDK_STRHASHTABLE * sizeof(stridx_t) + MIN(GDK_ELIMLIMIT, cap * GDK_VARALIGN); > if (HEAPalloc(d, size, 1) >= 0) { > - d->free = GDK_STRHASHTABLE * sizeof(var_t); > - h = (var_t *) d->base; > - for (e = h; e < h + GDK_STRHASHTABLE; e++) { > - *e = 0; > - } > + d->free = GDK_STRHASHTABLE * sizeof(stridx_t); > + memset(d->base, 0, d->free); > } > } > > @@ -1923,42 +1907,10 @@ > void > strCleanHash(Heap *h, int rebuild) > { > + (void) rebuild; > if (!GDK_ELIMDOUBLES(h)) { > /* flush hash table for security */ > memset(h->base, 0, GDK_STRHASHSIZE); > - } else if (rebuild) { > - var_t xx, cur = 0, end = 0; > - str hash = (str) h->base; > - > -/* int cnt[GDK_STRHASHTABLE]; */ > - /* collect all values in one big linked list */ > - for (xx = 0; xx < GDK_STRHASHTABLE; xx++) { > - var_t yy = ((var_t *) hash)[xx]; > - > -/* cnt[xx]=0; */ > - ((var_t *) hash)[xx] = 0; /* clear hash table slot */ > - > - if (end) { > - *(var_t *) (hash + end) = yy; > - } else { > - cur = yy; > - } > - for (; yy; yy = *(var_t *) (hash + yy)) > - end = yy; > - } > - > - /* process the linked list, inserting the values again */ > - for (; cur; cur = end) { > - str val = hash + cur; > - GDK_STRHASH(val + sizeof(var_t), xx); > - xx &= GDK_STRHASHMASK; > - end = *(var_t *) val; > - *(var_t *) val = ((var_t *) hash)[xx]; > - ((var_t *) hash)[xx] = cur; > -/* cnt[xx]++; */ > - } > -/* for(xx=0; xx<GDK_STRHASHTABLE; xx++) > - if(cnt[xx]) stream_printf(GDKout, "%5d %6d", xx, cnt[xx]); */ > } > } > > @@ -1970,90 +1922,63 @@ > var_t > strLocate(Heap *h, str v) > { > - var_t *htab = (var_t *) h->base; > - var_t *l, *e; > - BUN idx; > - > - GDK_STRHASH(v, idx); > - e = htab + (idx & GDK_STRHASHMASK); > - if (*e) { > - for (l = e; *l && *l < h->free; l = (var_t *) ((char *) h->base + *l)) { > - str x = (str) ((char *) h->base + *l + sizeof(var_t)); > - > - if (GDK_STRCMP(v, x) == 0) { > - return *l + (var_t) sizeof(var_t); > - } > - } > - } > - return 0; > -} > + stridx_t *ref, *next; > > -#if SIZEOF_SIZE_T == SIZEOF_INT > -#define normal_vart_SWAP(x) ((var_t) normal_int_SWAP((int)x)) > -#else > -#define normal_vart_SWAP(x) ((var_t) long_long_SWAP((lng)x)) > -#endif > + /* search hash-table, if double-elimination is still in place */ > + BUN off; GDK_STRHASH(v, off); > + off &= GDK_STRHASHMASK; > > -/* convert the integers in the implicit hash table structure */ > -void > -strHeapConvert(Heap *h, int dir) > -{ > - var_t *htab = (var_t *) h->base; > - var_t *l, i, j; > + /* should only use strLocate iff fully double eliminated */ > + assert(GDK_ELIMBASE(h->free) == 0); > > - if (dir == CONV_HTON) { > - for (i = 0; i < GDK_STRHASHTABLE; i++) { > - for (l = htab + i; (j = *l) != 0 && j < h->free; l = (var_t *) (h->base + j)) { > - *l = normal_vart_SWAP(j); > - } > - } > - } else { > - for (i = 0; i < GDK_STRHASHTABLE; i++) { > - for (l = htab + i; (j = *l) != 0 && j < h->free; l = (var_t *) (h->base + *l)) { > - *l = normal_vart_SWAP(j); > - } > - } > + /* search the linked list */ > + for (ref = ((stridx_t *) h->base) + off; *ref; ref = next) { > + next = (stridx_t*) (h->base + *ref); > + if (GDK_STRCMP(v, (str) (next+1)) == 0) > + return (sizeof(stridx_t) + *ref) >> GDK_VARSHIFT; /* found */ > } > + return 0; > } > > var_t > strPut(Heap *h, var_t *dst, str v) > { > - var_t *l; > - size_t i = GDK_STRLEN(v); > - BUN off; > - > - /* round up to var_t-byte alignment + var_t (next pointer) */ > - size_t len = ((i + sizeof(var_t) - 1) & ~(sizeof(var_t) - 1)) + sizeof(var_t); > - size_t elimlimit = GDK_ELIMBASE(h->free) + GDK_ELIMLIMIT; > + size_t elimbase = GDK_ELIMBASE(h->free); > + size_t pad = GDK_VARALIGN - (h->free & (GDK_VARALIGN-1)); > + size_t pos, len = GDK_STRLEN(v); > + stridx_t *bucket, *ref, *next; > > /* search hash-table, if double-elimination is still in place */ > - GDK_STRHASH(v, off); > + BUN off; GDK_STRHASH(v, off); > off &= GDK_STRHASHMASK; > - for (l = ((var_t *) h->base) + off; *l && *l < h->free; l = (var_t *) (h->base + *l)) { > - str x = (str) (h->base + *l + sizeof(var_t)); > + bucket = ((stridx_t *) h->base) + off; > > - if (GDK_STRCMP(v, x) == 0) { > - *dst = *l + (var_t) sizeof(var_t); /* already in heap; do not insert! */ > - if (GDK_ELIMDOUBLES(h) == 0 && GDK_STRHASHCREDIT(h)) > - GDK_STRHASHCREDIT(h) += 4; > - return *dst; > + if (elimbase == 0) { /* small string heap (<64KB) -- fully double eliminated */ > + for (ref = bucket; *ref; ref = next) { /* search the linked list */ > + next = (stridx_t*) (h->base + *ref); > + if (GDK_STRCMP(v, (str) (next+1)) == 0) { /* found */ > + pos = sizeof(stridx_t) + *ref; > + return *dst = (pos >> GDK_VARSHIFT); > + } > } > - } > - > - /* flush the hash table if it becomes too big (implies !GDK_ELIMDOUBLES) */ > - if (h->free + len >= elimlimit) { > - /* if we are not hash-inserting anymore, h->free may no longer be var_t aligned */ > - size_t mask = h->free & (sizeof(var_t) - 1); > - > - if (mask) > - h->free += sizeof(var_t) - mask; /* realign */ > - memset(h->base, 0, GDK_STRHASHSIZE); /* start over hash inserting in a pristine hash table */ > - GDK_STRHASHCREDIT(h) = 32; /* only tolerate limited misses in the future */ > + /* is there room for the next pointer in the padding space? */ > + if (pad < sizeof(stridx_t)) > + pad += GDK_VARALIGN; /* if not, pad more */ > + } else { > + /* large string heap (>=64KB) -- opportunistic/probabilistic double elimination */ > + pos = elimbase + *bucket; > + if (GDK_STRCMP(v, (str) (h->base + pos)) == 0) { > + return *dst = (pos >> GDK_VARSHIFT); /* already in heap; do not insert! */ > + } > +#if SIZEOF_VAR_T < SIZEOF_VOID_P > + pad &= (GDK_VARALIGN-1); /* a full VARALIGN pad can be omitted */ > +#else > + pad = 0; /* only 32-bits var_t in 64-bits needs padding */ > +#endif > } > > /* check heap for space (limited to a certain maximum after which nils are inserted) */ > - if (h->free + len >= h->size) { > + if (h->free + pad + len >= h->size) { > /* Something really strange happens here, In a special case > (Pentium II Klamath, gcc version 2.96 20000731, > GNU assembler version 2.10.90 using BFD version 2.10.0.18) > @@ -2064,11 +1989,15 @@ > */ > float batmargin = (float) BATMARGIN; > float hnewsize = h->size * batmargin; > - size_t newsize = len + (size_t) hnewsize; > + size_t newsize = pad + len + (size_t) hnewsize; > > assert(newsize); > > - if (h->free + len < h->maxsize) { > + if (h->free + pad + len >= (((size_t) VAR_MAX) << GDK_VARSHIFT)) { > + GDKerror("strPut: string heaps gets larger than %dGB.\n", (((size_t) VAR_MAX) << GDK_VARSHIFT) >> 30); > + return 0; > + } > + if (h->free + pad + len < h->maxsize) { > /* if there is reserved space, first use the reserved space */ > newsize = MIN(newsize, h->maxsize); > } > @@ -2077,32 +2006,27 @@ > } > /* fill should solve initialisation problems within valgrind */ > memset(h->base + h->free, 0, h->size - h->free); > - } > > - if (!GDK_ELIMDOUBLES(h)) { > - if (GDK_STRHASHCREDIT(h) == 0) { > - /* if credits are gone, we do not hash insert at all */ > - if (h->free > VAR_MAX) { > - GDKerror("strPut: string heaps gets larger than 2GB -- too large for 32-bits oids.\n"); > - return 0; > - } > - *dst = (var_t) h->free; > - memcpy(h->base + h->free, v, i); > - h->free += i; /* in this case, we do not round to var_t either */ > - return *dst; > - } > - GDK_STRHASHCREDIT(h)--; > + /* make bucket point into the enw heap */ > + bucket = ((stridx_t *) h->base) + off; > } > > - /* insert string in hash table and copy into the heap */ > - l = (var_t *) (h->base + h->free); > - *(l++) = ((var_t *) h->base)[off]; > - assert(h->free + sizeof(var_t) <= VAR_MAX); > - ((var_t *) h->base)[off] = (var_t) h->free; > - *dst = (var_t) (h->free + sizeof(var_t)); > - h->free += len; > - memcpy((char *) l, v, i); > + /* insert string */ > + pos = h->free + pad; > + *dst = pos >> GDK_VARSHIFT; > + memcpy(h->base + pos, v, len); > + h->free += pad + len; > + > + /* maintain hash table */ > + if (elimbase == 0) { /* small string heap: link the next pointer */ > + pos -= sizeof(stridx_t); /* the stridx_t next pointer directly precedes the string */ > + *(stridx_t*) (h->base + pos) = *bucket; > + } > + *bucket = (stridx_t) (pos - elimbase); /* set bucket to the new string */ > > + if (h->free >= elimbase + GDK_ELIMLIMIT) { > + memset(h->base, 0, GDK_STRHASHSIZE); /* flush hash table */ > + } > return *dst; > } > > > U gdk_bbp.mx > Index: gdk_bbp.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bbp.mx,v > retrieving revision 1.252 > retrieving revision 1.253 > diff -u -d -r1.252 -r1.253 > --- gdk_bbp.mx 9 Apr 2009 15:29:18 -0000 1.252 > +++ gdk_bbp.mx 13 Apr 2009 23:36:24 -0000 1.253 > @@ -2965,9 +2965,9 @@ > BATsetcount(&bs->B, 0); > > if (bs->H.vheap) > - memset(bs->H.vheap->base, 0, bs->H.vheap->free = GDK_STRHASHTABLE * sizeof(var_t)); > + memset(bs->H.vheap->base, 0, bs->H.vheap->free = GDK_STRHASHTABLE * sizeof(stridx_t)); > if (bs->T.vheap) > - memset(bs->T.vheap->base, 0, bs->T.vheap->free = GDK_STRHASHTABLE * sizeof(var_t)); > + memset(bs->T.vheap->base, 0, bs->T.vheap->free = GDK_STRHASHTABLE * sizeof(stridx_t)); > return bs; > } > BATDEBUG THRprintf(GDKout, "#BBPrecycle %d %d "SZFMT" (%d, %d %d %d) N2\n", ht, tt, cap, BATCACHE_NOTYPE(ht), BATCACHE_NOTYPE(tt), batcache_maxbuckets, bin); > > U gdk_batop.mx > Index: gdk_batop.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_batop.mx,v > retrieving revision 1.170 > retrieving revision 1.171 > diff -u -d -r1.170 -r1.171 > --- gdk_batop.mx 7 Jan 2009 14:13:48 -0000 1.170 > +++ gdk_batop.mx 13 Apr 2009 23:36:24 -0000 1.171 > @@ -1656,7 +1656,7 @@ > if (b->hkey == 0) { > /* sorted and key? */ > while (cur < end) { > - char *val = base + *(((var_t *)b->H->heap.base)+ cur); > + char *val = base + (*(((var_t *)b->H->heap.base)+ cur) << GDK_VARSHIFT); > > if (cmp(prv, val) @5= 0) { > key = FALSE; > @@ -1668,7 +1668,7 @@ > } > /* sorted? */ > while (cur < end) { > - char *val = base + *(((var_t *)b->H->heap.base)+ cur); > + char *val = base + (*(((var_t *)b->H->heap.base)+ cur) << GDK_VARSHIFT); > > if (cmp(prv, val) @5 0) { > /* record negative position info */ > > U gdk_utils.mx > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.246 > retrieving revision 1.247 > diff -u -d -r1.246 -r1.247 > --- gdk_utils.mx 9 Apr 2009 18:48:25 -0000 1.246 > +++ gdk_utils.mx 13 Apr 2009 23:36:24 -0000 1.247 > @@ -382,7 +382,7 @@ > #define GDK_MEM_NULLALLOWED > > #if SIZEOF_VOID_P==8 > -#define GDK_VM_MAXSIZE LL_CONSTANT(137438953472) /* :-) a 64-bit OS: 128 GB */ > +#define GDK_VM_MAXSIZE LL_CONSTANT(4398046511104) /* :-) a 64-bit OS: 4TB */ > #else > #define GDK_VM_MAXSIZE LL_CONSTANT(1610612736) /* :-| a 32-bit OS: 1.5GB */ > #endif > > U gdk_heap.mx > Index: gdk_heap.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v > retrieving revision 1.117 > retrieving revision 1.118 > diff -u -d -r1.117 -r1.118 > --- gdk_heap.mx 20 Mar 2009 11:59:57 -0000 1.117 > +++ gdk_heap.mx 13 Apr 2009 23:36:24 -0000 1.118 > @@ -75,8 +75,20 @@ > nice). > @{ > @c > -int > -HEAPalloc(Heap *h, size_t nitems, size_t itemsize) > +size_t HEAPmargin(size_t maxsize) { > + size_t ret; > +#if SIZEOF_VOID_P == 8 > + /* in 64-bits systems, try to enforce in-place realloc, but provoke the memcpy on 256MB, then 4GB */ > + size_t use = GDKvm_cursize(); > + ret = MIN(GDK_mem_maxsize, MAX(((size_t) 1) << 26, 16*maxsize)); > + if ((ret+ret) > (GDK_vm_maxsize-MIN(GDK_vm_maxsize,use))) /* only if room */ > +#endif > + ret = ((double) BATMARGIN)*maxsize - 1; /* do not waste VM on 32-bits */ > + return (1 + (MAX(maxsize,ret) >> 16)) << 16; /* round up to 64K */ > +} > + > +/* in 64-bits space, use very large margins to accomodate reallocations */ > +int HEAPalloc(Heap *h, size_t nitems, size_t itemsize) > { > char nme[PATHLENGTH], *ext = NULL; > > @@ -98,8 +110,7 @@ > /* when using anonymous vm we malloc we need 64K chunks, also we > * 20% extra malloc */ > if (h->size > GDK_mem_bigsize) { > - h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1; > - h->maxsize = (1 + (h->maxsize >> 16)) << 16; > + h->maxsize = HEAPmargin(h->maxsize); > } > if (h->filename == NULL || (h->size < GDK_mmap_minsize)) { > h->storage = STORE_MEM; > @@ -169,17 +180,14 @@ > /* extend a malloced heap, possibly switching over to file-mapped storage */ > Heap bak = *h; > int can_mmap = h->filename && (size >= GDK_mem_bigsize || h->newstorage != STORE_MEM); > - int must_mmap = can_mmap && (size >= GDK_mmap_minsize || h->newstorage != STORE_MEM); > + int small_cpy = (h->size*4 < size) && (size >= GDK_mmap_minsize); > + int must_mmap = can_mmap && (small_cpy || h->newstorage != STORE_MEM); > > h->size = size; > > if (can_mmap) { > /* in anonymous vm, if have to realloc anyway, we reserve some extra space */ > - if (size > h->maxsize) { > - h->maxsize = (size_t) ((double) size * BATMARGIN); > - } > - /* when using anonymous vm we malloc we need 64K chunks */ > - h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16; > + h->maxsize = HEAPmargin(MAX(size,h->maxsize)); > } else { > h->maxsize = size; /* for normal GDKmalloc, maxsize = size */ > } > @@ -514,9 +522,9 @@ > #define HEAPVERSION 20030408 > > typedef struct heapheader { > - var_t head; /* index to first free block */ > + size_t head; /* index to first free block */ > int alignment; /* alignment of objects on heap */ > - var_t firstblock; /* first block in heap */ > + size_t firstblock; /* first block in heap */ > int version; > int (*sizefcn) (ptr); /* ADT function to ask length */ > } HEADER32; > @@ -524,8 +532,8 @@ > typedef struct { > int version; > int alignment; > - var_t head; > - var_t firstblock; > + size_t head; > + size_t firstblock; > int (*sizefcn) (ptr); > } HEADER64; > > @@ -537,8 +545,8 @@ > typedef HEADER64 HEADER_OTHER; > #endif > typedef struct hfblock { > - var_t size; /* Size of this block in freelist */ > - var_t next; /* index of next block */ > + size_t size; /* Size of this block in freelist */ > + size_t next; /* index of next block */ > } CHUNK; > > @c > @@ -546,13 +554,13 @@ > #define roundup_4(x) (((x)+3)&~3) > #define blocksize(h,p) ((p)->size) > > -static INLINE var_t > -roundup_num(var_t number, int alignment) > +static INLINE size_t > +roundup_num(size_t number, int alignment) > { > - var_t rval; > + size_t rval; > > - rval = number + (var_t) alignment - 1; > - rval -= (rval % (var_t) alignment); > + rval = number + (size_t) alignment - 1; > + rval -= (rval % (size_t) alignment); > return rval; > } > > @@ -560,7 +568,7 @@ > HEAP_private(Heap *h) > { > (void) h; > - return (var_t) roundup_8(sizeof(HEADER)); > + return (var_t) (roundup_8(sizeof(HEADER)) >> GDK_VARSHIFT); > } > > #ifdef TRACE > @@ -568,7 +576,7 @@ > HEAP_printstatus(Heap *heap) > { > HEADER *hheader = HEAP_index(heap, 0, HEADER); > - var_t block, cur_free = hheader->head; > + size_t block, cur_free = hheader->head; > CHUNK *blockp; > > THRprintf(GDKout, "#HEAP has head " SZFMT " and alignment %d and size " SZFMT "\n", > @@ -591,7 +599,7 @@ > cur_free = blockp->next; > block += blockp->size; > } else { > - var_t size = blocksize(hheader, blockp); > + size_t size = blocksize(hheader, blockp); > > THRprintf(GDKout, "# block at " SZFMT " with size " SZFMT "\n", block, size); > block += size; > @@ -611,7 +619,7 @@ > /* > // Calculate position of first and only free block. > */ > - var_t head = roundup_num((var_t) (roundup_8(sizeof(HEADER)) + roundup_8(nprivate)), alignment); > + size_t head = roundup_num((size_t) (roundup_8(sizeof(HEADER)) + roundup_8(nprivate)), alignment); > CHUNK *headp = HEAP_index(heap, head, CHUNK); > > assert(roundup_8(sizeof(HEADER)) + roundup_8(nprivate) <= VAR_MAX); > @@ -629,7 +637,7 @@ > // Fill first free block. > */ > assert(heap->size - head <= VAR_MAX); > - headp->size = (var_t) (heap->size - head); > + headp->size = (size_t) (heap->size - head); > headp->next = 0; > #ifdef TRACE > THRprintf(GDKout, "#We created the following heap\n"); > @@ -669,9 +677,9 @@ > > > var_t > -HEAP_malloc(Heap *heap, var_t nbytes) > +HEAP_malloc(Heap *heap, size_t nbytes) > { > - var_t block, trail, ttrail; > + size_t block, trail, ttrail; > CHUNK *blockp; > CHUNK *trailp; > HEADER *hheader = HEAP_index(heap, 0, HEADER); > @@ -682,15 +690,9 @@ > > /* add space for size field */ > nbytes += hheader->alignment; > - if (hheader->alignment == 8) { > - nbytes = roundup_8(nbytes); > - } else if (hheader->alignment == 4) { > - nbytes = roundup_4(nbytes); > - } else { > - GDKfatal("HEAP_malloc: Heap structure corrupt\n"); > - } > + nbytes = roundup_8(nbytes); > if (nbytes < sizeof(CHUNK)) > - nbytes = (var_t) sizeof(CHUNK); > + nbytes = (size_t) sizeof(CHUNK); > > /* > // block -- points to block with acceptable size (if available). > @@ -718,12 +720,12 @@ > // If no block of acceptable size is found we try to enlarge the heap. > */ > if (block == 0) { > - var_t newsize; > + size_t newsize; > > assert(heap->free + MAX(heap->free, nbytes) <= VAR_MAX); > - newsize = (var_t) roundup_8(heap->free + MAX(heap->free, nbytes)); > + newsize = (size_t) roundup_8(heap->free + MAX(heap->free, nbytes)); > assert(heap->free <= VAR_MAX); > - block = (var_t) heap->free; /* current end-of-heap */ > + block = (size_t) heap->free; /* current end-of-heap */ > > #ifdef TRACE > THRprintf(GDKout, "#No block found\n"); > @@ -747,7 +749,7 @@ > > blockp->next = 0; > assert(heap->free - block <= VAR_MAX); > - blockp->size = (var_t) (heap->free - block); /* determine size of allocated block */ > + blockp->size = (size_t) (heap->free - block); /* determine size of allocated block */ > > /* > // Try to join the last block in the freelist and the newly allocated > @@ -778,7 +780,7 @@ > // TUNE: use different amount than 2*sizeof(CHUNK) > */ > if (blockp->size >= nbytes + 2 * sizeof(CHUNK)) { > - var_t newblock = block + nbytes; > + size_t newblock = block + nbytes; > CHUNK *newblockp = HEAP_index(heap, newblock, CHUNK); > > newblockp->size = blockp->size - nbytes; > @@ -800,17 +802,17 @@ > } > > block += hheader->alignment; > - return block; > + return block >> GDK_VARSHIFT; > } > > void > -HEAP_free(Heap *heap, var_t block) > +HEAP_free(Heap *heap, var_t mem) > { > HEADER *hheader = HEAP_index(heap, 0, HEADER); > CHUNK *beforep; > CHUNK *blockp; > CHUNK *afterp; > - var_t after, before; > + size_t after, before, block = mem << GDK_VARSHIFT; > > if (hheader->alignment != 8 && hheader->alignment != 4) { > GDKfatal("HEAP_free: Heap structure corrupt\n"); > @@ -884,10 +886,10 @@ > HEAP_check(Heap *heap, HeapRepair *hr) > { > HEADER *hheader = HEAP_index(heap, 0, HEADER); > - var_t head = hheader->head, alignshift = 2; > - var_t block, nwords = (var_t) ((heap->free - 1) >> 7); > + size_t head = hheader->head, alignshift = 2; > + size_t block, nwords = (size_t) ((heap->free - 1) >> 7); > int *freemask; > - var_t prevblock = 0; > + size_t prevblock = 0; > CHUNK *blockp; > > hr->alignment = hheader->alignment; > @@ -922,8 +924,8 @@ > // Walk the freelist; register them in freemask > */ > for (block = hheader->head; block != 0; block = HEAP_index(heap, block, CHUNK)->next) { > - var_t idx = block >> alignshift; > - var_t pos = idx >> 5; > + size_t idx = block >> alignshift; > + size_t pos = idx >> 5; > int mask = 1 << (idx & 31); > > if ((block <= prevblock) && (block != 0)) { > @@ -942,8 +944,8 @@ > */ > block = hheader->firstblock; > while (block < heap->free) { > - var_t idx = block >> alignshift; > - var_t pos = idx >> 5; > + size_t idx = block >> alignshift; > + size_t pos = idx >> 5; > int mask = 1 << (idx & 31); > > hr->validmask[pos] |= mask; > @@ -965,8 +967,8 @@ > // Check if there are left over free blocks > */ > for (block = hheader->head; block != 0; block = HEAP_index(heap, block, CHUNK)->next) { > - var_t idx = block >> alignshift; > - var_t pos = idx >> 5; > + size_t idx = block >> alignshift; > + size_t pos = idx >> 5; > int mask = 1 << (idx & 31); > > if (freemask[pos] & mask) { > @@ -1046,14 +1048,14 @@ > if (hheader->head > heap->free) { > hheader->head = 0; /* cut off free block */ > } else if (hheader->head) { > - var_t idx = hheader->head; > + size_t idx = hheader->head; > > while (idx) { > CHUNK *blk = HEAP_index(heap, idx, CHUNK); > > if (idx + blk->size > heap->free) { > assert(heap->free - idx <= VAR_MAX); > - blk->size = (var_t) (heap->free - idx); /* cut off illegal tail of block */ > + blk->size = (size_t) (heap->free - idx); /* cut off illegal tail of block */ > } > if (blk->next > heap->free || blk->next < (idx + blk->size) || (blk->next & (hheader->alignment - 1))) { > blk->next = 0; /* cut off next block */ > > U gdk_qsort.mx > Index: gdk_qsort.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_qsort.mx,v > retrieving revision 1.34 > retrieving revision 1.35 > diff -u -d -r1.34 -r1.35 > --- gdk_qsort.mx 7 Jan 2009 14:13:48 -0000 1.34 > +++ gdk_qsort.mx 13 Apr 2009 23:36:24 -0000 1.35 > @@ -86,6 +86,7 @@ > char *offst; /* NULL or start of varsized heap */ > int hshift; /* log2 of hs */ > int tshift; /* log2 of hs */ > + int vshift; /* shift to be applied on var_ offsets */ > unsigned hs; /* width of head record */ > unsigned ts; /* width of tail record */ > void *h; /* start of head buns */ > @@ -189,7 +190,7 @@ > #endif > #endif > > -#define offset(p) (buf->offst + *(var_t*) (p)) > +#define offset(p) (buf->offst + ((*(var_t*) (p)) << buf->vshift)) > #define direct(p) (p) > > #define any_LE(a,b) ((buf->cmp)(a,b) <= 0) > @@ -432,6 +433,7 @@ > buf.ts = (unsigned) ts; > buf.hshift = ATOMelmshift(hs); > buf.tshift = ATOMelmshift(ts); > + buf.vshift = ATOMvarsized(tpe)?GDK_VARSHIFT:0; > buf.h = h; > if (!t) > t = &x; > > U gdk_bat.mx > Index: gdk_bat.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bat.mx,v > retrieving revision 1.214 > retrieving revision 1.215 > diff -u -d -r1.214 -r1.215 > --- gdk_bat.mx 3 Apr 2009 09:14:52 -0000 1.214 > +++ gdk_bat.mx 13 Apr 2009 23:36:24 -0000 1.215 > @@ -434,7 +434,7 @@ > BAT * > BATextend(BAT *b, BUN newcap) > { > - size_t hheap_size, theap_size; > + size_t hheap_size = newcap, theap_size = newcap; > > assert(newcap <= BUN_MAX); > BATcheck(b, "BATextend"); > @@ -453,10 +453,10 @@ > > b->batCapacity = newcap; > > - hheap_size = Hsize(b) * newcap; > + hheap_size *= Hsize(b); > if (b->H->heap.base && HEAPextend(&b->H->heap, hheap_size) < 0) > return NULL; > - theap_size = Tsize(b) * newcap; > + theap_size *= Tsize(b); > if (b->T->heap.base && HEAPextend(&b->T->heap, theap_size) < 0) > return NULL; > HASHdestroy(b); > > U gdk_ssort.mx > Index: gdk_ssort.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_ssort.mx,v > retrieving revision 1.15 > retrieving revision 1.16 > diff -u -d -r1.15 -r1.16 > --- gdk_ssort.mx 7 Jan 2009 14:13:48 -0000 1.15 > +++ gdk_ssort.mx 13 Apr 2009 23:36:24 -0000 1.16 > @@ -203,8 +203,8 @@ > } \ > } while (0) > > -#define ISLT_any(X, Y, ms) (((ms)->heap ? (*(ms)->compare)((ms)->heap + * (var_t *) (X), (ms)->heap + * (var_t *) (Y)) : (*(ms)->compare)((X), (Y))) < 0) > -#define ISLT_any_rev(X, Y, ms) (((ms)->heap ? (*(ms)->compare)((ms)->heap + * (var_t *) (X), (ms)->heap + * (var_t *) (Y)) : (*(ms)->compare)((X), (Y))) > 0) > +#define ISLT_any(X, Y, ms) (((ms)->heap ? (*(ms)->compare)((ms)->heap + ((*(var_t*) (X)) << GDK_VARSHIFT), (ms)->heap + ((*(var_t*) (Y)) << GDK_VARSHIFT)) : (*(ms)->compare)((X), (Y))) < 0) > +#define ISLT_any_rev(X, Y, ms) (((ms)->heap ? (*(ms)->compare)((ms)->heap + ((*(var_t*) (X)) << GDK_VARSHIFT), (ms)->heap + ((*(var_t*) (Y)) << GDK_VARSHIFT)) : (*(ms)->compare)((X), (Y))) > 0) > @= islt > #define ISLT_@1@2(X, Y, ms) (* (@1 *) (X) @3 * (@1 *) (Y)) > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

5 4

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/scheduler run_octopus.mx, , 1.17, 1.18
by Romulo Goncalves 14 Apr '09

14 Apr '09

It was this check in who broke the compilation in SQL. Martin forgot to remove the calls for OCTOPUSdrop From sql/src/backend/monet5/sql.mx Romulo Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/scheduler > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv21275 > > Modified Files: > run_octopus.mx > Log Message: > A new round for the octopus scheduler. Code still in testing phase. > > > U run_octopus.mx > Index: run_octopus.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/scheduler/run_octopus.mx,v > retrieving revision 1.17 > retrieving revision 1.18 > diff -u -d -r1.17 -r1.18 > --- run_octopus.mx 29 Jan 2009 18:17:10 -0000 1.17 > +++ run_octopus.mx 13 Apr 2009 14:23:01 -0000 1.18 > @@ -30,17 +30,17 @@ > re-directing requests to multiple sites. If there are no sites known, > then the code is executed linearly as is. > > -The scheduler runs all tentacles asynchronously. > +The scheduler runs all tentacles asynchronously if possible. > To make our live easier, we assume that all tentacles are > grouped together in a guarded block as follows: > > @verbatim > -barrier (parallel,a):= scheduler.octopus(timeout); > -a:= octopus.tentacle_1(); > +barrier (parallel,version):= scheduler.octopus(timeout); > +a:= octopus.tentacle_1(sitename,fcnname,version); > ... > -b:= octopus.tentacle_n(); > -a:= mat.pack(a,...,b); > -exit (parallel,a); > +b:= octopus.tentacle_n(sitename,fcnname,version); > +exit (parallel,version); > +z:= mat.pack(a,...,b); > @end verbatim > > This way the MAL flow of control simplifies skipping to the end > @@ -50,27 +50,17 @@ > Allowing MAL instructions inbetween complicates our work, > because it would mean that we have to do a flow analysis. > > -To make this work the scheduler needs a list of database worker. > -For the time being, this is an explicitly administered list here. > -When the octopus scheduling is called, we check the connection with > -the remote site. If it is down, it is re-activated using Merovingian. > - > +To make this work the scheduler needs a list of databases to play with. > +For the time being this consists of all the database known > +and ending with the phrase 'sea'. > +This list is obtained through the remote module using the > +support of Merovingian. The default is to use the local > +database as a target. > @{ > @mal > -pattern scheduler.octopus(timeout:int)(:bit, :bat[:any_1,:any_2]) > +pattern scheduler.octopus(t:int)(:bit,version:int) > address OCTOPUSrun > -comment "Run the program block in parallel, but don't wait longer then t seconds"; > - > -pattern scheduler.worker(dbnme:str, usr:str, pw:str) > -address OCTOPUSworker > -comment "Add a new worker to the known list "; > -pattern scheduler.worker(dbnme:str, usr:str, pw:str, host:str, port:int) > -address OCTOPUSworker > -comment "Add a worker site to the known list "; > - > -pattern scheduler.drop(dbnme:str) > -address OCTOPUSdrop > -comment "Remove a worker from the list"; > +comment "Run the program block in parallel, but don't wait longer then t seconds. Also fix a consistent database version."; > @h > #ifndef _RUN_OCTOPUS > #define _RUN_OCTOPUS > @@ -78,7 +68,7 @@ > #include "mal_instruction.h" > #include "mal_client.h" > > -/*#define DEBUG_RUN_OCTOPUS to trace processing */ > +#define DEBUG_RUN_OCTOPUS /* to trace processing */ > > #ifdef WIN32 > #ifndef LIBRUN_OCTOPUS > @@ -91,8 +81,6 @@ > #endif > > octopus_export str OCTOPUSrun(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > -octopus_export str OCTOPUSworker(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > -octopus_export str OCTOPUSdrop(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p); > #endif /* MAL_RUN_OCTOPUS */ > > @+ Octopus scheduling implementation > @@ -108,155 +96,156 @@ > #include "remote.h" > #include "alarm.h" > > -#define SITEasleep 0 > -#define SITElocal 1 > -#define SITEremote 2 > +typedef struct REGMAL{ > + str fcn; > + struct REGMAL *nxt; > +} *Registry; > > typedef struct { > - str alias; > - str db; /* connection parameters */ > + str name; > str usr; > - str pw; > - str host; /* used when merovigian is not running */ > - int port; > - int status; /* asleep, local, remote */ > -} Site; > + str pwd; > + Registry nxt; /* list of registered queries */ > +} Sea; > > #define MAXSITES 2048 /* should become dynamic at some point */ > -static Site *sites; > -static int nrsites = 0; > +static Sea sea[MAXSITES]; > +static int nrsea = 0; > > static str > OCTOPUSdiscover(Client cntxt){ > bat b1 = 0, b2 = 0; > - BAT *b; > + BAT *l1, *l2; > + BUN p,q; > str msg = MAL_SUCCEED; > + BATiter bi; > > - (void) cntxt; > - (void) b2; > + sea[nrsea].usr = GDKstrdup("monetdb"); > + sea[nrsea].pwd = GDKstrdup("monetdb"); > + sea[nrsea++].name= GDKstrdup(GDKgetenv("gdk_dbname")); > /* determine if sites are reachable */ > msg = RMTgetList(&b1,&b2); > if ( msg != MAL_SUCCEED) > return msg; > - b = BATdescriptor(b1); > - if ( b == NULL) > + l1 = BATdescriptor(b1); > + if ( l1 == NULL) > throw(MAL,"octopus.discover","No database list available"); > - BBPunfix(b1); > + l2 = BATdescriptor(b2); > + if ( l2 == NULL){ > + BBPreleaseref(b1); > + throw(MAL,"octopus.discover","No database list available"); > + } > + /* add the databases to the working set */ > + bi= bat_iterator(l1); > + BATloop(l1,p,q){ > + str t= (str) BUNtail(bi,p); > + > + if (nrsea ==MAXSITES) break; > + if (strlen(t) >= 3 && strcmp("sea", t+strlen(t)-3) == 0){ > + sea[nrsea].usr = GDKstrdup("monetdb"); > + sea[nrsea].pwd = GDKstrdup("monetdb"); > + sea[nrsea++].name= GDKstrdup(t); > +#ifdef DEBUG_RUN_OCTOPUS > + stream_printf(cntxt->fdout,"Found site %s\n",t); > +#else > + (void) cntxt; > +#endif > + } > + } > +#ifdef DEBUG_RUN_OCTOPUS > + stream_printf(cntxt->fdout,"Seas %d\n",nrsea); > +#endif > + BBPreleaseref(b1); > + BBPreleaseref(b2); > return MAL_SUCCEED; > } > > @- > -The replica is identified by database name. The host and port > -should address a merovingian to ensure the database instance is > -started. The default is to contact the local merovingian at > -default port 50000. > +We first register the tentacle at all sites and keep > +a list of those already sent. > @c > -str > -OCTOPUSworker(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +static int > +OCTOPUSfind(Sea s, str qry){ > + Registry r; > + for ( r= s.nxt; r; r= r->nxt) > + if ( strcmp(qry, r->fcn)==0) > + return 1; > + return 0; > +} > + > +@- > +The functions called by the octopus.exec_qry are to be > +registered at all sites. > +@c > +static str > +OCTOPUSregister(Client cntxt, MalBlkPtr mb, InstrPtr p) > { > - int idx; > + int i; > + str conn, fname, msg = MAL_SUCCEED; > + > + fname = getVarConstant(mb,getArg(p,2)).val.sval; > + for ( i= 0; i< nrsea; i++){ > + msg= RMTconnect(&conn, &sea[i].name, &sea[i].usr, &sea[i].pwd); > + if (msg ){ > + stream_printf(cntxt->fdout,"!%s\n",msg); > + GDKfree(msg); > + msg = NULL; > + continue; > + } > + if( !OCTOPUSfind(sea[i], fname) ){ > + msg = RMTregisterInternal(cntxt, conn, octopusRef, fname); > > - (void) mb; > - if (nrsites == MAXSITES) > - throw(MAL,"scheduler.worker","Too many worker"); > - mal_set_lock(mal_contextLock,"scheduler.worker"); > - if (nrsites == 0) > - sites = (Site *) GDKzalloc(sizeof(Site) * MAXSITES); > - idx = nrsites++; > - sites[idx].alias = NULL; > - sites[idx].db = GDKstrdup(*(str*) getArgReference(stk,pci,1)); > - sites[idx].usr = GDKstrdup(*(str*) getArgReference(stk,pci,2)); > - sites[idx].pw = GDKstrdup(*(str*) getArgReference(stk,pci,3)); > - if (pci->argc > 4){ > - sites[idx].host = GDKstrdup(*(str*) getArgReference(stk,pci,4)); > - sites[idx].port = *(int*) getArgReference(stk,pci,5); > - } else { > - sites[idx].host = GDKstrdup("localhost"); > - sites[idx].port = 50000; > - } > - mal_unset_lock(mal_contextLock,"scheduler.worker"); > #ifdef DEBUG_RUN_OCTOPUS > - stream_printf(cntxt->fdout,"# added worker %s %s %s %s\n", > - sites[idx].alias, sites[idx].usr, sites[idx].pw); > - sites[idx].db, sites[idx].usr, sites[idx].pw); > + stream_printf(cntxt->fdout,"octopus.%s registered at site %s\n", > + fname,sea[i].name); > + stream_printf(cntxt->fdout,"reply: %s\n",msg?msg:"ok"); > #else > - (void) cntxt; > + (void) cntxt; > #endif > - return MAL_SUCCEED; > -} > -str > -OCTOPUSdrop(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > -{ > - int i,j; > - str alias = *(str*) getArgReference(stk,pci,1); > - > - (void) cntxt; > - (void) mb; > - mal_set_lock(mal_contextLock,"scheduler.drop"); > - for (i=j=0; i<nrsites; i++){ > - if( strcmp(sites[i].alias, alias) ==0) { > - GDKfree(sites[i].alias); > - GDKfree(sites[i].db); > - GDKfree(sites[i].usr); > - GDKfree(sites[i].pw); > - GDKfree(sites[i].host); > - continue; > + if ( msg == MAL_SUCCEED){ > + Registry r= (Registry) GDKzalloc(sizeof(struct REGMAL)); > + r->fcn = GDKstrdup(getFunctionId(p)); > + r->nxt = sea[i].nxt; > + sea[i].nxt = r; > + } > } > - sites[j++] = sites[i]; > } > - nrsites = j; > - mal_unset_lock(mal_contextLock,"scheduler.drop"); > - if ( i == j ) > - throw(MAL,"scheduler.drop","Site not found"); > - return MAL_SUCCEED; > + GDKfree(conn); > + return msg; > } > @- > -The policy to check for sites is a multiphase phase process. > -First, we try to re-use a site where the operation was ran before. > -If not available, we select a non-used worker. > -If all this fails, we pick a random site to execute the plan. > +The work division looks at the system opportunities and > +replaces the target site in all instructions. > +The first policy is to simply perform round robin. > +The more advanced way is to negotiat with the remote sites. > @c > static str > -OCTOPUSexec(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +OCTOPUSworkdivision(Client cntxt, MalBlkPtr mb, int pc) > { > - int i=0, tries= nrsites * 2; > + static int rr=0; > str msg = MAL_SUCCEED; > + InstrPtr p; > + ValPtr cst; > > -redo: > + for (; pc< mb->stop; pc++){ > + if ( nrsea >1 && rr == 0) rr++; /* ignore default */ > + p= getInstrPtr(mb,pc); > + if ( p->barrier == EXITsymbol) > + break; > + assert( isVarConstant(mb, getArg(p,1)) ); > + cst = &getVarConstant(mb, getArg(p,1)); > + if( cst->val.sval) > + GDKfree(cst->val.sval); > + cst->val.sval= GDKstrdup(sea[rr].name); > + cst->len = strlen(cst->val.sval); > #ifdef DEBUG_RUN_OCTOPUS > - stream_printf(cntxt->fdout,"octopus.exec site selected %d\n",i); > + stream_printf(cntxt->fdout,"octopus site selected %s\n",sea[rr].name); > + printInstruction(cntxt->fdout,mb,0,p,LIST_MAL_STMT); > #else > - (void) cntxt; > -#endif > - > - /* register the plan remotely */ > - msg = RMTregisterInternal(cntxt, sites[i].alias, > - getModuleId(pci), getFunctionId(pci)); > - > - /* ignore a duplicate definition */ > - if (msg != MAL_SUCCEED && !strstr(msg,"Function already defined")){ > -#ifdef DEBUG_RUN_OCTOPUS > - stream_printf(cntxt->fdout,"octopus.exec failed to register plan %s.%s at site %s\n",getModuleId(pci),getFunctionId(pci),sites[i].alias); > - stream_printf(cntxt->fdout,"reply: %s\n",msg); > -#endif > - if (--tries <= 0) > - return msg; > - goto redo; > - } > - > - /* execute the plan as an independent process thread if it is local*/ > - /* otherwise activate it on the remote site passing parameters as well */ > - msg =runMALprocess(cntxt,mb,stk, getPC(mb,pci), getPC(mb,pci)+1); > - if ( msg != MAL_SUCCEED){ > -#ifdef DEBUG_RUN_OCTOPUS > - stream_printf(cntxt->fdout,"octopus.exec failed to run remote plan\n"); > + (void) cntxt; > #endif > - if (--tries <= 0) > - return msg; > - goto redo; > + rr= (rr+1) % nrsea; > } > - > - /* if it fails, we need to find another site */ > return msg; > } > @- > @@ -270,37 +259,52 @@ > We should be careful in accessing a site that runs out > of clients or any failure. It may cause the scheduler to > wait forever. > + > +The database version should indicate to the tentacles > +if it is time to refresh their caches. > +It should be obtained from the recycler where we > +know when updates have been taken place. > @c > str > OCTOPUSrun(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p) > { > int *res = (int*) getArgReference(stk,p,0); > + int *version = (int*) getArgReference(stk,p,1); > int timeout = *(int*) getArgReference(stk,p,2); > - int j,fnd, i = getPC(mb,p); > + int j,fnd, i = getPC(mb,p), threadcnt=0; > str msg = MAL_SUCCEED; > *res = 0; /* skip the block */ > > - if ( OCTOPUSdiscover(cntxt) == 0 ){ > + *version = 0; > + > + if ( (msg= OCTOPUSdiscover(cntxt)) ){ > #ifdef DEBUG_RUN_OCTOPUS > stream_printf(cntxt->fdout,"#Run in local serial mode\n"); > #endif > *res = 1; > - return MAL_SUCCEED; > + return msg; > + } > +@- > +Register the tentacle functions at all sites. > +@c > + if (nrsea > 1) { > + for (j= i+1; j<mb->stop ; j++){ > + p= getInstrPtr(mb,j); > + if ( p->barrier == EXITsymbol ) > + break; > + msg= OCTOPUSregister(cntxt,mb,p); > + if ( msg ) > + return msg; > + } > } > + msg= OCTOPUSworkdivision(cntxt,mb,i+1); > + if ( msg ) > + return msg; > + > /* do the actual parallel work */ > for (i++; i<mb->stop && msg == MAL_SUCCEED; i++){ > p= getInstrPtr(mb,i); > - /* don't do it remote if we need arguments */ > - if ( p->retc != p->argc){ > -#ifdef DEBUG_RUN_OCTOPUS > - stream_printf(cntxt->fdout,"#Run in local serial mode due to arguments\n"); > -#endif > - *res = 1; > - return MAL_SUCCEED; > - } > - if ( p->barrier == EXITsymbol ) > - break; > - if ( getModuleId(p) == matRef && getFunctionId(p) == packRef){ > + if ( p->barrier == EXITsymbol ){ > /* collect the results */ > do{ > fnd = 0; > @@ -311,10 +315,10 @@ > #endif > MT_sleep_ms(1000); > timeout--; > - } while ( fnd < p->argc-3 && timeout > 0 ); > + } while ( fnd < threadcnt && timeout > 0 ); > if (timeout <= 0) > - throw(MAL,"scheduler.pack","Execution time out"); > - return MATpackInternal(stk,p,1); > + throw(MAL,"scheduler.octopus","Execution time out"); > + break; > } > if ( getModuleId(p) != octopusRef) > throw(MAL,"scheduler.octopus","tentacle expected"); > @@ -323,7 +327,8 @@ > #else > (void) cntxt; > #endif > - msg = OCTOPUSexec(cntxt,mb,stk,p); > + msg =runMALprocess(cntxt,mb,stk, getPC(mb,p), getPC(mb,p)+1); > + threadcnt++; > } > return msg; > } > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/tools stethoscope.mx, , 1.35, 1.36
by Sjoerd Mullender 14 Apr '09

14 Apr '09

Should this have been done on the stable branch? Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/tools > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv30964 > > Modified Files: > stethoscope.mx > Log Message: > The arguments don't have alternatives to worry about. > > > U stethoscope.mx > Index: stethoscope.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/tools/stethoscope.mx,v > retrieving revision 1.35 > retrieving revision 1.36 > diff -u -d -r1.35 -r1.36 > --- stethoscope.mx 20 Mar 2009 13:41:48 -0000 1.35 > +++ stethoscope.mx 12 Apr 2009 20:35:37 -0000 1.36 > @@ -258,36 +258,26 @@ > }; > for(;;){ > int option_index=0; > - int c= getopt_long(argc, argv, "d:u:P:p:?:h", > + int c= getopt_long(argc, argv, "d:u:P:p:?:h:g", > long_options, &option_index); > if( c == -1) > break; > switch(c){ > case 'd': > - if (strcmp(long_options[option_index].name, "dbname") == 0) { > - dbname= optarg; > - break; > - } > + dbname= optarg; > + break; > case 'u': > - if (strcmp(long_options[option_index].name, "user") == 0) { > - user= optarg; > - break; > - } > + user= optarg; > + break; > case 'P': > - if (strcmp(long_options[option_index].name, "password") == 0) { > - password= optarg; > - break; > - } > + password= optarg; > + break; > case 'p': > - if (strcmp(long_options[option_index].name, "port") == 0) { > - portnr= atol(optarg); > - break; > - } > + portnr= atol(optarg); > + break; > case 'g': > - if (strcmp(long_options[option_index].name, "gnuplot") == 0) { > - gnuplot= TRUE; > - break; > - } > + gnuplot= TRUE; > + break; > case 'h': > if (strcmp(long_options[option_index].name, "help") == 0) { > printf("help\n"); > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_session.mx, , 1.186, 1.187
by Sjoerd Mullender 14 Apr '09

14 Apr '09

Should this have been done on the stable branch? Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv30687 > > Modified Files: > mal_session.mx > Log Message: > On the console it could happen that a MAL syntax error caused a crash > in the next statement due to incomplete cleanup. > > > U mal_session.mx > Index: mal_session.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_session.mx,v > retrieving revision 1.186 > retrieving revision 1.187 > diff -u -d -r1.186 -r1.187 > --- mal_session.mx 2 Apr 2009 17:51:41 -0000 1.186 > +++ mal_session.mx 11 Apr 2009 07:07:48 -0000 1.187 > @@ -495,10 +495,14 @@ > table may have grown in size. > > @= restoreState > - if( c->listing) > - printFunction(c->fdout,c->curprg->def, 0, c->listing); > - MSresetVariables(c,c->curprg->def, c->glb, oldstate.vtop); > - resetMalBlk(c->curprg->def,oldstate.stop); > +{ int i; > + for ( i=1; i<c->curprg->def->vtop; i++) > + freeVariable(c->curprg->def,i); > + GDKfree(c->glb); > + c->glb = NULL; > + c->curprg->def->vtop = 1; > + MSinitClientPrg(c,"user", "main"); > +} > @c > str > MALinitClient(Client c) > @@ -511,15 +515,9 @@ > str > MALexitClient(Client c) > { > - int i; > if (c->glb && c->curprg->def->errors == 0) > garbageCollector(c, c->curprg->def, c->glb,TRUE); > - for ( i=1; i<c->curprg->def->vtop; i++) > - freeVariable(c->curprg->def,i); > - GDKfree(c->glb); > - c->glb = NULL; > - c->curprg->def->vtop = 1; > - MSinitClientPrg(c,"user", "main"); > + @:restoreState@ > return NULL; > } > > @@ -547,18 +545,20 @@ > str > MALparser(Client c) > { > - InstrPtr p; > MalBlkRecord oldstate; > > c->curprg->def->errors = 0; > oldstate = *c->curprg->def; > > prepareMalBlk(c->curprg->def, CURRENT(c)); > - if (parseMAL(c, c->curprg) || c->curprg->def->errors) { > + if (parseMAL(c, c->curprg) || c->curprg->def->errors || > + getInstrPtr(c->curprg->def, 0)->token != FUNCTIONsymbol ){; > /* just complete it for visibility */ > pushEndInstruction(c->curprg->def); > /* caught errors */ > showErrors(c); > + if( c->listing) > + printFunction(c->fdout,c->curprg->def, 0, c->listing); > @:restoreState@ > /* now the parsing is done we should advance the stream */ > c->fdin->pos += c->yycur; > @@ -577,15 +577,12 @@ > if (c->curprg->def->stop == 1) > return MAL_SUCCEED; > > - p = getInstrPtr(c->curprg->def, 0); > - if (p->token != FUNCTIONsymbol) { > - @:restoreState@ > - throw(MAL, "MAL.parser", "Function signature missing"); > - } > pushEndInstruction(c->curprg->def); > chkProgram(c->nspace, c->curprg->def); > if (c->curprg->def->errors) { > showErrors(c); > + if( c->listing) > + printFunction(c->fdout,c->curprg->def, 0, c->listing); > @:restoreState@ > throw(MAL, "MAL.parser", "Semantic errors detected"); > } > @@ -619,6 +616,7 @@ > > if (prg->def->errors > 0) { > showErrors(c); > + /* constructed MAL programs may not have been shown */ > if (c->listing ) > printFunction(c->fdout, c->curprg->def, 0, c->listing); > @:restoreState@ > @@ -651,8 +649,6 @@ > str place = getExceptionPlace(msg); > showException(getExceptionType(msg), place, getExceptionMessage(msg)); > GDKfree(place); > - if (!c->listing) > - printFunction(c->fdout, c->curprg->def, 0, c->listing); > showErrors(c); > } > MSresetVariables(c,prg->def, c->glb, 0); > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

[Monetdb-developers] 32GB string heaps for 64-bits/oid32
by Peter Boncz 14 Apr '09

14 Apr '09

MonetDB developers, As my Easter contribution, I just checked in changes that introduce a change in which strings are stored in BATs, that affects only the (not ver often used?) setting of 64-bits compilation with 32-bits oids. This is achieved with the configure options --enable-bits=64 --enable-oid32. By default, the oid width (used to identify tuples) follows the addressing bit with (hence 64-bits pointers and 64-bits oids). The motivation to use 32-bits oids is that many intermediates (often consisting of oid columns) and also the hash tables and the string offsets (which use the type var_t that follows oid in its width) become smaller. For those 64-bit users that are interested in reducing memory consumption, 32-bits oids will likely reduce the MonetDB memory footprint by 20-40%. Especially those who are just running out of space may benefit. Using 32-bits oids has the effect that only 4G tuples may be stored in a single (intermediate) table. The vaste majority of database problems run on a single box (as MonetDB does) have this characteristic. Regrettably, in the past, an additional resitriction (due to var_t) was that each individual string column could hold max 4GB as well. Note that MonetDB does best-effort string de-duplication in a column, so your tring data might be smaller than you think. By the way, there is no good way to get precise stats on the physical size of a column, but you can scan for .theap files in the bat/ directory of your database installation -- those are the string heaps. So, the change checked in interprets the var_t no longer as a byte offset, but an index that is multiplied by 8 (GDK_VARSHIFT=3 for 64bits/oid32, zero otherwise -- so then it is a no-op removed by the compiler even). It also means that strings in the heap are aligned on 8-byte boundaries, then. Note that strings often are aligned to 4-byte boundaries (32-bits) or 8-byte boundaries (64-bits) anyway, unless the duplicate elimination really gave up, which happens if only a very low duplicate locality exists. Also, the string heaps additionally stored pointers in the heap for the de-duplication hash-table which could lead to an avg overhead of 4 (half of 8 byte alignment) + 8 (pointer) = 12 bytes per string. The change also improves the non GDK_VARSHIFT compilation mods in reducing this overhead to 2 bytes only. For 64-bits/oid32, the overhead is 2+4, due to the forced alignment. So, in general, string storage got more compact across the board (the effect will depend on your string distribution), notehowever that for 32/32 and 64/64 the binary BAT format has *NOT* changed. No need to export and re-import. As for my original goal, for 64-bits/oid32 the effect is that a single column can now contain 32GB of (de-duplicated) string data. That aligns much better with the 4G tuple limit than the previous 4GB. I made this change actually mostly because I think oid32 reduces memory consumption, not even so much the var_t (string) offset width. Still as var_t and oid move together, I had to do something. As an alternative, we could make var_t independently configurable (such that we have 32-bits oids and 64-bits var_t). This still remains an option for the future. Another potential option for the future is to set the string alignment on a bat-for-bat (per-column) basis. This is also feasible, but would require more API changes. A final point of attention is the stability/fallout of this change. I have tried to be complete -- my changes span all our CVS repositories. However, I did not that in general the support for 64-bits/oid32 is not as perfect as it should be, as evidenced by the TestWeb. Strinking examples are queries in the src/benchmarks/ATIS (especially the simple join query seems easy to debug), and TPC-H. But in general the 64/32 column shows more red than other compilation settings. I think t=it is an interesting setting, we have used it consistently for our XMark XQuery benchmarking, and I think it also serves well for TPC-H (up to SF300) and e.g. the open streetmap data. Therefore, I think it would be worthwhile if some of these long standing M5/SQL issues that are specific to 64/32 would be fixed. thanks for your attention, Peter

1 0

[Monetdb-developers] FreeBSD port of MonetDB Feb2009-SP1 release
by George Cox 13 Apr '09

13 Apr '09

Hi, I've just finished my first attempt at creating some ports files for the installation of MonetDB on FreeBSD. I've put them up at: http://www.gjvc.com/software/freebsd/ports/monetdb/ Please have a look and let me know what you think. Things to do include (but not limited to :-)) are: * checking the packing list for each component * adding rc.d scripts * adding monetdb user best; gjvc

2 3

[Monetdb-developers] Another big group by query
by Stefan de Konink 13 Apr '09

13 Apr '09

Hi All, Might be better to go by the mailinglist instead of adding many people to the CC. Currently I am running the latest CVS code disabled mitosis and am running the following code: CREATE TABLE "sys"."way_nds" ( "way" int NOT NULL, "idx" int NOT NULL, "to_node" int, CONSTRAINT "pk_way_nds" PRIMARY KEY ("way", "idx") ); select count(way), way, to_node from way_nds group by way, to_node having count(way) > 1; The table contains about 362420451 rows, and I realised too late there could be many results comming from it. top - 03:12:15 up 1 day, 17:28, 8 users, load average: 1.00, 1.00, 1.00 Tasks: 193 total, 1 running, 192 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 94.6%id, 5.2%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 66113464k total, 39353192k used, 26760272k free, 326968k buffers Swap: 134215032k total, 94284k used, 134120748k free, 8489128k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11856 konink 20 0 51.5g 34g 5.9g S 0.3 54.6 45:34.93 mserver5 But again the load is low, the memory empty and it seems to do so few. Is there something that would be interesting to look at? Stefan

3 7

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_bbp.mx, , 1.251, 1.252 gdk_utils.mx, , 1.244, 1.245
by Sjoerd Mullender 09 Apr '09

09 Apr '09

Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv27450 > > Modified Files: > gdk_bbp.mx gdk_utils.mx > Log Message: > gdk_utils.mx: > - performance fix: do not use malloc() on Linux either > its malloc() implementation appears to give sizable private arenas to > each new thread that performs significant malloc work; this leads > to huge memory fragmentation (especially when used with mitosis) > > gdk_bbp.mx: > - BBPtrim() should not be allowed to trim logical views > this may lead to BBPdecref errors!! Is this a bug fix that should also be applied to the stable branch? > > > U gdk_bbp.mx > Index: gdk_bbp.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bbp.mx,v > retrieving revision 1.251 > retrieving revision 1.252 > diff -u -d -r1.251 -r1.252 > --- gdk_bbp.mx 31 Mar 2009 13:42:54 -0000 1.251 > +++ gdk_bbp.mx 9 Apr 2009 15:29:18 -0000 1.252 > @@ -1600,7 +1600,7 @@ > @{ > @h > #define BBPswappable(b) ((b) && BBP_refs((b)->batCacheid) == 0) > -#define BBPtrimmable(b) (BBPswappable(b) && !isVIEW(b) && (BBP_status((b)->batCacheid)&BBPWAITING) == 0) > +#define BBPtrimmable(b) (BBPswappable(b) && b->P->lview == 0 && isVIEW(b) == 0 && (BBP_status((b)->batCacheid)&BBPWAITING) == 0) > @- > The @%BBP_ref@ contains the amount of live references to a BAT. > These might be in recursive BATs, C or MIL variables. The count is > > U gdk_utils.mx > Index: gdk_utils.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v > retrieving revision 1.244 > retrieving revision 1.245 > diff -u -d -r1.244 -r1.245 > --- gdk_utils.mx 7 Apr 2009 14:22:55 -0000 1.244 > +++ gdk_utils.mx 9 Apr 2009 15:29:18 -0000 1.245 > @@ -1053,7 +1053,6 @@ > #endif > } > size = (size + 7) & ~7; /* round up to a multiple of eight */ > -#ifndef HAVE_POSIX_FADVISE > if (size > GDK_mem_bigsize) { > size_t newsize = size + sizeof(size_t) + sizeof(size_t); > size_t newmax = MAX(*maxsize, newsize); > @@ -1068,7 +1067,6 @@ > *maxsize = newmax - (sizeof(size_t) + sizeof(size_t)); > return (void *) s; > } > -#endif > CHKMEM(size, 0); > GDKmalloc_prefixsize(s, size); > if (s == NULL) { > @@ -1205,7 +1203,6 @@ > } > } > #ifndef GDK_MEM_MISALIGN > -#ifndef HAVE_POSIX_FADVISE > else if (size <= GDK_mem_bigsize) { > void *oldblk = blk; > size_t newsize = size + 8; > @@ -1239,7 +1236,6 @@ > return blk; > } > } > -#endif > /* alloc&copy due to failed realloc (may be very big heap that needs vm) */ > #else > /* alloc&copy because we cannot trust realloc due to misalignment */ > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

2 2