- developers-list - monetdb.org

Re: [Monetdb-developers] MonetDB: xid - BATderiveHeadProps & COLsettrivprop: do not set/d...
by Stefan Manegold 25 Jul '12

25 Jul '12

yep. thanks! testing just reminded me as well ;-) fix on its way. Sorry for premature checking! Stefan ----- Original Message ----- > On 2012-07-25 09:24, Stefan Manegold wrote: > > Changeset: 4da710a71d6f for MonetDB > > URL: > > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4da710a71d6f > > Modified Files: > > gdk/gdk.h > > gdk/gdk_bat.c > > Branch: xid > > Log Message: > > > > BATderiveHeadProps & COLsettrivprop: do not set/derive properties > > for xid-compressed BATs > > their content is known and designed to not match their properties > > see also changesets 805110fd0096 & b33eba6cfb09 > > > > > > diffs (36 lines): > > > > diff --git a/gdk/gdk.h b/gdk/gdk.h > > --- a/gdk/gdk.h > > +++ b/gdk/gdk.h > > @@ -1699,8 +1699,10 @@ gdk_export void GDKqsort_rev(void *h, vo > > #define BATtkey(b) (b->tkey != FALSE || BATtdense(b)) > > > > /* set some properties that are trivial to deduce */ > > +/* but not for xid-compressed BATs */ > > #define COLsettrivprop(b, col) \ > > do { \ > > + if (b->htype == TYPE_oid && b->H->heap.xidcompressed) { \ > > Shouldn't that be: > if (b->htype != TYPE_oid || !b->H->heap.xidcompressed) { > ? > You want the properties to be set if the heap is *not* compressed. > > > if ((col)->type == TYPE_void) { \ > > if ((col)->seq == oid_nil) { \ > > if (!(col)->nil && (b)->batCount >= 1) { \ > > @@ -1772,6 +1774,7 @@ gdk_export void GDKqsort_rev(void *h, vo > > (b)->batDirtydesc = 1; \ > > } \ > > } \ > > + } \ > > } while (0) > > #define BATsettrivprop(b) \ > > do { \ > > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > > --- a/gdk/gdk_bat.c > > +++ b/gdk/gdk_bat.c > > @@ -3096,6 +3096,10 @@ BATderiveHeadProps(BAT *b, int expensive > > assert(b != NULL); > > if (b == NULL) > > return; > > + if (b->htype == TYPE_oid && b->H->heap.xidcompressed) { > > + /* avoid property deriving on xid-compressed BATs */ > > + return; > > + } > > assert((b->hkey & BOUND2BTRUE) == 0); > > COLsettrivprop(b, b->H); > > cmpf = BATatoms[b->htype].atomCmp; > > _______________________________________________ > > Checkin-list mailing list > > Checkin-list(a)monetdb.org > > http://mail.monetdb.org/mailman/listinfo/checkin-list > > > > > -- > Sjoerd Mullender <sjoerd(a)monetdb.org> > MonetDB: Column store pioneers <http://www.monetdb.org/> > > > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list >

1 0

Re: [Monetdb-developers] MonetDB: xid - BATassertHeadProps: avoid property checking on xi...
by Sjoerd Mullender 24 Jul '12

24 Jul '12

On 2012-07-24 22:13, Stefan Manegold wrote: > Changeset: 805110fd0096 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=805110fd0096 > Modified Files: > gdk/gdk_bat.c > Branch: xid > Log Message: > > BATassertHeadProps: avoid property checking on xid-compressed BATs > their content is not plain valid data, > and their properties are those of the uncompressed BAT > > > diffs (13 lines): > > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -2892,6 +2892,9 @@ BATassertHeadProps(BAT *b) > return; > } > > + > + /* no property checking on xid-compressed BATs */ > + if (b->htype != TYPE_oid || !b->H->heap.xidcompressed) > PROPDEBUG { /* only do a scan if property checking is requested */ > if (b->hsorted || b->hrevsorted || !b->hkey) { > /* if sorted (either way), or we don't have to I really don't like this layout. Please consider using return after the if statement (with complementary test). I would also find that clearer. -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] MonetDB: default - Cleanup of logger code to split head and tail...
by Niels Nes 18 Jul '12

18 Jul '12

On Wed, Jul 18, 2012 at 12:55:31PM +0200, Niels Nes wrote: > Changeset: 982fdf69e13b for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=982fdf69e13b > Modified Files: > gdk/gdk_logger.c > gdk/gdk_logger.h > sql/server/rel_optimizer.c > sql/storage/bat/bat_logger.c > sql/storage/restrict/restrict_logger.c > Branch: default > Log Message: > > Cleanup of logger code to split head and tail bat into 2 void headed bats This checkin breaks the current on disk format, ie if your using the defaul branch a reload is needed. Before the (september) release we need to add updating functions for atleast SQL. Niels > > > diffs (truncated from 723 to 300 lines): > > diff --git a/gdk/gdk_logger.c b/gdk/gdk_logger.c > --- a/gdk/gdk_logger.c > +++ b/gdk/gdk_logger.c > @@ -73,7 +73,10 @@ > #include "gdk_logger.h" > #include <string.h> > > - > +static BUN BUNfndT( BAT *b, ptr v) > +{ > + return BUNfnd(BATmirror(b), v); > +} > /* > * @- > * The log record encoding is geared at reduced storage space, but > @@ -218,12 +221,11 @@ la_bat_clear(logger *lg, logaction *la) > BAT *b; > > /* do we need to skip these old updates */ > - if (BATcount(lg->snapshots)) { > - BUN p = BUNfnd(lg->snapshots, &bid); > + if (BATcount(lg->snapshots_bid)) { > + BUN p = BUNfndT(lg->snapshots_bid, &bid); > > if (p != BUN_NONE) { > - BATiter i = bat_iterator(lg->snapshots); > - int tid = *(int *) BUNtloc(i, p); > + int tid = *(int *) Tloc(lg->snapshots_tid, p); > > if (lg->tid <= tid) > return; > @@ -244,16 +246,18 @@ static int > log_read_seq(logger *lg, logformat *l) > { > int seq = l->nr; > - lng id; > + lng val; > + BUN p; > > - if (!mnstr_readLng(lg->log, &id)) > + if (!mnstr_readLng(lg->log, &val)) > return LOG_ERR; > > - if (BUNfnd(lg->seqs, &seq) != BUN_NONE) { > - BUNdelHead(lg->seqs, &seq, FALSE); > + if ((p = BUNfndT(lg->seqs_id, &seq)) != BUN_NONE) { > + BUNdelete(lg->seqs_id, p, FALSE); > + BUNdelete(lg->seqs_val, p, FALSE); > } > - BUNins(lg->seqs, &seq, &id, FALSE); > - > + BUNappend(lg->seqs_id, &seq, FALSE); > + BUNappend(lg->seqs_val, &val, FALSE); > return LOG_OK; > } > > @@ -404,12 +408,11 @@ la_bat_updates(logger *lg, logaction *la > return; /* ignore bats no longer in the catalog */ > > /* do we need to skip these old updates */ > - if (BATcount(lg->snapshots)) { > - BUN p = BUNfnd(lg->snapshots, &bid); > + if (BATcount(lg->snapshots_bid)) { > + BUN p = BUNfndT(lg->snapshots_bid, &bid); > > if (p != BUN_NONE) { > - BATiter i = bat_iterator(lg->snapshots); > - int tid = *(int *) BUNtloc(i, p); > + int tid = *(int *) Tloc(lg->snapshots_tid, p); > > if (lg->tid <= tid) > return; > @@ -477,10 +480,14 @@ la_bat_destroy(logger *lg, logaction *la > log_bid bid = logger_find_bat(lg, la->name); > > if (bid) { > + BUN p; > + > logger_del_bat(lg, bid); > - if (BUNfnd(lg->snapshots, &bid) != BUN_NONE) { > - BUNdelHead(lg->snapshots, &bid, FALSE); > - BUNins(lg->snapshots, &bid, &lg->tid, FALSE); > + if ((p = BUNfndT(lg->snapshots_bid, &bid)) != BUN_NONE) { > + BUNdelete(lg->snapshots_bid, p, FALSE); > + BUNdelete(lg->snapshots_tid, p, FALSE); > + BUNappend(lg->snapshots_bid, &bid, FALSE); > + BUNappend(lg->snapshots_tid, &lg->tid, FALSE); > } > } > } > @@ -564,15 +571,19 @@ la_bat_use(logger *lg, logaction *la) > { > log_bid bid = la->nr; > BAT *b = BATdescriptor(bid); > + BUN p; > > if (!b) { > GDKerror("logger: could not use bat (%d) for %s\n", (int) bid, la->name); > return; > } > logger_add_bat(lg, b, la->name); > - if (BUNfnd(lg->snapshots, &b->batCacheid) != BUN_NONE) > - BUNdelHead(lg->snapshots, &b->batCacheid, FALSE); > - BUNins(lg->snapshots, &b->batCacheid, &lg->tid, FALSE); > + if ((p = BUNfndT(lg->snapshots_bid, &b->batCacheid)) != BUN_NONE) { > + BUNdelete(lg->snapshots_bid, p, FALSE); > + BUNdelete(lg->snapshots_tid, p, FALSE); > + } > + BUNappend(lg->snapshots_bid, &b->batCacheid, FALSE); > + BUNappend(lg->snapshots_tid, &lg->tid, FALSE); > logbat_destroy(b); > } > > @@ -875,17 +886,23 @@ static int > logger_commit(logger *lg) > { > int id = LOG_SID; > + BUN p; > > if (lg->debug & 1) > fprintf(stderr, "logger_commit\n"); > > - BUNdelHead(lg->seqs, &id, FALSE); > - BUNins(lg->seqs, &id, &lg->id, FALSE); > + p = BUNfndT(lg->seqs_id, &id); > + BUNdelete(lg->seqs_id, p, FALSE); > + BUNdelete(lg->seqs_val, p, FALSE); > + BUNappend(lg->seqs_id, &id, FALSE); > + BUNappend(lg->seqs_val, &lg->id, FALSE); > > /* cleanup old snapshots */ > - if (BATcount(lg->snapshots)) { > - BATclear(lg->snapshots, FALSE); > - BATcommit(lg->snapshots); > + if (BATcount(lg->snapshots_bid)) { > + BATclear(lg->snapshots_bid, FALSE); > + BATclear(lg->snapshots_tid, FALSE); > + BATcommit(lg->snapshots_bid); > + BATcommit(lg->snapshots_tid); > } > return bm_commit(lg); > } > @@ -919,34 +936,34 @@ check_version(logger *lg, FILE *fp) > } > > static int > -bm_subcommit(BAT *list, BAT *catalog, BAT *extra, int debug) > +bm_subcommit(BAT *list_bid, BAT *list_nme, BAT *catalog_bid, BAT *catalog_nme, BAT *extra, int debug) > { > BUN p, q; > - BUN nn = 2 + (list->batFirst > list->batDeleted ? list->batFirst - list->batDeleted : 0) + BATcount(list) + (extra ? BATcount(extra) : 0); > + BUN nn = 3 + (list_bid->batFirst > list_bid->batDeleted ? list_bid->batFirst - list_bid->batDeleted : 0) + BATcount(list_bid) + (extra ? BATcount(extra) : 0); > bat *n = (bat*)GDKmalloc(sizeof(bat) * nn); > int i = 0; > - BATiter iter = bat_iterator(list); > + BATiter iter = (list_nme)?bat_iterator(list_nme):bat_iterator(list_bid); > int res; > > n[i++] = 0; /* n[0] is not used */ > > /* first loop over deleted then over current and new */ > - for (p = list->batDeleted; p < list->batFirst; p++) { > - bat col = *(log_bid *) BUNhead(iter, p); > + for (p = list_bid->batDeleted; p < list_bid->batFirst; p++) { > + bat col = *(log_bid *) Tloc(list_bid, p); > > if (debug & 1) > fprintf(stderr, "commit deleted %s (%d) %s\n", > BBPname(col), col, > - (list == catalog) ? BUNtail(iter, p) : "snapshot"); > + (list_bid == catalog_bid) ? BUNtail(iter, p) : "snapshot"); > n[i++] = ABS(col); > } > - BATloop(list, p, q) { > - bat col = *(log_bid *) BUNhead(iter, p); > + BATloop(list_bid, p, q) { > + bat col = *(log_bid *) Tloc(list_bid, p); > > if (debug & 1) > fprintf(stderr, "commit new %s (%d) %s\n", > BBPname(col), col, > - (list == catalog) ? BUNtail(iter, p) : "snapshot"); > + (list_bid == catalog_bid) ? BUNtail(iter, p) : "snapshot"); > n[i++] = ABS(col); > } > if (extra) { > @@ -957,14 +974,16 @@ bm_subcommit(BAT *list, BAT *catalog, BA > if (debug & 1) > fprintf(stderr, "commit extra %s %s\n", > name, > - (list == catalog) ? BUNtail(iter, p) : "snapshot"); > + (list_bid == catalog_bid) ? BUNtail(iter, p) : "snapshot"); > n[i++] = ABS(BBPindex(name)); > } > } > /* now commit catalog, so it's also up to date on disk */ > - n[i++] = ABS(catalog->batCacheid); > + n[i++] = ABS(catalog_bid->batCacheid); > + n[i++] = ABS(catalog_nme->batCacheid); > assert((BUN) i <= nn); > - BATcommit(catalog); > + BATcommit(catalog_bid); > + BATcommit(catalog_nme); > res = TMsubcommit_list(n, i); > GDKfree(n); > return res; > @@ -992,8 +1011,8 @@ logger_new(int debug, char *fn, char *lo > FILE *fp; > char filename[BUFSIZ]; > char bak[BUFSIZ]; > - log_bid seqs = 0; > - bat catalog; > + log_bid seqs_id = 0; > + bat catalog_bid, catalog_nme; > > if (lg == NULL) > return NULL; > @@ -1022,9 +1041,12 @@ logger_new(int debug, char *fn, char *lo > lg->postfuncp = postfuncp; > lg->log = NULL; > lg->end = 0; > - lg->catalog = NULL; > - lg->snapshots = NULL; > - lg->seqs = NULL; > + lg->catalog_bid = NULL; > + lg->catalog_nme = NULL; > + lg->snapshots_bid = NULL; > + lg->snapshots_tid = NULL; > + lg->seqs_id = NULL; > + lg->seqs_val = NULL; > > snprintf(filename, BUFSIZ, "%s%s", lg->dir, LOGFILE); > snprintf(bak, BUFSIZ, "%s.bak", filename); > @@ -1040,10 +1062,10 @@ logger_new(int debug, char *fn, char *lo > } > fp = fopen(filename, "r"); > > - snprintf(bak, BUFSIZ, "%s_catalog", fn); > - catalog = BBPindex(bak); > + snprintf(bak, BUFSIZ, "%s_catalog_bid", fn); > + catalog_bid = BBPindex(bak); > > - if (catalog == 0) { > + if (catalog_bid == 0) { > log_bid bid = 0; > > /* catalog does not exist, so the log file also shouldn't exist */ > @@ -1055,16 +1077,24 @@ logger_new(int debug, char *fn, char *lo > goto error; > } > > - lg->catalog = logbat_new(TYPE_int, TYPE_str, BATSIZE); > + lg->catalog_bid = logbat_new(TYPE_void, TYPE_int, BATSIZE); > + lg->catalog_nme = logbat_new(TYPE_void, TYPE_str, BATSIZE); > if (debug) > fprintf(stderr, "create %s catalog\n", fn); > > - bid = lg->catalog->batCacheid; > + /* Make persistent */ > + bid = lg->catalog_bid->batCacheid; > + BBPincref(bid, TRUE); > + BATmode(lg->catalog_bid, PERSISTENT); > + snprintf(bak, BUFSIZ, "%s_catalog_bid", fn); > + BBPrename(lg->catalog_bid->batCacheid, bak); > > /* Make persistent */ > + bid = lg->catalog_nme->batCacheid; > BBPincref(bid, TRUE); > - BATmode(lg->catalog, PERSISTENT); > - BBPrename(lg->catalog->batCacheid, bak); > + BATmode(lg->catalog_nme, PERSISTENT); > + snprintf(bak, BUFSIZ, "%s_catalog_nme", fn); > + BBPrename(lg->catalog_nme->batCacheid, bak); > > if (!GDKcreatedir(filename)) { > logger_fatal("logger_new: cannot create directory for log file %s\n", > @@ -1082,7 +1112,7 @@ logger_new(int debug, char *fn, char *lo > fclose(fp); > fp = NULL; > > - if (bm_subcommit(lg->catalog, lg->catalog, NULL, lg->debug) != 0) { > + if (bm_subcommit(lg->catalog_bid, lg->catalog_nme, lg->catalog_bid, lg->catalog_nme, NULL, lg->debug) != 0) { > /* cannot commit catalog, so remove log */ > unlink(filename); > goto error; > @@ -1092,8 +1122,11 @@ logger_new(int debug, char *fn, char *lo > require a logical reference we also add a logical > reference for the persistent bats */ > BUN p, q; > - BAT *b = BATdescriptor(catalog); > - BATiter bi = bat_iterator(b); > + BAT *b = BATdescriptor(catalog_bid), *n; > + > + snprintf(bak, BUFSIZ, "%s_catalog_nme", fn); > + catalog_nme = BBPindex(bak); > + n = BATdescriptor(catalog_nme); > > /* the catalog exists, and so should the log file */ > if (fp == NULL) { > @@ -1107,42 +1140,63 @@ logger_new(int debug, char *fn, char *lo > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

Re: [Monetdb-developers] MonetDB: default - Partial de-mx batmtime
by Stefan Manegold 18 Jul '12

18 Jul '12

Just wondering ... (and trying to learn): On Wed, Jul 18, 2012 at 10:41:04AM +0200, Martin Kersten wrote: > Changeset: 29b4294d4bea for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=29b4294d4bea > Modified Files: > monetdb5/modules/kernel/batmtime.mx > Branch: default > Log Message: > > Partial de-mx batmtime > Rest should be taken care of by shell scripting > > > diffs (truncated from 316 to 300 lines): > > diff --git a/monetdb5/modules/kernel/batmtime.mx b/monetdb5/modules/kernel/batmtime.mx > --- a/monetdb5/modules/kernel/batmtime.mx > +++ b/monetdb5/modules/kernel/batmtime.mx > @@ -30,28 +30,25 @@ All Rights Reserved. > * In line with the batcalc module, we assume that > * if two bat operands are provided that they are already > * aligned on the head. Moreover, the head of the BATs > - * are limited to :void, which can be cheaply realized using > + * are limited to :oid, which can be cheaply realized using > * the GRPsplit operation. > */ > @= compareOp > command batcalc.@2( l:bat[:oid,:@1], r:bat[:oid,:@1]) :bat[:oid,:bit] > -address MTIMEbat_@1_@3; > - > -command batcalc.@2( l:bat[:void,:@1], r:bat[:void,:@1]) :bat[:void,:bit] > address MTIMEbat_@1_@3 > comment "Compare a bat of @1 against each other"; > command batcalc.@2( l:bat[:oid,:@1], r:@1) :bat[:oid,:bit] > address MTIMEbat_@1_@3_cst; > -command batcalc.@2( l:bat[:void,:@1], r:@1) :bat[:void,:bit] > +command batcalc.@2(r:@1, l:bat[:oid,:@1]) :bat[:oid,:bit] Is the above signature change (swapping left & right operand) intended? If so, what is the reason for it? > address MTIMEbat_@1_cst_@3 > comment "Compare a bat of @1 against a singleton"; > > -command batcalc.@2(a:bat[:void,:bit], l:bat[:void,:@1], r:bat[:void,:@1]) :bat[:void,:bit] > +command batcalc.@2(a:bat[:oid,:bit], l:bat[:oid,:@1], r:bat[:oid,:@1]) :bat[:oid,:bit] > address MTIMEaccum_@1_@3 > comment "Compare a bat of @1 against each other"; > -command batcalc.@2(a:bat[:void,:bit], l:bat[:oid,:@1], r:@1) :bat[:oid,:bit] > +command batcalc.@2(a:bat[:oid,:bit], l:bat[:oid,:@1], r:@1) :bat[:oid,:bit] > address MTIMEaccum_@1_@3_cst; > -command batcalc.@2(a:bat[:void,:bit], l:bat[:void,:@1], r:@1) :bat[:void,:bit] > +command batcalc.@2(a:bat[:oid,:bit], r:@1, l:bat[:oid,:@1]) :bat[:oid,:bit] Is the above signature change (swapping left & right operand) intended? If so, what is the reason for it? > address MTIMEaccum_@1_cst_@3 > comment "Compare a bat of @1 against a singleton"; > > @@ -64,24 +61,35 @@ comment "Compare a bat of @1 against a s > @:compareOp(@1,>,GT)@ > @:compareOp(@1,>=,GE)@ > > -@= unaryISNIL > - command batcalc.isnil(b:bat[:any_1,:@1]):bat[:any_1,:bit] > - address MTIMEunaryISNIL > - comment "Unary check for nil over the tail of the bat"; > - > - command batcalc.isnil(b:bat[:void,:@1]):bat[:oid,:bit] > - address MTIMEunaryISNIL > - comment "Unary check for nil over the tail of the bat"; > - > -@ > @mal > @:compareGrp(date)@ > @:compareGrp(daytime)@ > @:compareGrp(timestamp)@ > > -@:unaryISNIL(date)@ > -@:unaryISNIL(daytime)@ > -@:unaryISNIL(timestamp)@ > +command batcalc.isnil(b:bat[:any_1,:date]):bat[:any_1,:bit] > +address MTIMEunaryISNIL > +comment "Unary check for nil over the tail of the bat"; > + > +command batcalc.isnil(b:bat[:oid,:date]):bat[:oid,:bit] > +address MTIMEunaryISNIL > +comment "Unary check for nil over the tail of the bat"; > + > +command batcalc.isnil(b:bat[:any_1,:daytime]):bat[:any_1,:bit] > +address MTIMEunaryISNIL > +comment "Unary check for nil over the tail of the bat"; > + > +command batcalc.isnil(b:bat[:oid,:daytime]):bat[:oid,:bit] > +address MTIMEunaryISNIL > +comment "Unary check for nil over the tail of the bat"; > + > +command batcalc.isnil(b:bat[:any_1,:timestamp]):bat[:any_1,:bit] > +address MTIMEunaryISNIL > +comment "Unary check for nil over the tail of the bat"; > + > +command batcalc.isnil(b:bat[:oid,:timestamp]):bat[:oid,:bit] > +address MTIMEunaryISNIL > +comment "Unary check for nil over the tail of the bat"; I see a pattern of three pairs of signatures. What is the difference between, say, command batcalc.isnil(b:bat[:any_1,:date]):bat[:any_1,:bit] and command batcalc.isnil(b:bat[:oid,:date]):bat[:oid,:bit] ? Isn't the latter fully covered by the former? Thanks! Stefan ps: I not study the remainder, yet ... > > @h > /* > @@ -112,9 +120,6 @@ comment "Compare a bat of @1 against a s > * head column. This is not checked and may be mis-used to deploy the > * implementation for shifted window arithmetic as well. > */ > -@= chkSize > - if( BATcount(@1) != BATcount(@2) ) > - throw(MAL, "batcalc.@3", ILLEGAL_ARGUMENT " Requires bats of identical size"); > > @= implDef > batmtime_export str MTIMEbat_@1_@2(int *ret, int *l, int *r); > @@ -124,68 +129,18 @@ batmtime_export str MTIMEaccum_@1_@2(int > batmtime_export str MTIMEaccum_@1_@2_cst(int *ret, int *r, int *l, @1 *cst); > batmtime_export str MTIMEaccum_@1_cst_@2(int *ret, int *r, @1 *cst, int *l); > > -@ > - * @- Comparison operator implementation for temporal objects > - * The structure of this code is copied from the arithmetic scheme. > - * @- > -@= voidresultBAT > - bn = BATnew(TYPE_void, @1, BATcount(b)); > - BATseqbase(bn, b->hseqbase); > - if (bn == NULL) { > - throw(MAL, @2, MAL_MALLOC_FAIL); > - } > - bn->hsorted = b->hsorted; > - bn->hrevsorted = b->hrevsorted; > - bn->tsorted = b->tsorted; > - bn->trevsorted = b->trevsorted; > - bn->H->nonil = 1; > +@c > +#define resultBAT(Type,Error)\ > + bn = BATnew(TYPE_void, Type, BATcount(b));\ > + BATseqbase(bn, b->hseqbase);\ > + if (bn == NULL) \ > + throw(MAL, Error, MAL_MALLOC_FAIL);\ > + bn->hsorted = b->hsorted;\ > + bn->hrevsorted = b->hrevsorted;\ > + bn->tsorted = b->tsorted;\ > + bn->trevsorted = b->trevsorted;\ > + bn->H->nonil = 1;\ > bn->T->nonil = b->T->nonil; > -@= unarybaselineCode > -str MTIMEbat_@1_@2_@3(int *ret, int *l) > -{ > - BAT *bn, *b; > - @1 *p, *q, *t; > - bit *o; > - > - if ((b = BATdescriptor(*l)) == NULL) > - throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > - @:voidresultBAT(@3,"batcalc.@2")@ > - > - o = (bit*)Tloc(bn,BUNfirst(bn)); > - p = (@1*)Tloc(b,BUNfirst(b)); > - q = (@1*)Tloc(b,BUNlast(b)); > - t = (@1*)Tloc(right,BUNfirst(right)); > - > - BATaccessBegin(b, USE_TAIL, MMAP_SEQUENTIAL); > - BATaccessBegin(right, USE_TAIL, MMAP_SEQUENTIAL); > - if (b->T->nonil) > - for (;p<q; p++, t++, o++) > - @1@2(o,p,t); > - else > - for (;p<q; p++, t++, o++) > - if ( @1_isnil(*p)) > - *o = bit_nil; > - else @1@2(o,p,t); > - > - BATaccessEnd(right, USE_TAIL, MMAP_SEQUENTIAL); > - BATaccessEnd(b, USE_TAIL, MMAP_SEQUENTIAL); > - if (!(bn->batDirty&2)) bn = BATsetaccess(bn, BAT_READ); > - > - BATsetcount(bn, BATcount(b)); > - if (b->htype != bn->htype) { > - BAT *r = VIEWcreate(b,bn); > - > - BBPreleaseref(bn->batCacheid); > - bn = r; > - } > - > - bn->tsorted = 0; > - bn->trevsorted = 0; > - BATkey(BATmirror(bn),FALSE); > - BBPkeepref(*ret= bn->batCacheid); > - BBPreleaseref(b->batCacheid); > - return MAL_SUCCEED; > -} > > @= baselineCode > str MTIMEbat_@1_@2(int *ret, int *l, int *r) > @@ -201,8 +156,9 @@ str MTIMEbat_@1_@2(int *ret, int *l, int > BBPreleaseref(b->batCacheid); > throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > } > - @:chkSize(b,right,batcalc.@2)@ > - @:voidresultBAT(TYPE_bit,"batcalc.@2")@ > + if( BATcount(b) != BATcount(right) ) > + throw(MAL, "batcalc.@2", ILLEGAL_ARGUMENT " Requires bats of identical size"); > + resultBAT(TYPE_bit,"batcalc.@2") > > o = (bit*)Tloc(bn,BUNfirst(bn)); > p = (@1*)Tloc(b,BUNfirst(b)); > @@ -270,8 +226,10 @@ str MTIMEaccum_@1_@2(int *ret, int *d, i > BBPreleaseref(b->batCacheid);BBPreleaseref(right->batCacheid); > throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > } > - @:chkSize(bn,b,batcalc.@2)@ > - @:chkSize(bn,right,batcalc.@2)@ > + if( BATcount(bn) != BATcount(b) ) > + throw(MAL, "batcalc.@2", ILLEGAL_ARGUMENT " Requires bats of identical size"); > + if( BATcount(bn) != BATcount(right) ) > + throw(MAL, "batcalc.@2", ILLEGAL_ARGUMENT " Requires bats of identical size"); > > o = (bit*)Tloc(bn,BUNfirst(bn)); > p = (@1*)Tloc(b,BUNfirst(b)); > @@ -322,7 +280,7 @@ str MTIMEbat_@1_@2_cst(int *ret, int *l, > > if ((b = BATdescriptor(*l)) == NULL) > throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > - @:voidresultBAT(TYPE_bit,"batcalc.@2")@ > + resultBAT(TYPE_bit,"batcalc.@2") > BATsetcount(bn, BATcount(b)); > if (!(bn->batDirty&2)) bn = BATsetaccess(bn, BAT_READ); > > @@ -371,7 +329,7 @@ str MTIMEbat_@1_cst_@2(int *ret, @1 *cst > > if ((b = BATdescriptor(*l)) == NULL) > throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > - @:voidresultBAT(TYPE_bit,"batcalc.@2")@ > + resultBAT(TYPE_bit,"batcalc.@2") > BATsetcount(bn, BATcount(b)); > > o = (bit*)Tloc(bn,BUNfirst(bn)); > @@ -423,7 +381,8 @@ str MTIMEaccum_@1_@2_cst(int *ret, int * > @4; > throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > } > - @:chkSize(bn,b,batcalc.@2)@ > + if( BATcount(bn) != BATcount(b) ) > + throw(MAL, "batcalc.@2", ILLEGAL_ARGUMENT " Requires bats of identical size"); > > o = (bit*)Tloc(bn,BUNfirst(bn)); > p = (@1*)Tloc(b,BUNfirst(b)); > @@ -464,7 +423,8 @@ str MTIMEaccum_@1_cst_@2(int *ret, int * > BBPreleaseref(bn->batCacheid); > throw(MAL, "batcalc.@2", RUNTIME_OBJECT_MISSING); > } > - @:chkSize(bn,b,batcalc.@2)@ > + if( BATcount(bn) != BATcount(b) ) > + throw(MAL, "batcalc.@2", ILLEGAL_ARGUMENT " Requires bats of identical size"); > > o = (bit*)Tloc(bn,BUNfirst(bn)); > p = (@1*)Tloc(b,BUNfirst(b)); > @@ -493,9 +453,6 @@ str MTIMEaccum_@1_cst_@2(int *ret, int * > return MAL_SUCCEED; > } > > -@ > - * @- > - * The implementation section. > @= implCmpGrpDef > @:implDef(@1,EQ)@ > @:implDef(@1,NEQ)@ > @@ -532,20 +489,6 @@ batmtime_export str MTIMEunaryISNIL(int > @:baselineCode(timestamp,GT)@ > @:baselineCode(timestamp,GE)@ > > -@= accumUnaryISNIL > -{ > - @1 *p = (@1*)Tloc(b,BUNfirst(b)); > - @1 *q = (@1*)Tloc(b,BUNlast(b)); > - BATaccessBegin(b, USE_TAIL, MMAP_SEQUENTIAL); > - if (b->T->nonil){ > - for(; p<q; p++, o++ ) > - *o = FALSE; > - } else > - for(; p<q; p++, o++ ) > - *o = @1_isnil(*p); > - BATaccessEnd(b, USE_TAIL, MMAP_SEQUENTIAL); > -} > -@ > @c > str MTIMEunaryISNIL(int *ret, int *bid) > { > @@ -554,18 +497,41 @@ str MTIMEunaryISNIL(int *ret, int *bid) > > if ((b = BATdescriptor(*bid)) == NULL) > throw(MAL, "batcalc.isnil", RUNTIME_OBJECT_MISSING); > - @:voidresultBAT(TYPE_bit,"batcalc.isnil")@ > + resultBAT(TYPE_bit,"batcalc.isnil") > > o = (bit*)Tloc(bn,BUNfirst(bn)); > + BATaccessBegin(b, USE_TAIL, MMAP_SEQUENTIAL); > if ( BATttype(b) == TYPE_date) { > - @:accumUnaryISNIL(date)@ > + date *p = (date*)Tloc(b,BUNfirst(b)); > + date *q = (date*)Tloc(b,BUNlast(b)); > + if (b->T->nonil){ > + for(; p<q; p++, o++ ) > + *o = FALSE; > + } else > + for(; p<q; p++, o++ ) > + *o = date_isnil(*p); > } else > if ( BATttype(b) == TYPE_daytime) { > - @:accumUnaryISNIL(daytime)@ > + daytime *p = (daytime*)Tloc(b,BUNfirst(b)); > + daytime *q = (daytime*)Tloc(b,BUNlast(b)); > + if (b->T->nonil){ > + for(; p<q; p++, o++ ) > + *o = FALSE; > + } else > + for(; p<q; p++, o++ ) > + *o = daytime_isnil(*p); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > > -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 2

[Monetdb-developers] php_monetdb.dll
by suchitradaemon 17 Jul '12

17 Jul '12

Hi I want to be able to use all the SQL related functions using PHP on monetdb. As far as I understand, the newer versions of monetdb state that the PHP interface is completely written in PHP and need not be built. Is this interface same as the php_mapi.in and the php_monetdb.php? If so, I haven't seen any support for all the SQL related function like query_params. Please correct me if I am wrong. I am new to MonetDB. I had initially stumbled upon one of the old server reference manuals which had indicated of an option to build monetdb for php_monetdb.dll php extension. But then I started off with the incorrect version of the source (the tarball) and hit all sorts of errors. Then I was told to use the HG version. At which point I checked the windowsbuild.rst which stated that the PHP need not be built. So I just needed a confirmation whether building php_monetdb.dll is supported even in the latest releases or I need to make use of the php_monetdb.php file or there's any other way of getting all the SQL related functions? If php_monetdb.php is the answer, please let me know how to make use of it. Thank you Suchitra

4 4

[Monetdb-developers] Announcement: New Jul2012 Feature release of MonetDB suite
by Fabian Groffen 17 Jul '12

17 Jul '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2012 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2012 feature release Build Environment * Created seperate RPM and DEB packages for MonetDB/JAQL. * Removed --enable-bits option from configure. * Split the MonetDB-client-ruby RPM package into two and named them in accordance with the Fedora packaging guidelines as rubygem-<gem-name>. SQL * COPY INTO now accepts optional parenthesis for file argument. Binary COPY INTO now requires 'COPY BINARY INTO'. Client Package * ODBC: Fixed a bug where SQLNativeSql expected a statment handle instead of a connection handle. MonetDB5 Server * Crackers code has been removed. Development continues in the holindex branch. Merovingian * Fixed misc memory leaks, which caused monetdbd to grow in memory size over time. * Removed erroneously (re-)added master and slave properties, this functionality is currently not working. MonetDB Common * Fixed intermittent problem that joins and selects return incorrect results and possibly inconsistent databases. The problems only occurred after a series of queries and updates, therefore it was hard to reproduce. * The sorted property, which was used to maintain whether a column in a BAT was sorted or reverse sorted, has been replaced by a pair of properties, sorted and revsorted. These new properties can be set independently (unlike the old sorted property), and so if both are set, the column must be constant. In addition, internal property checking has been overhauled. Now, when a property is set incorrectly, and when assertions are enabled, an assertion will go off. There is also a function which can derive properties. * Implemented proper overflow checking on all arithmetic operations. Bug Fixes * 3016: Overflowing double multiplication leads to incorrect result * 3091: Since Apr2012-SP1 !TypeException:user.s11_1[59]:'bat.reverse' undefined in: _89:any := bat.reverse(_86:wrd) * 3099: ambiguous attributes in query with WITH clause and EXISTS clause * 3100: charset problem with the odbc driver for windows * 3103: display_size and internal_size of Cursor.description or incorrect * 3105: Leaking using Python * 3109: Wrong order in autoload mal script causes warning on startup * 3111: memory leak, monetdb set/get command * 3112: Median over temporal domains * 3113: Cannot set value of indexed column when doing UPDATE * 3114: Foreign Key prblems * 3116: Segfault in SQL

1 0

[Monetdb-developers] Why is the sourcecode 'de-mx-ed'?
by Stefan de Konink 16 Jul '12

16 Jul '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 The MonetDB checkins show over time a lot of de-mx operations. Why is the preprocessor language being abandoned in favor of 'native' macro's? Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAlAEDa8ACgkQYH1+F2Rqwn2t3gCgiPrdrWFcxtBLIBWpVi/OTcx2 pywAoIdjsB8qXqGDjQqou0OKs5Esim8Q =xXNR -----END PGP SIGNATURE-----

2 1

Re: [Monetdb-developers] MonetDB: default - Add HEADLESS warning flag
by Stefan Manegold 15 Jul '12

15 Jul '12

FYI: "headless" means *dense* BAT heads (unless they are void-nil). S. ----- Original Message ----- > Changeset: 3be4152afaa3 for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3be4152afaa3 > Modified Files: > gdk/gdk.h > gdk/gdk_bat.c > gdk/gdk_bat.h > monetdb5/mal/mal_dataflow.c > monetdb5/mal/mal_interpreter.c > Branch: default > Log Message: > > Add HEADLESS warning flag > During conversion towards :oid and col it would be nice to > trap all cases where an 'alien' bat is created. > A free bit in the debug flag is used for that purpose. > > > diffs (71 lines): > > diff --git a/gdk/gdk.h b/gdk/gdk.h > --- a/gdk/gdk.h > +++ b/gdk/gdk.h > @@ -2334,6 +2334,8 @@ VALptr(const ValRecord *v) > */ > #define PARMASK (1<<7) > #define PARDEBUG if (GDKdebug & PARMASK) > +#define HEADLESSMASK (1<<8) > +#define HEADLESSDEBUG if ( GDKdebug & HEADLESSMASK) > /* TRGMASK not used anymore > #define TRGMASK (1<<8) > #define TRGDEBUG if (GDKdebug & TRGMASK) > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -89,6 +89,11 @@ BATcreatedesc(int ht, int tt, int heapna > > if (bs == NULL) > return NULL; > + HEADLESSDEBUG { > + if ( ht != TYPE_void && ht != TYPE_oid) > + fprintf(stderr, "#headless violation in BATcreatedesc %d\n", ht); > + > + } > /* > * assert needed in the kernel to get symbol eprintf resolved. > * Else modules using assert fail to load. > @@ -2799,7 +2804,7 @@ BATmode(BAT *b, int mode) > * is not actually used anywhere, but it is checked. */ > > #ifndef NDEBUG > -void > +static void > BATassertHeadProps(BAT *b) > { > BATiter bi = bat_iterator(b); > diff --git a/gdk/gdk_bat.h b/gdk/gdk_bat.h > --- a/gdk/gdk_bat.h > +++ b/gdk/gdk_bat.h > @@ -23,7 +23,6 @@ > gdk_export BUN void_replace_bat(BAT *b, BAT *u, bit force); > gdk_export int void_inplace(BAT *b, oid id, const void *val, bit > force); > gdk_export BAT *BATattach(int tt, const char *heapfile); > -gdk_export void BATassertHeadProps(BAT *b); > > extern int default_ident(char *s); > extern oid MAXoid(BAT *i); > diff --git a/monetdb5/mal/mal_dataflow.c > b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -618,7 +618,7 @@ DFLOWstep(FlowTask *t, FlowStatus fs) > #if 0 /* when needed, enable */ > if (i < pci->retc && bid){ > BAT *b = BATdescriptor(bid); > - BATassertHeadProps(b); > + BATassertProps(b); > BBPunfix(b->batCacheid); > } > #endif > diff --git a/monetdb5/mal/mal_interpreter.c > b/monetdb5/mal/mal_interpreter.c > --- a/monetdb5/mal/mal_interpreter.c > +++ b/monetdb5/mal/mal_interpreter.c > @@ -860,7 +860,7 @@ str runMALsequence(Client cntxt, MalBlkP > #if 0 /* when needed, enable */ > if (i < pci->retc && bid){ > BAT *b = BATdescriptor(bid); > - BATassertHeadProps(b); > + BATassertProps(b); > BBPunfix(b->batCacheid); > } > #endif > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list >

1 0

Re: [Monetdb-developers] MonetDB: default - Debugging routine should be globally known
by Stefan Manegold 15 Jul '12

15 Jul '12

Shouldn't BATassertProps() (have) be(en) used instead of BATassertHeadProps()? The former is already globally known and calls the latter on both, head & tail of a BAT ... Stefan ----- Original Message ----- > Changeset: 6a5b3889c41e for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=6a5b3889c41e > Modified Files: > gdk/gdk_bat.c > gdk/gdk_bat.h > Branch: default > Log Message: > > Debugging routine should be globally known > > > diffs (23 lines): > > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -2799,7 +2799,7 @@ BATmode(BAT *b, int mode) > * is not actually used anywhere, but it is checked. */ > > #ifndef NDEBUG > -static void > +void > BATassertHeadProps(BAT *b) > { > BATiter bi = bat_iterator(b); > diff --git a/gdk/gdk_bat.h b/gdk/gdk_bat.h > --- a/gdk/gdk_bat.h > +++ b/gdk/gdk_bat.h > @@ -23,6 +23,7 @@ > gdk_export BUN void_replace_bat(BAT *b, BAT *u, bit force); > gdk_export int void_inplace(BAT *b, oid id, const void *val, bit > force); > gdk_export BAT *BATattach(int tt, const char *heapfile); > +gdk_export void BATassertHeadProps(BAT *b); > > extern int default_ident(char *s); > extern oid MAXoid(BAT *i); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list >

1 0

Re: [Monetdb-developers] MonetDB: default - Include debugging
by Stefan Manegold 15 Jul '12

15 Jul '12

In the light of :oid- or even dense-headed BATs (and eventually COLs), what is the reason to only check/assert head properties (trivial once dense), but not tail properties? Stefan ----- Original Message ----- > Changeset: f96a7db93e54 for MonetDB > URL: > http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=f96a7db93e54 > Modified Files: > monetdb5/mal/mal_dataflow.c > monetdb5/mal/mal_interpreter.c > Branch: default > Log Message: > > Include debugging > We have to trace for illegal property settings. It is an expensive > test, that may isolate a few deeply hidden errors. > > Moreover, initialize the cpu affinity > > > diffs (61 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c > b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -614,6 +614,16 @@ DFLOWstep(FlowTask *t, FlowStatus fs) > if (isaBatType(getArgType(mb, pci, i))) { > bat bid = stk->stk[a].val.bval; > > +#ifndef NDEBUG > +#if 0 /* when needed, enable */ > + if (i < pci->retc && bid){ > + BAT *b = BATdescriptor(bid); > + BATassertHeadProps(b); > + BBPunfix(b->batCacheid); > + } > +#endif > +#endif > + /* Check output properties */ > /* update the bigfoot information only if we need to gc */ > if (cntxt->flags & bigfootFlag) > updateBigFoot(cntxt, bid, TRUE); > @@ -811,8 +821,12 @@ runDFLOWworker(void *t) > Thread thr; > int i, local = 0, last = 0; > long usec=0; > + //int cpu; > > thr = THRnew(MT_getpid(), "DFLOWworker"); > + > + //cpu = MT_set_affinity(task->id); > + > GDKsetbuf(GDKmalloc(GDKMAXERRLEN)); /* where to leave errors */ > GDKerrbuf[0] = 0; > while (task) { > @@ -915,6 +929,7 @@ runDFLOWworker(void *t) > } > GDKfree(GDKerrbuf); > GDKsetbuf(0); > + //MT_unset_affinity(cpu); > THRdel(thr); > } > > diff --git a/monetdb5/mal/mal_interpreter.c > b/monetdb5/mal/mal_interpreter.c > --- a/monetdb5/mal/mal_interpreter.c > +++ b/monetdb5/mal/mal_interpreter.c > @@ -856,6 +856,16 @@ str runMALsequence(Client cntxt, MalBlkP > if (isaBatType(getArgType(mb, pci, i))) { > bat bid = stk->stk[a].val.bval; > > +#ifndef NDEBUG > +#if 0 /* when needed, enable */ > + if (i < pci->retc && bid){ > + BAT *b = BATdescriptor(bid); > + BATassertHeadProps(b); > + BBPunfix(b->batCacheid); > + } > +#endif > +#endif > + > /* update the bigfoot information only if we need to gc */ > if (cntxt->flags & bigfootFlag) > updateBigFoot(cntxt, bid, TRUE); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list >

1 0