August 2013 - developers-list

Re: MonetDB: Feb2013 - fixed bug 3232, use dayofweek, starting at Mo...
by Sjoerd Mullender 27 Aug '13

27 Aug '13

This checkin is not allowed on the release branch: it changes the ABI by introducing a function MTIMEdate_extract_dayofworkweek. What's wrong with changing the implementation of MTIMEdate_extract_dayofweek to do what MTIMEdate_extract_dayofworkweek does now (and dropping MTIMEdate_extract_dayofworkweek)? On 2013-08-25 16:23, Niels Nes wrote: > Changeset: 0620e9df2003 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0620e9df2003 > Added Files: > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.sql > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.err > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.out > Modified Files: > clients/mapiclient/mclient.c > monetdb5/modules/atoms/mtime.c > monetdb5/modules/atoms/mtime.h > monetdb5/modules/atoms/mtime.mal > sql/backends/monet5/sql.mx > sql/common/sql_types.c > sql/scripts/mtime.sql > sql/server/rel_select.c > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > sql/test/BugTracker-2013/Tests/All > sql/test/leaks/Tests/check1.stable.out > sql/test/leaks/Tests/check2.stable.out > sql/test/leaks/Tests/check3.stable.out > sql/test/leaks/Tests/check4.stable.out > sql/test/leaks/Tests/check5.stable.out > sql/test/mapi/Tests/php_monetdb.stable.out > Branch: Feb2013 > Log Message: > > fixed bug 3232, use dayofweek, starting at Monday > fixed bug 2962, convert timestamp with time zone to local time zone, when used as input to the extract functions. > fixed bug 2781, use correct time zone in mclient > > > diffs (truncated from 761 to 300 lines): > > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -2559,6 +2559,7 @@ set_timezone(Mapi mid) > tmp = gmtime(&t); > gt = mktime(tmp); > tmp = localtime(&t); > + tmp->tm_isdst=0; /* We need the difference without dst */ > lt = mktime(tmp); > assert((lng) gt - (lng) lt >= (lng) INT_MIN && (lng) gt - (lng) lt <= (lng) INT_MAX); > tzone = (int) (gt - lt); > diff --git a/monetdb5/modules/atoms/mtime.c b/monetdb5/modules/atoms/mtime.c > --- a/monetdb5/modules/atoms/mtime.c > +++ b/monetdb5/modules/atoms/mtime.c > @@ -427,6 +427,7 @@ parse_substr(int *ret, str s, int min, s > return j; > } > > +/* Sunday = 1, Saturday = 7 */ > static int > date_dayofweek(date v) > { > @@ -434,6 +435,17 @@ date_dayofweek(date v) > return v <= 0 ? v + 7 : v; > } > > +/* Monday = 1, Sunday = 7 */ > +static int > +date_dayofworkweek(date v) > +{ > + int res = date_dayofweek(v); > + > + if (res >= 2) > + return res -1; > + return 7; > +} > + > #define SKIP_DAYS(d,w,i) d += i; w = (w + i)%7; if (w <= 0) w += 7; > > static date > @@ -1310,10 +1322,10 @@ date_extract_weekofyear(int *ret, date * > > fromdate((int) *v, &dummy, &dummy, &year); > year_jan_1 = todate(1, 1, year); > - dayofweek = date_dayofweek(year_jan_1); > + dayofweek = date_dayofworkweek(year_jan_1); > > if (dayofweek <= 4) { > - /* week of jan 1 belongs to this year */ > + /* 4 or more days in first week, ie week of jan 1 belongs to this year */ > *ret = (int) (1 + (*v - year_jan_1 + dayofweek - 1) / 7); > } else if (*v - year_jan_1 > 7 - dayofweek) { > /* week of jan 1 belongs to last year; but this is a later week */ > @@ -1340,6 +1352,18 @@ date_extract_dayofweek(int *ret, date *v > return MAL_SUCCEED; > } > > +/* Returns the current day of the week where 1=monday, .., 7=sunday */ > +static str > +date_extract_dayofworkweek(int *ret, date *v) > +{ > + if (*v == date_nil) { > + *ret = int_nil; > + } else { > + *ret = date_dayofworkweek(*v); > + } > + return MAL_SUCCEED; > +} > + > /* extracts hour from daytime (value between 0 and 23) */ > static str > daytime_extract_hours(int *ret, daytime *v) > @@ -1959,6 +1983,12 @@ MTIMEdate_extract_dayofweek(int *ret, da > } > > str > +MTIMEdate_extract_dayofworkweek(int *ret, date *v) > +{ > + return date_extract_dayofworkweek(ret, v); > +} > + > +str > MTIMEdaytime_extract_hours(int *ret, daytime *v) > { > return daytime_extract_hours(ret, v); > diff --git a/monetdb5/modules/atoms/mtime.h b/monetdb5/modules/atoms/mtime.h > --- a/monetdb5/modules/atoms/mtime.h > +++ b/monetdb5/modules/atoms/mtime.h > @@ -163,6 +163,7 @@ mtime_export str MTIMEdate_extract_day(i > mtime_export str MTIMEdate_extract_dayofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_weekofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_dayofweek(int *ret, date *v); > +mtime_export str MTIMEdate_extract_dayofworkweek(int *ret, date *v); > mtime_export str MTIMEdaytime_extract_hours(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_minutes(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_seconds(int *ret, daytime *v); > diff --git a/monetdb5/modules/atoms/mtime.mal b/monetdb5/modules/atoms/mtime.mal > --- a/monetdb5/modules/atoms/mtime.mal > +++ b/monetdb5/modules/atoms/mtime.mal > @@ -633,6 +633,10 @@ command dayofweek(d:date) :int > address MTIMEdate_extract_dayofweek > comment "Returns the current day of the week > where 1=sunday, .., 7=saturday"; > +command dayofworkweek(d:date) :int > +address MTIMEdate_extract_dayofworkweek > +comment "Returns the current day of the week > + where 1=monday, .., 7=sunday"; > > command add(v:timestamp, msecs:lng) :timestamp > address MTIMEtimestamp_add > diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx > --- a/sql/backends/monet5/sql.mx > +++ b/sql/backends/monet5/sql.mx > @@ -714,6 +714,14 @@ comment "cast @1 to @1 and check for ove > @:mal_casttime(timestamp)@ > @:mal_casttime(daytime)@ > > +pattern current_time() :daytime > +address SQLcurrent_daytime > +comment "Get the clients current daytime"; > + > +pattern current_timestamp() :timestamp > +address SQLcurrent_timestamp > +comment "Get the clients current timestamp"; > + > command calc.date( v:void ) :date > address nil_2_date > comment "cast to date"; > @@ -6381,6 +6389,8 @@ sql5_export str second_interval_daytime( > sql5_export str second_interval_2_daytime( daytime *res, lng *s, int *d); > sql5_export str timestamp_2_daytime( daytime *res, timestamp *v, int *d); > sql5_export str date_2_timestamp( timestamp *res, date *v, int *d); > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > @c > str > second_interval_daytime( lng *res, daytime *s, int *d, int *sk ) > @@ -6415,6 +6425,36 @@ second_interval_daytime( lng *res, dayti > return MAL_SUCCEED; > } > > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + daytime t, *res = (daytime *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_time(&t)) == MAL_SUCCEED) > + *res = t+m->timezone; > + return msg; > +} > + > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + timestamp t, *res = (timestamp *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_timestamp(&t)) == MAL_SUCCEED) { > + lng offset = m->timezone; > + return MTIMEtimestamp_add(res, &t, &offset); > + } > + return msg; > +} > + > /* str dump_cache(int *r); */ > str > dump_cache(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > diff --git a/sql/common/sql_types.c b/sql/common/sql_types.c > --- a/sql/common/sql_types.c > +++ b/sql/common/sql_types.c > @@ -1457,8 +1457,8 @@ sqltypeinit( sql_allocator *sa) > sql_create_func(sa, "curtime", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_time", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_timestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMPTZ, SCALE_NONE); > - sql_create_func(sa, "localtime", "mtime", "current_time", NULL, NULL, TME, SCALE_NONE); > - sql_create_func(sa, "localtimestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > + sql_create_func(sa, "localtime", "sql", "current_time", NULL, NULL, TME, SCALE_NONE); > + sql_create_func(sa, "localtimestamp", "sql", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > > sql_create_func(sa, "sql_sub", "mtime", "date_sub_msec_interval", DTE, SECINT, DTE, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "date_sub_month_interval", DTE, MONINT, DTE, SCALE_FIX); > @@ -1470,6 +1470,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "sql_sub", "mtime", "diff", DTE, DTE, INT, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMP, TMESTAMP, LNG, SCALE_FIX); > + sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMPTZ, TMESTAMPTZ, LNG, SCALE_NONE); > > sql_create_func(sa, "sql_add", "mtime", "date_add_msec_interval", DTE, SECINT, DTE, SCALE_NONE); > sql_create_func(sa, "sql_add", "mtime", "addmonths", DTE, MONINT, DTE, SCALE_NONE); > @@ -1498,6 +1499,15 @@ sqltypeinit( sql_allocator *sa) > /* fix result type */ > f->res.scale = 3; > > + sql_create_func(sa, "year", "mtime", "year", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "month", "mtime", "month", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "day", "mtime", "day", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "hour", "mtime", "hours", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "minute", "mtime", "minutes", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + f = sql_create_func(sa, "second", "mtime", "sql_seconds", TMESTAMPTZ, NULL, DEC, SCALE_NONE); > + /* fix result type */ > + f->res.scale = 3; > + > sql_create_func(sa, "year", "mtime", "year", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "month", "mtime", "month", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "day", "mtime", "day", SECINT, NULL, LNG, SCALE_NONE); > @@ -1507,7 +1517,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "dayofyear", "mtime", "dayofyear", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "weekofyear", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > - sql_create_func(sa, "dayofweek", "mtime", "dayofweek", DTE, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "dayofweek", "mtime", "dayofworkweek", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "dayofmonth", "mtime", "day", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "week", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > > diff --git a/sql/scripts/mtime.sql b/sql/scripts/mtime.sql > --- a/sql/scripts/mtime.sql > +++ b/sql/scripts/mtime.sql > @@ -29,16 +29,16 @@ create function "current_timestamp"( ) r > external name mtime."current_timestamp"; > > create function "localtime"( ) returns TIME > - external name mtime."current_time"; > + external name sql."current_time"; > + > create function "localtimestamp"( ) returns TIMESTAMP > - external name mtime."current_timestamp"; > + external name sql."current_timestamp"; > > create type second_interval as interval day to sec; > > create function local_timezone( ) returns second_interval > external name mtime.local_timezone; > > - > create function sql_sub( date, second_interval ) returns date > external name mtime.date_sub_sec_interval; > create function sql_sub( date, month_interval ) returns date > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -3290,6 +3290,17 @@ rel_unop(mvc *sql, sql_rel **rel, symbol > f = sql_bind_func(sql->sa, s, fname, t, NULL, F_AGGR); > if (f && IS_AGGR(f->func)) > return _rel_aggr(sql, rel, 0, s, fname, l->next, fs); > + > + if (f && type_has_tz(t) && f->func->fix_scale == SCALE_FIX) { > + /* set timezone (using msec) */ > + sql_subtype *intsec = sql_bind_subtype(sql->sa, "sec_interval", 10 /*hour to second */, 0); > + sql_exp *tz = exp_atom_lng(sql->sa, sql->timezone); > + > + tz = exp_convert(sql->sa, tz, exp_subtype(tz), intsec); > + e = rel_binop_(sql, e, tz, NULL, "sql_add", ek.card); > + if (!e) > + return NULL; > + } > return rel_unop_(sql, e, s, fname, ek.card); > } > > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > @@ -30,9 +30,9 @@ stderr of test 'predicate_select.Bug-309 > # 18:53:13 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=niels" "--port=32323" > # 18:53:13 > > > -MAPI = (monetdb) /var/tmp/mtest-16462/.s.monetdb.36324 > +MAPI = (monetdb) /var/tmp/mtest-4932/.s.monetdb.34449 > QUERY = select * from tables where name; > -ERROR = !conversion of string 'r' to type bit failed. > +ERROR = !conversion of string 'storagemodelinput' to type bit failed. > > # 18:53:13 > > # 18:53:13 > "Done." > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > @@ -47,19 +47,19 @@ Ready. > [ 2130, "idxs", 2102, NULL, 0, true, 2, false, 0 ] > [ 2135, "triggers", 2102, NULL, 0, true, 2, false, 0 ] > [ 2146, "objects", 2102, NULL, 0, true, 2, false, 0 ] > -[ 5060, "tables", 2000, "SELECT * FROM (SELECT p.*, 0 AS \"temporary\" FROM \"sys\".\"_tables\" AS p UNION ALL SELECT t.*, 1 AS \"temporary\" FROM \"tmp\".\"_tables\" AS t) AS tables where tables.type <> 2;", 1, true, 0, false, 0 ] > -[ 5070, "columns", 2000, "SELECT * FROM (SELECT p.* FROM \"sys\".\"_columns\" AS p UNION ALL SELECT t.* FROM \"tmp\".\"_columns\" AS t) AS columns;", 1, true, 0, false, 0 ] > -[ 5090, "db_user_info", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5098, "users", 2000, "SELECT u.\"name\" AS \"name\", ui.\"fullname\", ui.\"default_schema\" FROM db_users() AS u LEFT JOIN \"sys\".\"db_user_info\" AS ui ON u.\"name\" = ui.\"name\" ;", 1, true, 0, false, 0 ] > -[ 5102, "user_role", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5105, "auths", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5109, "privileges", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5262, "queryhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5276, "callhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5293, "querylog", 2000, "create view querylog as\nselect qd.*, ql.ctime, ql.arguments, ql.exec, ql.result, ql.foot, ql.memory, ql.tuples, ql.inblock, ql.oublock from queryhistory qd, callhistory ql\nwhere qd.id = ql.id;", 1, true, 0, false, 0 ] > -[ 6059, "storagemodelinput", 2000, NULL, 0, true, 0, false, 0 ] > -[ 6100, "tablestoragemodel", 2000, "-- A summary of the table storage requirement is is available as a table view.\n-- The auxillary column denotes the maximum space if all non-sorted columns\n-- would be augmented with a hash (rare situation)\ncreate view sys.tablestoragemodel\nas select \"schema\",\"table\",max(count) as \"count\",\n\tsum(columnsize) as columnsize,\n\tsum(heapsize) as heapsize,\n\tsum(indices) as indices,\n\tsum(case when sorted = false then 8 * count else 0 end) as auxillary\nfrom sys.storagemodel() group by \"schema\",\"table\";", 1, true, 0, false, 0 ] > -[ 6122, "systemfunctions", 2000, NULL, 0, true, 0, false, 0 ] > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

Re: MonetDB: Feb2013 - Organize parallelism per worker pool
by Stefan Manegold 21 Aug '13

21 Aug '13

Martin, could you please also add sql/test/BugTracker-2013/Tests/nestedcalls.sql ? And could you plase also add the test scripts and stable output for tests sql/test/BugTracker-2013/Tests/oid_handling sql/test/BugTracker-2013/Tests/constraint_checking.Bug_3335 sql/test/BugTracker-2013/Tests/pivot.Bug-3339 sql/test/BugTracker-2013/Tests/recursion or alternatively remove them from sql/test/BugTracker-2013/Tests/All Thanks! In any case, propagation of this will checking will result in (expected/unavoidable) conflicts in monetdb5/mal/mal_dataflow.[ch] and (avoidable) conflicts in sql/test/BugTracker-2013/Tests All of these will need to be solved by hand ... Stefan ----- Original Message ----- > Changeset: 489815265a61 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=489815265a61 > Added Files: > sql/test/BugTracker-2013/Tests/nestedcalls.stable.err > sql/test/BugTracker-2013/Tests/nestedcalls.stable.out > Modified Files: > monetdb5/mal/mal_dataflow.c > monetdb5/mal/mal_dataflow.h > monetdb5/modules/mal/language.c > monetdb5/modules/mal/language.h > sql/test/BugTracker-2013/Tests/All > Branch: Feb2013 > Log Message: > > Organize parallelism per worker pool > Each (recursive) dataflow block is handed a worker pool now. > Once we run out of worker pools or fail to create a worker, > we continue in sequential mode. > > Cleaning up the threads is implicit, like all other non-pool > interpreters that might be active at system shutdown. > > > diffs (truncated from 762 to 300 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -35,6 +35,7 @@ > * The flow graphs should be organized such that parallel threads can > * access it mostly without expensive locking. > */ > +#include "monetdb_config.h" > #include "mal_dataflow.h" > #include "mal_client.h" > > @@ -82,10 +83,10 @@ typedef struct DATAFLOW { > Queue *done; /* instructions handled */ > } *DataFlow, DataFlowRec; > > -#define MAXQ 1024 > -static MT_Id workers[THREADS] = {0}; > -static int workerqueue[THREADS] = {0}; /* maps workers towards the todo > queues */ > -static Queue *todo[MAXQ] = {0}; /* pending instructions organized by user > MAXTODO > #users */ > +#define MAXQ 256 > +static Queue *todos[MAXQ] = {0}; /* pending instructions organized by > dataflow block */ > +static bit occupied[MAXQ]={0}; /* worker pool is in use? */ > +static int volatile exiting = 0; > > /* > * Calculate the size of the dataflow dependency graph. > @@ -108,9 +109,8 @@ DFLOWgraphSize(MalBlkPtr mb, int start, > */ > > static Queue* > -q_create(int sz) > +q_create(int sz, const char *name) > { > - const char* name = "q_create"; > Queue *q = (Queue*)GDKmalloc(sizeof(Queue)); > > if (q == NULL) > @@ -208,6 +208,8 @@ q_dequeue(Queue *q) > > assert(q); > MT_sema_down(&q->s, "q_dequeue"); > + if (exiting) > + return NULL; > MT_lock_set(&q->l, "q_dequeue"); > assert(q->last > 0); > if (q->last > 0) { > @@ -255,25 +257,23 @@ DFLOWworker(void *t) > { > DataFlow flow; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > - int wq; > Thread thr; > str error = 0; > - > - int i; > - lng usec = 0; > + Queue *todo = *(Queue **) t; > + int i,last; > > thr = THRnew("DFLOWworker"); > > GDKsetbuf(GDKmalloc(GDKMAXERRLEN)); /* where to leave errors */ > GDKerrbuf[0] = 0; > while (1) { > - assert(workerqueue[id] > 0); > - wq = workerqueue[id] - 1; > if (fnxt == 0) > - fe = q_dequeue(todo[wq]); > + fe = q_dequeue(todo); > else > fe = fnxt; > + if (exiting) { > + break; > + } > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -285,22 +285,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo[wq], fe); > + q_requeue(todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > - fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > + fe->pc, (int)((Queue **)t - todos), fe->argclaim, fe->hotclaim, > error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > MALadmission(-fe->argclaim, -fe->hotclaim); > @@ -331,8 +329,8 @@ DFLOWworker(void *t) > InstrPtr p = getInstrPtr(flow->mb, fe->pc); > assert(p); > fe->hotclaim = 0; > - for (i = 0; i < p->retc; i++) > - fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, fe->pc, i, FALSE); > + //for (i = 0; i < p->retc; i++) > + //fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, p, i, FALSE); > } > #endif > MT_lock_set(&flow->flowlock, "MALworker"); > @@ -351,56 +349,64 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > } > } > GDKfree(GDKerrbuf); > GDKsetbuf(0); > - workerqueue[wq] = 0; > - workers[wq] = 0; > THRdel(thr); > } > > /* > - * Create a set of DFLOW interpreters. > + * Create an interpreter pool. > * One worker will adaptively be available for each client. > * The remainder are taken from the GDKnr_threads argument and > - * typically is equal to the number of cores > + * typically is equal to the number of cores. > + * A recursive MAL function call would make for one worker less, > + * which limits the number of cores for parallel processing. > * The workers are assembled in a local table to enable debugging. > + * > + * BEWARE, failure to create a new worker thread is not an error > + * but would lead to serial execution. > */ > -static void > -DFLOWinitialize(int index) > +static int > +DFLOWinitialize(void) > { > - int i, worker, limit; > + int i, threads, grp; > + MT_Id worker; > > - assert(index >= 0); > - assert(index < THREADS); > + threads = GDKnr_threads ? GDKnr_threads : 1; > MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo[index]) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + for(grp = 0; grp< MAXQ; grp++) > + if ( occupied[grp] == FALSE){ > + occupied[grp] = TRUE; > + break; > + } > + MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + if (grp > THREADS) { > + // continue non-parallel > + return -1; > } > - todo[index] = q_create(2048); > - assert(todo[index]); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - assert(limit <= THREADS); > - for (worker = 0, i = 0; i < limit; i++){ > - for (; worker < THREADS; worker++) > - if( workers[worker] == 0) > - break; > - assert(worker < THREADS); > - if (worker < THREADS) { > - assert(workers[worker] == 0); > - MT_create_thread(&workers[worker], DFLOWworker, (void *) > &workers[worker], MT_THR_JOINABLE); > - assert(workers[worker] > 0); > - workerqueue[worker] = index + 1; > + if ( todos[grp] ) > + return grp; > + > + todos[grp] = q_create(2048, "todo"); > + if (todos[grp] == NULL) > + return -1; > + > + // associate a set of workers with the pool > + for (i = 0; grp>= 0 && i < threads; i++){ > + if (MT_create_thread(&worker, DFLOWworker, (void *) &todos[grp], > MT_THR_JOINABLE) < 0) { > + //Can not create interpreter thread > + grp = -1; > + } > + if (worker == 0) { > + //Failed to create interpreter thread > + grp = -1; > } > } > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + return grp; > } > > /* > @@ -409,18 +415,28 @@ DFLOWinitialize(int index) > * For each instruction we keep a list of instructions whose > * blocking counter should be decremented upon finishing it. > */ > -static void > +static str > DFLOWinitBlk(DataFlow flow, MalBlkPtr mb, int size) > { > int pc, i, j, k, l, n, etop = 0; > int *assign; > InstrPtr p; > > + if (flow == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with flow == NULL"); > + if (mb == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with mb == NULL"); > PARDEBUG printf("Initialize dflow block\n"); > assign = (int *) GDKzalloc(mb->vtop * sizeof(int)); > + if (assign == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Failed to allocate assign"); > etop = flow->stop - flow->start; > for (n = 0, pc = flow->start; pc < flow->stop; pc++, n++) { > p = getInstrPtr(mb, pc); > + if (p == NULL) { > + GDKfree(assign); > + throw(MAL, "dataflow", "DFLOWinitBlk(): getInstrPtr() returned NULL"); > + } > > /* initial state, ie everything can run */ > flow->status[n].flow = flow; > @@ -501,6 +517,7 @@ DFLOWinitBlk(DataFlow flow, MalBlkPtr mb > #ifdef USE_MAL_ADMISSION > memorypool = memoryclaims = 0; > #endif > + return MAL_SUCCEED; > } > > /* > @@ -528,18 +545,17 @@ static void showFlowEvent(DataFlow flow, > */ > > static str > -DFLOWscheduler(DataFlow flow) > +DFLOWscheduler(DataFlow flow, Queue *todo) > { > int last; > int i; > #ifdef USE_MAL_ADMISSION > - int j; > + //int j; > InstrPtr p; > #endif > int tasks=0, actions; > str ret = MAL_SUCCEED; > FlowEvent fe, f = 0; > - int wq; > > if (flow == NULL) > throw(MAL, "dataflow", "DFLOWscheduler(): Called with flow == NULL"); > @@ -549,19 +565,19 @@ DFLOWscheduler(DataFlow flow) > /* initialize the eligible statements */ > fe = flow->status; > > - if (fe[0].flow->cntxt->flags & timerFlag) > - fe[0].flow->cntxt->timer = GDKusec(); > - > MT_lock_set(&flow->flowlock, "MALworker"); > - wq = flow->cntxt->idx; > for (i = 0; i < actions; i++) > if (fe[i].blocks == 0) { > #ifdef USE_MAL_ADMISSION > p = getInstrPtr(flow->mb,fe[i].pc); > - for (j = p->retc; j < p->argc; j++) > - fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > + if (p == NULL) { > + MT_lock_unset(&flow->flowlock, "MALworker"); > + throw(MAL, "dataflow", "DFLOWscheduler(): getInstrPtr(flow->mb,fe[i].pc) > returned NULL"); > + } > + //for (j = p->retc; j < p->argc; j++) > + //fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, p, j, > FALSE); > #endif > - q_enqueue(todo[wq], flow->status + i); > + q_enqueue(todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -571,6 +587,10 @@ DFLOWscheduler(DataFlow flow) > > while (actions != tasks ) { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

default vs. Feb2013: opt_evaluate.c & mal_dataflow.c
by Stefan Manegold 19 Aug '13

19 Aug '13

Martin, I just merged recent changes from Feb2013 into default, and while doing so resolved --- hopefully correctly; see below --- some conflicts that occurred in opt_evaluate.c & mal_dataflow.c . In fact, (1) I first merged up until before your changeset that introduced the new multiple worker pools, cf., http://dev.monetdb.org/hg/MonetDB/rev/10125a9c3976 (2) then merged only that very changeset, cf., http://dev.monetdb.org/hg/MonetDB/rev/1eea7fa4cc7c (3) and finally merge the rest, cf., http://dev.monetdb.org/hg/MonetDB/rev/3e3d7d5f8dc9 During (1), I resolved some conflicts in opt_evaluate.c due to http://dev.monetdb.org/hg/MonetDB/rev/62ed11377998 I hope, I did this correctly. However, there are still some differences in opt_evaluate.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/optimizer/opt_evaluate.c Could you please check, whether these are OK? During (2), I resolved some conflicts in mal_dataflow.c due to http://dev.monetdb.org/hg/MonetDB/rev/4c47797192f4 I hope, I did this correctly. However, there are still some differences in mal_dataflow.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/mal/mal_dataflow.c Could you please check, whether these are OK? It appears, that after (2), some tests in sql/jdbc/tests/Tests/ run into a timeout (only when run multi-threaded, i.e., not when run single-threaded) --- I have not yet been able to find out why ... Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Report correct 100% sample size
by Niels Nes 16 Aug '13

16 Aug '13

On Fri, Aug 16, 2013 at 09:56:49PM +0200, Martin Kersten wrote: > Changeset: afa6da5dd7bd for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=afa6da5dd7bd > Modified Files: > sql/backends/monet5/sql_statistics.c > Branch: default > Log Message: > > Report correct 100% sample size > > > diffs (12 lines): > > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -169,7 +169,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > } > snprintf(query,8192, > "insert into sys.statistics values('%s','%s','%s','%s',%d,now(),"LLFMT","LLFMT","LLFMT","LLFMT",'%s','%s',%s);", > - b->name, bt->name, bc->name, c->type.type->sqlname,width, samplesize, sz, uniq, nils, minval,maxval, sorted?"true":"false"); > + b->name, bt->name, bc->name, c->type.type->sqlname,width, (samplesize?samplesize:sz), sz, uniq, nils, minval,maxval, sorted?"true":"false"); > #ifdef DEBUG_SQL_STATISTICS > mnstr_printf(cntxt->fdout,"%s\n",dquery); > mnstr_printf(cntxt->fdout,"%s\n",query); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list Martin Volgens mij wil je de volgende verzameling van analyze calls: analyze([opt_sample_size]) -- full db analyze(schema, [opt_sample_size]) -- full schema analyze(schema, table, [opt_sample_size]) -- full table analyze(schema, table, column, [opt_sample_size]) -- full table ie dus niet analyze(table, ...) voor die is het onduidelijk welk schema. Niels -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

Re: MonetDB: Feb2013 - New thread group per dataflow block
by Stefan Manegold 16 Aug '13

16 Aug '13

Martin, this checkins appears to make several tests run into timeouts; I have not yet been able to analyze why ... A complete Mtest.py run results in * 142 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 * 6 out of 1863 tests ran into timeout * sql/benchmarks/ATIS/load * sql/benchmarks/arno/insert_ATOM * sql/benchmarks/arno/insert_BOND * sql/benchmarks/arno_flt/init * sql/benchmarks/moa/load * sql/benchmarks/wisconsin/load 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 while with the version before this checkin, it results in 41 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 Best, Stefan ----- Original Message ----- > Changeset: 3669ddd28bf0 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3669ddd28bf0 > Modified Files: > monetdb5/mal/mal_dataflow.c > Branch: Feb2013 > Log Message: > > New thread group per dataflow block > The centralized worker thread group could lead to an unacceptable > situation. If a user is heavily processing complex queries, then > no other user could even log into the system, for its MAL statements > ended up at the end of the shared queues. > > The problem has been resolved by introducing a thread group per dataflow > block. This may lead to a large number of processes, whose resources > are managed by the OS. > > It solves bug 3258 > > > diffs (235 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -78,12 +78,12 @@ typedef struct DATAFLOW { > int *nodes; /* dependency graph nodes */ > int *edges; /* dependency graph */ > MT_Lock flowlock; /* lock to protect the above */ > + queue *todo; /* pending instructions */ > queue *done; /* instructions handled */ > + int threads; /* worker threads active */ > + MT_Id workers[THREADS]; > } *DataFlow, DataFlowRec; > > -static MT_Id workers[THREADS]; > -static queue *todo = 0; /* pending instructions */ > - > /* > * Calculate the size of the dataflow dependency graph. > */ > @@ -138,7 +138,6 @@ q_destroy(queue *q) > static void > q_enqueue_(queue *q, FlowEvent d) > { > - assert(d); > if (q->last == q->size) { > q->size <<= 1; > q->data = GDKrealloc(q->data, sizeof(FlowEvent) * q->size); > @@ -214,7 +213,6 @@ q_dequeue(queue *q) > */ > > MT_lock_unset(&q->l, "q_dequeue"); > - assert(r); > return r; > } > > @@ -239,14 +237,15 @@ q_dequeue(queue *q) > static void > DFLOWworker(void *t) > { > - DataFlow flow; > + DataFlow flow = (DataFlow) t; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > + MT_Id id = MT_getpid(); > + int last = 0; > Thread thr; > str error = 0; > > int i; > - lng usec = 0; > + //lng usec = 0; > > thr = THRnew("DFLOWworker"); > > @@ -254,8 +253,10 @@ DFLOWworker(void *t) > GDKerrbuf[0] = 0; > while (1) { > if (fnxt == 0) > - fe = q_dequeue(todo); > + fe = q_dequeue(flow->todo); > else fe = fnxt; > + if ( fe == 0) > + break; > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -266,20 +267,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > + //usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - if (todo->last == 0) > + if (flow->todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo, fe); > + q_requeue(flow->todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > - PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > + PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= "SZFMT" claim= " > LLFMT "," LLFMT " %s\n", > fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > @@ -330,12 +331,15 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - if (todo->last == 0) > + if (flow->todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > + //else > + //MALresourceFairness(NULL, NULL, usec); > } > } > + for( i = 0; i< flow->threads; i++) > + if ( flow->workers[i] == id) > + flow->workers[i] = 0; > GDKfree(GDKerrbuf); > GDKsetbuf(0); > THRdel(thr); > @@ -349,22 +353,51 @@ DFLOWworker(void *t) > * The workers are assembled in a local table to enable debugging. > */ > static void > -DFLOWinitialize(void) > +DFLOWinitialize(DataFlow flow, int size) > { > - int i, limit; > + int i; > > - MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + MT_lock_init(&flow->flowlock, "DFLOWworker"); > + flow->todo = q_create(size); > + flow->done = q_create(size); > + flow->threads = GDKnr_threads ? GDKnr_threads :1; > + for (i = 0; i < flow->threads; i++){ > + MT_create_thread(&flow->workers[i], DFLOWworker, (void *) flow, > MT_THR_JOINABLE); > + /* upon failure of starting threads we reduce the count */ > + if ( flow->workers[i]== 0){ > + flow->threads --; > + i--; > + } > } > - todo = q_create(2048); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - for (i = 0; i < limit; i++) > - MT_create_thread(&workers[i], DFLOWworker, (void *) &workers[i], > MT_THR_JOINABLE); > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > } > > +static str > +DFLOWfinalize(DataFlow flow) > +{ > + int i, cnt= flow->threads, runs =0; > + > + for( i = 0; i< cnt; i++) > + q_enqueue(flow->todo, 0); > + /* time out when threads are already killed */ > + do{ > + runs++; > + cnt = 0; > + MT_sleep_ms(1); > + for( i = 0; i < flow->threads; i++) > + cnt += flow->workers[i] ==0; > + } while( cnt != flow->threads && runs <5000); > + > + if ( runs == 5000) > + throw(MAL,"dataflow","Timeout on thread termination"); > + GDKfree(flow->status); > + GDKfree(flow->edges); > + GDKfree(flow->nodes); > + q_destroy(flow->done); > + q_destroy(flow->todo); > + MT_lock_destroy(&flow->flowlock); > + GDKfree(flow); > + return MAL_SUCCEED; > +} > /* > * The dataflow administration is based on administration of > * how many variables are still missing before it can be executed. > @@ -518,7 +551,7 @@ DFLOWscheduler(DataFlow flow) > for (j = p->retc; j < p->argc; j++) > fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > #endif > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -543,7 +576,7 @@ DFLOWscheduler(DataFlow flow) > if (flow->status[i].blocks == 1 ) { > flow->status[i].state = DFLOWrunning; > flow->status[i].blocks--; > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > PARDEBUG > mnstr_printf(GDKstdout, "#enqueue pc=%d claim= " LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } else { > @@ -579,13 +612,11 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > assert(stoppc > startpc); > > - /* check existence of workers */ > - if (workers[0] == 0) > - DFLOWinitialize(); > - assert(workers[0]); > - assert(todo); > - > flow = (DataFlow)GDKzalloc(sizeof(DataFlowRec)); > + > + DFLOWinitialize(flow, stoppc- startpc +1); > + assert(flow->todo); > + assert(flow->done); > > flow->cntxt = cntxt; > flow->mb = mb; > @@ -596,9 +627,6 @@ runMALdataflow(Client cntxt, MalBlkPtr m > flow->start = startpc + 1; > flow->stop = stoppc; > > - MT_lock_init(&flow->flowlock, "DFLOWworker"); > - flow->done = q_create(stoppc- startpc+1); > - > flow->status = (FlowEvent)GDKzalloc((stoppc - startpc + 1) * > sizeof(FlowEventRec)); > size = DFLOWgraphSize(mb, startpc, stoppc); > size += stoppc - startpc; > @@ -608,11 +636,9 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > ret = DFLOWscheduler(flow); > > - GDKfree(flow->status); > - GDKfree(flow->edges); > - GDKfree(flow->nodes); > - q_destroy(flow->done); > - MT_lock_destroy(&flow->flowlock); > - GDKfree(flow); > + if( ret == MAL_SUCCEED) > + ret = DFLOWfinalize(flow); > + else (void) > + DFLOWfinalize(flow); > return ret; > } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Kernel Function-level Profiling
by ajonwa ajonwa 15 Aug '13

15 Aug '13

Hello All, I am new to MonetDB and would like to profile the MonetDB kernel to find out the functions that are most frequently executed while it runs TPCH. So far, I have tried compiling with "--enable-profile" but it seems that gmon.out files are not generated. Is there a recommend way of doing function level profiling in MonetDB? Thanks, Geoffrey

2 1

Re: MonetDB: SciQL-2 - handle non-rectangular array tiles
by Stefan Manegold 15 Aug '13

15 Aug '13

Hoi Jennie, > Changeset: 58e31783ae0f for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=58e31783ae0f > Modified Files: > monetdb5/modules/mal/array.mx > sql/server/rel_dump.c > sql/server/rel_exp.c > sql/server/rel_select.c > sql/test/sciql/Tests/teleios_noa_bsm.sql > Branch: SciQL-2 > Log Message: > > handle non-rectangular array tiles > > - accept either interval tile range, or point tile ragnes, no mix of them > - multiple tile ranges are appended to e_column->f, thus e_column->f list can > have more than 3 items. > - to distinguish a point range from an interval range: a point range only has > one expr., while an interval range has three Thanks! > array.mx: > - save the values for the error message, since the lists are freed in > AGGR_CLEANUP Nice catch ;-) --- Thanks for fixing! ... and sorry for overlooking it myself ;-) > - disable the check on (max - min + 1 = size), since it only holds for > stepsize = 1. > maybe we should just remove this check, since we don't have step size in > the > tiling aggr impl. functions, do we? IMHO, our current tiled aggregation implementation inherently assumes (and thus requires) that all dimensions of the given array have step-size == 1. Since ARRAYtiles*() does not get the step sizes of the given array as parameters, the check (max - min + 1 = size) was/is IMHO the only way to check/ensure that we are indeed working on an array where all dimensions have step-size == 1. > teleios_noa_bsm.sql: > now we can get rid of the workarounds > > more tests needed... Indeed --- I'll resume me test tonight or tomorrow ... Best, Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

2 1

UTF-8 support for column names?
by Satoshi Nagayasu 12 Aug '13

12 Aug '13

Hi, Is there any plan to support UTF-8 charactors for column names (or object identifiers) in MonetDB? I think it's not supported so far. Right? I'm trying to port my database from a row-store one to MonetDB. Now, we are using UTF-8 charactors (Japanese) in column names, so we need to convert them manually. Obviously, it is possible, but if MonetDB can support UTF-8 charactors in column name, it would be easier to get porting our database to MonetDB. And I found the related thread and feature request on this. [MonetDB-users] 答复: Does Monetdb supports UTF-8 characters when I use 'as' clause? http://mail.monetdb.org/pipermail/users-list/2012-February/005397.html Bug 3010 - Does not currently support non-ASCII names in identifiers http://bugs.monetdb.org/show_bug.cgi?id=3010#c2 Is there any particluar plan? Is it difficult to implement? Regards, -- Satoshi Nagayasu <snaga(a)uptime.jp> Uptime Technologies, LLC. http://www.uptime.jp

1 0

Re: MonetDB repository problem!
by Stefan Manegold 07 Aug '13

07 Aug '13

In sort, there is a slight problem with a server side check to prevent users from pushing in unresolved merge conflicts. The problem materializes such that one can checkin such conflicts, but then not (easily) remove them, again. Having you .hgrc up to date as "requested", in particular the whitespace hook [hooks] pretxncommit.whitespace = python:<path-to-check_whitespace-directory>/check_whitespace.py:hook prevent the checkin of unresolved unresolved merge conflicts (already) on the client side. (We'll fully fix the server side once Sjoerd is back from vacation.) Stefan ----- Original Message ----- > Hi stefan, > > can you please share some more information about the problem? how can > configurations of local files on the client side affect the server/main > repository? I am asking for purely academic reasons. Is there some kind of > security problem in hg? > > ----- Original Message ----- > > Dear all, > > > > the MonetDB Mercurial (HG) repository should be working, again. > > > > IMPORTANT NOTE: > > > > To prevent further problems, it is mandatory that you do consciously read > > http://www.monetdb.org/Assets/MonetDB-Mercurial.html > > and follow all instructions carefully. > > > > In particular, your .hgrc must contain all of the "Important Configuration" > > mention in above document. > > > > You must double-check your .hgrc before your next hg pull, hg commit or hg > > push, > > which ever comes first. > > > > Thanks! > > Stefan > > > > ----- Original Message ----- > > > Dear all, > > > > > > there seems to be a problem with the MonetDB HG (Mercurial) repository. > > > Please refrain from pushing to it until we have sorted-out the problem. > > > > > > Thanks! > > > Stefan > > > > -- > > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > > > > > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

MonetDB repository problem!
by Stefan Manegold 06 Aug '13

06 Aug '13

Dear all, there seems to be a problem with the MonetDB HG (Mercurial) repository. Please refrain from pushing to it until we have sorted-out the problem. Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 1