April 2017 - developers-list

Re: MonetDB: viewless - remove BATslice of candidate lists from gdk_...
by Roberto Cornacchia 28 Apr '17

28 Apr '17

No idea what viewless branch is aiming at, but my 5 1/2th sense tells me there is something interesting here.. ;) On 27 Apr 2017 5:56 pm, "lsidir" <commits+lsidir=gmail.com(a)monetdb.org> wrote: > Changeset: 63c8d8dbed74 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=63c8d8dbed74 > Modified Files: > gdk/gdk_firstn.c > Branch: viewless > Log Message: > > remove BATslice of candidate lists from gdk_firstn > > > diffs (45 lines): > > diff --git a/gdk/gdk_firstn.c b/gdk/gdk_firstn.c > --- a/gdk/gdk_firstn.c > +++ b/gdk/gdk_firstn.c > @@ -140,7 +140,7 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > if (n >= (BUN) (candend - cand)) { > /* trivial: return the candidate list (the > * part that refers to b, that is) */ > - return BATslice(s, > + return CANDslice(s, > (BUN) (cand - (const oid *) > Tloc(s, 0)), > (BUN) (candend - (const oid *) > Tloc(s, 0))); > } > @@ -163,12 +163,12 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > /* return copy of first relevant part > * of candidate list */ > i = (BUN) (cand - (const oid *) Tloc(s, > 0)); > - return BATslice(s, i, i + n); > + return CANDslice(s, i, i + n); > } > /* return copy of last relevant part of > * candidate list */ > i = (BUN) (candend - (const oid *) Tloc(s, 0)); > - return BATslice(s, i - n, i); > + return CANDslice(s, i - n, i); > } > bn = COLnew(0, TYPE_void, n, TRANSIENT); > if (bn == NULL) > @@ -374,7 +374,7 @@ BATfirstn_unique_with_groups(BAT *b, BAT > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - return BATslice(s, 0, n); > + return CANDslice(s, 0, n); > } > > CANDINIT(b, s, start, end, cnt, cand, candend); > @@ -916,7 +916,7 @@ BATfirstn_grouped_with_groups(BAT **topn > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - bn = BATslice(s, 0, n); > + bn = CANDslice(s, 0, n); > gn = gids ? BATslice(g, 0, n) : NULL; > if (bn == NULL || (gids != NULL && gn == NULL)) { > BBPreclaim(bn); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 2

New Jul2017 stable branch created
by Sjoerd Mullender 24 Apr '17

24 Apr '17

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2017. So in each of your stable checkout directories you can do hg pull hg update -rJul2017 to update to this new branch. -- Sjoerd Mullender

3 2

MonetDB Intra-Query Checkpointing
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi all & Stefan, I mentioned my intra-query checkpointing work in my query about database sizing, Stefan expressed interest so I thought I'd give a brief overview in a separate topic for anyone interested... This work was inspired by: - the observation that current database HA doesn't do much to protect in-flight queries, and - the existence of databases such as MonetDB, and SAP HANA that materialize their results and have graph-like query plans. My work aims to leverage potential lower cost of checkpointing these materialized intermediates (since there is no materialization required), as well as the graph like structure of query plans to provide HA for queries. This is intended to benefit long running queries, since the cost to simply repeating short queries is low. The first aspect of my work is algorithms to produce checkpoint plans from query plan graphs. Currently this planning is an offline step, based on the output of query profiling. This work so far has discovered some interesting counter-intuitive but useful placement of checkpoints. The second aspect is my implementation for evaluating the general mechanism, and specific algorithms. I have implemented this in MonetDB. Some details: - I produce the graph structure for my algorithms from the exported MAL for a query. - Clients connect through a load balancer ("Persephone") that I have written for MonetDB. - This load balancer also orchestrates the mechanism it: o injects checkpoint plans into the database o receives checkpoints o detects failures o initiates replays on another database node by sending it a replay plan (checkpoints + skipped operations) - This process is transparent to clients and they require no modifications. - My modifications to MonetDB: (which shift most of the decision making to Persephone) o add a side-channel connection facilitating this orchestration. o Modify the mal interpreter: ? When a query matches a checkpoint plan, the selected intermediates will be transmitted using the side-channel connection to Persephone. ? To follow replay plans: using previously checkpointed intermediates, and entirely skipping operations that are now redundant due to checkpoints Hopefully this answers some of your questions Stefan. I'm happy to answer questions if anyone is interested. Thanks, ~ Daniel.

1 0

Sizing benchmarks
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi, I was wondering is there any guidance or rules of thumb for setting the parameters (database size, number of simultaneous users) of benchmarks in MonetDB in relation to system properties (RAM, CPU, Disk, Network)? Specifically I'm looking to run microbenchmarks with TPC-H, looking at performance of each query in isolation, so for my setup a RAM vs database size relationship is probably most important. (This is to evaluate the impact of intra-query checkpointing capabilities I've added to a modified version of MonetDB as part of my PhD work.) Thanks, ~ Daniel.

2 2