- developers-list - monetdb.org

Re: MonetDB: Dec2016 - fix for bug 6310, ie properly use the left pa...
by Martin Kersten 13 May '17

13 May '17

That's quick. I am using Jul2017 for testing sqlsmith please roll forward and I'll hunt for the next one. FYI on rocks006 /export/scratch1/home/mk/ssmonetdb you find the current output of ?160K queries (still running) grep ERROR output|sort -u gives the list that triggers the one I found. There are a few more to be investigated (after grocery shopping), e.g. ERROR = !Can not access column ERROR = !can not access index column ERROR = !Can not create object ERROR = !exist operator on type clob missing ERROR = !exist operator on type int missing ERROR = !exist operator on type varchar missing ERROR = !Illegal argument ERROR = !Object not found On 13/05/2017 11:00, Niels Nes wrote: > Changeset: fea5dccda850 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fea5dccda850 > Modified Files: > sql/server/rel_select.c > Branch: Dec2016 > Log Message: > > fix for bug 6310, ie properly use the left part of a lateral join, > when the left part isn't used in the right part. > > > diffs (29 lines): > > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -4902,7 +4902,7 @@ rel_query(mvc *sql, sql_rel *rel, symbol > sql_rel *fnd = NULL; > > for (n = fl->h; n ; n = n->next) { > - int lateral = check_is_lateral(n->data.sym); > + int lateral = check_is_lateral(n->data.sym), lateral_used = 0; > > fnd = table_ref(sql, NULL, n->data.sym, 0); > if (!fnd && (rel || lateral) && sql->session->status != -ERR_AMBIGUOUS) { > @@ -4926,6 +4926,7 @@ rel_query(mvc *sql, sql_rel *rel, symbol > fnd = table_ref(sql, res, n->data.sym, lateral); > if (fnd && is_project(fnd->op)) > fnd->exps = list_merge(fnd->exps, pre_exps, (fdup)NULL); > + lateral_used = 1; > } else { > fnd = table_ref(sql, rel, n->data.sym, 0); > } > @@ -4934,7 +4935,7 @@ rel_query(mvc *sql, sql_rel *rel, symbol > > if (!fnd) > break; > - if (res && !lateral) > + if (res && !lateral_used) > res = rel_crossproduct(sql->sa, res, fnd, op_join); > else > res = fnd; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Announcement: New Dec2016-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 01 May '17

01 May '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP4 bugfix release MonetDB5 Server * Fixed a bug causing a crash during cleanup when mserver5 is stopped with monetdb stop database. * Some memory leaks were plugged. MonetDB Common * A potential deadlock was fixed in order index creation. * A bug that could happen during recovery of the write-ahead log (WAL) was fixed. See changeset 98ad79c555cc for details. SQL * Some memory leaks were plugged. * Upgrade code was added for an old change in the sys.settimeout function. * A bug was fixed with the automatic "vacuum" operation on system tables. Bug Fixes * 6110: cast of a SQL boolean value to a string or clob or (var)char is wrong * 6254: Crash (and assertion failure) after querying a view which uses a correlated subquery in the select-list * 6256: Assertion Trigger on FULL OUTER JOIN with more than two BETWEEN clauses * 6257: wrong count values (1 instead of 0) for correlated aggregation queries * 6258: Vulnerability in FITS and NETCDF data vaults * 6259: crash on select query from sqlitelogictests * 6260: Sqlitelogictest crash * 6288: Function cannot find column in merge table * 6295: msqldump writes unescaped timestamp values when using inserts

1 0

Re: MonetDB: viewless - remove BATslice of candidate lists from gdk_...
by Roberto Cornacchia 28 Apr '17

28 Apr '17

No idea what viewless branch is aiming at, but my 5 1/2th sense tells me there is something interesting here.. ;) On 27 Apr 2017 5:56 pm, "lsidir" <commits+lsidir=gmail.com(a)monetdb.org> wrote: > Changeset: 63c8d8dbed74 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=63c8d8dbed74 > Modified Files: > gdk/gdk_firstn.c > Branch: viewless > Log Message: > > remove BATslice of candidate lists from gdk_firstn > > > diffs (45 lines): > > diff --git a/gdk/gdk_firstn.c b/gdk/gdk_firstn.c > --- a/gdk/gdk_firstn.c > +++ b/gdk/gdk_firstn.c > @@ -140,7 +140,7 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > if (n >= (BUN) (candend - cand)) { > /* trivial: return the candidate list (the > * part that refers to b, that is) */ > - return BATslice(s, > + return CANDslice(s, > (BUN) (cand - (const oid *) > Tloc(s, 0)), > (BUN) (candend - (const oid *) > Tloc(s, 0))); > } > @@ -163,12 +163,12 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > /* return copy of first relevant part > * of candidate list */ > i = (BUN) (cand - (const oid *) Tloc(s, > 0)); > - return BATslice(s, i, i + n); > + return CANDslice(s, i, i + n); > } > /* return copy of last relevant part of > * candidate list */ > i = (BUN) (candend - (const oid *) Tloc(s, 0)); > - return BATslice(s, i - n, i); > + return CANDslice(s, i - n, i); > } > bn = COLnew(0, TYPE_void, n, TRANSIENT); > if (bn == NULL) > @@ -374,7 +374,7 @@ BATfirstn_unique_with_groups(BAT *b, BAT > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - return BATslice(s, 0, n); > + return CANDslice(s, 0, n); > } > > CANDINIT(b, s, start, end, cnt, cand, candend); > @@ -916,7 +916,7 @@ BATfirstn_grouped_with_groups(BAT **topn > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - bn = BATslice(s, 0, n); > + bn = CANDslice(s, 0, n); > gn = gids ? BATslice(g, 0, n) : NULL; > if (bn == NULL || (gids != NULL && gn == NULL)) { > BBPreclaim(bn); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 2

New Jul2017 stable branch created
by Sjoerd Mullender 24 Apr '17

24 Apr '17

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2017. So in each of your stable checkout directories you can do hg pull hg update -rJul2017 to update to this new branch. -- Sjoerd Mullender

3 2

MonetDB Intra-Query Checkpointing
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi all & Stefan, I mentioned my intra-query checkpointing work in my query about database sizing, Stefan expressed interest so I thought I'd give a brief overview in a separate topic for anyone interested... This work was inspired by: - the observation that current database HA doesn't do much to protect in-flight queries, and - the existence of databases such as MonetDB, and SAP HANA that materialize their results and have graph-like query plans. My work aims to leverage potential lower cost of checkpointing these materialized intermediates (since there is no materialization required), as well as the graph like structure of query plans to provide HA for queries. This is intended to benefit long running queries, since the cost to simply repeating short queries is low. The first aspect of my work is algorithms to produce checkpoint plans from query plan graphs. Currently this planning is an offline step, based on the output of query profiling. This work so far has discovered some interesting counter-intuitive but useful placement of checkpoints. The second aspect is my implementation for evaluating the general mechanism, and specific algorithms. I have implemented this in MonetDB. Some details: - I produce the graph structure for my algorithms from the exported MAL for a query. - Clients connect through a load balancer ("Persephone") that I have written for MonetDB. - This load balancer also orchestrates the mechanism it: o injects checkpoint plans into the database o receives checkpoints o detects failures o initiates replays on another database node by sending it a replay plan (checkpoints + skipped operations) - This process is transparent to clients and they require no modifications. - My modifications to MonetDB: (which shift most of the decision making to Persephone) o add a side-channel connection facilitating this orchestration. o Modify the mal interpreter: ? When a query matches a checkpoint plan, the selected intermediates will be transmitted using the side-channel connection to Persephone. ? To follow replay plans: using previously checkpointed intermediates, and entirely skipping operations that are now redundant due to checkpoints Hopefully this answers some of your questions Stefan. I'm happy to answer questions if anyone is interested. Thanks, ~ Daniel.

1 0

Sizing benchmarks
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi, I was wondering is there any guidance or rules of thumb for setting the parameters (database size, number of simultaneous users) of benchmarks in MonetDB in relation to system properties (RAM, CPU, Disk, Network)? Specifically I'm looking to run microbenchmarks with TPC-H, looking at performance of each query in isolation, so for my setup a RAM vs database size relationship is probably most important. (This is to evaluate the impact of intra-query checkpointing capabilities I've added to a modified version of MonetDB as part of my PhD work.) Thanks, ~ Daniel.

2 2

Announcement: New Dec2016-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Mar '17

31 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP3 bugfix release MonetDB Common * Fixed a bug when appending string bats that are fully duplicate eliminated. It could happend that the to-be-appended bat had an empty string at an offset and at that same offset in the to-be-appended-to bat there happened to be a (sequence of) NULL(s). Then this offset would be used, even though it might nog be the right offset for the empty string in the to-be-appended-to bat. This would result in multiple offsets for the empty string, breaking the promise of being duplicate eliminated. Bug Fixes * 6138: Weak duplicate elimination in string heaps > 64KB * 6183: ResultSet returns double quoted column name if name contains space characters * 6216: Assertion raised (sqlsmith) * 6219: Crash in rel_optimizer (sqlsmith) * 6227: Monetdb fails on remote tables * 6228: mclient crashes if real column is multiplied by it itself * 6229: ANALYZE, unexpected end of input * 6230: ANALYZE, syntax error * 6237: semijoin with empty right bat does not return immediately * 6242: Crash on rel_reduce_groupby_exps (sqlsmith) * 6243: Static optimization gives wrong result (1 + NULL = -127) * 6245: Nested query crashes all versions of MonetDB or gives wrong result starting from Dec2016-SP2 * 6246: update statements: references to a table do not bind to its alias * 6247: Type analysis issue (sqlsmith) * 6248: update statements: the semantic stage does not resolve the relation in the from clause * 6250: Assertion failure when querying a Blob column with order by DESC * 6251: Crash after adding an ordered index on sys.statistics column and querying sys.statistics * 6253: FITS Data Vaults does not work when using user/pw and other than sys schema name

1 0

BBP.dir file in MonetDB
by Senthil Vidhiyakar 31 Mar '17

31 Mar '17

Can someone tell me the need and file structure of BBP.dir file and the kind of information it contains.? Thank you, Senthil Vidhiyakar. S

2 1

File system in MonetDB
by Senthil Vidhiyakar 30 Mar '17

30 Mar '17

I want to know the file system or how the data is stored in MonetDB. I know data for each column is stored in separate files. But I want to know what those .*thead *and .*tail *files inside BAT folder of the monetDB data folder mean and how data is stored in those files. Please help me ASAP. Thank you.

2 1

Columnar Storage layout of MonetDB
by Senthil Vidhiyakar 21 Mar '17

21 Mar '17

I want to know the file layout of columnar data storage in MonetDB. I know each column is stored in separate files. Actually I want to know the exact layout as in a block diagram to get a clear picture of the storage layout. Helps are appreciated. Regards, Senthil Vidhiyakar. S

4 5