- developers-list - monetdb.org

Re: MonetDB: viewless - remove BATslice of candidate lists from gdk_...
by Roberto Cornacchia 28 Apr '17

28 Apr '17

No idea what viewless branch is aiming at, but my 5 1/2th sense tells me there is something interesting here.. ;) On 27 Apr 2017 5:56 pm, "lsidir" <commits+lsidir=gmail.com(a)monetdb.org> wrote: > Changeset: 63c8d8dbed74 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=63c8d8dbed74 > Modified Files: > gdk/gdk_firstn.c > Branch: viewless > Log Message: > > remove BATslice of candidate lists from gdk_firstn > > > diffs (45 lines): > > diff --git a/gdk/gdk_firstn.c b/gdk/gdk_firstn.c > --- a/gdk/gdk_firstn.c > +++ b/gdk/gdk_firstn.c > @@ -140,7 +140,7 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > if (n >= (BUN) (candend - cand)) { > /* trivial: return the candidate list (the > * part that refers to b, that is) */ > - return BATslice(s, > + return CANDslice(s, > (BUN) (cand - (const oid *) > Tloc(s, 0)), > (BUN) (candend - (const oid *) > Tloc(s, 0))); > } > @@ -163,12 +163,12 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > /* return copy of first relevant part > * of candidate list */ > i = (BUN) (cand - (const oid *) Tloc(s, > 0)); > - return BATslice(s, i, i + n); > + return CANDslice(s, i, i + n); > } > /* return copy of last relevant part of > * candidate list */ > i = (BUN) (candend - (const oid *) Tloc(s, 0)); > - return BATslice(s, i - n, i); > + return CANDslice(s, i - n, i); > } > bn = COLnew(0, TYPE_void, n, TRANSIENT); > if (bn == NULL) > @@ -374,7 +374,7 @@ BATfirstn_unique_with_groups(BAT *b, BAT > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - return BATslice(s, 0, n); > + return CANDslice(s, 0, n); > } > > CANDINIT(b, s, start, end, cnt, cand, candend); > @@ -916,7 +916,7 @@ BATfirstn_grouped_with_groups(BAT **topn > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - bn = BATslice(s, 0, n); > + bn = CANDslice(s, 0, n); > gn = gids ? BATslice(g, 0, n) : NULL; > if (bn == NULL || (gids != NULL && gn == NULL)) { > BBPreclaim(bn); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 2

New Jul2017 stable branch created
by Sjoerd Mullender 24 Apr '17

24 Apr '17

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2017. So in each of your stable checkout directories you can do hg pull hg update -rJul2017 to update to this new branch. -- Sjoerd Mullender

3 2

MonetDB Intra-Query Checkpointing
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi all & Stefan, I mentioned my intra-query checkpointing work in my query about database sizing, Stefan expressed interest so I thought I'd give a brief overview in a separate topic for anyone interested... This work was inspired by: - the observation that current database HA doesn't do much to protect in-flight queries, and - the existence of databases such as MonetDB, and SAP HANA that materialize their results and have graph-like query plans. My work aims to leverage potential lower cost of checkpointing these materialized intermediates (since there is no materialization required), as well as the graph like structure of query plans to provide HA for queries. This is intended to benefit long running queries, since the cost to simply repeating short queries is low. The first aspect of my work is algorithms to produce checkpoint plans from query plan graphs. Currently this planning is an offline step, based on the output of query profiling. This work so far has discovered some interesting counter-intuitive but useful placement of checkpoints. The second aspect is my implementation for evaluating the general mechanism, and specific algorithms. I have implemented this in MonetDB. Some details: - I produce the graph structure for my algorithms from the exported MAL for a query. - Clients connect through a load balancer ("Persephone") that I have written for MonetDB. - This load balancer also orchestrates the mechanism it: o injects checkpoint plans into the database o receives checkpoints o detects failures o initiates replays on another database node by sending it a replay plan (checkpoints + skipped operations) - This process is transparent to clients and they require no modifications. - My modifications to MonetDB: (which shift most of the decision making to Persephone) o add a side-channel connection facilitating this orchestration. o Modify the mal interpreter: ? When a query matches a checkpoint plan, the selected intermediates will be transmitted using the side-channel connection to Persephone. ? To follow replay plans: using previously checkpointed intermediates, and entirely skipping operations that are now redundant due to checkpoints Hopefully this answers some of your questions Stefan. I'm happy to answer questions if anyone is interested. Thanks, ~ Daniel.

1 0

Sizing benchmarks
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi, I was wondering is there any guidance or rules of thumb for setting the parameters (database size, number of simultaneous users) of benchmarks in MonetDB in relation to system properties (RAM, CPU, Disk, Network)? Specifically I'm looking to run microbenchmarks with TPC-H, looking at performance of each query in isolation, so for my setup a RAM vs database size relationship is probably most important. (This is to evaluate the impact of intra-query checkpointing capabilities I've added to a modified version of MonetDB as part of my PhD work.) Thanks, ~ Daniel.

2 2

Announcement: New Dec2016-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Mar '17

31 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP3 bugfix release MonetDB Common * Fixed a bug when appending string bats that are fully duplicate eliminated. It could happend that the to-be-appended bat had an empty string at an offset and at that same offset in the to-be-appended-to bat there happened to be a (sequence of) NULL(s). Then this offset would be used, even though it might nog be the right offset for the empty string in the to-be-appended-to bat. This would result in multiple offsets for the empty string, breaking the promise of being duplicate eliminated. Bug Fixes * 6138: Weak duplicate elimination in string heaps > 64KB * 6183: ResultSet returns double quoted column name if name contains space characters * 6216: Assertion raised (sqlsmith) * 6219: Crash in rel_optimizer (sqlsmith) * 6227: Monetdb fails on remote tables * 6228: mclient crashes if real column is multiplied by it itself * 6229: ANALYZE, unexpected end of input * 6230: ANALYZE, syntax error * 6237: semijoin with empty right bat does not return immediately * 6242: Crash on rel_reduce_groupby_exps (sqlsmith) * 6243: Static optimization gives wrong result (1 + NULL = -127) * 6245: Nested query crashes all versions of MonetDB or gives wrong result starting from Dec2016-SP2 * 6246: update statements: references to a table do not bind to its alias * 6247: Type analysis issue (sqlsmith) * 6248: update statements: the semantic stage does not resolve the relation in the from clause * 6250: Assertion failure when querying a Blob column with order by DESC * 6251: Crash after adding an ordered index on sys.statistics column and querying sys.statistics * 6253: FITS Data Vaults does not work when using user/pw and other than sys schema name

1 0

BBP.dir file in MonetDB
by Senthil Vidhiyakar 31 Mar '17

31 Mar '17

Can someone tell me the need and file structure of BBP.dir file and the kind of information it contains.? Thank you, Senthil Vidhiyakar. S

2 1

File system in MonetDB
by Senthil Vidhiyakar 30 Mar '17

30 Mar '17

I want to know the file system or how the data is stored in MonetDB. I know data for each column is stored in separate files. But I want to know what those .*thead *and .*tail *files inside BAT folder of the monetDB data folder mean and how data is stored in those files. Please help me ASAP. Thank you.

2 1

Columnar Storage layout of MonetDB
by Senthil Vidhiyakar 21 Mar '17

21 Mar '17

I want to know the file layout of columnar data storage in MonetDB. I know each column is stored in separate files. Actually I want to know the exact layout as in a block diagram to get a clear picture of the storage layout. Helps are appreciated. Regards, Senthil Vidhiyakar. S

4 5

Announcement: New Dec2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Mar '17

03 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP2 bugfix release Bug Fixes * 4034: argnames array in rapi.c has fixed length (that was too short) * 6080: mserver5: rel_bin.c:2391: rel2bin_project: Assertion `0' failed. * 6081: Segmentation fault (core dumped) * 6082: group.subgroup is undefined if group by is used on an expression involving only constants * 6111: Maximum number of digits for hge decimal is listed as 39 in sys.types. Should be 38 as DECIMAL(39) is not supported. * 6112: Crash upgrading Jul2015->Jun2016 * 6130: Query rewriter crashes on a NULL pointer when having a correlated subquery * 6133: A crash occurs when preparing an INSERT on joined tables during the query semantic phase * 6141: Getting an error message regarding a non-GROUP-BY column rather than an unknown identifier * 6177: Server crashes * 6186: Null casting causes no results (silent server crash?) * 6189: Removing a NOT NULL constraint from a PKey column should NOT be allowed * 6190: CASE query crashes database * 6191: MT_msync failed with "Cannot allocate memory" * 6192: Numeric column stores wrong values after adding large numbers * 6193: converting to a smaller precision (fewer or no decimals after decimal point) should round/truncate consistently * 6194: splitpart returns truncated last part if it contains non ascii caracters * 6195: Cast from huge decimal type to smaller returns wrong results * 6196: Database crashes after generate_series query * 6198: COALESCE could be more optimized * 6201: MonetDB completely giving up on certain queries - no error and no result * 6202: querying a table with an ordered index on string/varchar column crashes server and makes server unrestartable! * 6203: copy into: Failed to import table Leftover data 'False' * 6205: Integer addition overflow * 6206: casting strings with more than one trailing zero ('0') to decimal goes wrong * 6209: Aggregation over complex OR expressions produce wrong results * 6210: Upgrading a database from Jun2015 or older crashes the server * 6213: SQLsmith causes server to crash * 6217: Segfault in rel_optimizer (sqlsmith) * 6218: grouped quantiles with all null group causes following groups to return null * 6224: mal_parser: cannot refer to types containing an underscore

1 0

WARNING for users of the default branch
by Sjoerd Mullender 02 Mar '17

02 Mar '17

Databases created with the default branch before changeset f054b7310be7 (Mar 1, 2017) are *not* compatible with the default branch from that changeset on. The changeset is a merge of fixes from the Dec2016 branch, and one of the fixes introduced a version number update of the BBP.dir file. This update was incompatible with an earlier update of the version number that was exclusive to the default branch. There are two possible solutions: - dump your database(s) with a version from before the chnangeset, and restore with a version after. - manually update the version number in the BBP.dir file and ignore the possibility that the bug that was fixed may be in your database. Manually fixing the BBP.dir file involves incrementing the number on the first line of the file from 25118 to 25119. Needless to say, this is not recommended. Remember, there are no guarantees when using the default branch. In fact, there are also no guarantees when using any other branch. The only thing we promise is that we will do our best to make sure the actual releases work and can be upgraded properly. -- Sjoerd Mullender

1 0