- developers-list - monetdb.org

Re: Get rid of MCstartMaintenance and MCexitMaintenance.
by aris 19 Sep '19

19 Sep '19

On 19-09-19 12:00, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Re: MonetDB: mosaic - Get rid of MCstartMaintenance and > MCexitMainte... (Martin Kersten) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 18 Sep 2019 17:13:33 +0200 > From: Martin Kersten <martin.kersten(a)cwi.nl> > To: developers-list(a)monetdb.org > Subject: Re: MonetDB: mosaic - Get rid of MCstartMaintenance and > MCexitMainte... > Message-ID: <90f5c40e-3c1c-5299-83cb-621660f3960c(a)cwi.nl> > Content-Type: text/plain; charset=utf-8; format=flowed > > I suggest to keep these functions available in the code base. > > They may be needed soon and are easier to control then > using monetdb stop <db> Sure but not in the mosaic feature branch. They are not needed there as I am now using proper GDK style bat level locks. But I guess it wouldn't hurt if somebody cherry picks the corresponding change sets directly into the default branch. But I am too busy currently :) > > > > On 17/09/2019 17:41, Aris Koning wrote: >> Changeset: b9c74a8bec29 for MonetDB >> URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b9c74a8bec29 >> Modified Files: >> clients/Tests/exports.stable.out >> monetdb5/mal/mal_client.c >> monetdb5/mal/mal_client.h >> Branch: mosaic >> Log Message: >> >> Get rid of MCstartMaintenance and MCexitMaintenance. >> >> >> diffs (99 lines): >> >> diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out >> --- a/clients/Tests/exports.stable.out >> +++ b/clients/Tests/exports.stable.out >> @@ -1449,7 +1449,6 @@ str MBMuniform(bat *ret, oid *base, lng >> int MCactiveClients(void); >> str MCawakeClient(int id); >> void MCcloseClient(Client c); >> -str MCexitMaintenance(Client cntxt); >> Client MCforkClient(Client father); >> Client MCgetClient(int id); >> bool MCinit(void); >> @@ -1457,7 +1456,6 @@ Client MCinitClient(oid user, bstream *f >> int MCinitClientThread(Client c); >> int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); >> int MCshutdowninprogress(void); >> -str MCstartMaintenance(Client cntxt, lng timeout, int abort); >> void MCstopClients(Client c); >> str MCsuspendClient(int id); >> int MCvalid(Client c); >> diff --git a/monetdb5/mal/mal_client.c b/monetdb5/mal/mal_client.c >> --- a/monetdb5/mal/mal_client.c >> +++ b/monetdb5/mal/mal_client.c >> @@ -518,64 +518,6 @@ MCawakeClient(int id) >> } >> >> /* >> - * Server maintenance modes without stopping it completery calls for care. >> - * Any new client connection should be temporily be refused >> - * and all clients should stop at a MAL instruction. >> - * In rare cases, all running queries should be aborted. >> - */ >> -str >> -MCstartMaintenance(Client cntxt, lng timeout, int abort) >> -{ >> - int active=1; >> - Client c = mal_clients; >> - (void) abort; >> - >> - MT_lock_set(&mal_contextLock); >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( cntxt != c) >> - c-> itrace = 'S'; >> - // wait for all running users to stop >> - while (active && timeout > 0){ >> - active = 0; >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( cntxt!= c && c->mode == RUNCLIENT){ >> - if (abort ){ >> - c->mode = FINISHCLIENT; >> - active++; >> - } else >> - if( cntxt != c) >> - active += c->itrace == 'S'; >> - } else >> - if (cntxt!= c && c->mode == FINISHCLIENT) >> - active++; >> - if( active == 0) >> - break; >> - MT_sleep_ms(1000); >> - timeout--; >> - } >> - if( timeout == 0 && active){ >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( c->itrace == 'W' || c->itrace == 'S') >> - c->itrace = 0; >> - throw(MAL,"clients.startmaintenance","timeout"); >> - } >> - return MAL_SUCCEED; >> -} >> - >> -str >> -MCexitMaintenance(Client cntxt) >> -{ >> - Client c = mal_clients; >> - (void) cntxt; >> - >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( c->itrace == 'W' || c->itrace == 'S') >> - c->itrace = 0; >> - MT_lock_unset(&mal_contextLock); >> - return MAL_SUCCEED; >> -} >> - >> -/* >> * Input to be processed is collected in a Client specific buffer. It >> * is filled by reading information from a stream, a terminal, or by >> * scheduling strings constructed internally. The latter involves >> diff --git a/monetdb5/mal/mal_client.h b/monetdb5/mal/mal_client.h >> --- a/monetdb5/mal/mal_client.h >> +++ b/monetdb5/mal/mal_client.h >> @@ -201,8 +201,6 @@ mal_export str MCsuspendClient(int i >> mal_export str MCawakeClient(int id); >> mal_export int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); >> mal_export int MCvalid(Client c); >> -mal_export str MCstartMaintenance(Client cntxt, lng timeout, int abort); >> -mal_export str MCexitMaintenance(Client cntxt); >> >> mal_export str PROFinitClient(Client c); >> mal_export str PROFexitClient(Client c); >> _______________________________________________ >> checkin-list mailing list >> checkin-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/checkin-list >> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list > > > ------------------------------ > > End of developers-list Digest, Vol 78, Issue 8 > **********************************************

1 0

Re: MonetDB: mosaic - Get rid of MCstartMaintenance and MCexitMainte...
by Martin Kersten 18 Sep '19

18 Sep '19

I suggest to keep these functions available in the code base. They may be needed soon and are easier to control then using monetdb stop <db> On 17/09/2019 17:41, Aris Koning wrote: > Changeset: b9c74a8bec29 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b9c74a8bec29 > Modified Files: > clients/Tests/exports.stable.out > monetdb5/mal/mal_client.c > monetdb5/mal/mal_client.h > Branch: mosaic > Log Message: > > Get rid of MCstartMaintenance and MCexitMaintenance. > > > diffs (99 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1449,7 +1449,6 @@ str MBMuniform(bat *ret, oid *base, lng > int MCactiveClients(void); > str MCawakeClient(int id); > void MCcloseClient(Client c); > -str MCexitMaintenance(Client cntxt); > Client MCforkClient(Client father); > Client MCgetClient(int id); > bool MCinit(void); > @@ -1457,7 +1456,6 @@ Client MCinitClient(oid user, bstream *f > int MCinitClientThread(Client c); > int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); > int MCshutdowninprogress(void); > -str MCstartMaintenance(Client cntxt, lng timeout, int abort); > void MCstopClients(Client c); > str MCsuspendClient(int id); > int MCvalid(Client c); > diff --git a/monetdb5/mal/mal_client.c b/monetdb5/mal/mal_client.c > --- a/monetdb5/mal/mal_client.c > +++ b/monetdb5/mal/mal_client.c > @@ -518,64 +518,6 @@ MCawakeClient(int id) > } > > /* > - * Server maintenance modes without stopping it completery calls for care. > - * Any new client connection should be temporily be refused > - * and all clients should stop at a MAL instruction. > - * In rare cases, all running queries should be aborted. > - */ > -str > -MCstartMaintenance(Client cntxt, lng timeout, int abort) > -{ > - int active=1; > - Client c = mal_clients; > - (void) abort; > - > - MT_lock_set(&mal_contextLock); > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( cntxt != c) > - c-> itrace = 'S'; > - // wait for all running users to stop > - while (active && timeout > 0){ > - active = 0; > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( cntxt!= c && c->mode == RUNCLIENT){ > - if (abort ){ > - c->mode = FINISHCLIENT; > - active++; > - } else > - if( cntxt != c) > - active += c->itrace == 'S'; > - } else > - if (cntxt!= c && c->mode == FINISHCLIENT) > - active++; > - if( active == 0) > - break; > - MT_sleep_ms(1000); > - timeout--; > - } > - if( timeout == 0 && active){ > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( c->itrace == 'W' || c->itrace == 'S') > - c->itrace = 0; > - throw(MAL,"clients.startmaintenance","timeout"); > - } > - return MAL_SUCCEED; > -} > - > -str > -MCexitMaintenance(Client cntxt) > -{ > - Client c = mal_clients; > - (void) cntxt; > - > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( c->itrace == 'W' || c->itrace == 'S') > - c->itrace = 0; > - MT_lock_unset(&mal_contextLock); > - return MAL_SUCCEED; > -} > - > -/* > * Input to be processed is collected in a Client specific buffer. It > * is filled by reading information from a stream, a terminal, or by > * scheduling strings constructed internally. The latter involves > diff --git a/monetdb5/mal/mal_client.h b/monetdb5/mal/mal_client.h > --- a/monetdb5/mal/mal_client.h > +++ b/monetdb5/mal/mal_client.h > @@ -201,8 +201,6 @@ mal_export str MCsuspendClient(int i > mal_export str MCawakeClient(int id); > mal_export int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); > mal_export int MCvalid(Client c); > -mal_export str MCstartMaintenance(Client cntxt, lng timeout, int abort); > -mal_export str MCexitMaintenance(Client cntxt); > > mal_export str PROFinitClient(Client c); > mal_export str PROFexitClient(Client c); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Re: ALTER TABLE ALTER COLUMN SET STORAGE
by aris 14 Sep '19

14 Sep '19

Hi Dan, On 12-09-19 11:01, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Re: ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) > 2. Re: ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 11 Sep 2019 13:53:27 +0300 > From: Daniel Zvinca <daniel.zvinca(a)logbis.com> > To: "Communication channel for developers of the MonetDB suite." > <developers-list(a)monetdb.org> > Subject: Re: ALTER TABLE ALTER COLUMN SET STORAGE > Message-ID: > <CALxHsPEdkfOUKjy-xV5GDYHFry=SAst5qzZM5rk8JvFHqk1FNQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Thank you so much for your answer, Aris. > > The good news is that compression is considered and is going to be part of > MonetDB in two releases or so (one year). First release is probably much closer to half a year. > > I will try to see if I can use the current code in a custom build, I am > quite curious how that will affect performance. > To be honest I don't expect performance issues, nowadays even on SSD era, > it is still faster to read compressed data and decompress in memory if the > right ratio is there, of course. And I do expect decent compression ratio > on most of the data. The idea is that between every memory fetch you have quite some CPU cycles at your disposal which can be used to perform either (potentially) more expensive algebraic operations directly on compressed data or to just decompress data. This is the general principle that implies that at least lightweight compression can be used to accelerate queries. > > However, at this stage the MOSAIC's dual compressed - uncompressed > storage will obviously not give me the gain I need. Yet, it is interesting > to understand at least how query performance might look in the future. Again this feature is still under discussion. But I don't think it is unlikely that it will enter into Mosaic. > > Bad news is of course the compression feature is going to happen in ... one > year. But if it comes also with support for compressing in memory results > (I know, it wasnt promised), it might worth to wait. We're working it ☺. More remarks below on your other email... > > Best regards, > > > > > > > On Wed, Sep 11, 2019 at 12:18 PM aris <aris.koning(a)monetdbsolutions.com> > wrote: > >> >> >> -------- Forwarded Message -------- >> Subject: Re: ALTER TABLE ALTER COLUMN SET STORAGE >> Date: Tue, 10 Sep 2019 15:15:09 +0200 >> From: aris <aris.koning(a)monetdbsolutions.com> >> <aris.koning(a)monetdbsolutions.com> >> To: developers-list-request(a)monetdb.org >> >> Hi Daniel, >> >> On 10-09-19 12:00, developers-list-request(a)monetdb.org wrote: >> >> Send developers-list mailing list submissions to >> developers-list(a)monetdb.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://www.monetdb.org/mailman/listinfo/developers-list >> or, via email, send a message with subject or body 'help' to >> developers-list-request(a)monetdb.org >> >> You can reach the person managing the list at >> developers-list-owner(a)monetdb.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of developers-list digest..." >> >> >> Today's Topics: >> >> 1. ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 10 Sep 2019 12:40:22 +0300 >> From: Daniel Zvinca <daniel.zvinca(a)logbis.com> <daniel.zvinca(a)logbis.com> >> To: developers-list(a)monetdb.org >> Subject: ALTER TABLE ALTER COLUMN SET STORAGE >> Message-ID: >> <CALxHsPHEEExozCgdp_srxQm2tFtFOqGOGNqBTw2UhTBwWZu93g(a)mail.gmail.com> >> <CALxHsPHEEExozCgdp_srxQm2tFtFOqGOGNqBTw2UhTBwWZu93g(a)mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hello, >> >> I am interested to find out more about ALTER TABLE ALTER COLUMN SET STORAGE >> feature and how is that related to compression. >> >> As far as I understood this is related to an active development branch, >> called MOSAIC which was never merged with any of the previous MonetDB >> versions. Obviously, compression is an important feature columnar databases >> are providing for data storage and manipulation. A module like MOSAIC that >> seems to allow several compression techniques, would be an interesting >> option. >> >> Yes, compression a.k.a. Mosaic is a going to be a new feature in MonetDB. >> Although the feature won't be included in the upcoming November release. >> Most likely, you can expect the feature in the first release after the >> November release. But Mosaic is a somewhat big undertaking. our current >> road map is probably covering multiple future MonetDB releases before all >> envisioned compression features are available in MonetDB. The first >> milestone in the current road map is to apply a single compression >> technique on an entire column. But one of the next milestone is to >> partition a column into variable-sized compression blocks. Within each >> block a particular compression is applied. >> >> >> First question I have: Can MOSAIC extension be used (sources added and >> custom compiled) with success for any of its proposed codecs with any of >> the newest versions (Apr2019 +). I mean without affecting any of embedded, >> capi, rapi and pyapi modules, which all exchange data with external >> libraries. >> >> If by this you mean you want to import the mosaic module as an external >> library into an existing release out of the box, then the answer is no. >> There are some slight modifications in the GDK layer to accommodate the >> Mosaic module. And to interact with it from SQL, there are also some code >> changes in the SQL layer. But besides those dependencies, I don't expect >> any issue with the particular (x)api frameworks. But nothing is guaranteed >> obviously. It sounds like you want to hack-back port it into custom builds >> of earlier releases. I wouldn't give it a zero change of success but I do >> wish you much luck :) >> >> >> A quick read of MOSAIC code made me understand that this compression can be >> applied only on readonly PERSISTENT columns. That means that I would loose >> the major benefit of compression that I mostly need during importing stage. >> Sure I can imagine a controlled batching import scheme that would append >> data to tables and when it reaches certain threshold table is made >> readonly, then compressed, then added to a merged table, but this looks >> quite of a scenario. Am I wrong, can MOSAIC be used in a different >> scenario? >> >> Your observation about the joint life cycle of a Mosaic structure and its >> original column file is correct: currently Mosaic adds a compressed >> representation next to the existing uncompressed column. For the first >> milestone on the Mosaic road map we want to successfully apply compression >> on READ-ONLY pre-existing columns where the purpose of compression is to >> potentially accelerate analytical queries on these columns. However we are >> still looking into potentially freeing the uncompressed column once a >> compressed Mosaic heap is available. This would accommodate compression for >> the more traditional sake of limiting memory- and/or disk footprint. >> >> >> I can understand reasons behind compressing only PERSISTENT bats, yet I am >> wondering if TRANSIENT bats can also benefit from it especially for 1. >> result building stage (server-client or embedded version) or 2. for remote >> connections when data is transferred for merging operations. >> >> Regarding to above question, are there any chances that you would consider >> keeping compressed results in memory? Sure I can use instead disk >> temporary tables for subsequent manipulation, but for performance reasons >> in memory compressed results would be way faster. Actually, when embedded >> version provides a result set, it stays valid till the user releases it, >> why not to be able to also use that for possible subsequent SQL operations >> that do not fit into a CTE scenario. That would provide superior >> flexibility and memory management to CTE mechanism. Temporary results can >> be developed in steps, they can be accessed directly at any time as >> convenient as temporary views in CTE, but without the burden of possible >> temporary bats that are not released till one CTE ends. >> >> I think it is an interesting idea. But I think it is part of a more >> general goal/problem of how to handle updates on compressed data. There are >> internal discussions on this topic. But whatever the outcome, this will be >> only relevant for a much later milestone on the road map. >> >> >> >> Thank you, >> Dan >> >> >> Hope it helps. >> >> Kind regards, >> >> Aris >> >> >>

2 1

Re: ALTER TABLE ALTER COLUMN SET STORAGE
by aris 12 Sep '19

12 Sep '19

-------- Forwarded Message -------- Subject: Re: ALTER TABLE ALTER COLUMN SET STORAGE Date: Tue, 10 Sep 2019 15:15:09 +0200 From: aris <aris.koning(a)monetdbsolutions.com> To: developers-list-request(a)monetdb.org Hi Daniel, On 10-09-19 12:00, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 10 Sep 2019 12:40:22 +0300 > From: Daniel Zvinca <daniel.zvinca(a)logbis.com> > To: developers-list(a)monetdb.org > Subject: ALTER TABLE ALTER COLUMN SET STORAGE > Message-ID: > <CALxHsPHEEExozCgdp_srxQm2tFtFOqGOGNqBTw2UhTBwWZu93g(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I am interested to find out more about ALTER TABLE ALTER COLUMN SET > STORAGE > feature and how is that related to compression. > > As far as I understood this is related to an active development branch, > called MOSAIC which was never merged with any of the previous MonetDB > versions. Obviously, compression is an important feature columnar > databases > are providing for data storage and manipulation. A module like MOSAIC that > seems to allow several compression techniques, would be an interesting > option. Yes, compression a.k.a. Mosaic is a going to be a new feature in MonetDB. Although the feature won't be included in the upcoming November release. Most likely, you can expect the feature in the first release after the November release. But Mosaic is a somewhat big undertaking. our current road map is probably covering multiple future MonetDB releases before all envisioned compression features are available in MonetDB. The first milestone in the current road map is to apply a single compression technique on an entire column. But one of the next milestone is to partition a column into variable-sized compression blocks. Within each block a particular compression is applied. > > First question I have: Can MOSAIC extension be used (sources added and > custom compiled) with success for any of its proposed codecs with any of > the newest versions (Apr2019 +). I mean without affecting any of embedded, > capi, rapi and pyapi modules, which all exchange data with external > libraries. If by this you mean you want to import the mosaic module as an external library into an existing release out of the box, then the answer is no. There are some slight modifications in the GDK layer to accommodate the Mosaic module. And to interact with it from SQL, there are also some code changes in the SQL layer. But besides those dependencies, I don't expect any issue with the particular (x)api frameworks. But nothing is guaranteed obviously. It sounds like you want to hack-back port it into custom builds of earlier releases. I wouldn't give it a zero change of success but I do wish you much luck :) > > A quick read of MOSAIC code made me understand that this compression > can be > applied only on readonly PERSISTENT columns. That means that I would loose > the major benefit of compression that I mostly need during importing > stage. > Sure I can imagine a controlled batching import scheme that would append > data to tables and when it reaches certain threshold table is made > readonly, then compressed, then added to a merged table, but this looks > quite of a scenario. Am I wrong, can MOSAIC be used in a different > scenario? Your observation about the joint life cycle of a Mosaic structure and its original column file is correct: currently Mosaic adds a compressed representation next to the existing uncompressed column. For the first milestone on the Mosaic road map we want to successfully apply compression on READ-ONLY pre-existing columns where the purpose of compression is to potentially accelerate analytical queries on these columns. However we are still looking into potentially freeing the uncompressed column once a compressed Mosaic heap is available. This would accommodate compression for the more traditional sake of limiting memory- and/or disk footprint. > > I can understand reasons behind compressing only PERSISTENT bats, yet I am > wondering if TRANSIENT bats can also benefit from it especially for 1. > result building stage (server-client or embedded version) or 2. for remote > connections when data is transferred for merging operations. > > Regarding to above question, are there any chances that you would consider > keeping compressed results in memory? Sure I can use instead disk > temporary tables for subsequent manipulation, but for performance reasons > in memory compressed results would be way faster. Actually, when embedded > version provides a result set, it stays valid till the user releases it, > why not to be able to also use that for possible subsequent SQL operations > that do not fit into a CTE scenario. That would provide superior > flexibility and memory management to CTE mechanism. Temporary results can > be developed in steps, they can be accessed directly at any time as > convenient as temporary views in CTE, but without the burden of possible > temporary bats that are not released till one CTE ends. I think it is an interesting idea. But I think it is part of a more general goal/problem of how to handle updates on compressed data. There are internal discussions on this topic. But whatever the outcome, this will be only relevant for a much later milestone on the road map. > > > Thank you, > Dan Hope it helps. Kind regards, Aris >

2 2

ALTER TABLE ALTER COLUMN SET STORAGE
by Daniel Zvinca 10 Sep '19

10 Sep '19

Hello, I am interested to find out more about ALTER TABLE ALTER COLUMN SET STORAGE feature and how is that related to compression. As far as I understood this is related to an active development branch, called MOSAIC which was never merged with any of the previous MonetDB versions. Obviously, compression is an important feature columnar databases are providing for data storage and manipulation. A module like MOSAIC that seems to allow several compression techniques, would be an interesting option. First question I have: Can MOSAIC extension be used (sources added and custom compiled) with success for any of its proposed codecs with any of the newest versions (Apr2019 +). I mean without affecting any of embedded, capi, rapi and pyapi modules, which all exchange data with external libraries. A quick read of MOSAIC code made me understand that this compression can be applied only on readonly PERSISTENT columns. That means that I would loose the major benefit of compression that I mostly need during importing stage. Sure I can imagine a controlled batching import scheme that would append data to tables and when it reaches certain threshold table is made readonly, then compressed, then added to a merged table, but this looks quite of a scenario. Am I wrong, can MOSAIC be used in a different scenario? I can understand reasons behind compressing only PERSISTENT bats, yet I am wondering if TRANSIENT bats can also benefit from it especially for 1. result building stage (server-client or embedded version) or 2. for remote connections when data is transferred for merging operations. Regarding to above question, are there any chances that you would consider keeping compressed results in memory? Sure I can use instead disk temporary tables for subsequent manipulation, but for performance reasons in memory compressed results would be way faster. Actually, when embedded version provides a result set, it stays valid till the user releases it, why not to be able to also use that for possible subsequent SQL operations that do not fit into a CTE scenario. That would provide superior flexibility and memory management to CTE mechanism. Temporary results can be developed in steps, they can be accessed directly at any time as convenient as temporary views in CTE, but without the burden of possible temporary bats that are not released till one CTE ends. Thank you, Dan

1 0

New Nov2019 stable branch created
by Sjoerd Mullender 05 Sep '19

05 Sep '19

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Nov2019. So in each of your stable checkout directories you can do hg pull hg update -rNov2019 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Apr2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Sep '19

02 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019-SP1 bugfix release (11.33.11) Build Environment * Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched. MonetDB Common * We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available. SQL Frontend * Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries. * Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR. Bug Fixes * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6697: Duplicate expressions not eliminated with long CASE statement * 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed * 6703: SQL optimizer enters loop and goes into stack overflow * 6706: prepare doesn't recognize merge statement * 6712: Where clause with cast ignores sub-select * 6713: COPY INTO FROM 'file.xz' does not work * 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit) * 6715: Assertion failure in rel_bin.c for MERGE INTO command * 6716: COPY INTO does not load UTF8 encoded text * 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash) * 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation] * 6721: Possibly incorrect call to pcre_exec * 6725: Prepare statement on non-existing function crashes * 6728: SELECT DISTINCT not removing duplicates * 6729: Creating a table with duplicate column name should not be possible * 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always" * 6736: Unexpected query result with merge tables and view after upgrade * 6738: issue with window functions and hugeint type coercion * 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi * 6744: CREATE SEQUENCE seq; fails * 6748: MonetDB "forgets" columns after ALTER TABLE statements * 6749: mserver5 restart aborts/segfaults after dropping column with constraint * 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL

1 0

Alter question about MonetDB
by 角 06 Aug '19

06 Aug '19

Hi,All: I am a MonetDB user from China. I have some troubles when I was using MonetDB. I use mclient and enter this sql : "alter table t1 rename to t2；", while this mclient is alive,I use anthor mclient to connect the same database as the former and I enter this sql : "select * from t2" then I get an error : no such table. So how can I use alter command to make all mclients take effect?

1 0

Re: developers-list Digest, Vol 76, Issue 1
by Odin Loch 20 Jun '19

20 Jun '19

Where?! Need more details.. ________________________________ From: developers-list <developers-list-bounces+odin.loch=outlook.com.au(a)monetdb.org> on behalf of developers-list-request(a)monetdb.org <developers-list-request(a)monetdb.org> Sent: Thursday, June 20, 2019 8:00:02 PM To: developers-list(a)monetdb.org Subject: developers-list Digest, Vol 76, Issue 1 Send developers-list mailing list submissions to developers-list(a)monetdb.org To subscribe or unsubscribe via the World Wide Web, visit https://www.monetdb.org/mailman/listinfo/developers-list or, via email, send a message with subject or body 'help' to developers-list-request(a)monetdb.org You can reach the person managing the list at developers-list-owner(a)monetdb.org When replying, please edit your Subject line so it is more specific than "Re: Contents of developers-list digest..." Today's Topics: 1. UTF-8 data: copy into fails (Arjen P. de Vries) 2. Re: UTF-8 data: copy into fails (Panagiotis Koutsourakis) 3. Re: UTF-8 data: copy into fails (Arjen P. de Vries) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Jun 2019 23:06:48 +0200 From: "Arjen P. de Vries" <arjen(a)acm.org> To: developers-list(a)monetdb.org Subject: UTF-8 data: copy into fails Message-ID: <CAM_um4c2Qyab7Y9E9efQqdgfGPBmSr7bns=49t+-goGCqiyNZw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" I filed a bug report but did not see any response so maybe better mention it here: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6716 Definitely would be nice to know if you can reproduce the bug as reported. Groetjes, Arjen _____ Please excuse brevity - sent from a mobile.

1 0

UTF-8 data: copy into fails
by Arjen P. de Vries 20 Jun '19

20 Jun '19

I filed a bug report but did not see any response so maybe better mention it here: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6716 Definitely would be nice to know if you can reproduce the bug as reported. Groetjes, Arjen _____ Please excuse brevity - sent from a mobile.

3 2