Memory bloat on concurrency - Any fix to release memory post query run - users-list

newer
(lastest version) monetdb source...

Memory bloat on concurrency - Any fix to release memory post query run

Sharma, Sreejith

10 Aug 2017 10 Aug '17

11:26 a.m.

Hi, I have been using monetdb (11.25.3) in CentOS for one of application. I have been working on a PSR testing for this application and have noticed on high concurrency and on same pattern of query running again and again causing memory to bloat up and doesn’t release once the query run completes. I understand that the data is lifted into memory and the result set is also cached, however I assume this should be release once the system is idle which I don’t see it as of now. Is there any way or parameter to release the unused memory blocks? I have asked this before, but couldn't get to the root to fix this. Regards, Sreejith

Show replies by date

Ying Zhang

10 Aug 10 Aug

5:05 p.m.

Hai Sreejith, There is no way to manually force MonetDB to release memory. However, what you described is not supposed to happen. Not very long ago (but should be before the version you use), we have put quite some efforts in release unused memory earlier (i.e. free a variable as soon as it’s not needed, instead of waiting until the end of query execution to release all variables at once) so as to reduce MonetDB’s virtual memory usage. Can you please upgrade to the latest release Jul2017-SP1 (11.27.5)? The version you’re using is a bit old. In the bug-fix releases following that, several memory leaks were fixed [1]. If your problem persists, can you please give us the necessary data sample, schema, queries and script to reproduce your problem? Regards, Jennie [1] https://www.monetdb.org/OldReleaseNotes/Dec2016

...

On 10 Aug 2017, at 11:26, Sharma, Sreejith wrote:

Hi,

I have been using monetdb (11.25.3) in CentOS for one of application. I have been working on a PSR testing for this application and have noticed on high concurrency and on same pattern of query running again and again causing memory to bloat up and doesn’t release once the query run completes. I understand that the data is lifted into memory and the result set is also cached, however I assume this should be release once the system is idle which I don’t see it as of now.

Is there any way or parameter to release the unused memory blocks? I have asked this before, but couldn't get to the root to fix this.

Regards, Sreejith _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Sharma, Sreejith

5:18 p.m.

New subject: [EXTERNAL] Re: Memory bloat on concurrency - Any fix to release memory post query run

Hi Jennie, Apologies, I have MonetDB-11.25.23 installed. I guess this version has the memory leaks issues fixed. Let me know if there is any other version of SP available. Meanwhile I will send across the details. Regards, Sreejith -----Original Message----- From: users-list [mailto:users-list-bounces+sreejith.sharma=harman.com@monetdb.org] On Behalf Of Ying Zhang Sent: Thursday, August 10, 2017 8:36 PM To: Communication channel for MonetDB users Subject: [EXTERNAL] Re: Memory bloat on concurrency - Any fix to release memory post query run Hai Sreejith, There is no way to manually force MonetDB to release memory. However, what you described is not supposed to happen. Not very long ago (but should be before the version you use), we have put quite some efforts in release unused memory earlier (i.e. free a variable as soon as it’s not needed, instead of waiting until the end of query execution to release all variables at once) so as to reduce MonetDB’s virtual memory usage. Can you please upgrade to the latest release Jul2017-SP1 (11.27.5)? The version you’re using is a bit old. In the bug-fix releases following that, several memory leaks were fixed [1]. If your problem persists, can you please give us the necessary data sample, schema, queries and script to reproduce your problem? Regards, Jennie [1] https://clicktime.symantec.com/a/1/2SmZnQfkzCIOWj3BYqEvQrPIz-CbpXjoikixwTcM6ZQ=?d=THHydlHeoPzbgOMuBtR7AYATbzUPaY-BBAZqe-6pJRMPmHEokVbZu0xxGL0mYpM6NvA8bNhnFJZKh8P3r2TOyW9S_yUzbvEU2RtDvtSooFyoDqeoVg8QWehPciigNFMKlqALFQmrJ3HmcGcVQU1ENt3jrCdH8hP6B6i9-BE9-vtWR5DYOeEOUMDkvDVLwEcwPaYAmL-QdKsFrQV_Hj8D5M0HzCzufne3VuxVi0YiqteCCVKXoqcmH-iLccvYmesa4KDyTeeo7lUUlaoYlPJ2mqytsgEd3KR9ZB-jLGfA3t0mVqPMgBzjoFbl3fdrjlD2yXMbpKlyUfGSMH8p1pil2Pupq2Q-HyzHQNYn7VsPuy6qJqF2PjutuzKMWGgz-f-ecv6p4QWArvqhrTpz-7R3rN575pdQ4hD_jsRqJtoXLzVHq8u4nR_kCq43FaCrC7mVey8du9nf1zk9_Ck2-OUHhNLG4sVXCw%3D%3D&u=https%3A%2F%2Fwww.monetdb.org%2FOldReleaseNotes%2FDec2016

...

On 10 Aug 2017, at 11:26, Sharma, Sreejith wrote:

Hi,

I have been using monetdb (11.25.3) in CentOS for one of application. I have been working on a PSR testing for this application and have noticed on high concurrency and on same pattern of query running again and again causing memory to bloat up and doesn’t release once the query run completes. I understand that the data is lifted into memory and the result set is also cached, however I assume this should be release once the system is idle which I don’t see it as of now.

Is there any way or parameter to release the unused memory blocks? I have asked this before, but couldn't get to the root to fix this.

Regards, Sreejith _______________________________________________ users-list mailing list users-list@monetdb.org https://clicktime.symantec.com/a/1/4arWuVyzE5LkZgTnAIJx3HZzMUb40yXaiaLm4ghdQpw=?d=THHydlHeoPzbgOMuBtR7AYATbzUPaY-BBAZqe-6pJRMPmHEokVbZu0xxGL0mYpM6NvA8bNhnFJZKh8P3r2TOyW9S_yUzbvEU2RtDvtSooFyoDqeoVg8QWehPciigNFMKlqALFQmrJ3HmcGcVQU1ENt3jrCdH8hP6B6i9-BE9-vtWR5DYOeEOUMDkvDVLwEcwPaYAmL-QdKsFrQV_Hj8D5M0HzCzufne3VuxVi0YiqteCCVKXoqcmH-iLccvYmesa4KDyTeeo7lUUlaoYlPJ2mqytsgEd3KR9ZB-jLGfA3t0mVqPMgBzjoFbl3fdrjlD2yXMbpKlyUfGSMH8p1pil2Pupq2Q-HyzHQNYn7VsPuy6qJqF2PjutuzKMWGgz-f-ecv6p4QWArvqhrTpz-7R3rN575pdQ4hD_jsRqJtoXLzVHq8u4nR_kCq43FaCrC7mVey8du9nf1zk9_Ck2-OUHhNLG4sVXCw%3D%3D&u=https%3A%2F%2Fwww.monetdb.org%2Fmailman%2Flistinfo%2Fusers-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://clicktime.symantec.com/a/1/4arWuVyzE5LkZgTnAIJx3HZzMUb40yXaiaLm4ghdQpw=?d=THHydlHeoPzbgOMuBtR7AYATbzUPaY-BBAZqe-6pJRMPmHEokVbZu0xxGL0mYpM6NvA8bNhnFJZKh8P3r2TOyW9S_yUzbvEU2RtDvtSooFyoDqeoVg8QWehPciigNFMKlqALFQmrJ3HmcGcVQU1ENt3jrCdH8hP6B6i9-BE9-vtWR5DYOeEOUMDkvDVLwEcwPaYAmL-QdKsFrQV_Hj8D5M0HzCzufne3VuxVi0YiqteCCVKXoqcmH-iLccvYmesa4KDyTeeo7lUUlaoYlPJ2mqytsgEd3KR9ZB-jLGfA3t0mVqPMgBzjoFbl3fdrjlD2yXMbpKlyUfGSMH8p1pil2Pupq2Q-HyzHQNYn7VsPuy6qJqF2PjutuzKMWGgz-f-ecv6p4QWArvqhrTpz-7R3rN575pdQ4hD_jsRqJtoXLzVHq8u4nR_kCq43FaCrC7mVey8du9nf1zk9_Ck2-OUHhNLG4sVXCw%3D%3D&u=https%3A%2F%2Fwww.monetdb.org%2Fmailman%2Flistinfo%2Fusers-list

Anton Kravchenko

5:27 p.m.

Hi Jennie,

...

we have put quite some efforts in release unused memory earlier (i.e. free a variable as soon as it’s not needed, instead of waiting until the end of query execution to release all variables at once)

Have it been tested against a query that is stopped by "call sys.stop(qtag)" and would be a memory properly released for this case scenario? Regards, Anton On Thu, Aug 10, 2017 at 8:05 AM, Ying Zhang wrote:

...

Hai Sreejith,

There is no way to manually force MonetDB to release memory. However, what you described is not supposed to happen. Not very long ago (but should be before the version you use), we have put quite some efforts in release unused memory earlier (i.e. free a variable as soon as it’s not needed, instead of waiting until the end of query execution to release all variables at once) so as to reduce MonetDB’s virtual memory usage.

Can you please upgrade to the latest release Jul2017-SP1 (11.27.5)? The version you’re using is a bit old. In the bug-fix releases following that, several memory leaks were fixed [1].

If your problem persists, can you please give us the necessary data sample, schema, queries and script to reproduce your problem?

Regards, Jennie

[1] https://www.monetdb.org/OldReleaseNotes/Dec2016

...
On 10 Aug 2017, at 11:26, Sharma, Sreejith wrote:

Hi,

I have been using monetdb (11.25.3) in CentOS for one of application. I have been working on a PSR testing for this application and have noticed on high concurrency and on same pattern of query running again and again causing memory to bloat up and doesn’t release once the query run completes. I understand that the data is lifted into memory and the result set is also cached, however I assume this should be release once the system is idle which I don’t see it as of now.

Is there any way or parameter to release the unused memory blocks? I have asked this before, but couldn't get to the root to fix this.

Regards, Sreejith _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Ying Zhang

11 Aug 11 Aug

4:38 p.m.

...

On 10 Aug 2017, at 17:27, Anton Kravchenko wrote:

Hi Jennie,

...
we have put quite some efforts in release unused memory earlier (i.e. free a variable as soon as it’s not needed, instead of waiting until the end of query execution to release all variables at once)

Have it been tested against a query that is stopped by "call sys.stop(qtag)" and would be a memory properly released for this case scenario?

Hai Anton, These two things are orthogonal to each other. The earlier-release of memory happens *during* the query execution, so this will be done no matter how a query is stopped (i.e. by a sys.stop() or because its execution has finished). After the end of a query execution, either because it has finished or because sys.stop() was called on this query, MonetDB should free up whatever memory used for this query. If that’s not the case, there’s probably a bug we should fix. However, please be aware MonetDB cannot stop a query execution when it’s in the middel of processing a MAL statement. A sys.stop() will only stop MonetDB from executing the next MAL statement. So, if MonetDB is executing a long running MAL statement, it can take some time before sys.stop() is effective. Best, Jennie

...

Regards, Anton

On Thu, Aug 10, 2017 at 8:05 AM, Ying Zhang wrote: Hai Sreejith,

There is no way to manually force MonetDB to release memory. However, what you described is not supposed to happen. Not very long ago (but should be before the version you use), we have put quite some efforts in release unused memory earlier (i.e. free a variable as soon as it’s not needed, instead of waiting until the end of query execution to release all variables at once) so as to reduce MonetDB’s virtual memory usage.

Can you please upgrade to the latest release Jul2017-SP1 (11.27.5)? The version you’re using is a bit old. In the bug-fix releases following that, several memory leaks were fixed [1].

If your problem persists, can you please give us the necessary data sample, schema, queries and script to reproduce your problem?

Regards, Jennie

[1] https://www.monetdb.org/OldReleaseNotes/Dec2016

...
On 10 Aug 2017, at 11:26, Sharma, Sreejith wrote:

Hi,

I have been using monetdb (11.25.3) in CentOS for one of application. I have been working on a PSR testing for this application and have noticed on high concurrency and on same pattern of query running again and again causing memory to bloat up and doesn’t release once the query run completes. I understand that the data is lifted into memory and the result set is also cached, however I assume this should be release once the system is idle which I don’t see it as of now.

Is there any way or parameter to release the unused memory blocks? I have asked this before, but couldn't get to the root to fix this.

Regards, Sreejith _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

2703

Age (days ago)

2704

Last active (days ago)

List overview

Download

4 comments

3 participants

participants (3)

Anton Kravchenko
Sharma, Sreejith
Ying Zhang