[MonetDB-users] monetdb do not release the memory
hi, everyone i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter? Also, i found the same problem after query several times with large amount of records returned;
Hi liangmeng, On 15-06-2012 09:10:25 +0800, liangmeng wrote:
i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter?
MonetDB is a *main-memory* database. It will basically keep as much in memory as it can.
I think it is a good monetdb practice to manage the memory monetd is allocated at the OS level. Typically on Linux, you will have to use ulimit. Don't know how (if) your Windows flavour can handle that, but AFAIK, this is the way to go. Franck Le 15/06/2012 08:37, Fabian Groffen a écrit :
Hi liangmeng,
On 15-06-2012 09:10:25 +0800, liangmeng wrote:
i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter? MonetDB is a *main-memory* database. It will basically keep as much in memory as it can.
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
To be a little more precise, MonetDB is designed from a main-memory perspective and works happily with multiple GB databases. It aggressively use the memory available to keep a hotset readily available in RAM. During query processing, the virtual memory allocated by MonetDB may be much larger then your RAM, because it memory-maps columns. The system keeps an eye on not over-stressing the RAM footprint of the OS platform using operator scheduling. regards, Martin On 6/15/12 8:44 AM, Franck Routier wrote:
I think it is a good monetdb practice to manage the memory monetd is allocated at the OS level. Typically on Linux, you will have to use ulimit. Don't know how (if) your Windows flavour can handle that, but AFAIK, this is the way to go.
Franck
Le 15/06/2012 08:37, Fabian Groffen a écrit :
Hi liangmeng,
On 15-06-2012 09:10:25 +0800, liangmeng wrote:
i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter? MonetDB is a *main-memory* database. It will basically keep as much in memory as it can.
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Ouch... Martin, could elaborate a bit or give me pointers ? I must admit that your last sentence lost me... "The system keeps an eye on not over-stressing the RAM footprint of the OS platform using operator scheduling." By the system, do you mean Monetdb ? And is "operator scheduling" a defined concept ? I must admit my ignorance, and would gladly accept any pointer :-) Regards, Franck Le 15/06/2012 09:30, Martin Kersten a écrit :
To be a little more precise, MonetDB is designed from a main-memory perspective and works happily with multiple GB databases. It aggressively use the memory available to keep a hotset readily available in RAM. During query processing, the virtual memory allocated by MonetDB may be much larger then your RAM, because it memory-maps columns. The system keeps an eye on not over-stressing the RAM footprint of the OS platform using operator scheduling.
regards, Martin
On 6/15/12 8:44 AM, Franck Routier wrote:
I think it is a good monetdb practice to manage the memory monetd is allocated at the OS level. Typically on Linux, you will have to use ulimit. Don't know how (if) your Windows flavour can handle that, but AFAIK, this is the way to go.
Franck
Le 15/06/2012 08:37, Fabian Groffen a écrit :
Hi liangmeng,
On 15-06-2012 09:10:25 +0800, liangmeng wrote:
i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter? MonetDB is a *main-memory* database. It will basically keep as much in memory as it can.
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Ouch... Did not want to hurt you, thanks for responding to the question initially.
Martin, could elaborate a bit or give me pointers ? I must admit that your last sentence lost me...
On 6/15/12 10:04 AM, Franck Routier wrote: pointers... the code base ;)
"The system keeps an eye on not over-stressing the RAM footprint of the OS platform using operator scheduling."
By the system, do you mean Monetdb ? And is "operator scheduling" a defined concept ? I must admit my ignorance, and would gladly accept any pointer :-)
System is indeed MonetDB. If you look at EXPLAIN sqlquery you see the plan being executed by the kernel. Multiple of such instructions are executed concurrently. The kernel scheduler makes sure that it does not overrun the memory resources, as this might cause page thrashing (recognizable as a MonetDB server with 0% cpu use and 0% io) regards, Martin
Regards,
Franck
Le 15/06/2012 09:30, Martin Kersten a écrit :
To be a little more precise, MonetDB is designed from a main-memory perspective and works happily with multiple GB databases. It aggressively use the memory available to keep a hotset readily available in RAM. During query processing, the virtual memory allocated by MonetDB may be much larger then your RAM, because it memory-maps columns. The system keeps an eye on not over-stressing the RAM footprint of the OS platform using operator scheduling.
regards, Martin
On 6/15/12 8:44 AM, Franck Routier wrote:
I think it is a good monetdb practice to manage the memory monetd is allocated at the OS level. Typically on Linux, you will have to use ulimit. Don't know how (if) your Windows flavour can handle that, but AFAIK, this is the way to go.
Franck
Le 15/06/2012 08:37, Fabian Groffen a écrit :
Hi liangmeng,
On 15-06-2012 09:10:25 +0800, liangmeng wrote:
i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter?
MonetDB is a *main-memory* database. It will basically keep as much in memory as it can.
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
thanks, Martin Kersten, but my database might grow up to TBs, is MonetDB suitable? From: Martin Kersten Date: 2012-06-15 15:30 To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] monetdb do not release the memory To be a little more precise, MonetDB is designed from a main-memory perspective and works happily with multiple GB databases. It aggressively use the memory available to keep a hotset readily available in RAM. During query processing, the virtual memory allocated by MonetDB may be much larger then your RAM, because it memory-maps columns. The system keeps an eye on not over-stressing the RAM footprint of the OS platform using operator scheduling. regards, Martin On 6/15/12 8:44 AM, Franck Routier wrote:
I think it is a good monetdb practice to manage the memory monetd is allocated at the OS level. Typically on Linux, you will have to use ulimit. Don't know how (if) your Windows flavour can handle that, but AFAIK, this is the way to go.
Franck
Le 15/06/2012 08:37, Fabian Groffen a écrit :
Hi liangmeng,
On 15-06-2012 09:10:25 +0800, liangmeng wrote:
i'am new here, i install monetdb of windows 64bit version, after copy into table from several very large files(about 10G each), i found that monetdb occupied too much momery, almost 100% of total 32G , and it keeps occupying, after 24hours, it still don't release the memory, what's the matter? MonetDB is a *main-memory* database. It will basically keep as much in memory as it can.
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (4)
-
Fabian Groffen -
Franck Routier -
liangmeng -
Martin Kersten