[MonetDB-users] Limit memory
Hello, I'm trying to limit the memory that monetdb can use during its executions. I'm using mal language and I've found this operation in order to do it. command mem_maxsize(v:lng) :void address set_mem_maxsize comment "Set the maximum usable amount of physical swapspace in KB."; When I use this operation the connection is terminated. I let here some examples: (I'm working with a machine with 32 GB) mal>status.mem_maxsize(4294967296:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(4294967296:lng); ERROR = !Connection terminated mal>status.mem_maxsize(1073741824:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(1073741824:lng); ERROR = !Connection terminated mal>status.mem_maxsize(1048576:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(1048576:lng); ERROR = !Connection terminated Looking forward to hearing from you. Thank you. Mike
Mike, please file a bug report via http://bugs.monetdb.org/ Thanks! Stefan On Fri, Jun 11, 2010 at 02:02:14PM +0200, Miguel Ángel Águila Lorente wrote:
Hello,
I'm trying to limit the memory that monetdb can use during its executions. I'm using mal language and I've found this operation in order to do it.
command mem_maxsize(v:lng) :void address set_mem_maxsize comment "Set the maximum usable amount of physical swapspace in KB.";
When I use this operation the connection is terminated. I let here some examples: (I'm working with a machine with 32 GB)
mal>status.mem_maxsize(4294967296:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(4294967296:lng); ERROR = !Connection terminated
mal>status.mem_maxsize(1073741824:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(1073741824:lng); ERROR = !Connection terminated
mal>status.mem_maxsize(1048576:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(1048576:lng); ERROR = !Connection terminated
Looking forward to hearing from you. Thank you.
Mike
------------------------------------------------------------------------------ ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |
Hi Miguel, On 11-06-2010 14:02:14 +0200, Miguel Ángel Águila Lorente wrote:
I'm trying to limit the memory that monetdb can use during its executions. I'm using mal language and I've found this operation in order to do it.
command mem_maxsize(v:lng) :void address set_mem_maxsize comment "Set the maximum usable amount of physical swapspace in KB.";
When I use this operation the connection is terminated. I let here some examples: (I'm working with a machine with 32 GB)
This operation is most probably not doing what you want. The crash you reported is not dependant on the value you supply, it cannot be avoided. In short, it's flawed, so don't use it. If you want to limit monetdb's memory usage, you better go with ulimit.
Hello, I'm trying to use ulimit to limit memory but it doesn't work. My steps are: 1. Open a shell 2. write ulimit -m 4194304 3. and start mserver5 When I'm in a client I execute a query and with comand top I can see that is using more than 4194304 kbytes. After several trials I did the same but limiting the cpu time with: ulimit -t 3 and it works fine, after 3 seconds the server was killed. I don't understand what is happening, the only thing I can supouse is that ulimit can't be used with monet. I hope a solution. Thank you. Mike El lun, 14-06-2010 a las 14:33 +0200, Fabian Groffen escribió:
Hi Miguel,
On 11-06-2010 14:02:14 +0200, Miguel Ángel Águila Lorente wrote:
I'm trying to limit the memory that monetdb can use during its executions. I'm using mal language and I've found this operation in order to do it.
command mem_maxsize(v:lng) :void address set_mem_maxsize comment "Set the maximum usable amount of physical swapspace in KB.";
When I use this operation the connection is terminated. I let here some examples: (I'm working with a machine with 32 GB)
This operation is most probably not doing what you want. The crash you reported is not dependant on the value you supply, it cannot be avoided. In short, it's flawed, so don't use it. If you want to limit monetdb's memory usage, you better go with ulimit.
------------------------------------------------------------------------------ ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 17-06-2010 11:20:59 +0200, Miguel Ángel Águila Lorente wrote:
Hello,
I'm trying to use ulimit to limit memory but it doesn't work. My steps are:
1. Open a shell 2. write ulimit -m 4194304 3. and start mserver5
When I'm in a client I execute a query and with comand top I can see that is using more than 4194304 kbytes.
After several trials I did the same but limiting the cpu time with:
ulimit -t 3
and it works fine, after 3 seconds the server was killed.
I don't understand what is happening, the only thing I can supouse is that ulimit can't be used with monet.
you're probably using Linux, it is known for not respecting it.
On Thu, 17 Jun 2010, Fabian Groffen wrote:
I don't understand what is happening, the only thing I can supouse is that ulimit can't be used with monet.
you're probably using Linux, it is known for not respecting it.
Pragmatically the user would be helped if MonetDB provided an option to specify a maximum memory use. Since Windows doesn't have ulimit either more than half of the userbase is missing this feature. The consequence of not having the automatic malloc failures provided by ulimit, would be greatly outperformed by using the same information to compute the memory pressure. Since there is no hard limit, it will not fail, and given a conservative approach: the user gets what he wants. Stefan
On Thu, Jun 17, 2010 at 11:45:28AM +0200, Fabian Groffen wrote:
On 17-06-2010 11:20:59 +0200, Miguel Ángel Águila Lorente wrote:
Hello,
I'm trying to use ulimit to limit memory but it doesn't work. My steps are:
1. Open a shell 2. write ulimit -m 4194304 3. and start mserver5
When I'm in a client I execute a query and with comand top I can see that is using more than 4194304 kbytes.
After several trials I did the same but limiting the cpu time with:
ulimit -t 3
and it works fine, after 3 seconds the server was killed.
I don't understand what is happening, the only thing I can supouse is that ulimit can't be used with monet.
you're probably using Linux, it is known for not respecting it.
`ulimit -d` (not `ulimit -m` seems to work as expected on my Fedora 12 machine: ======== $ ulimit -a -------- core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited ^^^^^^^^^ scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 73216 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ======== $ mserver5 --version -------- MonetDB server v5.18.6 (64-bit), based on kernel v1.36.6 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Found 7.8GiB available memory, 4 available cpu cores ^^^^^^ Configured for prefix: /ufs/manegold/_/scratch0/Monet/HG/Feb2010/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0 29 Mar 2010 (compiled with OpenSSL 1.0.0-fips 29 Mar 2010) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: manegold@rig.ins.cwi.nl (x86_64-unknown-linux-gnu) Compilation: gcc -Wall -Wextra -fno-strict-aliasing -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 Linking : ld -IPA -m elf_x86_64 ======== $ ulimit -d $[1024*1024] -------- ======== $ ulimit -a -------- core file size (blocks, -c) 0 data seg size (kbytes, -d) 1048576 ^^^^^^^ scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 73216 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ======== $ mserver5 --version -------- MonetDB server v5.18.6 (64-bit), based on kernel v1.36.6 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Found 1.0GiB available memory, 4 available cpu cores ^^^^^^ Configured for prefix: /ufs/manegold/_/scratch0/Monet/HG/Feb2010/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 1.0.0 29 Mar 2010 (compiled with OpenSSL 1.0.0-fips 29 Mar 2010) libxml2: 2.7.6 (compiled with 2.7.6) Compiled by: manegold@rig.ins.cwi.nl (x86_64-unknown-linux-gnu) Compilation: gcc -Wall -Wextra -fno-strict-aliasing -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 Linking : ld -IPA -m elf_x86_64 ======== Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |
participants (4)
-
Fabian Groffen -
Miguel Ángel Águila Lorente -
Stefan de Konink -
Stefan Manegold