Hi, there has been a thread back in 2010 on that subject, tittled "Virtual Memory Requirement". Here is an extensive quote of an explaination by Stefan Manegold on Monetdb memory usage : quote: "MonetDB requires all data that needs to be active at any given point in time to fit into the address space --- and of course to fit on the storage device, i.e., your disk system). On 32-bit systems, the address space is at most 32-bit, i.e., at most 4 GB; in practice, it is actually limited to 3 GB or even 2 GB on most systems. On 64-bit systems, the address space can theoritcally be 64-bit, but in pactice is often "limited" to 48-bit or so --- not that that makes any difference ... MonetDB excessively uses main memory for processing, but does not require that all data fit in the available physical memory. To handle dataset that exceed the available physical memory, MonetDB does not (only) rely on the available swap space, but (also) uses memory-mapped files to exploit disk storage beyond the swap space as virtual memory. For example, while bulk-loading data (preferably via a COPY INTO statements from a (possibly compressed) CSV file), MonetDB need to have all column of the table that is currently being loaded "active", i.e., accessable in the address space. However, during loading, parts of the data are continuously written to the persisten files on disk, i.e., the whole table does not have to fit into main memory. E.g., loading a 100 GB table works fine on a system with 8 GB RAM and 16 GB swap -- provided there is sufficient free disk space. During query processing, MonetDB requires for each operation during the query execution that all its inputs, its outputs, and possible temporary data structure fit in the address space. MonetDB automatically resorts to virtual memory and memory mapped files for large intermedate results. Also (large) persitent tables are accessed using memory mapping. While running, you might see your mserver5 process' virtual size grow well beyond the available physical memory and possibly also well beyond your swap space. In principle, this is no problem at all. Most of this virtual size is due to the fact that large base tables (or intermediate results) that rised as files on disk are memory-mapped into the address space. However, those parts of the data that are currently not accessed do not consume any physical memory (except possible for caching purposes). However, if individual columns of your table(s) and/or indivdual columns of intermediate results exceed the size of the available physical memory, the performance of MonetDB might (will) decrease due to increased I/O requirements. In your case, you indeed limit the virtual size (including memory-mapped files) of your mserver5 to 32 GB, and hence cannot load your 115 GB table. Did you also try without limiting the virtual size? Did that work? If that does not work, either, could you please report in detail what happends (errors? crashes?) and share some details about your data, in particular the schema of your table (number and types of columns; number of records to be loaded; whether there are any constraints (primary-/foreign-keys, etc.) defined on the table, etc.) --- knowing your "create table" and "copy into" statements would help us a lot, then." Then it was suggested to limit the data memory segment using ulimit -d on the parent database process (merovingian), but with no luck for the guy starting the thread... quote : "On Fri, Mar 19, 2010 at 12:01 PM, Stefan Manegold wrote:

$ ulimit -d $[1024*1024]

$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) 1048576 ^^^^^^^^^^^^^ ^^^^^^^ [...]

$ mserver5 --version MonetDB server v5.18.2 (64-bit), based on kernel v1.36.2 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visithttp://monetdb.cwi.nl/ for further information Found 1.0GiB available memory, 4 available cpu cores ^^^^^^

I tried this, but I get different results. Do I also need to limit vmemory? $ ulimit -d 409600 $ ulimit -a time(cpu-seconds) unlimited file(blocks) unlimited coredump(blocks) 0 data(kbytes) 409600 stack(kbytes) 8192 lockedmem(kbytes) 64 memory(kbytes) unlimited nofiles(descriptors) 1024 processes unlimited vmemory(kbytes) unlimited flocks unlimited $ monetdb start aoflash starting database 'aoflash'... done $ tail -20 /var/log/merovingian.log | grep -A 3 "based on kernel " | sed s/.* # /# / # MonetDB server v5.16.2, based on kernel v1.34.2 # Serving database 'aoflash', using 1 thread # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked # Detected 481.391 MiB (504774656 bytes) main-memory. $ uname -a Linux mydomain.com 2.6.28-17-server #58-Ubuntu SMP Tue Dec 1 19:58:28 UTC 2009 i686 GNU/Linux" Maybe you could give this a try a come back to the list with your results... Franck Le 08/08/2012 13:37, Christian Braun a écrit :

Hello,

i would like to limit the memory usage of MonetDB to half of the physical available memory. This is on Debian 6.0 64-bit with 48GB memory. I added to /etc/default/monetdb: ulimit -v 25165824

On startup MonetDB still reports: 2012-08-08 13:21:30 MSG db[15678]: # Found 47.262 GiB available main-memory.

cat /proc/15687/limits shows the limit: Limit Soft Limit Hard Limit Units Max address space 25769803776 25769803776 bytes

Unfortunately on the first query i run the database gets killed: 2012-08-08 13:21:54 MSG merovingian[15663]: database 'db' (15678) was killed by signal SIGSEGV

Is there a way to limit the memory usage of MonetDB on Linux?

Package: monetdb5-server Version: 11.11.5-20120710

Regards and thanks, Christian.

------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users