[MonetDB-users] maximum memory usage
Is there a method of controlling the maximum amount of memory that MonetDB can allocate? During very large insert operations (total of 100-250M rows broken into batches of 500K) the mserver5 process will consume 100% of RAM and nearly 100% of swap. I'm running on Linux 64bit with 9GB RAM and 2GB of SWAP.
Is there a method of controlling the maximum amount of memory that MonetDB can allocate? No, not really. MonetDB already has a parallel thread
Colin Foss wrote: that frees memory already aggressively.
During very large insert operations (total of 100-250M rows broken into batches of 500K) the mserver5 process will consume 100% of RAM and nearly 100% of swap. I'm running on Linux 64bit with 9GB RAM and 2GB of SWAP.
One cause of errors in this area is that people forget to announce the number of records that will be copied. This really makes a difference. This problem should be solved in the upcoming release (and CURRENT) since a few weeks. It now samples the load file before BATs are allocated.
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
--- Martin Kersten
During very large insert operations (total of
rows broken into batches of 500K) the mserver5
100-250M process
will consume 100% of RAM and nearly 100% of swap. I'm running on Linux 64bit with 9GB RAM and 2GB of SWAP. One cause of errors in this area is that people forget to announce the number of records that will be copied. This really makes a difference.
This problem should be solved in the upcoming release (and CURRENT) since a few weeks. It now samples the load file before BATs are allocated.
Indeed it made a big difference. From the documentation it seemed simply optional to leave off the number of records being copied. By adding it into my load script the memory exhaustion issue has been solved. MonetDB5 doesn't use swap now and performance is predictable again. I'm running a nightly build from the stable branch, BTW.
participants (2)
-
Colin Foss -
Martin Kersten