--- Martin Kersten
During very large insert operations (total of
rows broken into batches of 500K) the mserver5
100-250M process
will consume 100% of RAM and nearly 100% of swap. I'm running on Linux 64bit with 9GB RAM and 2GB of SWAP. One cause of errors in this area is that people forget to announce the number of records that will be copied. This really makes a difference.
This problem should be solved in the upcoming release (and CURRENT) since a few weeks. It now samples the load file before BATs are allocated.
Indeed it made a big difference. From the documentation it seemed simply optional to leave off the number of records being copied. By adding it into my load script the memory exhaustion issue has been solved. MonetDB5 doesn't use swap now and performance is predictable again. I'm running a nightly build from the stable branch, BTW.