I am still trying to get better performance when doing TPC-H tests for a database more than twice the size of physical memory. I am not sure how MonetDB handles its virtual memory, but it sure seems to be going to disk an awful lot. Does it swap out entire BATs? Or just smaller blocks? What type of caching algorithms does it use? Also, one of the things I wanted to try is to setup a really big swap partition on a separate device with write-back cache enabled and have MonetDB use it (Basically, see if Linux will virtual memory better than MonetDB does, and take advantage of putting swap on a separate device for speed). When I try and do that though, MonetDB seems to just be trying to stay within the confines of physical memory available. Is there something I can configure to try and get it to use system virtual memory? Has anyone tried out anything like that? -Rahul

...

...

I am trying to run some TPC-H queries to get a sense for how >

large of databases MonetDB can handle. I am currently trying a > factor 20 database. I notice that on query 9, it is taking over

...

two hours (still running). In "top" it claims that virtual memory > for the process started at 27 GB or so, and is now down to 10 GB. > What is noteworthy is that CPU usage is pretty small, at 2%. I > previously observed 80-100% with a factor 1 database.

Does this mean you got the sf-20 loaded? What fixed your problem with loading?

Yes, I got it loaded. Since I had broken up the file into smaller chunks, I just restarted on the last chunk that it failed on before the crash. Eventually, it went through all of the tables.

...

The low cpu usage indicates that MonetDB is waiting for IO. I'll later this week try to repeat your experiment on a similar system. Then I'll have more input on how to improve things.

After writing the email I noticed that system CPU wait time was at 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is waiting on IO. Perhaps it is just my hardware.

I did disable the 2 GB swap partition and rerun. After doing so, execution time did improve somewhat. I am not sure if this means one should always disable swap on a MonetDB system, since the system drive is slower than the data drives. I will try another experiment and let you know the results.

= College For Financial Planning Nation's most successful CFP® Program with 35+ years of experience. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=0badaaf35b41b0f3ca7504bf... -- Powered by Outblaze