Charles Samuels wrote:
Hi,
I have read previously that monetdb is intended to be used only when the data can fit in memory. I'm not sure how to interpret this statement - does this mean that the entire database must fit in memory, or does it mean that only the data being operated on when selected from the database must fit in memory?
MonetDB does *not* limit your database to fit in main memory. However, it uses main memory techniques to address the elements in tables loaded for processing. This means that the hot-set accessed during query execution is limited when you run on 32-bit machines. For 64-bit architectures you don't have this 'problem'.
We are wanting to store *huge* amounts of data, ultimately several TBs. The actual data being select at any given time will only be several hundred MBs, however. Is this a reasonable usage of MonetDB?
The largest database we use internally to experiment with is a 6TB astronomical database. It contains two large tables, one with 400M rows of 500 columns and another of 20 B rows and 6 columns. Both tables are larger then 1TB each. This runs on a Linux dual quadcore with 64G Ram and lots of disk space. Ofcourse, at some point you well notice IO behavior ;)
But most importantly, I have noticed that all the row ID indexes are "int" - why the 32 bit limitation? This just seems like a symptom of the above problem; that the entire database must fit in memory.
You can use the system with both 32- and 64- bit oids. regards, Martin
Thanks a lot for your advice,
Charles