Hi,
I have read previously that monetdb is intended to be used only when the data can fit in memory. I'm not sure how to interpret this statement - does
Hi, I think that is a quite interesting point. We tested MonetDB (MonetDB5-SQL-Installer-x86_64-20090708.msi) on a 64Bit Windows Server 2003 machine with 2GB ram and sufficient available HDD memory and always got "runtime error R6031 - Attempt to initialize the CRT more than once. This indicates a bug in your application." when we tried to load more than about 3.4 GB of data during a single transaction into the tables. After extending the memory to 4GB we were able to load the data into the tables, but complex crossjoins with order-by statements caused again the crashing of the server or the error mentioned above. We were able to reconstruct this error and got to know that a second after the free ram drops under 200MB, the CPU usage stops and the monetdb server application responds with the error above or freezes. The database is always fully loaded to the memory and there seem not to be any mechanism which clear the memory?! So it would be indeed very interesting to know, whether monetdb on a windows machine is able to administer a database which is significant bigger than the available ram and if the mentioned problem is a know bug. For the possibility to reconstruct the issue: We used the star-schema-benchmark described here http://www.cs.umb.edu/~xuedchen/research/publications/StarSchemaB.PDF with this data generator http://www.tpc.org/tpch/ and a scale factor 10 [dbgen -s 1 -T a (SF = 10)] (about 5.6 GB of data). Thank you in advance for your answers. Regards, Sebastian Bräuer -----Ursprüngliche Nachricht----- Von: Martin Kersten [mailto:Martin.Kersten@cwi.nl] Gesendet: Mittwoch, 22. Juli 2009 21:25 An: Communication channel for MonetDB users Betreff: Re: [MonetDB-users] Database size Charles Samuels wrote: this
mean that the entire database must fit in memory, or does it mean that only the data being operated on when selected from the database must fit in memory?
MonetDB does *not* limit your database to fit in main memory. However, it uses main memory techniques to address the elements in tables loaded for processing. This means that the hot-set accessed during query execution is limited when you run on 32-bit machines. For 64-bit architectures you don't have this 'problem'.
We are wanting to store *huge* amounts of data, ultimately several TBs.
The
actual data being select at any given time will only be several hundred MBs, however. Is this a reasonable usage of MonetDB?
The largest database we use internally to experiment with is a 6TB astronomical database. It contains two large tables, one with 400M rows of 500 columns and another of 20 B rows and 6 columns. Both tables are larger then 1TB each. This runs on a Linux dual quadcore with 64G Ram and lots of disk space. Ofcourse, at some point you well notice IO behavior ;)
But most importantly, I have noticed that all the row ID indexes are "int"
-
why the 32 bit limitation? This just seems like a symptom of the above problem; that the entire database must fit in memory. You can use the system with both 32- and 64- bit oids.
regards, Martin
Thanks a lot for your advice,
Charles
---------------------------------------------------------------------------- -- _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users