Monetdb performance improvement with more RAM
Hello All, We are evaluating monetdb for use as a tick database. We are storing all the data into a single table. Our table has a column for timestamp and a few other fields. We wish to run queries against it, the most useful of which is to return all entries within a given timestamp range. This query takes a long time to execute for even a 15 minute interval (about 40 to 60 seconds). We have noted that the system RAM usage goes up to the maximum (almost 8GB) while we run such queries on monetdb. We wanted to understand whether installing more RAM (say, we upgrade to 16 GB) will result in a significant performance improvement for such queries on monetdb. Also, if this is the case, we would like to know, how much RAM is monetdb optimised for? The following is the specification of our server: Intel Xeon X3430 2.4Ghz 64Bit 8Gb Ram 500Gb HDD - Raid0 Running on Windows Server 2008 R2. We are using the 20130212 build of monetdb. Thank you, Anshuman
Hi Anshuman, MonetDB aims at using all physically available memory, provided that is required for the given data and workload. Thus, more RAM will speed up query processing until all data (base data and intermediate results) easily fit in memory, as then (expensive) I/O can be avoided. Alternatively using fast SSD's with a capable RAID controller might help --- we get almost 4 GB/s raw bandwidth out of an Raid0 of 8 500 MB/s SSD's ... For a general documentation see http://www.monetdb.org/Documentation/UserGuide/resources Also, range-selections will be faster on sorted data, as then MonetDB can exploit binary search. I must admit, though, that while we do provide Windows builds of MonetDB, we do not have much (hardly any) experience about MonetDB's performance on Windows systems as our primary research, development, testing and experimentation platforms are Linux (Fedora) systems, and we are (thus) no core Windows experts. Best, Stefan ----- Original Message -----
Hello All,
We are evaluating monetdb for use as a tick database. We are storing all the data into a single table. Our table has a column for timestamp and a few other fields. We wish to run queries against it, the most useful of which is to return all entries within a given timestamp range. This query takes a long time to execute for even a 15 minute interval (about 40 to 60 seconds). We have noted that the system RAM usage goes up to the maximum (almost 8GB) while we run such queries on monetdb. We wanted to understand whether installing more RAM (say, we upgrade to 16 GB) will result in a significant performance improvement for such queries on monetdb.
Also, if this is the case, we would like to know, how much RAM is monetdb optimised for?
The following is the specification of our server:
Intel Xeon X3430 2.4Ghz
64Bit
8Gb Ram
500Gb HDD - Raid0
Running on Windows Server 2008 R2.
We are using the 20130212 build of monetdb.
Thank you,
Anshuman
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
participants (2)
-
Anshuman Kumar -
Stefan Manegold