On Tue, 18 Oct 2011, Jeremy OSullivan wrote:
till system crash when data volume exceeds total capacity of memory and swap.
Since MonetDB uses memory mapping any "data volume" is in essence 'swap', it is up to the operation system to load these parts in memory that it is processing on.
So the high performance of MMDB relies on sufficient memory supply. "
Is this a fair assessment ?
I don't think so; it is about selectivity. First of all MonetDB is a column store thus only the columns that are used in a query plan have to be in memory, either in paralel or in sequence. The total amount data is therefore not the bottle neck, the selectivity of the query is. Since meta data on columns exist data can be read from a specific offset. The memory mapping allows a specific region te be load into memory by the Operating System. Obviously this will not really help with a sequential scan on a column that is bigger than the available memory of the host system, but given that scenario the actual performance issue is in the bandwidth of storage to the memory, and not the amount of memory: the data is a supply by itself - swap space. The only thing regarding to 'system crash' I can currently think of: if the intermediate results exceed the amount available memory (in physical RAM and in swap)... Stefan