Re: growing size of database

On Wed, Nov 14, 2012 at 11:04:06AM -0500, Medha Atre wrote:

On Wed, Nov 14, 2012 at 10:55 AM, Stefan Manegold wrote: [...]

...

- You're talking about "merge-joins"; did you verify that MonetDB indeed uses merge-joins? Prerequisite would be that the data is sorted and MonetDB "knows" about that ...

That was just my assumption based on previously published papers.. but yes, I do not know of internal details of MonetDB and nor have I touched MonetDB code while running my data/queries on it.

In case your join attributes are both sorted, MonetDB will use merge-join; otherwise, it will use hash-join, unless you join columns exceed physical memory size.

...

- During query processing, dbfarm size can (will) grow due to intermediate results; given that we (MonetDB) use memory-mapped files for virtual memory, it depends on the intermediate result sizes and you manchines physical memory size, and your OS, whether or not the data actually gets written to disk or not ...

Yes, that is what even I understood, but which is why I mentioned that the size I measured was after the server was shutdown completely, and I presumed that like any other server, MonetDB would not maintain cache or intermediate results "across" multiple reboots of the server (however, my knowledge of internals of large-scale servers is limited).

Clean-up of "left-overs" is done at server (re-)start, not at shutdown. Of course, no longer used intermediates are clean-up instantly (while server is running), unless deemed useful for reuse via the heap-cache.

...

- Do I understand you loop correctly, that you bulk-load your data 6000 times???

No. Data is loaded only once, and 6000 distinct queries are run on it one after another, with shutting down MonetDB every 1000 queries (to relieve memory leaks, if there are any).

Thanks --- I mis-read "bulk_q_file" for "bulk_load_file" ... Stefan [...] -- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list