Hi Peter, Peter Boncz wrote:
Like Martin indicated, MonetDB runs out of memory here, when trying to obtain a contiguous area of 14GB.
But is a contiguous area of 14GB required for this? That is much memory!
The error messages show that MonetDB is pulling all the stops to try to get this memory from the system. That is, a so-called "trim" buffer manager thread tries to unload all unpinned other tables. To no avail here. You see the actvity of the trim thread only when a GDKmalloc has failed already, and the system tries again after unloading what it can.
I'm now trying to import the relative 'smaller' integer only tables without references; but it ends up also in: #BBPTRIM_ENTER: memsize=180224,vmsize=4443275264 #BBPTRIM: memtarget=0 vmtarget=0 #BBPTRIM_EXIT: memsize=14925824,vmsize=4443275264 #BBPTRIM_ENTER: memsize=180224,vmsize=4443275264 #BBPTRIM: memtarget=0 vmtarget=0 #BBPTRIM_EXIT: memsize=14925824,vmsize=4443275264 #GDKmalloc(1374238248) fail => BBPtrim(enter) usage[mem=180224,vm=4443275264] #BBPTRIM_ENTER: memsize=180224,vmsize=4443275264 #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 #BBPTRIM_EXIT: memsize=14925824,vmsize=4443275264 #GDKmalloc(1374238248) fail => BBPtrim(ready) usage[mem=180224,vm=4443275264] !ERROR: GDKmallocmax: failed for 1374238248 bytes !ERROR: GDKload: failed name=31/3141, ext=tail
The fact that though it tries to GDKmalloc you see the GDKload error message indicates that it actually tries to get the 14GB by creating a file and memory mapping on it.
The cause for failure can thus be: (1) your file system is full (test just after the error message appears, because a MonetDB restarts first clears out all temp files it created on the last run)
MonetDB runs on 95GB of free data. Resulting in 16GB in MonetDB5, having inserted 19GB worth of textual content.
(2) you try to process such large loads on a 32-bts machine (naughty naughty)
I'm not naughty, only have sick mind to try to run this amount of data on only 2GB of RAM. Linux srv1 2.6.27-gentoo-r4 #2 SMP Sat Nov 22 18:34:42 CET 2008 x86_64 Intel(R) Pentium(R) Dual CPU E2180 @ 2.00GHz GenuineIntel GNU/Linux Stefan