Dear Stefan, Let's explain as far as possible. Stefan de Konink wrote:
#GDKmalloc(14777222432) fail => BBPtrim(enter) usage[mem=14840896,vm=27527806976] #BBPTRIM_ENTER: memsize=14840896,vmsize=27527806976 #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976
The system is trying to malloc space for (in this case) a hash table, but that instruction fails. Then it attempts to free up memory by swapping out tables. Given the fact that after this sweep it still fails, indicates that your swap memory has become fragmented, such that there is no consecutive area of 14 GB left. For large table loads (relative to the swap size) it is advisable to load in multiple steps. First load the data into the tables without (foreign) key checks. After this step ALTER the tables to respect the (foreign) keys. Better solutions are being worked on.
#GDKmalloc(14777222432) fail => BBPtrim(ready) usage[mem=11538120,vm=27527806976] #BATpropcheck: BAT tmp_1260(-688): could not allocate hash table for key test #GDKmalloc(14777222432) fail => BBPtrim(enter) usage[mem=11537664,vm=27527806976] #BBPTRIM_ENTER: memsize=11537664,vmsize=27527806976 #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 #GDKmalloc(14777222432) fail => BBPtrim(ready) usage[mem=10416208,vm=27527806976] #BATpropcheck: BAT tmp_1135(-605): could not allocate hash table for key test MAPI = monetdb@localhost:50000 QUERY = COPY 773410980 RECORDS INTO node_tags from '/mnt/data2/csv/node_tags.csv' USING DELIMITERS ',', '\n', ''''; ERROR = !MALException: GDKmallocmax: failed for 1892320544 bytes !ERROR: GDKload: failed name=12/1252, ext=thash !ERROR: GDKmallocmax: failed for 1892320544 bytes !ERROR: GDKload: failed name=12/1253, ext=thash
What kind of voodoo is this?
I'm trying to import a 42GB csv file using copy into, on a native system, 2GB of memory, 4GB of swap. Latest CVS version, this time compiled for CoreDuo, without debuging, with optimise.
Stefan
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers