Re: Failed to load 128GB CSV file using Oct2012

Hai Stefan, Before loading, I have 414GB free disk space, which should be sufficient (I didn't check in Sep. how large the database was after a successful loading. According to the SSDB benchmark definition, it should be about 100G.). Database sizes just after the failure are as followings: $ duh Disk usage of directory /export/scratch2/zhang/dbfarm 0.015625 MB: merovingian.log 179464.648438 MB: ssdb/ $ cd ssdb/ $ duh Disk usage of directory /export/scratch2/zhang/dbfarm/ssdb 0.000000 MB: 31c67d12-147b-4dd1-bc1b-c6c0e4b3edbf 179464.609375 MB: bat/ 0.003906 MB: box/ 0.011719 MB: sql_logs/ There is still 239GB free on the disk. mserver5 didn't crash, my connection was just lost. Thanks! Jennie On Dec 05, 2012, at 10:06, Stefan Manegold wrote:

Jennie,

just to be sure: are you sure there is sufficient free disk space on your machine, say, twice the expected database size before starting the load?

How large is your database (and sql_logs) just after the failure while the server is still running? How much disk space is still free at that time?

Stefan

Ying Zhang wrote:

Hai all,

I try to load a CVS file of ~128GB in the ssdb branch (based on Oct2012), but after quite sometime, it failed with:

#GDKrealloc(19598973840) fails, try to free up space [memory in use=22695528216,virtual memory in use=138875848472] #GDKrealloc(19598973840) result [mem=22695528216,vm=138875848472]

In the last Sep., I was able to execute the same query with the same data. But now it takes much longer and ends in a crash. For INS1-ers, the command to start the server:

/export/scratch2/zhang/monet-install/ssdb/debug/bin/mserver5 --set gdk_dbfarm=/export/scratch2/zhang/dbfarm --dbname=ssdb --set mapi_usock=/export/scratch2/zhang/dbfarm/ssdb/.mapi.sock --set monet_vault_key=/export/scratch2/zhang/dbfarm/ssdb/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set mapi_port=60000

The command to start the query:

cd /ufs/zhang/papers/ssdb/monetdb ./run.sh loadimg small output

Does anyone have an idea? If anymore information is needed, please let me know. I'm re-running the query to get a GDB stack trace.

Thanks!

Jennie _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list