[MonetDB-users] Data Load Issue

Hi, I am trying to load a 12.8 Million row table using COPY command. The table has about 256 columns. After 4 or 5 hours of running, the server terminated abruptly. I am running this on a AMD 64 Bit running Redhat Linux with 3GB of memory and lots of swap space. Any help in identifying the issue would be appreciated. Thanks, - Venks

Hi, This short message is hard to interpret. In general there may be several causes for a long load. - the size of the table in Mbs (we load 100Gb in 5 hours) - the scheme contains foreign key dependencies that should be checked at some point - the database contains a lot of strings, whose distribution bites the underlying hash function and creates long collision chains - the system can not properly determine the size of BATs (eg due to variable strings) In loading a database it helps to plan your experiments. Start with a small load file and a few more on the road of your target size (e.g. 10K,100k 500K,1M tuples) This gives you enough information to predict how long the complete load takes. Venks wrote:
Hi,
I am trying to load a 12.8 Million row table using COPY command. The table has about 256 columns. After 4 or 5 hours of running, the server terminated abruptly. I am running this on a AMD 64 Bit running Redhat Linux with 3GB of memory and lots of swap space. Any help in identifying the issue would be appreciated.
Thanks,
- Venks
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (2)
-
Martin Kersten
-
Venks