I have similar problem when loading large dataset. I'm trying to load
650million by 10. It gets stuck after ~350million. Interestingly not always
at the same point...
On Wed, Apr 8, 2009 at 6:10 AM, dariuszs
Hi, Can anybody help me with this issue? Thanks. Dariusz.
dariuszs wrote:
Hi, Specs: MonetDB Server 5.10.0, based on kernel v1.28.0 Windows x64 Server 2003 Enterprise Intel Dual Xenon 5470 server, SAS drives, 48GB of memory / 220GB swap space
CREATE TABLE xxx( C1 VARCHAR(16), C2 VARCHAR(15), C3 VARCHAR(1), C4 VARCHAR(4), C5 VARCHAR(1), C6 VARCHAR(10), C7 VARCHAR(2), C8 VARCHAR(1), C9 VARCHAR(1), C10 VARCHAR(1), C11 VARCHAR(1), C12 VARCHAR(1), C13 VARCHAR(2), C14 VARCHAR(1), C15 VARCHAR(2), C16 VARCHAR(30), C17 VARCHAR(16), C18 VARCHAR(2), C19 VARCHAR(5), C20 VARCHAR(4), C21 VARCHAR(3), C22 VARCHAR(2), C23 VARCHAR(1), C24 VARCHAR(2), C25 VARCHAR(1), C26 VARCHAR(2), C27 VARCHAR(1), C28 VARCHAR(2), C29 VARCHAR(3), C30 VARCHAR(4), C31 VARCHAR(10), C32 VARCHAR(1), C33 VARCHAR(2), C34 VARCHAR(4), C35 VARCHAR(1), C36 VARCHAR(1), C37 VARCHAR(6), C38 VARCHAR(6), C39 VARCHAR(2), C40 VARCHAR(6), C41 VARCHAR(1), C42 VARCHAR(6), C43 VARCHAR(2), C44 VARCHAR(1), C45 VARCHAR(1), C46 VARCHAR(1), C47 VARCHAR(1), C48 VARCHAR(1), C49 VARCHAR(1), C50 VARCHAR(1), C51 VARCHAR(2), C52 VARCHAR(6), C53 VARCHAR(2), C54 VARCHAR(2), C55 VARCHAR(6), C56 VARCHAR(2), C57 VARCHAR(1), C58 VARCHAR(1), C59 VARCHAR(1), C60 VARCHAR(1), C61 VARCHAR(1), C62 VARCHAR(1), C63 VARCHAR(1), C64 VARCHAR(4), C65 VARCHAR(1), C66 VARCHAR(1), C67 VARCHAR(1), C68 VARCHAR(3), C69 VARCHAR(2), C70 VARCHAR(2), C71 VARCHAR(1), C72 VARCHAR(1), C73 VARCHAR(1), C74 VARCHAR(1), C75 VARCHAR(2), C76 VARCHAR(8), C77 VARCHAR(8), C78 VARCHAR(8), C79 VARCHAR(1), C80 VARCHAR(1), C81 VARCHAR(1), C82 VARCHAR(1), C83 VARCHAR(6), C84 VARCHAR(6), C85 VARCHAR(6), C86 VARCHAR(1), C87 VARCHAR(1), C88 VARCHAR(1), C89 VARCHAR(1), C90 VARCHAR(1), C91 VARCHAR(1), C92 VARCHAR(1), C93 VARCHAR(1), C94 VARCHAR(1), C95 VARCHAR(1), C96 VARCHAR(1), C97 VARCHAR(1), C98 VARCHAR(1), C99 VARCHAR(1), C100 VARCHAR(1), C101 VARCHAR(1), C102 VARCHAR(1), C103 VARCHAR(1), C104 VARCHAR(4), C105 VARCHAR(4), C106 VARCHAR(6), C107 VARCHAR(8), C108 VARCHAR(8), C109 VARCHAR(1), C110 VARCHAR(1), C111 VARCHAR(8), C112 VARCHAR(6), C113 VARCHAR(1), C114 VARCHAR(1), C115 VARCHAR(1), C116 VARCHAR(1), C117 VARCHAR(1), C118 VARCHAR(1), C119 VARCHAR(1), C120 VARCHAR(3), C121 VARCHAR(1), C122 VARCHAR(1), C123 VARCHAR(1), C124 VARCHAR(6), C125 VARCHAR(6), C126 VARCHAR(1), C127 VARCHAR(1), C128 VARCHAR(1), C129 VARCHAR(22), C130 VARCHAR(27)); copy 160000000 records into xxx from 'm:/monetdb_data/xxx.txt'; commit;
OK, so the server loads the file for about 4 hours and I see CPU activity, I see memory usage going up (47GB) and down several times, but then after 4 hours everything sits - also I see that server creates a files in the sql_logs directory 'log.179' which grew up to 47GB and then it stopped.
I was able to successfully load 130 million records using the same file and the same system in about 3 hours. Thanks. Dariusz.
Stefan Manegold wrote:
Which version of MonetDB? Which OS? Which HW arcitecture? Which table schema? What happens during "sits" (CPU bound, I/O bound, ...)? How do you "load"?
Stefan
On Fri, Apr 03, 2009 at 02:40:10PM -0400, dariuszs wrote:
Hi, I've got a table with 130 fields. When I load 130 million records into that table (about 50G) - it's not a problem, but when I try to load 160 million records (about 60G of data) into it this thing just sits for hours with no error messages. Can somebody help me? Thanks. Dariusz.
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users