Re: [MonetDB-users] * loading big table

Hi, How much memory do you have? Dariusz. Yue Sheng wrote:

I have similar problem when loading large dataset. I'm trying to load 650million by 10. It gets stuck after ~350million. Interestingly not always at the same point...

On Wed, Apr 8, 2009 at 6:10 AM, dariuszs mailto:dariuszs@svnp.com> wrote:

Hi, Can anybody help me with this issue? Thanks. Dariusz.

dariuszs wrote: > Hi, > Specs: > MonetDB Server 5.10.0, based on kernel v1.28.0 > Windows x64 Server 2003 Enterprise > Intel Dual Xenon 5470 server, SAS drives, 48GB of memory / 220GB swap > space > > CREATE TABLE xxx( > C1 VARCHAR(16), > C2 VARCHAR(15), > C3 VARCHAR(1), > C4 VARCHAR(4), > C5 VARCHAR(1), > C6 VARCHAR(10), > C7 VARCHAR(2), > C8 VARCHAR(1), > C9 VARCHAR(1), > C10 VARCHAR(1), > C11 VARCHAR(1), > C12 VARCHAR(1), > C13 VARCHAR(2), > C14 VARCHAR(1), > C15 VARCHAR(2), > C16 VARCHAR(30), > C17 VARCHAR(16), > C18 VARCHAR(2), > C19 VARCHAR(5), > C20 VARCHAR(4), > C21 VARCHAR(3), > C22 VARCHAR(2), > C23 VARCHAR(1), > C24 VARCHAR(2), > C25 VARCHAR(1), > C26 VARCHAR(2), > C27 VARCHAR(1), > C28 VARCHAR(2), > C29 VARCHAR(3), > C30 VARCHAR(4), > C31 VARCHAR(10), > C32 VARCHAR(1), > C33 VARCHAR(2), > C34 VARCHAR(4), > C35 VARCHAR(1), > C36 VARCHAR(1), > C37 VARCHAR(6), > C38 VARCHAR(6), > C39 VARCHAR(2), > C40 VARCHAR(6), > C41 VARCHAR(1), > C42 VARCHAR(6), > C43 VARCHAR(2), > C44 VARCHAR(1), > C45 VARCHAR(1), > C46 VARCHAR(1), > C47 VARCHAR(1), > C48 VARCHAR(1), > C49 VARCHAR(1), > C50 VARCHAR(1), > C51 VARCHAR(2), > C52 VARCHAR(6), > C53 VARCHAR(2), > C54 VARCHAR(2), > C55 VARCHAR(6), > C56 VARCHAR(2), > C57 VARCHAR(1), > C58 VARCHAR(1), > C59 VARCHAR(1), > C60 VARCHAR(1), > C61 VARCHAR(1), > C62 VARCHAR(1), > C63 VARCHAR(1), > C64 VARCHAR(4), > C65 VARCHAR(1), > C66 VARCHAR(1), > C67 VARCHAR(1), > C68 VARCHAR(3), > C69 VARCHAR(2), > C70 VARCHAR(2), > C71 VARCHAR(1), > C72 VARCHAR(1), > C73 VARCHAR(1), > C74 VARCHAR(1), > C75 VARCHAR(2), > C76 VARCHAR(8), > C77 VARCHAR(8), > C78 VARCHAR(8), > C79 VARCHAR(1), > C80 VARCHAR(1), > C81 VARCHAR(1), > C82 VARCHAR(1), > C83 VARCHAR(6), > C84 VARCHAR(6), > C85 VARCHAR(6), > C86 VARCHAR(1), > C87 VARCHAR(1), > C88 VARCHAR(1), > C89 VARCHAR(1), > C90 VARCHAR(1), > C91 VARCHAR(1), > C92 VARCHAR(1), > C93 VARCHAR(1), > C94 VARCHAR(1), > C95 VARCHAR(1), > C96 VARCHAR(1), > C97 VARCHAR(1), > C98 VARCHAR(1), > C99 VARCHAR(1), > C100 VARCHAR(1), > C101 VARCHAR(1), > C102 VARCHAR(1), > C103 VARCHAR(1), > C104 VARCHAR(4), > C105 VARCHAR(4), > C106 VARCHAR(6), > C107 VARCHAR(8), > C108 VARCHAR(8), > C109 VARCHAR(1), > C110 VARCHAR(1), > C111 VARCHAR(8), > C112 VARCHAR(6), > C113 VARCHAR(1), > C114 VARCHAR(1), > C115 VARCHAR(1), > C116 VARCHAR(1), > C117 VARCHAR(1), > C118 VARCHAR(1), > C119 VARCHAR(1), > C120 VARCHAR(3), > C121 VARCHAR(1), > C122 VARCHAR(1), > C123 VARCHAR(1), > C124 VARCHAR(6), > C125 VARCHAR(6), > C126 VARCHAR(1), > C127 VARCHAR(1), > C128 VARCHAR(1), > C129 VARCHAR(22), > C130 VARCHAR(27)); > copy 160000000 records into xxx from 'm:/monetdb_data/xxx.txt'; > commit; > > OK, so the server loads the file for about 4 hours and I see CPU > activity, I see memory usage going up (47GB) and down several times, > but then after 4 hours everything sits - also I see that server > creates a files in the sql_logs directory 'log.179' which grew up to > 47GB and then it stopped. > > I was able to successfully load 130 million records using the same > file and the same system in about 3 hours. Thanks. Dariusz. > > > > > Stefan Manegold wrote: >> Which version of MonetDB? >> Which OS? >> Which HW arcitecture? >> Which table schema? >> What happens during "sits" (CPU bound, I/O bound, ...)? >> How do you "load"? >> >> Stefan >> >> On Fri, Apr 03, 2009 at 02:40:10PM -0400, dariuszs wrote: >> >>> Hi, >>> I've got a table with 130 fields. When I load 130 million records >>> into that table (about 50G) - it's not a problem, but when I try to >>> load 160 million records (about 60G of data) into it this thing just >>> sits for hours with no error messages. Can somebody help me? >>> Thanks. Dariusz. >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> MonetDB-users mailing list >>> MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/monetdb-users >>> >>> >> >> > >

------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com ------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users