Re: [MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows

Anyone has any tips please? This is halting our whole development. Please help! Thanks! On Mon, Mar 16, 2009 at 9:58 AM, Yue Sheng wrote:

625GB available. Way bigger than the data set. So disk space no problem

On Mon, Mar 16, 2009 at 9:56 AM, Yue Sheng wrote:

...

could it be due to sql config settings? need bigger log file size perhaps? just guessing now...

On Sat, Mar 14, 2009 at 8:56 PM, Yue Sheng wrote:

...

I don't have any indexing on the table.

On Thu, Mar 12, 2009 at 1:24 AM, Stefan Manegold

...

wrote:

...

How exactly do you load your data (via RJDBC)? A single COPY 600000000 RECORDS INTO <table> FROM <file> USING DELIMITERS <...>; or a single INSERT statement per row?

Does your table have constraints (keys, NOT NULL, ...) defined?

Stefan

On Thu, Mar 12, 2009 at 01:07:00AM -0700, Yue Sheng wrote:

...

I have a new 1TB on the Machine, it has more than 800GB free; much more than what's needed... So that cannot be the problem.

On Thu, Mar 12, 2009 at 12:26 AM, Martin Kersten < Martin.Kersten@cwi.nl>wrote:

...

Yue Sheng wrote: > I'm been trying to insert from file for over 322 files, totally 12GB > with over 600 million rows. The RJDBC connection hangs after 350 million > row. I noticed the VSIZE in Activity Monitor is 30.85GB, closer to 32GB > system memory. Is this the reason that I cannot connect to the database? > What should I do? > > Error message I get from MonetDB server: > > # MonetDB server v5.8.4, based on kernel v1.26.4 > # Serving database 'bigdata', using 8 threads > # Compiled for i386-apple-darwin9.6.0/64bit with 64bit OIDs dynamically > linked > # Copyright (c) 1993-July 2008 CWI. > # Copyright (c) August 2008- MonetDB B.V., all rights reserved > # Visit http://monetdb.cwi.nl/ for further information > #warning: please don't forget to set your vault key! > #(see /Users/someuser/MonetDB64/etc/monetdb5.conf) > # Listening for connection requests on mapi:monetdb:// 127.0.0.1:50000/ > http://127.0.0.1:50000/ > # MonetDB/SQL module v2.26.4 loaded > >!ERROR: GDKsave: failed name=24/2460, ext=tail, mode 0 > !ERROR: GDKsave: failed name=23/2317, ext=tail, mode 0 > !ERROR: GDKsave: failed name=25/2503, ext=tail, mode 0 > !ERROR: GDKsave: failed name=26/2657, ext=tail, mode 0 > !ERROR: GDKsave: failed name=22/2274, ext=tail, mode 0 > !ERROR: GDKsave: failed name=22/2224, ext=tail, mode 0 > !ERROR: GDKsave: failed name=01/112, ext=head, mode 0 > !ERROR: logger_exit: could not open > /Users/someuser/MonetDB64/var/MonetDB5/sql_logs/bigdata/sql/log This looks as if the device on which you attempt to write is full. > > > > System: > MacPro Dual Quad Xenon > Leopard OS X > 32GB RAM > R 2.8.1 32Bit > MonetDB v5.8.4 complied for i386-apple-darmin9.6.0/64bit > > > Many thanks. > > Yuesheng > > >

---

...

...

> >

---

...

...

> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > > >

---

...

...

> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

---

...

...

Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

------------------------------------------------------------------------------

...

Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users