Re: [MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows

12 Mar 2009

      How exactly do you load your data (via RJDBC)?
A single 
	COPY 600000000 RECORDS INTO <table> FROM <file> USING DELIMITERS <...>;
or a single INSERT statement per row?

Does your table have constraints (keys, NOT NULL, ...) defined?

Stefan

On Thu, Mar 12, 2009 at 01:07:00AM -0700, Yue Sheng wrote:
...
I have a new 1TB on the Machine, it has more than 800GB free; much more than
what's needed... So that cannot be the problem.
On Thu, Mar 12, 2009 at 12:26 AM, Martin Kersten wrote:
...
Yue Sheng wrote:
...
I'm been trying to insert from file for over 322 files, totally 12GB
with over 600 million rows. The RJDBC connection hangs after 350 million
row. I noticed the VSIZE in Activity Monitor is 30.85GB, closer to 32GB
system memory. Is this the reason that I cannot connect to the database?
What should I do?
Error message I get from MonetDB server:
# MonetDB server v5.8.4, based on kernel v1.26.4
# Serving database 'bigdata', using 8 threads
# Compiled for i386-apple-darwin9.6.0/64bit with 64bit OIDs dynamically
linked
# Copyright (c) 1993-July 2008 CWI.
# Copyright (c) August 2008- MonetDB B.V., all rights reserved
# Visit http://monetdb.cwi.nl/ for further information
#warning: please don't forget to set your vault key!
#(see /Users/someuser/MonetDB64/etc/monetdb5.conf)
# Listening for connection requests on mapi:monetdb://127.0.0.1:50000/
http://127.0.0.1:50000/
# MonetDB/SQL module v2.26.4 loaded
...
!ERROR: GDKsave: failed name=24/2460, ext=tail, mode 0
!ERROR: GDKsave: failed name=23/2317, ext=tail, mode 0
!ERROR: GDKsave: failed name=25/2503, ext=tail, mode 0
!ERROR: GDKsave: failed name=26/2657, ext=tail, mode 0
!ERROR: GDKsave: failed name=22/2274, ext=tail, mode 0
!ERROR: GDKsave: failed name=22/2224, ext=tail, mode 0
!ERROR: GDKsave: failed name=01/112, ext=head, mode 0
!ERROR: logger_exit: could not open
/Users/someuser/MonetDB64/var/MonetDB5/sql_logs/bigdata/sql/log
This looks as if the device on which you attempt to write
is full.
System:
MacPro Dual Quad Xenon
Leopard OS X
32GB RAM
R 2.8.1 32Bit
MonetDB v5.8.4 complied for i386-apple-darmin9.6.0/64bit
Many thanks.
Yuesheng
------------------------------------------------------------------------
------------------------------------------------------------------------------
...
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
------------------------------------------------------------------------
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
...
------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |