Re: [MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows

On Thu, Mar 12, 2009 at 09:15:17AM -0700, Yue Sheng wrote:

This is the loop I used, as I'm inserting a file at a time.

drv <- JDBC("nl.cwi.monetdb.jdbc.MonetDriver", "/Users/someuser/monetDB64/share/MonetDB/lib/monetdb-1.9-jdbc.jar")

conn <- dbConnect(drv, "jdbc:monetdb://localhost/bigdata","username", "password")

for (i in 1:length(filenames)) {

str <- paste("copy into usrtable from '", wholename[i],"' using delimiters '|';", sep="")

try(dbSendUpdate(conn, str))

}

Could you try giving the first "copy into" a hint of how many records you plan to insert in total, e.g., COPY <N> RECORDS INTO ... where <N> is the sum of all records from all files you load, (and optionally give each subsequent "copy into" likewise a hint of how many records the respective file holds). This could help MonetDB memory management ... Stefan

where filesnames are file names and wholename are same as filenames but has directory info as well.

Thanks.

On Thu, Mar 12, 2009 at 3:27 AM, Tom H wrote:

...

Yue Sheng wrote:

...

> !ERROR: logger_exit: could not open > /Users/someuser/MonetDB64/var/MonetDB5/sql_logs/bigdata/sql/log This looks as if the device on which you attempt to write is full.

How big is the /Users partition on the disk? # df -h

Tom

------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |