Re: [MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows

(1) How do I tell MonetDB to use a lower bound for insert?(2) Do you think this might be the problem why I have problem inserting large files? Thanks. On Mon, Mar 16, 2009 at 11:08 AM, Stefan Manegold wrote:

On Mon, Mar 16, 2009 at 10:40:58AM -0700, Yue Sheng wrote:

...

I could give an upper bound but cannot give absolute. Because I used error suppression in my R code such that it "skips" row that give insert error (such as missing column). This suppression method seems to work well for all other inserts. So shouldn't be the cause of problem

Would it be possible to tell MonteDB to "pre-allocate" more memory then it currently does? the files have a upper bound of how much rows it will have.

giving an upper bound is possibile --- it might be counter-productive, though, in case it is significantly larger than the actual value; in that case, a close lower bound (if possible) might do a better job ...

Stefan

...

Thanks.

On Mon, Mar 16, 2009 at 10:20 AM, Stefan Manegold < Stefan.Manegold@cwi.nl>wrote:

...

On Thu, Mar 12, 2009 at 09:15:17AM -0700, Yue Sheng wrote:

...

This is the loop I used, as I'm inserting a file at a time.

drv <- JDBC("nl.cwi.monetdb.jdbc.MonetDriver", "/Users/someuser/monetDB64/share/MonetDB/lib/monetdb-1.9-jdbc.jar")

conn <- dbConnect(drv, "jdbc:monetdb://localhost/bigdata","username", "password")

for (i in 1:length(filenames)) {

str <- paste("copy into usrtable from '", wholename[i],"' using delimiters '|';", sep="")

try(dbSendUpdate(conn, str))

}

Could you try giving the first "copy into" a hint of how many records you plan to insert in total, e.g.,

COPY <N> RECORDS INTO ...

where <N> is the sum of all records from all files you load,

(and optionally give each subsequent "copy into" likewise a hint of how many records the respective file holds).

This could help MonetDB memory management ...

Stefan

...

where filesnames are file names and wholename are same as filenames

but has

...

directory info as well.

Thanks.

On Thu, Mar 12, 2009 at 3:27 AM, Tom H wrote:

...

Yue Sheng wrote:

...

> !ERROR: logger_exit: could not open >

/Users/someuser/MonetDB64/var/MonetDB5/sql_logs/bigdata/sql/log

...

This looks as if the device on which you attempt to write is full.

How big is the /Users partition on the disk? # df -h

Tom

---

...

...

...

...

Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

---

...

...

...

Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

---

...

...

Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

------------------------------------------------------------------------------

...

Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users