Re: [MonetDB-users] 64bit MonetDB, JDBC Insert via RJDBC, >300 million rows

Yue Sheng wrote:

Table currently has around 500 million records. Now it hangs even if I try to run a single copy in command line:

./mclient -lsql -s "copy 3000000 records into... "

no activity in CPU monitor, Vsize constant at 40.25GB, R Size at 759MB. no error from server. does 'top' tell mserver is not consuming cpu? or just minor? does 'iostat' is telling there is no io? does 'ps' tell that the OS has put the process on the lowest priority?

i can't even kill the server.

victimized by the OS, swapping out all other processes, which then have to be brought back in.

this is very frustrating.

could someone please tell me:

(1) if the DB can handle rows that are in the billions?

the largest database running at our site has a table of 12B rows 6 cols and 400M rows 500 columns. Loaded using a schema without (referential) constraints, which are added after the fact using an ALTER statement.

(2) if the config file needs to be tweaked? no

thanks.

On Mon, Mar 16, 2009 at 5:25 PM, Stefan Manegold mailto:Stefan.Manegold@cwi.nl> wrote:

On Mon, Mar 16, 2009 at 04:56:31PM -0700, Yue Sheng wrote: > no i/o, swapping, no activity in CPU monitor > 32GB > > haven't tried. manual loading >600 files then see when it break is an > option, albert a painful one...

assiming your files are in .../DIR/*.csv:

rows="`cat .../DIR/*.csv | wc -l`" for i in .../DIR/*.csv ; do if [ "$rows" ] ; then mclient -lsql -s"copy $ROWS records into usrtable from '$i' using delimiters '|';" rows='' else mclient -lsql -s"copy `cat $i | wc -l` records into usrtable from '$i' using delimiters '|';" fi done

Stefan

> ------------------------------------------------------------------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users