PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE

3634 top 3.1% 0:02.37 1 18 29 1048K 188K 1640K 18M

3625 bash 0.0% 0:00.00 1 14 19 308K 632K 944K 18M

3624 login 0.0% 0:00.01 1 17 55 308K 268K 1080K 19M

3613 mdworker 0.0% 0:00.11 3 66 35 620K 2188K 2136K 31M

3612 SecurityAg 0.0% 0:00.56 5 113 166 1396K 13M 6252K 610M

3611 authorizat 0.0% 0:00.01 1 31 39 476K 1152K 1560K 29M

3610 mdworker 0.0% 0:00.12 3 67 43 272K 2540K 2272K 30M

2457 mclient 0.0% 0:00.00 1 14 35 268K 424K 804K 21M

2105 AppleSpell 0.0% 0:00.01 1 22 35 404K 2840K 1908K 34M

2013 JavaApplic 1.5% 11:11.84 20 868 470 78M 18M 97M 894M

1295 pmTool 0.0% 3:50.93 1 24 29 440K 1128K 1332K 27M

1291 Activity M 0.3% 6:01.13 11 137 326 11M 22M 24M+ 670M

1204 Safari 2.6% 62:04.13 19 185 2026 243M 14M 161M 997M

114 R 0.2% 2:16.16 18 196+ 321 13M+ 42M 50M+ 671M+

113 mserver5 0.0% 30:22.73 14 47 614 18G 428K 2224M 44G

102 bash 0.0% 0:00.00 1 14 20 280K 632K 940K 18M

101 login 0.0% 0:00.01 1 17 57 268K 268K 1080K 19M

100 Terminal 1.4% 0:01.22 3 99- 153 2368K 7688K 9796K 617M

89 Finder 0.0% 0:06.15 7 157+ 224 1952K+ 12M 13M+ 625M+

88 SystemUISe 0.0% 0:01.09 14 298 345 2484K 6856K 9424K 615M

87 coreaudiod 0.0% 0:00.05 2 139 57 536K 340K 1728K 20M

86 pboard 0.0% 0:00.00 1 15 27 116K 184K 588K 19M

85 ATSServer 0.0% 0:00.72 2 99 131 976K 5644K 4552K 73M

84 Dock 0.0% 0:00.64 3 126 288 2052K 42M 11M 648M

83 UserEventA 0.0% 0:00.13 2 158 118 364K 1724K 2240K 533M

82 Spotlight 0.0% 0:00.10 8 86 102 768K 2960K 4124K 551M

78 AirPort Ba 0.0% 0:00.05 1 59 99 604K 3268K 2944K 579M

62 launchd 0.0% 0:00.35 3 153 26 192K 296K 524K 18M

50 WindowServ 1.3% 3:37.31 11 249 1122 16M 77M 102M+ 743M

45 blued 0.0% 0:00.20 1 91 45 488K 1624K 2156K 28M

44 coreservic 0.0% 0:00.62 2 140 101 1360K 17M 11M 29M

41 distnoted 0.0% 0:00.45 1 49 22 236K 184K 736K 18M

39 socketfilt 0.0% 0:00.46 3 35 29 568K 196K 1548K 18M

37 autofsd 0.0% 0:00.01 1 21 22 104K 184K 668K 18M

34 configd 0.0% 0:00.68 3 177 79 252K 304K 1892K 19M

33 CoreRAIDSe 0.0% 2:53.29 2 65 43 260K 1132K 2216K 30M

32 DirectoryS 0.0% 0:03.88 5 66 41 1560K 188K 3300K 21M

31 diskarbitr 0.0% 0:00.23 1 112 20 328K 184K 964K 18M

iostat:

disk0 cpu load average

KB/t tps MB/s us sy id 1m 5m 15m

514.68 16 8.10 2 1 97 0.10 0.07 0.12

ps:

PID TTY TIME CMD

102 ttys000 0:00.01 -bash

113 ttys000 30:22.74 ./mserver5 --dbname=usrdb --dbinit include sql;

3625 ttys001 0:00.01 -bash

Thanks.

Good to know that Monet can handle 12B rows. Now just need to work out why it keeps failing when it reaches a certain size...

On Mon, Mar 16, 2009 at 10:57 PM, Martin Kersten <Martin.Kersten@cwi.nl> wrote:

Yue Sheng wrote:
> Table currently has around 500 million records. Now it hangs even if I
> try to run a single copy in command line:
>
> ./mclient -lsql -s "copy 3000000 records into... "
>
> no activity in CPU monitor, Vsize constant at 40.25GB, R Size at 759MB.
> no error from server.

does 'top' tell mserver is not consuming cpu? or just minor?
does 'iostat' is telling there is no io?
does 'ps' tell that the OS has put the process on the lowest priority?

>
> i can't even kill the server.

victimized by the OS, swapping out all other processes, which
then have to be brought back in.

>
> this is very frustrating.
>
> could someone please tell me:
>
> (1) if the DB can handle rows that are in the billions?

the largest database running at our site has a table of 12B rows 6 cols
and 400M rows 500 columns.
Loaded using a schema without (referential) constraints, which
are added after the fact using an ALTER statement.

> (2) if the config file needs to be tweaked?

no

>
> thanks.
>
> On Mon, Mar 16, 2009 at 5:25 PM, Stefan Manegold <Stefan.Manegold@cwi.nl

> <mailto:Stefan.Manegold@cwi.nl>> wrote:
>
> On Mon, Mar 16, 2009 at 04:56:31PM -0700, Yue Sheng wrote:
> > no i/o, swapping, no activity in CPU monitor
> > 32GB
> >
> > haven't tried. manual loading >600 files then see when it break is an
> > option, albert a painful one...
>
> assiming your files are in .../DIR/*.csv:
>
> rows="`cat .../DIR/*.csv | wc -l`"
> for i in .../DIR/*.csv ; do
> if [ "$rows" ] ; then
> mclient -lsql -s"copy $ROWS records into usrtable
> from '$i' using delimiters '|';"
> rows=''
> else
> mclient -lsql -s"copy `cat $i | wc -l` records into
> usrtable from '$i' using delimiters '|';"
> fi
> done
>
> Stefan
>
> >
> ------------------------------------------------------------------------------
> > Apps built with the Adobe(R) Flex(R) framework and Flex
> Builder(TM) are
> > powering Web 2.0 with engaging, cross-platform capabilities.
> Quickly and
> > easily build your RIAs with Flex Builder, the Eclipse(TM)based
> development
> > software that enables intelligent coding and step-through debugging.
> > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> > _______________________________________________
> > MonetDB-users mailing list
> > MonetDB-users@lists.sourceforge.net

> <mailto:MonetDB-users@lists.sourceforge.net>

> > https://lists.sourceforge.net/lists/listinfo/monetdb-users
>
>
> --
> | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl
> <mailto:Stefan.Manegold@cwi.nl> |
> | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
> | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
> | The Netherlands | Fax : +31 (20) 592-4312 |
>
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based
> development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users@lists.sourceforge.net

> <mailto:MonetDB-users@lists.sourceforge.net>

> https://lists.sourceforge.net/lists/listinfo/monetdb-users
>
>
>

> ------------------------------------------------------------------------

>
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users