That's great, would you mind illustrating how to  deal with 
(1) having mixture of text and .gz files in the same directory
(2) dealing with error (mostly the rare encounter of "non-standard" columns)

both (1) & (2) are currently taken care of in R (for (2), i just ignore the error message and press on), but I'm not sure how to deal with them in shell script. hints also welcome. Just need pointers on where to start.

thanks.

On Mon, Mar 16, 2009 at 5:25 PM, Stefan Manegold <Stefan.Manegold@cwi.nl> wrote:
On Mon, Mar 16, 2009 at 04:56:31PM -0700, Yue Sheng wrote:
> no i/o, swapping, no activity in CPU monitor
> 32GB
>
> haven't tried. manual loading >600 files then see when it break is an
> option, albert a painful one...

assiming your files are in .../DIR/*.csv:

rows="`cat .../DIR/*.csv | wc -l`"
for i in .../DIR/*.csv ; do
       if [ "$rows" ] ; then
               mclient -lsql -s"copy $ROWS records into usrtable from '$i' using delimiters '|';"
               rows=''
       else
               mclient -lsql -s"copy `cat $i | wc -l` records into usrtable from '$i' using delimiters '|';"
       fi
done

Stefan

> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users


--
| Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users