[MonetDB-users] copy/select * from /into - very slow
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.
On Mon, Feb 09, 2009 at 12:30:34PM -0500, dariuszs wrote:
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz. Did MonetDB run out of memory, ie do you see any swap activity?
Niels
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Hi, Not at all - I've got 48GB of memory and the memory goes up slowly - right now about 9GB, cpu usage is about 6%, page file usage is about 2.6GB. When I select let's say 50 fields memory goes up quickly and cpu usage goes to 100 in no time. Dariusz. Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:30:34PM -0500, dariuszs wrote:
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.
Did MonetDB run out of memory, ie do you see any swap activity?
Niels
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Feb 09, 2009 at 12:56:56PM -0500, dariuszs wrote:
Hi, Not at all - I've got 48GB of memory and the memory goes up slowly - right now about 9GB, cpu usage is about 6%, page file usage is about 2.6GB. When I select let's say 50 fields memory goes up quickly and cpu usage goes to 100 in no time. Dariusz. strange. Do you see io activity (iostat or iotop)?
Niels
Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:30:34PM -0500, dariuszs wrote:
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.
Did MonetDB run out of memory, ie do you see any swap activity?
Niels
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Hi, I'm running this on Windows x64 and yes there is activity it's just writing out like 5MB/s. Dariusz. Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:56:56PM -0500, dariuszs wrote:
Hi, Not at all - I've got 48GB of memory and the memory goes up slowly - right now about 9GB, cpu usage is about 6%, page file usage is about 2.6GB. When I select let's say 50 fields memory goes up quickly and cpu usage goes to 100 in no time. Dariusz.
strange. Do you see io activity (iostat or iotop)?
Niels
Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:30:34PM -0500, dariuszs wrote:
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.
Did MonetDB run out of memory, ie do you see any swap activity?
Niels
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Feb 09, 2009 at 01:29:04PM -0500, dariuszs wrote:
Hi, I'm running this on Windows x64 and yes there is activity it's just writing out like 5MB/s. Dariusz.
Windows isn't exactly our day to day platform. But clearly we should do some heavy debugging/testing on Windows. Maybe I can get some tests running tomorrow. Niels
Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:56:56PM -0500, dariuszs wrote:
Hi, Not at all - I've got 48GB of memory and the memory goes up slowly - right now about 9GB, cpu usage is about 6%, page file usage is about 2.6GB. When I select let's say 50 fields memory goes up quickly and cpu usage goes to 100 in no time. Dariusz.
strange. Do you see io activity (iostat or iotop)?
Niels
Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:30:34PM -0500, dariuszs wrote:
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.
Did MonetDB run out of memory, ie do you see any swap activity?
Niels
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Hi, Here is the log of processing just 10 records for all the fields - I'm a newbie to this software but what this looks like is that for every record the software is scanning all columns and then joins them? Please take a look. Thanks. Dariusz. Niels Nes wrote:
On Mon, Feb 09, 2009 at 01:29:04PM -0500, dariuszs wrote:
Hi, I'm running this on Windows x64 and yes there is activity it's just writing out like 5MB/s. Dariusz.
Windows isn't exactly our day to day platform. But clearly we should do some heavy debugging/testing on Windows. Maybe I can get some tests running tomorrow.
Niels
Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:56:56PM -0500, dariuszs wrote:
Hi, Not at all - I've got 48GB of memory and the memory goes up slowly - right now about 9GB, cpu usage is about 6%, page file usage is about 2.6GB. When I select let's say 50 fields memory goes up quickly and cpu usage goes to 100 in no time. Dariusz.
strange. Do you see io activity (iostat or iotop)?
Niels
Niels Nes wrote:
On Mon, Feb 09, 2009 at 12:30:34PM -0500, dariuszs wrote:
Hi, Could you please help me here. I've got 300 fields, 100 million records, 800 bytes per row in a table. Exporting 60 fields into a table takes about 1 minute and selecting * from table takes very long time, and I don't see lots of cpu usage when exporting all records. Thanks. Dariusz.
Did MonetDB run out of memory, ie do you see any swap activity?
Niels
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
:0[0]:W
Strace copy select * from trw100 limit 10 into 'd:/x1.txt';
:3297[0]:R
[ 0 usec mdb.setTimer(true); ]
[ 32000 usec _1 := sql.bind("sys","trw100","f1",0); ]
[ 15000 usec _6 := sql.bind("sys","trw100","f1",1); ]
[ 16000 usec _8 := algebra.kunion(