On Wed, Jun 18, 2008 at 10:47:18AM +0100, Sam Mason wrote:
On Tue, Jun 17, 2008 at 08:20:58PM +0200, Niels Nes wrote:
On Tue, Jun 17, 2008 at 07:06:24PM +0100, Sam Mason wrote:
COPY INTO source_livestock FROM '/mnt/scratch/smason/source_livestock' DELIMITERS ','; !SQLException:importTable:failed to import table !ERROR: GDKload: cannot mmap(): name=16/1634, ext=theap !OS: Cannot allocate memory !ERROR: GDKload failed: name=16/1634, ext=theap
Would you expect me to get this still? and hence should I provide more details or should I be building something else?
For this problem I indeed need more info. How much memory does your system have and how much swap. Also what are the sizes of the to be loaded table (nr of rows, columns, data types) ?
Sorry, it's almost the same as my original post. The table now has 10 columns, 2 integer, 4 varchars and 4 dates. I'm trying to pull in about 38million rows (text file is about 3GB) into a 32bit build of monetdb. RAM and swap space are 2GB each, i.e. I've got 4GB in total. I must have mist your 32 bit build in your previous mail. The 32 bit adress space will cause problems as that limits you to 2G databases. 38 * 10*4 (int) (already about 1.5G) + what ever string data. This may just not fit, leading to the GDKload etc error.
The solution is to switch to a 64 bit machine + build. So indeed the memory/swap isn't the limiting factor (other then speed) but the adress space is. Niels
Why would ram/swap space matter for anything but performance? If they do, I'm assuming monet wouldn't work with files larger than the memory addressable by a single process? This file is going into one of the smaller tables that I want to work with. If this is going to be a problem how much of a change is it going to be to the code to support out-of-core style operations.
Sam
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl