On Wed, Jun 18, 2008 at 11:54:46AM +0200, Niels Nes wrote:
On Wed, Jun 18, 2008 at 10:47:18AM +0100, Sam Mason wrote:
The table now has 10 columns, 2 integer, 4 varchars and 4 dates. I'm trying to pull in about 38million rows (text file is about 3GB) into a 32bit build of monetdb. RAM and swap space are 2GB each, i.e. I've got 4GB in total. I must have mist your 32 bit build in your previous mail.
I'm not sure I thought of specifying it, every other database I've used hasn't been limited by address space size in this way :)
The 32 bit adress space will cause problems as that limits you to 2G databases. 38 * 10*4 (int) (already about 1.5G) + what ever string data. This may just not fit, leading to the GDKload etc error.
What does monet actually try to fit in memory at any one time? I guess it depends on the operation, but does it say try and mmap the entire source file when you're doing a COPY or does it stream that in? What about where it's being stored, is that mmapped in as well?
The solution is to switch to a 64 bit machine + build. So indeed the memory/swap isn't the limiting factor (other then speed) but the adress space is.
OK. It'll be a bit of a fiddle installing a 64bit OS on this machine so I'd prefer not to unless it's going to be a benefit. I've split the data up and it all imports OK now. I tried adding a primary key to the table and this causes the server to thrash, implying that it's not using particularly swap friendly algorithms. How much testing has this use case had? The dbfarm (what's the correct terminology?) takes up a couple of gig on disk if that helps. Sam