Having recompiled with 32bit OID's I was able to load the 6 million records in 4h23m. Resource usage on the VM IS - 7.5GB VIRT (I always think this is virtual memory, but it's virtual address space correct?) - 840MB RES System memory - PHYSICAL - 1.2GB used - VIRTUAL - 190MB I imagine the reason why the load was taking so long is due to the VM using 190MB virtual memory, which will kill performance. Still intrigued by the 64bit OID speed issue. But I guess double the VM usage space can result in exponential performance degradation. Cronje Cronje Fourie wrote:
Not sure what you mean with "Total VM usage or just mserver5 process"
What I can tell you today is the following: - Created a VMWare workstation system under KUbuntu 64bit 8.04 - latest updates - VM Has - 1Processor, 2GB memory, 30GB disk space - OS = Ubuntu Server 64bit 8.04 with latest updates - Compiled monet from Jun Super Tar with --enable-optimise --enable-sql --nightly-stable - Created database with exact same scripts as used for 32bit systems - started copy into process using script... <code> copy 6157114 records into PrimalPool from '/home/cfourie/temp/PrimalPool.csv' delimiters '|', '\n' null as '' GO --score copy 12240201 records into score from '/home/cfourie/temp/score.csv' delimiters '|', '\n' null as '' GO --mailpot copy 10078100 records into mailpot from '/home/cfourie/temp/mailpot.csv' delimiters '|', '\n' null as '' GO </code> - Got back to the office this morning. The first copy of 6mil records was still running, mserver5 consuming 19GB VM according to top
I'm stumped :)
Next test - - keep existing ubuntu server install within VM - recompile monet same as before BUT with 32bitOID (2 billion record limit, I think we'll be ok for now :) - recreate DB same as before - attempt to load data again
As a point of interest I'll enable memory debug and major IO mask. Maybe that gives me a pointer.
Also related to this...Is there a known limit of the number of columns in a monetdb table? Could this be an issue? That will be my next step ;)
-- View this message in context: http://www.nabble.com/COPY-INTO-on-Ubuntu-64bit-is-slow-tp18833879p18887127.... Sent from the monetdb-users mailing list archive at Nabble.com.