Nozhup wrote:
Since the FEB2010+SP1 release we're encountering loading issues.
With the FEB2010 release we were able to load records until the HDD was full (500+ million records) on PCs with 2,3 or 24 GB ram memory without going to the SWAP. This was done by loading CSV-files with 1 million records each through the COPY INTO command (in the mclient and through a c-script).
Since the FEB2010+SP1 release the system starts to SWAP after loading the first couple of million records. This results in increased loadtimes per loaded CSV-file:
1th million: 14 sec 2nd million: 26 sec 3rd million: 44 sec 4th million: 50 sec 5th million: 69 sec etc.
We have tested this on several PCs and all PCs encounter the same problem. Test files used are the same for both releases as is the OS (Ubuntu 9.10). The only variable that has changed is MonetDB.
To analyse the situation, the schema should be known, in particular if there are any constraints defined over it. This problem could not been reproduced on the ontime benchmark database (120Mrecords in batches of 0.5M) Investigation still going... Martin