Re: [MonetDB-users] COPY TO corrupting data
Stefan Manegold wrote:
32-bit system on 8 GB machines does not make much sense, does it? A 32-bit systems can (at least per process) address at most 4 GB, usually only 2 GB in practice...
True, but doesn't seem like I am hitting the maximum process memory limit (not even approaching 1.5GB).
With 9M+ records of >= 438 byte each, your data is >=3.9 GB in size. I'm not completely sure whether MonetDB temprarily need to hold all data concurrently in its address sapce during bulk loading; but if so, your data size might simply exceed the 32-bit address space. (Obviously, MonetDB should better give an error than crash ...)
I am bulk loading only 250K rows at a time, at which point they are copied from staging table into permament final table. am I allowed to have a table with total data size larger than max process memory size?
Did you try on a 64-bit systems using 64-bit MonetDB?
I have, but hitting another - higher limit - 100M rows ending with corruption on 64 bits, see first message on this thread.
participants (1)
-
mobigital-nbl@yahoo.com