On Fri, Jan 23, 2009 at 04:52:54PM +0100, Jens Oehlschlägel wrote:
Thanks Stefan,
with loading ~1.5 GB on a 32 bit machine, there is a reasonable chance that the loading process (that needs to be able to address all data concurrently) comes close to or even hits the 32-bit (i.e., max. 4 GB, possibly only 2 GB) address space limit.
Does this limitation refers to the size of the data that I am adding per COPY INTO (was 0.2 GB) or the final size of the table (was at 1.9). Assuming the latter this would give a hard limit for table size under Win32.
Indeed, the limitation refers to the final table size, as all COPY INTO queries add data to the same table --- and the limitation of course holds for all 32-bit systems, not only for Win32 (although Win32 often appear to be particularly sensitive to memory / address space fragmentation) ...
If this is the case, a workaround could be partioning the data into several tables. What is more efficient in MonetDB: chunks of rows at full columns width or chunks of columns at full row length?
Indeed, partitioning the data into multiple tables would help with the loading, and there is should not matter (much) whether you partition horizontally or vertically. Of course, for querying the data, it depends on your query load, which partitioning is more efficient. Basically, the partitioning that requires the least the partitions per query would be the most efficient one. But also the way that your queries re-combine the partition that are required --- we don't have generic automatic partitining *yet*, hence, when partitioning your table, you also need to "partition" your queries accordingly --- has an impact on the performance. It is hard to make a general statement, here. We might be able to give more advice for more concrete cases, i.e., if your could provide us with some more details about your typical / anticipated query load, we could have a look at that.
Would it be possible that you could provide us with your data? That would help us a lot to analyze, locate and hopefully fix the crash...
I give you access by seperate mail, you can also create the test data file using the R script snippet in my previous mail.
Ok, I dind't realize that. Thanks. We'll try to have a closer look at the problem next week. Have a nice weekend! Stefan
Have a nice weekend
Jens
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |