Hi Erwin, as my experiments on Linux showed, shredding 1 GB XML file with MonetDB/XQuery should not require more than 1 GB of (virtual) memory, however, in the process, MonetDB/XQuery requires (at least once) to be able to allocate a consecutive memory area of 436731904 bytes (i.e., ~417 MB). In particular this means, that a *consecutive* area of ~417 MB needs to be free in the very process' address space. While this seems to work fine on Linux, it does not seem to work on Windows. The reasons are (as far as I know) twofold, (1) 32-bit Windows only provides an address space of only 2 GB, while Linux usually provides 3 GB (theoretically, 32-bit systems could allow up to 4 GB); hence, even if the physical memory or Windows page file is large than 2 GB, each process can at most allocate up to 2 GB. (2) Windows seems to be more vulnerable to address space fragmentation than Linux, i.e., while there might still be more than 417 MB free in total in the process' address space, none of the *consecutive* free blocks is large enough to accommodate the requested 417 MB. Lacking a Windows machine, I unfortunately cannot verify this practically. Could any "Windows expert" help? The document in question is available from http://www.cwi.nl/~manegold/Public/DC1000catalog.xml.zip Concerning MonetDB/XQuery, I'll try to analyze, why/whether we indeed need to allocate that 417 MB block while shredding this 1 GB document, but I guess we cannot avoid this, because this might be required by MonetDB processing model. Hence, there's quite a chance that MonetDB/XQuery represents quite a challenge to the above mentioned address space limitations of 32-bit Windows systems. The short-term solutions (unless we unexpectedly find out that this is caused by a bug in MonetDB/XQuery) are to either use a different operating system (e.g., Linux) or "at least" to use a 64-bit Windows system. My apologies for the inconvenience! Stefan On Fri, Sep 08, 2006 at 10:11:06AM +0800, Erwin Leonardi wrote:
Hi there,
I tried to shred 1GB XML data into MonetDB. However, I faced the same problem as Klarinda. The error is as follows.
-------------- Error --------------- ERROR = !ERROR: MT_mmap: MapViewOfFile(68c, 2, 0, 0, 436731904, 0) failed !OS: Not enough space !GDKmmap(436731904) fail => BBPtrim(enter) usage[mem=212972396,vm=803209216] ---------- End Error ---------------
I use a PC with 1GB RAM, more than 5GB free hard disk space, and WinXP SP2 as OS. I have tried to set Windows' page file to 2GB. I read the previous postings regarding the same problem. I notice that Stefan has done similar experiment using his laptop (Pentium M (Dothan) 1.5 GHz, 2 GB RAM, 2.1 GB free disk space; running Fedora Core 5 Linux (Kernel 2.6.17)).
My question is whether I need more than 1GB RAM if I want to shred 1GB XML data.
Thanks Erwin Leonardi
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |