On Sat, Oct 02, 2010 at 01:20:50AM +0200, Christian Grün wrote:
Thanks for the quick feedback, Stefan,
In case of doubt, we recomment to use a 64-bit version of MonetDB (on a 64-bit system); cf., http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html
On this page, I found the information that "The default 64-bits MonetDB/XQuery binaries are built with 32-bits object identifiers (OIDs)". I noticed that the "--disable-oid32" isn't recognized by monetdb-install.sh; but this is probably negligible, as my Mserver instance shows me the following information:
monetdb-install.sh is to build from a source tar ball on unix-like systems; i.e., is not at all related to the binary builds we provide for Windows. The default is (and has always been) to build with 32-bit OIDs on 32-bit systems and with 64-it OIDs on 64-bit systems. Alternatively, there is an option to build with 32-bit OIDs on 64-bit systems to get a slightly smaller footprint / memory requirement --- at the expense of scalability: BATs (tables) can then contain at most 2^31 BUNs (tuples) rather than 2^63. You can enable that option by running ("plain") configure with "--enable-oid32" (see `monetdb-install.sh --devhelp` for expert help on monetdb-install.sh). We did once decide to build the 64-bit Windows installer with that option to cope with 64-bit Windows machines with (rather) smalle amounts of physical memory and their inherent vulnarability to address space fragmentation. We are currently considering to also provide the 64-bit Windows build with default 64-bit OIDs in the future.
# MonetDB Server v4.38.5 # based on GDK v1.38.5 # release Jun2010-SP2 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-2010, MonetDB B.V.. All rights reserved. # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs; dynamically linked.
…so I guess 64bit is now default?
As said above, not only now.
Next, I've now successfully created an 1GB XMark instance on a 64bit Linux machine with 32GB RAM; but when I try to shred an 11GB instance, I get the following output:
mclient -tlx pfadddoc.xq #GDKmmap(1336803328) fails, try to free up space [memory in use=33105304,virtual memory in use=36277911552] #GDKmmap(1336803328) result [mem=32625976,vm=35689922560] #GDKmmap: recovery ok. Continuing.. #GDKmmap(1336803328) fails, try to free up space [memory in use=32626608,virtual memory in use=36971282432] #GDKmmap(1336803328) result [mem=23481048,vm=36963221504] MAPI = monetdb@localhost:50000 QUERY = pf:add-doc("11gb.xml", "xmark") ERROR = !ERROR: HEAPalloc: Insufficient space for HEAP of 1336803328 bytes. !ERROR: CMDleftjoin: operation failed. !ERROR: interpret_params: reverse(param 1): evaluation error. Timer 720853.104 msec
Your Mserver is already using ~36 GB (virtual) address space --- I haven't recently loaded an 11 GB XMark doc., but that might be reasonable ---, and then (unexpectedly?) fails to allocate an other ~1.3 GB. How much free disk space it there on the partition that holds your dbfarm?
Do you have an idea how to tweak MonetDB, or my system, to parse documents of that size? Currently, no other processes are running on the machine, and space on hard disk should be enough as well.
What does "space on hard disk should be enough as well" mean? Is there (considerably) more than 36 GB free (on the partition that holds your dbfarm)? if so, please consider filing a bug report via http://bugs.monetdb.org/ Stefan
Thanks again, Christian
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |