Hi Christian, quick and simple answer, on a 32-bit Windows system, you are running out of address space, which is in practice limited to 2GB or at most 3GB if you configured your system accordingly (cf., http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html http://zone.ni.com/reference/en-XX/help/371361D-01/lvhowto/enable_lrg_ad_awa... ), and know to be very vulnarable to address space fragmentation. I haven't tried it recently, but you might be able to load the 1 GB XMark document on a 32-bit Unix-like system. A 64-bit build of MonetDB/XQuery (on a 64-bit system) should not have any problems loading a 1 GB XMark document. Stefan On Fri, Oct 01, 2010 at 07:13:53PM +0200, Christian Grün wrote:
Hi everyone,
is it still possible to shred large XML instances with MonetDB/XQuery, or do I need more memory for shredding? I was just trying to the 1GB instance of XMark with WinXP 32Bit and 2GB RAM, and this is what I got:
MonetDB>shred_doc("1gb.xml", "xmark"); #GDKmmap(536870912) fails, try to free up space [memory in use=36143780,virtual memory in use=1015480320] #GDKmmap(536870912) result [mem=26016940,vm=1005387776] !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0@0. !ERROR: [shred_url]: first error was: !ERROR: HEAPextend: failed to extend to 536870912 for 06\632theap !ERROR: shredBAT_append_str: APPEND-STR[_prop_text]( follow murders prov shoes p resent eager delivered defy bend viol here marquis powers complexion pillar cran ts agate ), BUNappend fails !ERROR: CMDshred_url: operation failed.
pf:add-doc(...) produces the same output; the XML document itself is well-formed.
Thanks, Christian
------------------------------------------------------------------------------ Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |