Re: [Monetdb-developers] Large XML instances with MonetDB/XQuery 4.38.5

On Fri, Oct 01, 2010 at 09:35:26PM +0200, Christian Grün wrote:

To continue… Do you have some information on the current theoretical limits for storing XML instances in MonetDB/XQuery?

The theoretical and pratical limit are the available address space and free disk space on your system, as well as the vulnerability of your OS to address space fragmentation. In practice, not only the plain serialized document size is a factor, but also the number of nodes in your document(s). In case of doubt, we recomment to use a 64-bit version of MonetDB (on a 64-bit system); cf., http://monetdb.cwi.nl/XQuery/Documentation/Scalability.html Stefan

Thanks, Christian

On Fri, Oct 1, 2010 at 7:13 PM, Christian Grün wrote:

...

Hi everyone,

is it still possible to shred large XML instances with MonetDB/XQuery, or do I need more memory for shredding? I was just trying to the 1GB instance of XMark with WinXP 32Bit and 2GB RAM, and this is what I got:

MonetDB>shred_doc("1gb.xml", "xmark"); #GDKmmap(536870912) fails, try to free up space [memory in use=36143780,virtual memory in use=1015480320] #GDKmmap(536870912) result [mem=26016940,vm=1005387776] !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0@0. !ERROR: [shred_url]: first error was: !ERROR: HEAPextend: failed to extend to 536870912 for 06\632theap !ERROR: shredBAT_append_str: APPEND-STR[_prop_text]( follow murders prov shoes p resent eager delivered defy bend viol here marquis powers complexion pillar cran ts agate ), BUNappend fails !ERROR: CMDshred_url: operation failed.

pf:add-doc(...) produces the same output; the XML document itself is well-formed.

Thanks, Christian

------------------------------------------------------------------------------ Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |