Re: [MonetDB-users] Searching for possibilities to reduce the fluctuation in insertion times.

Hi, Your correct: with insertion I mean a pf:add-doc() and not updating an existing document. The document is shredded as read-only and there is ~170GB available on the disk (out of 400GB) The system I/O is caused by copying a 40GB file from USB drive to disk (but dd if=/dev/zero gave the same result) and originated from a separate process. I'll look into the OS I/O scheduling (for Fedora Core 10) and let you know if I find anything. Regards, John Lefteris wrote:

Hi,

by insertion of the document, you are referring to shredding (aka. pf:add-doc) and not inserting (updating) with new elements on the already shredded document, right?

What kind of I/O is the system doing? You mean an I/O originated from some other process that just take up all the resources of your system? (like copying some hundreds of gigs data to the disk?) If that is the case, then the problem is just from the OS and the scheduling algorithms that uses for managing I/O requests. On the other hand, if this I/O is started because of shredding this small XML document, then that should not happen. Do you shredded as an updatable document? How full is your disk? It might be because of large fragmentation on the disk and not enough free space.

lefteris

On Mon, Feb 2, 2009 at 10:15 AM, John van Schie (DT) wrote:

...

Hi,

We use MonetDB v4.26.4 (Nov 2008-sp2) for our application with the XQuery frontend. Often, queries are executed very fast, but sometimes, execution of the same query takes significantly more time.

For example: We insert a XML document of 1.4Mb. with a maximum depth of 4. The document is structured as follows: <rootelement> <item> <child1> <sha1>sha1-value</sha1> <md5>md5-value</md5> </child1> </item> ... </rootelement> We have 8k 'item' elements.

Often the insertion of this document is performed in ~0.5s, which is fine. But when the system is performing IO (e.g. copying a large file), the insertion can take up to 6 minutes.

So I have two questions: 1) What is the cause of this large difference in execution times; 2) Is there a way to configure MonetDB (or the server) in such a way that the difference between the 'fast' and 'slow' is smaller.

Regards,

John van Schie

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users