Re: [MonetDB-users] Searching for possibilities to reduce the fluctuation in insertion times.

It is only natural if you overload the system with some other process, such as copying 40G with dd, other processes will become slower. It has nothing to do with MonetDB, your favorite web browser will also take 3 minutes to open (USB interface eats up CPU, and dd eats up the IO channel). Especially if you run a DBMS on a machine, which is already resource consuming, the system will slow done alot even with small unrelated processes. Thats is why a dedicate server is usually better. I don't think you will be able to find anything about the IO scheduler of Fedora. Actually this is part of the Linux Kernel and not configurable. The only suggestion that I can make (which I don't know if it works for IO), if you *must* copy 40gigs while you are shredding documents, is to run the copy command with "nice", that is: bash$ nice copy /path/a path/b lefteris On Mon, Feb 2, 2009 at 10:51 AM, John van Schie (DT) wrote:

Hi,

Your correct: with insertion I mean a pf:add-doc() and not updating an existing document. The document is shredded as read-only and there is ~170GB available on the disk (out of 400GB)

The system I/O is caused by copying a 40GB file from USB drive to disk (but dd if=/dev/zero gave the same result) and originated from a separate process.

I'll look into the OS I/O scheduling (for Fedora Core 10) and let you know if I find anything.

Regards,

John

Lefteris wrote:

...

Hi,

by insertion of the document, you are referring to shredding (aka. pf:add-doc) and not inserting (updating) with new elements on the already shredded document, right?

What kind of I/O is the system doing? You mean an I/O originated from some other process that just take up all the resources of your system? (like copying some hundreds of gigs data to the disk?) If that is the case, then the problem is just from the OS and the scheduling algorithms that uses for managing I/O requests. On the other hand, if this I/O is started because of shredding this small XML document, then that should not happen. Do you shredded as an updatable document? How full is your disk? It might be because of large fragmentation on the disk and not enough free space.

lefteris

On Mon, Feb 2, 2009 at 10:15 AM, John van Schie (DT) wrote:

...

Hi,

We use MonetDB v4.26.4 (Nov 2008-sp2) for our application with the XQuery frontend. Often, queries are executed very fast, but sometimes, execution of the same query takes significantly more time.

For example: We insert a XML document of 1.4Mb. with a maximum depth of 4. The document is structured as follows: <rootelement> <item> <child1> <sha1>sha1-value</sha1> <md5>md5-value</md5> </child1> </item> ... </rootelement> We have 8k 'item' elements.

Often the insertion of this document is performed in ~0.5s, which is fine. But when the system is performing IO (e.g. copying a large file), the insertion can take up to 6 minutes.

So I have two questions: 1) What is the cause of this large difference in execution times; 2) Is there a way to configure MonetDB (or the server) in such a way that the difference between the 'fast' and 'slow' is smaller.

Regards,

John van Schie

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users