Hi Stefan I just talked to our admins and they told me that zone reclaim already set to 0 and the query is not the first or so. we run thousands of queries per hour and just one or two take more than 0.2 seconds (up to 70 seconds by test from today). We have a database of around 10gb and about 40 tables. The servers have 32gb ram, 180 gb ssd hard drives and 24 cpus. So this should be ok. We are running a single query per client. Client is a php script over apache server making the requests. Changing from normal connection to pconnection didn't make any difference. During the hick up we had no other process running except apache and monetdb and both didn't use to much cpu and the system also didn't use to much cpu. Any other suggestion would be appreciated. Thanks sebastian Am 10.12.13 23:06 schrieb "Stefan Manegold" unter :

Hi Sebastian,

given your brief description of the problem, I don't expect that MonetDB itself triggers/causes the problem, but rather suspect that it is related to the system you're running on.

If I understand you correctly, your machine has ample CPU and RAM resources to execute your queries, right?

Are you running a single query / client at a time, or are multiple queries / clients active concurrently?

Can / did you monitor the system behavior/activity during the "hick-ups"? E.g., are there other ("heavy") processes running? Is there (unexpected) I/O activity? Is there unexpectedly high CPU system (rather than user) load?

Please also see the experiences shared at in this blog post: http://engineering.linkedin.com/performance/optimizing-linux-memory-manage ment-low-latency-high-throughput-databases and this monetdb user mailing list thread: https://www.monetdb.org/pipermail/users-list/2013-December/thread.html#702 4

Best, Stefan

----- Original Message -----

...

HI,

We are using monetdb for some statistical analysis.

The queries are very fast (0.2 sec), but some times (not regularly) the queries get peeks of up to 32 sec for the same query.

CPU and RAM don't change and are way higher than actually needed.

Does any one have this same situation? Any idea what could be doing this?

Best regards and thanks to any help.

sebastian

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

Hi, Thanks for your suggestion. Unfortunately this seems not possible with monetdb and we are getting out of options. :( Best sebastian Am 11.12.13 11:43 schrieb "Sjoerd Mullender" unter :

On 2013-12-11 11:14, Sebastian Kielmann wrote:

...

Hi Stefan

I just talked to our admins and they told me that zone reclaim already set to 0 and the query is not the first or so. we run thousands of queries per hour and just one or two take more than 0.2 seconds (up to 70 seconds by test from today).

We have a database of around 10gb and about 40 tables. The servers have 32gb ram, 180 gb ssd hard drives and 24 cpus. So this should be ok.

We are running a single query per client. Client is a php script over apache server making the requests. Changing from normal connection to pconnection didn't make any difference.

During the hick up we had no other process running except apache and monetdb and both didn't use to much cpu and the system also didn't use to much cpu.

Any other suggestion would be appreciated.

Here's a suggestion: run your server with the added argument: --set gdk_vm_trim=no I'm not sure whether this can be done using the monetdb program, so you may have to run the server "by hand" to try this. You can extract the exact invocation of the server from the log file.

gdk_vm_trim is default "yes" and then enables a thread that every once in a while sees whether it needs to free up memory. When it does, it may remove from memory (not from disk) any BATs that are currently unused, and also destroy their hash tables. Those hash tables get created automatically when needed, but creating them can take a significant amount of time.

...

Thanks

sebastian

Am 10.12.13 23:06 schrieb "Stefan Manegold" unter :

...

Hi Sebastian,

given your brief description of the problem, I don't expect that MonetDB itself triggers/causes the problem, but rather suspect that it is related to the system you're running on.

If I understand you correctly, your machine has ample CPU and RAM resources to execute your queries, right?

Are you running a single query / client at a time, or are multiple queries / clients active concurrently?

Can / did you monitor the system behavior/activity during the "hick-ups"? E.g., are there other ("heavy") processes running? Is there (unexpected) I/O activity? Is there unexpectedly high CPU system (rather than user) load?

Please also see the experiences shared at in this blog post:

http://engineering.linkedin.com/performance/optimizing-linux-memory-mana ge ment-low-latency-high-throughput-databases and this monetdb user mailing list thread:

https://www.monetdb.org/pipermail/users-list/2013-December/thread.html#7 02 4

Best, Stefan

----- Original Message -----

...

HI,

We are using monetdb for some statistical analysis.

The queries are very fast (0.2 sec), but some times (not regularly) the queries get peeks of up to 32 sec for the same query.

CPU and RAM don't change and are way higher than actually needed.

Does any one have this same situation? Any idea what could be doing this?

Best regards and thanks to any help.

sebastian

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

-- Sjoerd Mullender