Oops.. My bad.. Thanks Brian.. Yes I was indeed looking for connection pools... Something like maximum thread concurrency option in innodb... Is there any such available with MonetDB? I had gone through this option... mserver5 --set max_clients=<num> Is this the only way to control parallel threads?? Thanks & Regards, Vijayakrishna.P. Mobile : (+91) 9500402305. On Mon, Aug 10, 2015 at 7:48 PM, Brian Hood wrote:

Hi Vijay,

I think the words your looking for a connection pooling.

InnoDB Buffer pool in MySQL is the amount of data stored in memory which you set to a certain size and has nothing to do with network traffic.

Regards,

Brian Hood

On Mon, Aug 10, 2015 at 8:45 AM, Vijay Krishna wrote:

...

Hi,

Does MonetDB have any built in buffer pools to handle parallel requests?

I see the default number of parallel connections accepted and processed at a time by MonetDB is 64.. While I see that there are ways to increase it, is there anyway built in to pool incoming requests until the live 64 requests are processed?

For example, our application would receive 100 requests at a time, while we wish to limit the number of queries handled at a time by MonetDB to 50. Is there any way to automatically handle the remaining 50 (something like innodb buffer pool) ??

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305.

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Hi, indeed; see also `man mserver5` and `man monetdb` or https://www.monetdb.org/Documentation/mserver5-man-page and https://www.monetdb.org/Documentation/monetdb-man-page Stefan ----- On Aug 11, 2015, at 3:11 PM, Brian Hood brianh6854@googlemail.com wrote:

Hi Vijay,

Well if you take a look at your merovingian.log you can see the command that starts the database.

/usr/bin/mserver5 --dbpath=/data2/threatmonitor/threatmonitor --set merovingian_uri=mapi:monetdb://orville.lan:50000/threatmonitor --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data2/threatmonitor/threatmonitor/.mapi.sock --set monet_vault_key=/data2/threatmonitor/threatmonitor/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes

These 2 i think are the ones your looking for.

--set gdk_nr_threads=2 --set max_clients=64

Regards,

Brian Hood

On Tue, Aug 11, 2015 at 7:14 AM, Vijay Krishna < vijayakrishna55@gmail.com > wrote:

Oops.. My bad.. Thanks Brian..

Yes I was indeed looking for connection pools... Something like maximum thread concurrency option in innodb...

Is there any such available with MonetDB?

I had gone through this option... mserver5 --set max_clients=<num> Is this the only way to control parallel threads??

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305 .

On Mon, Aug 10, 2015 at 7:48 PM, Brian Hood < brianh6854@googlemail.com > wrote:

Hi Vijay,

I think the words your looking for a connection pooling.

InnoDB Buffer pool in MySQL is the amount of data stored in memory which you set to a certain size and has nothing to do with network traffic.

Regards,

Brian Hood

On Mon, Aug 10, 2015 at 8:45 AM, Vijay Krishna < vijayakrishna55@gmail.com > wrote:

Hi,

Does MonetDB have any built in buffer pools to handle parallel requests?

I see the default number of parallel connections accepted and processed at a time by MonetDB is 64.. While I see that there are ways to increase it, is there anyway built in to pool incoming requests until the live 64 requests are processed?

For example, our application would receive 100 requests at a time, while we wish to limit the number of queries handled at a time by MonetDB to 50. Is there any way to automatically handle the remaining 50 (something like innodb buffer pool) ??

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305 .

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

Hi Vijay, I have found some interesting info in the monetdb man page actually. --- This is interests me as it alllows you to limit the amount of clients that can connect to a database not just an mserver *nclients=**number* Sets the maximum amount of clients that can connect to this database at the same time. Setting this to a high value is discouraged. A multiplex-funnel may be more performant, see *MULTIPLEX-FUNNEL* below. Regards, Brian Hood On Wed, Aug 12, 2015 at 7:48 AM, Vijay Krishna wrote:

Thanks Stefan and Brian.

When I went through monetdb man page, the usage of nthreads (set as gdk_nr_threads) environmental variable was defined as

...

...

Defines how many worker threads the server should use to perform main processing. Normally, this number equals the number of available CPU cores in the system. Reducing this number forces the server to use less parallelism when executing queries, or none at all if set to *1*. I am a bit confused in understanding this statement. The variable "nthreads" controls

1. the number of threads processing a single query using multiple cores. 2. the number of queries currently being executed over all databases by the mserver.

Please help me understand which of these two is correct.. This doubt arises because I have read from MonetDB home page that "MonetDB is designed for multi-core parallel execution on desktops to reduce response time for complex query processing".

If the variable "nthreads" controls the latter, then does monetdb have an in built queue to store the queries from remaining connections (controlled by the variable nclients) to the mserver?? I have read about multiplex funnel and a bit afraid over the transactional disadvantages mentioned in there.. Is there any other solution?

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305.

On Tue, Aug 11, 2015 at 6:56 PM, Stefan Manegold wrote:

...

Hi,

indeed;

see also `man mserver5` and `man monetdb` or https://www.monetdb.org/Documentation/mserver5-man-page and https://www.monetdb.org/Documentation/monetdb-man-page

Stefan

----- On Aug 11, 2015, at 3:11 PM, Brian Hood brianh6854@googlemail.com wrote:

...

Hi Vijay,

Well if you take a look at your merovingian.log you can see the command that starts the database.

/usr/bin/mserver5 --dbpath=/data2/threatmonitor/threatmonitor --set merovingian_uri=mapi:monetdb://orville.lan:50000/threatmonitor --set mapi_open=false --set mapi_port=0 --set mapi_usock=/data2/threatmonitor/threatmonitor/.mapi.sock --set monet_vault_key=/data2/threatmonitor/threatmonitor/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes

These 2 i think are the ones your looking for.

--set gdk_nr_threads=2 --set max_clients=64

Regards,

Brian Hood

On Tue, Aug 11, 2015 at 7:14 AM, Vijay Krishna < vijayakrishna55@gmail.com > wrote:

Oops.. My bad.. Thanks Brian..

Yes I was indeed looking for connection pools... Something like maximum thread concurrency option in innodb...

Is there any such available with MonetDB?

I had gone through this option... mserver5 --set max_clients=<num> Is this the only way to control parallel threads??

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305 .

On Mon, Aug 10, 2015 at 7:48 PM, Brian Hood < brianh6854@googlemail.com wrote:

Hi Vijay,

I think the words your looking for a connection pooling.

InnoDB Buffer pool in MySQL is the amount of data stored in memory which you set to a certain size and has nothing to do with network traffic.

Regards,

Brian Hood

On Mon, Aug 10, 2015 at 8:45 AM, Vijay Krishna < vijayakrishna55@gmail.com > wrote:

Hi,

Does MonetDB have any built in buffer pools to handle parallel requests?

I see the default number of parallel connections accepted and processed at a time by MonetDB is 64.. While I see that there are ways to increase it, is there anyway built in to pool incoming requests until the live 64 requests are processed?

For example, our application would receive 100 requests at a time, while we wish to limit the number of queries handled at a time by MonetDB to 50. Is there any way to automatically handle the remaining 50 (something like innodb buffer pool) ??

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305 .

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Hi All queries are taken into execution concurrently (and possibly will compete for resources!!). Internally, queues are used to handle the eligible MAL instructions ready for execution. regards, Martin On 13/08/15 08:44, Vijay Krishna wrote:

Thanks for the links Brian..

Dear all,

From this, I see that a database can be set to any number of clients, which gives me a question on number of queries processed at a time.

If I am using an 8 core machine and connecting 50 client threads (each executing many queries one by one) at a time to a database, will the mserver be processing queries from ALL the 50 clients at any given instance?? Or will that be restricted by the number of cores?? If this is restricted, then is there a default queue maintained by the server??

Put in simpler words, is the maximum number of queries processed by MonetDB at a given time equal to the number of client connections?? If not, then how is this handled (with a queue) ??

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305.

On Wed, Aug 12, 2015 at 6:11 PM, Brian Hood mailto:brianh6854@googlemail.com> wrote:

Hi Vijay,

I have found some interesting info in the monetdb man page actually.

--- This is interests me as it alllows you to limit the amount of clients that can connect to a database not just an mserver

*nclients=*/number/

Sets the maximum amount of clients that can connect to this database at the same time. Setting this to a high value is discouraged. A multiplex-funnel may be more performant, see /MULTIPLEX-FUNNEL/ below.

Regards,

Brian Hood

On Wed, Aug 12, 2015 at 7:48 AM, Vijay Krishna mailto:vijayakrishna55@gmail.com> wrote:

Thanks Stefan and Brian.

When I went through monetdb man page, the usage of nthreads (set as gdk_nr_threads) environmental variable was defined as

>> Defines how many worker threads the server should use to perform main processing. Normally, this number equals the number of available CPU cores in the system. Reducing this number forces the server to use less parallelism when executing queries, or none at all if set to *1*.

I am a bit confused in understanding this statement. The variable "nthreads" controls

1. the number of threads processing a single query using multiple cores. 2. the number of queries currently being executed over all databases by the mserver.

Please help me understand which of these two is correct.. This doubt arises because I have read from MonetDB home page that "MonetDB is designed for multi-core parallel execution on desktops to reduce response time for complex query processing".

If the variable "nthreads" controls the latter, then does monetdb have an in built queue to store the queries from remaining connections (controlled by the variable nclients) to the mserver?? I have read about multiplex funnel and a bit afraid over the transactional disadvantages mentioned in there.. Is there any other solution?

Thanks & Regards,

Vijayakrishna.P. Mobile : (+91) 9500402305 tel:%28%2B91%29%209500402305.

On Tue, Aug 11, 2015 at 6:56 PM, Stefan Manegold mailto:Stefan.Manegold@cwi.nl> wrote:

Hi,

indeed;

see also `man mserver5` and `man monetdb` or https://www.monetdb.org/Documentation/mserver5-man-page and https://www.monetdb.org/Documentation/monetdb-man-page

Stefan

----- On Aug 11, 2015, at 3:11 PM, Brian Hood brianh6854@googlemail.com mailto:brianh6854@googlemail.com wrote:

> Hi Vijay, > > Well if you take a look at your merovingian.log you can see the command that > starts the database. > > /usr/bin/mserver5 --dbpath=/data2/threatmonitor/threatmonitor --set > merovingian_uri=mapi:monetdb://orville.lan:50000/threatmonitor --set > mapi_open=false --set mapi_port=0 --set > mapi_usock=/data2/threatmonitor/threatmonitor/.mapi.sock --set > monet_vault_key=/data2/threatmonitor/threatmonitor/.vaultkey --set > gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set > monet_daemon=yes > > > These 2 i think are the ones your looking for. > > --set gdk_nr_threads=2 --set max_clients=64 > > Regards, > > Brian Hood > > > On Tue, Aug 11, 2015 at 7:14 AM, Vijay Krishna < vijayakrishna55@gmail.com mailto:vijayakrishna55@gmail.com > > wrote: > > > > Oops.. My bad.. Thanks Brian.. > > Yes I was indeed looking for connection pools... Something like maximum thread > concurrency option in innodb... > > Is there any such available with MonetDB? > > I had gone through this option... mserver5 --set max_clients=<num> > Is this the only way to control parallel threads?? > > > > > > Thanks & Regards, > > Vijayakrishna.P. > Mobile : (+91) 9500402305 tel:%28%2B91%29%209500402305 . > > On Mon, Aug 10, 2015 at 7:48 PM, Brian Hood < brianh6854@googlemail.com mailto:brianh6854@googlemail.com > wrote: > > > > Hi Vijay, > > I think the words your looking for a connection pooling. > > InnoDB Buffer pool in MySQL is the amount of data stored in memory which you set > to a certain size and has nothing to do with network traffic. > > Regards, > > Brian Hood > > On Mon, Aug 10, 2015 at 8:45 AM, Vijay Krishna < vijayakrishna55@gmail.com mailto:vijayakrishna55@gmail.com > > wrote: > > > > Hi, > > Does MonetDB have any built in buffer pools to handle parallel requests? > > I see the default number of parallel connections accepted and processed at a > time by MonetDB is 64.. > While I see that there are ways to increase it, is there anyway built in to pool > incoming requests until the live 64 requests are processed? > > For example, our application would receive 100 requests at a time, while we wish > to limit the number of queries handled at a time by MonetDB to 50. Is there any > way to automatically handle the remaining 50 (something like innodb buffer > pool) ?? > > Thanks & Regards, > > Vijayakrishna.P. > Mobile : (+91) 9500402305 tel:%28%2B91%29%209500402305 . > > _______________________________________________ > users-list mailing list > users-list@monetdb.org mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > > > _______________________________________________ > users-list mailing list > users-list@monetdb.org mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > > > _______________________________________________ > users-list mailing list > users-list@monetdb.org mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > > > > _______________________________________________ > users-list mailing list > users-list@monetdb.org mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list

-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ http://www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 tel:%2B31%20%280%2920%20592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org mailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org mailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org mailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list