Multiple MonetDB nodes behind HAProxy loadbalancer
Hi all, We're thinking about running multiple MonetDB servers with the same databases behind a single HAProxy loadbalancer (see attached diagram). Would this be a good idea? bad idea? Any experiences with this kind of setup? We are thinking about this route primarily so that we can bring a new dataset online during business hours without disrupting end-users who are running queries from our application. An additional benefit would be the ability to handle more load and enable horizontal scalability. Any insights would be much appreciated! Best regards, Dennis
Hi Dennis, very similar setups were the trigger for us to create monetdbd with it's funnel functionality; see the "MULTIPLEX-FUNNELS" section of the monetdbd manpage: https://www.monetdb.org/Documentation/monetdbd-man-page Admittedly, this multiplexes / broadcasts queries rather than doing active load balancing. For the required synchronisation of the multiple database copies, you might want to check the new --- so far neither documented nor released --- transaction-replication functionality of MonetDB ... Best, Stefan ----- On Aug 18, 2015, at 9:29 AM, Dennis Pallett dennis@pallett.nl wrote:
Hi all,
We're thinking about running multiple MonetDB servers with the same databases behind a single HAProxy loadbalancer (see attached diagram). Would this be a good idea? bad idea? Any experiences with this kind of setup?
We are thinking about this route primarily so that we can bring a new dataset online during business hours without disrupting end-users who are running queries from our application. An additional benefit would be the ability to handle more load and enable horizontal scalability.
Any insights would be much appreciated!
Best regards, Dennis
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Hi Dennis, There is a small manual on transaction-replication: https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/TransactionReplic... It is currently only available on the Default brach. Best, Dimitar
On 2015-Aug-18, at 10:15 , Stefan Manegold
wrote: Hi Dennis,
very similar setups were the trigger for us to create monetdbd with it's funnel functionality; see the "MULTIPLEX-FUNNELS" section of the monetdbd manpage: https://www.monetdb.org/Documentation/monetdbd-man-page
Admittedly, this multiplexes / broadcasts queries rather than doing active load balancing.
For the required synchronisation of the multiple database copies, you might want to check the new --- so far neither documented nor released --- transaction-replication functionality of MonetDB ...
Best, Stefan
----- On Aug 18, 2015, at 9:29 AM, Dennis Pallett dennis@pallett.nl wrote:
Hi all,
We're thinking about running multiple MonetDB servers with the same databases behind a single HAProxy loadbalancer (see attached diagram). Would this be a good idea? bad idea? Any experiences with this kind of setup?
We are thinking about this route primarily so that we can bring a new dataset online during business hours without disrupting end-users who are running queries from our application. An additional benefit would be the ability to handle more load and enable horizontal scalability.
Any insights would be much appreciated!
Best regards, Dennis
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hi Stefan & Dimitar, Thank you both for your replies. Good to hear that transaction-replication is coming to MonetDB. Fortunately though we don't require that at the moment because we generally use MonetDB as a read-only backend en do nightly batch updates from an external source. That is exactly why we want to run multiple MonetDB nodes behind a load-balancer so that we can also do (rolling) batch updates during business hours by taking nodes offline/online as needed. We'll be starting with a proof-of-concept soon to see if it's a feasible idea. Hopefully with the Jul2015 version of MonetDB :) Best regards, Dennis On Tue, Aug 18, 2015 at 1:05 PM, Dimitar Nedev < dimitar.nedev@monetdbsolutions.com> wrote:
Hi Dennis,
There is a small manual on transaction-replication:
https://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/TransactionReplic...
It is currently only available on the Default brach.
Best, Dimitar
On 2015-Aug-18, at 10:15 , Stefan Manegold
wrote: Hi Dennis,
very similar setups were the trigger for us to create monetdbd with it's funnel functionality; see the "MULTIPLEX-FUNNELS" section of the monetdbd manpage: https://www.monetdb.org/Documentation/monetdbd-man-page
Admittedly, this multiplexes / broadcasts queries rather than doing active load balancing.
For the required synchronisation of the multiple database copies, you might want to check the new --- so far neither documented nor released --- transaction-replication functionality of MonetDB ...
Best, Stefan
----- On Aug 18, 2015, at 9:29 AM, Dennis Pallett dennis@pallett.nl wrote:
Hi all,
We're thinking about running multiple MonetDB servers with the same databases behind a single HAProxy loadbalancer (see attached diagram). Would this be a good idea? bad idea? Any experiences with this kind of setup?
We are thinking about this route primarily so that we can bring a new dataset online during business hours without disrupting end-users who are running queries from our application. An additional benefit would be the ability to handle more load and enable horizontal scalability.
Any insights would be much appreciated!
Best regards, Dennis
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Dennis Pallett
-
Dimitar Nedev
-
Stefan Manegold