[MonetDB-users] Replication/Redundancy
We're using MonetDB quite heavily and are really enjoying it. We're now wanting to add replication/redundancy to ensure we have as little downtime as possible. We're also thinking of balancing reads if that is possible. I've heard this is possible, but I'm unsure on the best way to approach it. I've not been able to find many docs that talk about this either. Could someone point me in the right direction? Thanks, Joseph Brower
I'm not sure I understand. On 03/12/2012 10:55 AM, Stefan de Konink wrote:
On Mon, 12 Mar 2012, Joseph Brower wrote:
Could someone point me in the right direction? Naively running two instances on two different works quite well.
Stefan
------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, 12 Mar 2012, Joseph Brower wrote:
I'm not sure I understand.
Without any 'true' balancing or distributed queries I am running queries individually on multiple systems. So it is just round-robin over the available backend servers. This does require that your datasets are updated per system individually. Stefan
Ok. Let me describe our use case. We have various logs being written to the database. This is done via COPY INTO. This works very well. The problem comes when we have many many reads occuring to see things like averages, counts, and such from that dataset. The dataset needs to be synced between both environments. Is this possible? How would I go about setting this up? Thanks, Joseph Brower On 03/12/2012 11:41 AM, Stefan de Konink wrote:
On Mon, 12 Mar 2012, Joseph Brower wrote:
I'm not sure I understand. Without any 'true' balancing or distributed queries I am running queries individually on multiple systems. So it is just round-robin over the available backend servers.
This does require that your datasets are updated per system individually.
Stefan
------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 13-03-12 00:32, Joseph Brower wrote:
The dataset needs to be synced between both environments. Is this possible? How would I go about setting this up?
What you could do: - Retract half of the systems from the balance system (- Start these systems in normal mode) - update these systems (- Bring up these systems in readonly) - Switch between updated and old - Apply the same steps to the other systems If you don't really require exact synchronisation, you could just send you copy into to the foreign system over SSH, for example. Stefan
Thanks for all of the help! I think I'm beginning to understand. My concern is that it seems I can't be inserting every 1 minute like we are currently. We want to keep things up to date on a per minute basis. Thanks, Joseph Brower On 03/12/2012 05:44 PM, Stefan de Konink wrote:
On 13-03-12 00:32, Joseph Brower wrote:
The dataset needs to be synced between both environments. Is this possible? How would I go about setting this up? What you could do:
- Retract half of the systems from the balance system (- Start these systems in normal mode) - update these systems (- Bring up these systems in readonly)
- Switch between updated and old - Apply the same steps to the other systems
If you don't really require exact synchronisation, you could just send you copy into to the foreign system over SSH, for example.
Stefan
------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 13-03-12 00:47, Joseph Brower wrote:
Thanks for all of the help! I think I'm beginning to understand. My concern is that it seems I can't be inserting every 1 minute like we are currently. We want to keep things up to date on a per minute basis.
Actually you can, I am inserting records per second, but at very high transaction overhead costs. If you solve it with: Start Transaction; COPY INTO; or INSERT; Commit; You might get away with that. But indeed, if you want to synchronise two systems, it is much better to create these copy into files. And stream them to the foreign host. Stefan
So right now we're only doing a COPY INTO; . Would there be a positive impact to doing the Start Transaction; first? Also, how would I configure MonetDB to balance those connections? Is that something I'd need to do at the client level or is it possible to have a single "master" server the distributes that load down to the child nodes? Thanks, Joseph Brower On 03/12/2012 05:53 PM, Stefan de Konink wrote:
Thanks for all of the help! I think I'm beginning to understand. My concern is that it seems I can't be inserting every 1 minute like we are currently. We want to keep things up to date on a per minute basis. Actually you can, I am inserting records per second, but at very high
On 13-03-12 00:47, Joseph Brower wrote: transaction overhead costs. If you solve it with:
Start Transaction; COPY INTO;
or
INSERT; Commit;
You might get away with that. But indeed, if you want to synchronise two systems, it is much better to create these copy into files. And stream them to the foreign host.
Stefan
------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 13-03-12 01:00, Joseph Brower wrote:
So right now we're only doing a COPY INTO; . Would there be a positive impact to doing the Start Transaction; first?
If that is the only thing you do in a client connection, and then close it again: probably not.
Also, how would I configure MonetDB to balance those connections? Is that something I'd need to do at the client level or is it possible to have a single "master" server the distributes that load down to the child nodes?
I don't have experience with non-naive implementation on that. Maybe some of the MonetDB researchers could comment on this. Stefan
On 13-03-2012 01:10:53 +0100, Stefan de Konink wrote:
On 13-03-12 01:00, Joseph Brower wrote:
So right now we're only doing a COPY INTO; . Would there be a positive impact to doing the Start Transaction; first?
If that is the only thing you do in a client connection, and then close it again: probably not.
Also, how would I configure MonetDB to balance those connections? Is that something I'd need to do at the client level or is it possible to have a single "master" server the distributes that load down to the child nodes?
I don't have experience with non-naive implementation on that. Maybe some of the MonetDB researchers could comment on this.
monetdbd(1) allows amongst other things balancing per connection. See the "REMOTE DATABASES" section in the man-page [1]. Fabian [1] http://www.monetdb.org/Documentation/monetdbd-man-page
participants (4)
-
Fabian Groffen -
Joseph Brower -
Joseph Brower -
Stefan de Konink