[Monetdb-developers] mindmap
Updated version. (Should find a place in the MonetDB Repository for it) Inclusion of Peter's comments. - Mserver becomes a watchdog for local Mserver Instances (other name needed) Mguardian itself is a process to manage a community of Mservers, who have delegated some control. Please add comments. <map version="0.7.1"> <node TEXT="Mguardian"> <node TEXT="Introduction" POSITION="right"> <node TEXT="Automated crash recovery" FOLDED="true"> <node TEXT="Single Mserver failure survivals"/> <node TEXT="Fail-over in case of a hardware crash"/> </node> <node TEXT="Resource management" FOLDED="true"> <node TEXT="Dbfarm management" FOLDED="true"> <node TEXT="Creation of necessary directories"/> <node TEXT="Assurance of sufficient space"/> </node> <node TEXT="Load distribution on multiple servers" FOLDED="true"> <node TEXT="Forwarding query clients to slave"/> </node> <node TEXT="Automatic backup/recovery management" FOLDED="true"> <node TEXT="Day/weekly schedules"/> <node TEXT="Highwatermark scheduling"/> </node> <node TEXT="Disk volume management" FOLDED="true"> <node TEXT="Multi-hierarchical chkpoint stores"/> <node TEXT="log compression"/> </node> </node> <node TEXT="Monitoring" FOLDED="true"> <node TEXT="Lifeness of all Mserver instances in the convoy"/> <node TEXT="Resource usage" FOLDED="true"> <node TEXT="MonetDB client load"/> <node TEXT="Host resource load"/> </node> </node> <node TEXT="Authorization" FOLDED="true"> <node TEXT="User administration"/> <node TEXT="Granting access to services"/> </node> </node> <node TEXT="Design goals" POSITION="right"> <node TEXT="appearance"> <node TEXT="Transparent to users"> <node TEXT="Becoming a member of a guardian group 
is an option in the Mserver configuration file."/> </node> <node TEXT="Transparent to location"> <node TEXT="Cluster machines are supported"/> <node TEXT="loosely coupled servers through HTTP"/> </node> <node TEXT="Transparent to hw/sw"> <node TEXT="Platform independent code"/> <node TEXT="Python? Java?"/> </node> </node> <node TEXT="Convoys" FOLDED="true"> <node TEXT="The guardian manages a group of database services"/> <node TEXT="The guardian process acts as a servant to those requiring it"/> <node TEXT="It is a free association, i.e. a Mserver can at any time
join/leave the group. Depending on the way this is
realised, Mguardian may initiate an action"/> </node> <node TEXT="State management"> <node TEXT="Driven by XML configuration file and state file"/> <node TEXT="Ascii-based state management at transaction level"/> <node TEXT="Distributed transaction management"/> </node> <node TEXT="Multiple points of failure infrastructure" FOLDED="true"> <node TEXT="Multiple servers in a cluster"/> <node TEXT="Mguardian lifes outside the area of Mservers?"/> </node> <node TEXT="Web-based status monitoring" FOLDED="true"> <node TEXT="At the level of a single server"/> <node TEXT="At the level of managin multiple Mservers"/> </node> <node TEXT="Active warning system" FOLDED="true"> <node TEXT="Email warning to DBA"/> </node> </node> <node TEXT="Architecture" POSITION="right"> <node TEXT="Configuration files" FOLDED="true"> <node TEXT="XML-based structure of Mserver Farms"/> <node TEXT="Database specific management conditions"/> <node TEXT="Disaster recovery rules"/> </node> <node TEXT="Implementation"> <node TEXT="Watchdog function" FOLDED="true"> <node TEXT="A lightweight independent process that keeps that
keeps a specific Mserver instance alive"> <node TEXT="It 'behaves' as the main thread of Mserver"/> <node TEXT="It should maintain information for restart" FOLDED="true"> <node TEXT="how to know this ?"/> <node TEXT="By being part of Mserver thread group"/> </node> </node> <node TEXT="Recover policy is database specific"> <node TEXT="Simply restart Mserver using known script"/> <node TEXT="Rejoin the Mguardian convoy"/> <node TEXT="Set-back the latest chkpoint instance"/> </node> <node TEXT="Monitoring" FOLDED="true"> <node TEXT="Collect session information"/> <node TEXT="Obtain webbased state of a single Mserver"/> </node> <node TEXT="Connection server"> <node TEXT="Should take over Internet thread listener"/> <node TEXT="Authenticates valid connections"/> <node TEXT="Forward request to convoy if needed"/> </node> </node> <node TEXT="Management function" FOLDED="true"> <node TEXT="Independent MonetDB application" FOLDED="true"> <node TEXT="Runs on secure system and keeps overview"/> <node TEXT="Mknife access possible"/> </node> <node TEXT="Manages allocation of connection requests"/> <node TEXT="Revives the watchdogs or call for DBA help"/> <node TEXT="Monitors high-water marks and calls the DBA"/> </node> </node> <node TEXT="Mguardian GUI" FOLDED="true"> <node TEXT="A system wide control centre "> <node TEXT="Gui-based"/> <node TEXT="Controls instances in cluster"/> <node TEXT="Can collect information from autonomous nodes"/> </node> <node TEXT="It receives reports from the watchdogs" FOLDED="true"> <node TEXT="Incremental load levels"/> <node TEXT="Database farm size"/> <node TEXT="log sizes"/> <node TEXT="Number of concurrent clients"/> <node TEXT="Machine load level"/> </node> <node TEXT="supports remotely initiated actions" FOLDED="true"> <node TEXT="Watchdog is the proxy for local system actions"/> </node> </node> </node> <node TEXT="Implementation strategy" POSITION="right"> <node TEXT="Phase 1: Mserver becomes a watchdog"> <node TEXT="Performs rudimentary guardian actions 
before admitting a Mserver Instance"/> <node TEXT="Performs auto reboot of Mserver Instance"/> <node TEXT="Tell users about the language/instance port to use"/> <node TEXT="Keeps log of user sessions"/> </node> <node TEXT="Phase 2: Mserver monitoring"> <node TEXT="Mguardian database design"/> <node TEXT="Webservice to collect data" FOLDED="true"> <node TEXT="HW load"/> <node TEXT="Free log space"/> <node TEXT="Database space"/> </node> <node TEXT="Simple admin control to change properties"/> <node TEXT="Prototyping as Mknife scripts"/> </node> <node TEXT="Phase 3: Mguardian in control"> <node TEXT="Primary portal for client connectivity"/> <node TEXT="start/stop Mservers"/> </node> </node> </node> </map>
I would certainly add at Mguardian/Architecture/Implementation/Watchdog\ function/Connection\ server/"limits concurrent connections to a certain Mserver or the whole 'cluster' (prevent going down the drain due to DoS attack)" and Mguardian/Architecture/Implementation/Watchdog\ function/Connection\ server/"reserves a (small number of) connection(s) for administrator access" and Mguardian/Architecture/Implementation/Management\ function/"kills connections on request of DBA" Martin Kersten wrote:
Updated version. (Should find a place in the MonetDB Repository for it)
Inclusion of Peter's comments. - Mserver becomes a watchdog for local Mserver Instances (other name needed)
Mguardian itself is a process to manage a community of Mservers, who have delegated some control.
Please add comments.
------------------------------------------------------------------------
<map version="0.7.1"> <node TEXT="Mguardian"> <node TEXT="Introduction" POSITION="right"> <node TEXT="Automated crash recovery" FOLDED="true"> <node TEXT="Single Mserver failure survivals"/> <node TEXT="Fail-over in case of a hardware crash"/> </node> <node TEXT="Resource management" FOLDED="true"> <node TEXT="Dbfarm management" FOLDED="true"> <node TEXT="Creation of necessary directories"/> <node TEXT="Assurance of sufficient space"/> </node> <node TEXT="Load distribution on multiple servers" FOLDED="true"> <node TEXT="Forwarding query clients to slave"/> </node> <node TEXT="Automatic backup/recovery management" FOLDED="true"> <node TEXT="Day/weekly schedules"/> <node TEXT="Highwatermark scheduling"/> </node> <node TEXT="Disk volume management" FOLDED="true"> <node TEXT="Multi-hierarchical chkpoint stores"/> <node TEXT="log compression"/> </node> </node> <node TEXT="Monitoring" FOLDED="true"> <node TEXT="Lifeness of all Mserver instances in the convoy"/> <node TEXT="Resource usage" FOLDED="true"> <node TEXT="MonetDB client load"/> <node TEXT="Host resource load"/> </node> </node> <node TEXT="Authorization" FOLDED="true"> <node TEXT="User administration"/> <node TEXT="Granting access to services"/> </node> </node> <node TEXT="Design goals" POSITION="right"> <node TEXT="appearance"> <node TEXT="Transparent to users"> <node TEXT="Becoming a member of a guardian group 
is an option in the Mserver configuration file."/> </node> <node TEXT="Transparent to location"> <node TEXT="Cluster machines are supported"/> <node TEXT="loosely coupled servers through HTTP"/> </node> <node TEXT="Transparent to hw/sw"> <node TEXT="Platform independent code"/> <node TEXT="Python? Java?"/> </node> </node> <node TEXT="Convoys" FOLDED="true"> <node TEXT="The guardian manages a group of database services"/> <node TEXT="The guardian process acts as a servant to those requiring it"/> <node TEXT="It is a free association, i.e. a Mserver can at any time
join/leave the group. Depending on the way this is
realised, Mguardian may initiate an action"/> </node> <node TEXT="State management"> <node TEXT="Driven by XML configuration file and state file"/> <node TEXT="Ascii-based state management at transaction level"/> <node TEXT="Distributed transaction management"/> </node> <node TEXT="Multiple points of failure infrastructure" FOLDED="true"> <node TEXT="Multiple servers in a cluster"/> <node TEXT="Mguardian lifes outside the area of Mservers?"/> </node> <node TEXT="Web-based status monitoring" FOLDED="true"> <node TEXT="At the level of a single server"/> <node TEXT="At the level of managin multiple Mservers"/> </node> <node TEXT="Active warning system" FOLDED="true"> <node TEXT="Email warning to DBA"/> </node> </node> <node TEXT="Architecture" POSITION="right"> <node TEXT="Configuration files" FOLDED="true"> <node TEXT="XML-based structure of Mserver Farms"/> <node TEXT="Database specific management conditions"/> <node TEXT="Disaster recovery rules"/> </node> <node TEXT="Implementation"> <node TEXT="Watchdog function" FOLDED="true"> <node TEXT="A lightweight independent process that keeps that
keeps a specific Mserver instance alive"> <node TEXT="It 'behaves' as the main thread of Mserver"/> <node TEXT="It should maintain information for restart" FOLDED="true"> <node TEXT="how to know this ?"/> <node TEXT="By being part of Mserver thread group"/> </node> </node> <node TEXT="Recover policy is database specific"> <node TEXT="Simply restart Mserver using known script"/> <node TEXT="Rejoin the Mguardian convoy"/> <node TEXT="Set-back the latest chkpoint instance"/> </node> <node TEXT="Monitoring" FOLDED="true"> <node TEXT="Collect session information"/> <node TEXT="Obtain webbased state of a single Mserver"/> </node> <node TEXT="Connection server"> <node TEXT="Should take over Internet thread listener"/> <node TEXT="Authenticates valid connections"/> <node TEXT="Forward request to convoy if needed"/> </node> </node> <node TEXT="Management function" FOLDED="true"> <node TEXT="Independent MonetDB application" FOLDED="true"> <node TEXT="Runs on secure system and keeps overview"/> <node TEXT="Mknife access possible"/> </node> <node TEXT="Manages allocation of connection requests"/> <node TEXT="Revives the watchdogs or call for DBA help"/> <node TEXT="Monitors high-water marks and calls the DBA"/> </node> </node> <node TEXT="Mguardian GUI" FOLDED="true"> <node TEXT="A system wide control centre "> <node TEXT="Gui-based"/> <node TEXT="Controls instances in cluster"/> <node TEXT="Can collect information from autonomous nodes"/> </node> <node TEXT="It receives reports from the watchdogs" FOLDED="true"> <node TEXT="Incremental load levels"/> <node TEXT="Database farm size"/> <node TEXT="log sizes"/> <node TEXT="Number of concurrent clients"/> <node TEXT="Machine load level"/> </node> <node TEXT="supports remotely initiated actions" FOLDED="true"> <node TEXT="Watchdog is the proxy for local system actions"/> </node> </node> </node> <node TEXT="Implementation strategy" POSITION="right"> <node TEXT="Phase 1: Mserver becomes a watchdog"> <node TEXT="Performs rudimentary guardian actions 
before admitting a Mserver Instance"/> <node TEXT="Performs auto reboot of Mserver Instance"/> <node TEXT="Tell users about the language/instance port to use"/> <node TEXT="Keeps log of user sessions"/> </node> <node TEXT="Phase 2: Mserver monitoring"> <node TEXT="Mguardian database design"/> <node TEXT="Webservice to collect data" FOLDED="true"> <node TEXT="HW load"/> <node TEXT="Free log space"/> <node TEXT="Database space"/> </node> <node TEXT="Simple admin control to change properties"/> <node TEXT="Prototyping as Mknife scripts"/> </node> <node TEXT="Phase 3: Mguardian in control"> <node TEXT="Primary portal for client connectivity"/> <node TEXT="start/stop Mservers"/> </node> </node> </node> </map>
participants (2)
-
Fabian -
Martin Kersten