Our business at Continuent is development of database clusters with highly simplified management and the ability to be operated unattended for prolonged periods of time. As part of our Tungsten Clustering product we developed an extensible, low-latency, fault-tolerant management framework for database clusters built around a core of group communications and business rules. We have found that our system is easy to maintain and to extend. For example, an extension to switch virtual IP addresses in the event of a database node failure was implemented in an afternoon as a set of two rules and a single bash script. In our talk we will cover the following: * Basic architecture of a rules-based management framework for databases * Introduction to business rules, with code examples showing how they work to repair problems ranging from simple process failures to network partitions * A quick demo of business rules in operation. * Finally, some thoughts on the benefits of the approach and our experiences (good and bad) with autonomic management of database clusters. This is an approach to management that we believe will be of interest to anyone who cares about keeping important data highly available as well anyone interested in learning about rules technology.

1. Linas Virbalas
Continuent, Inc.
© Continuent 2010

2. / Introductions
/ What is Tungsten?
/ Architecture of a Rule–Based Management
Framework for Database Clusters
/ Demo of Business Rules in Operation
/ Business Rules in Source Code
/ Questions and Comments
© Continuent 2010

3. © Continuent 2010

4. / Our Mission: Continuent is the data reliability company
that keeps business in business
/ Our Solution:
• Continuent Tungsten (Master/Slave Database Replication)
/ Our Value:
• Enterprise class data management
• Open source cost
• Integrated solution
/ Our Technical Expertise
• Database replication
• Database cluster management
• Application connectivity
• Software-as-a-Service (SaaS)
© Continuent 2010

5. © Continuent 2010

6. What Is Tungsten?
/ Tungsten implements master/slave clusters to:
• Protect data
• Maintain high availability
• Improve resource utilization
• Raise performance
/ Install and set up in a few minutes
/ Integrated backup/restore and data integrity checks
/ Efficient failover operations
/ Distributed, rule-driven management
/ No/minimal application changes
/ Highly pluggable
/ No specialized hardware requirements
© Continuent 2010

7. What’s Inside Tungsten?
/ Replication - Making copies
• Tungsten Replicator -- Database-neutral, platform independent
master/slave replication
/ Connectivity -- Finding databases
• Tungsten Connector -- Fast MySQL/PostgreSQL client to JDBC
proxying
• Tungsten SQL Router --JDBC wrapper for high-performance and
transparent failover, load-balancing, and partitioning (no proxy
required)
/ Management -- Administering the database
• Tungsten Manager -- Distributed administration with autonomic,
rule-based configuration and no single point of failure
• Tungsten Monitor -- Track resource status and database liveness
© Continuent 2010

8. Application Server Application Server
SQL Router/ SQL Router/
Manager Manager
Connector Connector
Replicator Replicator
Monitor Monitor
Manager Manager
Management Management
Client Client
© Continuent 2010

9. Multiple Routes to Databases
Java App Server PHP Application
Tungsten SQL Router libpq.a
PostgreSQL JDBC Driver
Tungsten
Connector
Tungsten Cluster
© Continuent 2010

10. Tungsten on Open Source Databases
/ MySQL 5.0/5.1
• Read and replicate from MySQL binlog
• Works with Oracle, MySQL and look-alikes (Percona, MariaDB)
/ PostgreSQL < 9
• Warm Standby + WAL Shipping
• Good basic availability/fast failover
• Slaves open up for reads only after failover
/ PostgreSQL >= 9
• Hot Standby + Streaming Replication
• Slaves opened up for reads = Tungsten scaling facilities work
• Add Streaming Replication = minimal delay in replicating data
/ Sometime in the future: Read PG logs directly!
© Continuent 2010

11. Tungsten Added Value
/ 15 minute cluster installation
/ Single commands to:
• View cluster status
• Provision a new standby
• Confirm liveness of replication
• Switch servers safely for maintenance
• Failover a dead server to most current replica
/ Automatic discovery of new replicas
/ Automatic failover when databases fail
/ Simple procedures for provisioning
/ Transparent application routing
/ Easy scaling
© Continuent 2010

12. © Continuent 2010

13. Distributed Rule-Based Management
Admin Client
Manager
Manager
(Coordinator)
Admin Client
Manager
Admin Client
© Continuent 2010

14. Group Communications in Clustering
/ Tungsten Manager processes connect to each
other via Group Communications protocol
/ Group Communications provide essential
features for running multi-node environments:
• Membership detection
• Notification about joined/left/crashed members
• Reliable message broadcast (lost messages retransmitted)
• Ordered message broadcast (all members receive
messages in the same order)
/ Different transport protocols can be used
• IP Multicast – for open local networks
• Peer to Peer TCP – for firewalled local networks
• Centralized Gossip Server – for Cloud environments
© Continuent 2010

15. What is a Rule?
/ Trivial Rule:
/ Example (pseudo code):
© Continuent 2010

16. What is a Rule Engine?
© Continuent 2010

17. How does the Rule Engine Work?
Application
with a Rule
Engine
(Tungsten
Manager)
© Continuent 2010

18. How does the Rule Engine get its Facts?
Tungsten Tungsten
Monitor Replicator
Tungsten Manager
© Continuent 2010

19. Who is The Coordinator?
/ Only one Coordinator for a group (cluster)
• Only one Manager must act on important events like
failures
• Having more than one Coordinator would lead to race
conditions and eventually chaotic cluster behavior
/ Tungsten chooses Coordinator from the
Group Communications
• Coordinator is the oldest member of the group
• New members handshake with the Coordinator when
joining
• Upon Coordinator death, GC automatically promotes a
new one
© Continuent 2010

20. © Continuent 2010

21. © Continuent 2010

22. The Rules are “Layered”
/ Lower level rules react on events from
• Database servers
• Replicators
• Managers
/ Outcome: the state change of a concept
“data source”
/ Higher level rules react on
• Data source state changes
/ Outcome: actual operations to the cluster
components (eg. promote slave to a master)
© Continuent 2010

23. Scenario: Database Crashes
/ Failover is handled by a controlled chain reaction of
rules rather than a single rule:
1. "DETECT AND FENCE FAILED DATASERVER" rule
detects state transition of the master DB from
ONLINE to STOPPED
 It then 'fences' the failure by setting the data source of the stopped
DB server to FAILED
2. "AUTOMATIC FAILOVER" rule fires, because
DataSource(state == ResourceState.FAILED &&
role == "master”)
 It then calls a method in the ClusterManagementHelper to do the
actual failover, after which it refreshes the data source states
i If the data source was of a slave, it would stay in the FAILED state
and no further action would take place
© Continuent 2010

24. © Continuent 2010

25. © Continuent 2010

26. Conclusion
/ Business Rules are a very powerful and flexible
tool for automating otherwise complex tasks
/ They are especially useful in implementing
automated DB clusters
/ A robust top layer connecting all the
components underneath into a seamless
experience
© Continuent 2010

27. © Continuent 2010

28. © Continuent 2010

29. HQ and Americas EMEA and APAC
560 S. Winchester Blvd., Suite 500 Lars Sonckin kaari 16
San Jose, CA 95128 02600 Espoo, Finland
Tel (866) 998-3642 Tel +358 50 517 9059
Fax (408) 668-1009 Fax +358 9 863 0060
e-mail: sales@continuent.com, linas.virbalas@continuent.com
Continuent Web Site:
http://www.continuent.com
© Continuent 2010