Distributed Lock Manager
•
7 recomendaciones•1,742 vistas
What’s the problem? Possible Solutions Distributed Lock Manager - Redlock - Google Chubby - ZooKeeper Takeaways
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Distributed Lock Manager
Similar a Distributed Lock Manager (20)
Último
Último (20)
Distributed Lock Manager
- 2. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 Outline What’s topics • What’s the problem? • Possible Solutions • Distributed Lock Manager • Redlock • Google Chubby • ZooKeeper • Takeaways
- 3. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 Consistency What’s the real problem is? Order Ship the Order Cancel the Order Seller Buyer How to drive the State Machine? What’s the end state?
- 4. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 4 Order (Canceled) Ship the Order Cancel the Order Seller Buyer Consistency Why should we need the lock? Within Locked Transaction Ship the Order Order (Canceled) Waiting For Lock Within Locked Transaction Error Invalid State Order (Shipped) Ship the Order Cancel the Order Seller Buyer Within Locked Transaction Order (Return) Waiting For Lock Within Locked Transaction Cancel the Order Return Process
- 5. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 5 Locking Locking Solutions Shard Application Centralized Distributed Lock Manager Database Lock
- 6. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 6 DB Lock MySQL InnoDB Locking Solutions SELECT ... LOCK IN SHARE MODE SELECT ... FOR UPDATE Exclusive Lock Shared Lock Versioning Optimistic Lock SELECT id, data, version FROM table WHERE id = $ID UPDATE table SET data = $data version = version + 1 WHERE id = $ID AND version=$VER if {UPDATE} Failed, go to step 1 else Done! 1 3 4 2 { Business Code }
- 7. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 7 Shard HotSpots indices Operation for Order A Operation for Order A Shard Data Nodes Operation Gateway Operation Gateway forward f o r w a r d • Using the Shard data Solution • Forward all of the operations to a single node • Consistency can be easy to be guaranteed • Consistent Hash • Gossip the indices Operation for Order A Operation for Order A Operation for Order A forw ard forward fo rw a rd Uber Ringpop https://eng.uber.com/ringpop-open-source-nodejs-library/
- 8. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 8 The Problems of Shard Why the shard is not a good design? Hotspots Multiple Entities Transaction Node Failed Load is not balanced Transfer money from account A to account B Data Replication cause more consistent issues R+W > N CAP Theorem
- 9. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 9 Lock Manager Centralized Lock Manager? • Distributed Lock Manager (DLM) • https://en.wikipedia.org/wiki/Distributed_lock_manager • Provides a lock-leasing protocol • Transaction Service is stateless • Lock Manager must be designed for HA • No single point failure • The lock state can be persistent. • Dead Lock Detection • Lock expired • Keepalive heart beats DLM Storage Persistence Service lock unlock
- 10. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 10 DLM Lock Modes Six lock modes Mode Requesting Process Other Processes Null (NL) No access Read or write access Concurrent Read (CR) Read access only Read or write access Concurrent Write (CW) Read or write access Read or write access Protected Read (PR) Read access only Read access only Protected Write (PW) Read or write access Read access only Exclusive (EX) Read or write access No access
- 11. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 11 Redlock Distributed locks with Redis SET resource_name my_random_value NX PX 30000 Not Exist Expired of 30s Unique across all clients Algorithm 1. Client get the current time in milliseconds 2. Client tries to acquire the lock in all the N instances sequentially, using the same key name and random value in all the instances. 3. Successfully acquire the lock - majority Redis nodes accepts && elapsed time < lock validity time. 4. Actual lock validity time = initial validity time - elapsed time 5. If acquire the lock failed, try to unlock all instances https://redis.io/topics/distlock if redis.call(“get”,KEYS[1]) == ARGV[1] then redis.call("del",KEYS[1]) end
- 12. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 12 How to do distributed locking https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html Challenge Redlock Using time to solve consensus? 5 Redis nodes (A, B, C, D and E) -- 2 Clients (1, 2) A B C D E 1 2 OK OK OK OK OK OK 1. Node C time jumps à Lock expired 2. Client 1 hangs while Node C reply, and become live after Lock expired 3. Client 1 get very long network delay from Node C “the algorithm makes dangerous assumptions about timing and system clocks” Martin Kleppmann
- 13. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 13 Lock Expired How to deal with dead lock ?
- 14. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 14 Lock Expired Making the lock safe with fencing?
- 15. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 15 Is Redlock Safe ? A technical debates http://antirez.com/news/101 TL;DR • Fencing is great, with this, no need distributed lock! • Clock jump is a big problem, Redlock cannot work correctly under this problem. • For network delay & client hangs, they all will be fine.
- 16. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 16 Google Chubby Google Chubby lock service • Two Components – Master & Client SDK • Cluster – typically 5 nodes • Replicas - use Paxos protocol to elect a master and replicate the logs • Failover – Master fails, other replicas run the election • Session & Keeplive • Client request or end a session • Lease Interval and Lease Timeout • Performance is average Source: https://www.slideshare.net/romain_jacotin/the-google-chubby-lock-service-for-looselycoupled-distributed-systems (45s) (12s)
- 17. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 17 Zookeeper Lock How to use Zookeeper as lock service 1 2 Create `_locknode/lock-` with Sequence & Ephemeral flag https://zookeeper.apache.org/doc/r3.1.2/recipes.html#sc_recipes_Locks Get children to see to check if I am the lowest sequence number • Yes. Get the Lock • No. Set the watch locknode _c_8a27d9a2-d177-11eb-b8bc-0242ac130003-lock-0000000001 _c_8a27d9a2-d177-11eb-b8bc-0242ac130003-lock-0000000002 Client A Client B
- 18. 企业云原生架构解决方案提供商/用技术推动商业进步 MegaEase 企业云原生架构提供商 18 Takeaways Summary • Concurrent transaction need be synchronized • DB Lock is fine, but the Optimistic Lock is great. • Sharding the data cannot solve the all of problem • Distributed Lock Service need the following features: • High Availability • Data Replicas - strong consistent protocol – Pasox, Raft, zab • Master Failover – Leader election • Deadlock Detection • Keepalive & Lease Timeout