Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Distributed Locking in Kubernetes
Similar a Distributed Locking in Kubernetes (20)
Más de Rafał Leszko
Más de Rafał Leszko (15)
Último
Último (20)
Distributed Locking in Kubernetes
- 1. Distributed Locking in Kubernetes Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
- 2. About me ● Cloud-Native Team Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
- 3. About Hazelcast ● Distributed Company ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
- 4. ● Introduction ● Distributed Locking ○ Single-Instance ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 5. Introduction
- 6. Reasons for Distributed Locking ● Efficiency ○ prevent executing the same work more than once, e.g., performing some expensive calculation ○ failing lock results in some additional costs or some inconvenience ● Correctness ○ prevent data corruption, data loss, inconsistency ○ failing lock results in some serious problems with the system
- 7. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
- 8. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
- 9. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 10. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write()
- 11. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock()
- 12. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock()
- 13. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2
- 14. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2 write()
- 15. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 16. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
- 17. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 18. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 19. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 lease expired
- 20. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() lock() unlock() lease expired
- 21. Single-Instance
- 22. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
- 23. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Redis
- 24. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
- 25. Code: Single-Instance Distributed Locking var hazelcast = HazelcastClient. newHazelcastClient(); var lock = hazelcast.getCPSubsystem().getLock( "my-lock"); lock.lock(); writeToSharedResource(); lock.unlock();
- 26. Code: Single-Instance Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run hazelcast --image hazelcast/hazelcast --port 5701 --expose
- 27. Single-Instance Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 28. Single-Instance Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 29. Application 1 Single-Instance Distributed Locking Application 2 lock() lock() Hazelcast
- 30. Summary: Single-Instance Distributed Locking Efficiency Correctness Complexity Performance
- 31. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 32. Multi-Instance
- 33. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
- 34. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
- 35. Code: Multi-Instance Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast
- 36. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster
- 37. Application 1 Multi-Instance Distributed Locking Application 2 lock() lock() Hazelcast Cluster(s)
- 38. Summary: Multi-Instance Distributed Locking Efficiency Correctness Complexity Performance
- 39. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 40. RedLock
- 45. Code: RedLock var lock = new RedissonRedLock( client("redis-1").getLock("lock1"), client("redis-2").getLock("lock2"), client("redis-3").getLock("lock3") ); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run redis-1 --image redis:6.2.1 --port 6379 --expose kubectl run redis-2 --image redis:6.2.1 --port 6379 --expose kubectl run redis-3 --image redis:6.2.1 --port 6379 --expose
- 47. RedLock: Issues ● Server clocks need to be in sync ● Executing lock() and write() is not atomic
- 48. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
- 49. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1
- 50. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked
- 51. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock()
- 52. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock() write()
- 53. Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 write() write() lock() process blocked lock()
- 55. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 56. Consensus-Based
- 57. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 58. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Zookeeper Cluster
- 59. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 60. Code: Consensus-Based Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
- 61. Summary: Consensus-Based Distributed Locking Efficiency Correctness Complexity Performance
- 62. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 63. Application 1 Consensus-Based Distributed Locking Application 2 lock() lock() Zookeeper Cluster
- 65. Application 1 Kubernetes Native Distributed Locking Application 2 lock() lock() Kubernetes Cluster
- 66. Kubernetes Native Distributed Locking
- 68. Libraries for Kubernetes Native Distributed Locking ● Lockgate: cross-platform locking library with distributed locks using Kubernetes ● Kube-lock: simple library that implements a distributed lock using annotations on a Kubernetes resource
- 69. Summary: Kubernetes Native Distributed Locking Efficiency Correctness Complexity Performance
- 70. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ● Summary Agenda
- 71. Fenced
- 72. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() write() lock() process blocked
- 73. Application Fenced Distributed Locking token = lockAndGetToken() Hazelcast Cluster (CP Subsystem Enabled) Shared Resource token 1 2
- 74. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager
- 75. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock()
- 76. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11
- 77. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked
- 78. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock()
- 79. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12
- 80. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
- 81. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
- 82. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12 write() token: 11
- 83. Fenced Distributed Locking Shared Resource Application 2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() token: 11 write() token: 12 lock() process blocked token: 11 token: 12 rejected
- 84. Code: Fenced Distributed Locking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); long token = lock.lockAndGetFence(); writetoSharedResource(token); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
- 85. Summary: Fenced Distributed Locking Efficiency Correctness Complexity Performance
- 86. ● Introduction ✔ ● Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ✔ ● Summary Agenda
- 87. Summary
- 89. Resources ● Code for this presentation: https://github.com/leszko/distributed-locking ● How to do distributed locking: https://martin.kleppmann.com/2016/02/08/how-to-do-distrib uted-locking.htm ● Distributed Locks are Dead; Long Live Distributed Locks! https://hazelcast.com/blog/long-live-distributed-locks/