SlideShare una empresa de Scribd logo
1 de 69
Descargar para leer sin conexión
From Mainframe to Microservice 
An Introduction to 
Distributed Systems 
@tyler_treat 
Workiva
An Introduction to Distributed Systems 
❖ Building a foundation of understanding 
❖ Why distributed systems? 
❖ Universal fallacies 
❖ Characteristics and the CAP theorem 
❖ Common pitfalls 
❖ Digging deeper 
❖ Byzantine Generals Problem and consensus 
❖ Split-brain 
❖ Hybrid consistency models 
❖ Scaling shared data and CRDTs
“A distributed system is one in which the failure of 
a computer you didn't even know existed can 
render your own computer unusable.” 
–Leslie Lamport
Scale Up vs. Scale Out 
Vertical Scaling 
❖ Add resources to a node 
❖ Increases node capacity, load 
is unaffected 
❖ System complexity unaffected 
Horizontal Scaling 
❖ Add nodes to a cluster 
❖ Decreases load, capacity is 
unaffected 
❖ Availability and throughput w/ 
increased complexity
A distributed system 
is a collection of independent computers 
that behave as a single coherent system.
Why Distributed Systems? 
Availability 
Fault Tolerance 
Throughput 
Architecture 
Economics 
serve every request 
resilient to failures 
parallel computation 
decoupled, focused services 
scale-out becoming manageable/ 
cost-effective
oh shit…
“You have to design distributed systems with the 
expectation of failure.” 
–Ken Arnold
Distributed systems engineers are 
the world’s biggest pessimists.
Universal Fallacy #1 
The network is reliable. 
❖ Message delivery is never guaranteed 
❖ Best effort 
❖ Is it worth it? 
❖ Resiliency/redundancy/failover
Universal Fallacy #2 
Latency is zero. 
❖ We cannot defy the laws of physics 
❖ LAN to WAN deteriorates quickly 
❖ Minimize network calls (batch) 
❖ Design asynchronous systems
Universal Fallacy #3 
Bandwidth is infinite. 
❖ Out of our control 
❖ Limit message sizes 
❖ Use message queueing
Universal Fallacy #4 
The network is secure. 
❖ Everyone is out to get you 
❖ Build in security from day 1 
❖ Multi-layered 
❖ Encrypt, pentest, train developers
Universal Fallacy #5 
Topology doesn’t change. 
❖ Network topology is dynamic 
❖ Don’t statically address hosts 
❖ Collection of services, not nodes 
❖ Service discovery
Universal Fallacy #6 
There is one administrator. 
❖ May integrate with third-party systems 
❖ “Is it our problem or theirs?” 
❖ Conflicting policies/priorities 
❖ Third parties constrain; weigh the risk
Universal Fallacy #7 
Transport cost is zero. 
❖ Monetary and practical costs 
❖ Building/maintaining a network is not 
trivial 
❖ The “perfect” system might be too costly
Universal Fallacy #8 
The network is homogenous. 
❖ Networks are almost never homogenous 
❖ Third-party integration? 
❖ Consider interoperability 
❖ Avoid proprietary protocols
These problems apply to LAN and WAN systems 
(single-data-center and cross-data-center) 
No one is safe.
“Anything that can go 
wrong will go wrong.” 
–Murphy’s Law
Characteristics of a Reliable Distributed System 
Fault-tolerant 
Available 
Scalable 
Consistent 
Secure 
Performant 
nodes can fail 
serve all the requests, all the time 
behave correctly with changing 
topologies 
state is coordinated across nodes 
access is authenticated 
it’s fast!
Distributed systems are 
all about trade-offs.
CAP Theorem 
❖ Presented in 1998 by Eric 
Brewer 
❖ Impossible to guarantee 
all three: 
❖ Consistency 
❖ Availability 
❖ Partition tolerance
Consistency
Consistency 
❖ Linearizable - there exists a total order of all state 
updates and each update appears atomic 
❖ E.g. mutexes make operations appear atomic 
❖ When operations are linearizable, we can assign a unique 
“timestamp” to each one (total order) 
❖ A system is consistent if every node shares the same 
total order 
❖ Consistency which is both global and instantaneous is 
impossible
Consistency 
Eventual consistency 
replicas allowed to diverge, 
eventually converge 
Strong consistency 
replicas can’t diverge; 
requires linearizability
Availability 
❖ Every request received by a non-failing node must be 
served 
❖ If a piece of data required for a request is unavailable, 
the system is unavailable 
❖ 100% availability is a myth
Partition Tolerance 
❖ A partition is a split in the network—many causes 
❖ Partition tolerance means partitions can happen 
❖ CA is easy when your network is perfectly reliable 
❖ Your network is not perfectly reliable
Partition Tolerance
Common Pitfalls 
❖ Halting failure - machine stops 
❖ Network failure - network connection breaks 
❖ Omission failure - messages are lost 
❖ Timing failure - clock skew 
❖ Byzantine failure - arbitrary failure
Digging Deeper 
Exploring some higher-level concepts
Byzantine Generals Problem 
❖ Consider a city under siege by two allied armies 
❖ Each army has a general 
❖ One general is the leader 
❖ Armies must agree when to attack 
❖ Must use messengers to communicate 
❖ Messengers can be captured by defenders
Byzantine Generals Problem
Byzantine Generals Problem 
❖ Send 100 messages, attack no matter what 
❖ A might attack without B 
❖ Send 100 messages, wait for acks, attack if confident 
❖ B might attack without A 
❖ Messages have overhead 
❖ Can’t reliably make decision (provenly impossible)
Distributed Consensus 
❖ Replace 2 generals with N 
generals 
❖ Nodes must agree on data 
value 
❖ Solutions: 
❖ Multi-phase commit 
❖ State replication
Two-Phase Commit 
❖ Blocking protocol 
❖ Coordinator waits for 
cohorts 
❖ Cohorts wait for 
commit/rollback 
❖ Can deadlock
Three-Phase Commit 
❖ Non-blocking 
protocol 
❖ Abort on timeouts 
❖ Susceptible to 
network partitions
State Replication 
❖ E.g. Paxos, Raft protocols 
❖ Elect a leader (coordinator) 
❖ All changes go through leader 
❖ Each change appends log entry 
❖ Each node has log replica
State Replication 
❖ Must have quorum (majority) 
to proceed 
❖ Commit once quorum acks 
❖ Quorums mitigate partitions 
❖ Logs allow state to be rebuilt
Split-Brain
Split-Brain
Split-Brain
Split-Brain 
❖ Optimistic (AP) - let partitions work as usual 
❖ Pessimistic (CP) - quorum partition works, fence others
Hybrid Consistency Models 
❖ Weak == available, low latency, stale reads 
❖ Strong == fresh reads, less available, high latency 
❖ How do you choose a consistency model? 
❖ Hybrid models 
❖ Weaker models when possible (likes, followers, votes) 
❖ Stronger models when necessary 
❖ Tunable consistency models (Cassandra, Riak, etc.)
Scaling Shared Data 
❖ Sharing mutable data at large scale is difficult 
❖ Solutions: 
❖ Immutable data 
❖ Last write wins 
❖ Application-level conflict resolution 
❖ Causal ordering (e.g. vector clocks) 
❖ Distributed data types (CRDTs)
Scaling Shared Data 
Imagine a shared, global 
counter… 
“Get, add 1, and put” 
transaction will not 
scale
CRDT 
❖ Conflict-free Replicated Data Type 
❖ Convergent: state-based 
❖ Commutative: operations-based 
❖ E.g. distributed sets, lists, maps, counters 
❖ Update concurrently w/o writer coordination
CRDT 
❖ CRDTs always converge (provably) 
❖ Operations commute (order doesn’t matter) 
❖ Highly available, eventually consistent 
❖ Always reach consistent state 
❖ Drawbacks: 
❖ Requires knowledge of all clients 
❖ Must be associative, commutative, and idempotent
G-Counter
CRDT 
❖ Add to set is associative, commutative, idempotent 
❖ add(“a”), add(“b”), add(“a”) => {“a”, “b”} 
❖ Adding and removing items is not 
❖ add(“a”), remove(“a”) => {} 
❖ remove(“a”), add(“a”) => {“a”} 
❖ CRDTs require interpretation of common data 
structures w/ limitations
Two-Phase Set 
❖ Use two sets, one for adding, one for removing 
❖ Elements can be added once and removed once 
❖ { 
“a”: [“a”, “b”, “c”], 
“r”: [“a”] 
} 
❖ => {“b”, “c”} 
❖ add(“a”), remove(“a”) => {“a”: [“a”], “r”: [“a”]} 
❖ remove(“a”), add(“a”) => {“a”: [“a”], “r”: [“a”]}
Let’s Recap...
Distributed architectures allow us to build 
highly available, fault-tolerant systems.
We can't live in this fantasy land 
where everything works perfectly 
all of the time.
Shit happens — network partitions, 
hardware failure, GC pauses, 
latency, dropped packets…
Build resilient systems.
Design for failure.
kill -9
Consider the trade-off between 
consistency and availability.
Partition tolerance is not an option, 
it’s required. 
(if you’re building a distributed system)
Use weak consistency when possible, 
strong when necessary.
Sharing data at scale is hard, 
let’s go shopping. 
(or consider your options)
State is hell.
Further Readings 
❖ Jepsen series 
Kyle Kingsbury (aphyr) 
❖ A Comprehensive Study of Convergent and Commutative 
Replicated Data Types 
Shapiro et al. 
❖ In Search of an Understandable Consensus Algorithm 
Ongaro et al. 
❖ CAP Twelve Years Later 
Eric Brewer 
❖ Many, many more…
Thanks! 
@tyler_treat 
github.com/tylertreat 
bravenewgeek.com

Más contenido relacionado

La actualidad más candente

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Dvir Volk
 

La actualidad más candente (20)

Under The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database ArchitectureUnder The Hood Of A Shard-Per-Core Database Architecture
Under The Hood Of A Shard-Per-Core Database Architecture
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafka
 
Hello, kafka! (an introduction to apache kafka)
Hello, kafka! (an introduction to apache kafka)Hello, kafka! (an introduction to apache kafka)
Hello, kafka! (an introduction to apache kafka)
 
DNS Security Presentation ISSA
DNS Security Presentation ISSADNS Security Presentation ISSA
DNS Security Presentation ISSA
 
Microservices Architecture Part 2 Event Sourcing and Saga
Microservices Architecture Part 2 Event Sourcing and SagaMicroservices Architecture Part 2 Event Sourcing and Saga
Microservices Architecture Part 2 Event Sourcing and Saga
 
Domain Driven Design (DDD)
Domain Driven Design (DDD)Domain Driven Design (DDD)
Domain Driven Design (DDD)
 
APACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsAPACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka Streams
 
Cassandra Introduction & Features
Cassandra Introduction & FeaturesCassandra Introduction & Features
Cassandra Introduction & Features
 
Envoy and Kafka
Envoy and KafkaEnvoy and Kafka
Envoy and Kafka
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
Secrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on KubernetesSecrets of Performance Tuning Java on Kubernetes
Secrets of Performance Tuning Java on Kubernetes
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
 
CQRS: Command/Query Responsibility Segregation
CQRS: Command/Query Responsibility SegregationCQRS: Command/Query Responsibility Segregation
CQRS: Command/Query Responsibility Segregation
 
Zookeeper Architecture
Zookeeper ArchitectureZookeeper Architecture
Zookeeper Architecture
 
Distributed Transactions: Saga Patterns
Distributed Transactions: Saga PatternsDistributed Transactions: Saga Patterns
Distributed Transactions: Saga Patterns
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Producer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache KafkaProducer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache Kafka
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
Grokking TechTalk #33: High Concurrency Architecture at TIKI
Grokking TechTalk #33: High Concurrency Architecture at TIKIGrokking TechTalk #33: High Concurrency Architecture at TIKI
Grokking TechTalk #33: High Concurrency Architecture at TIKI
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 

Similar a From Mainframe to Microservice: An Introduction to Distributed Systems

Design Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databasesDesign Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databases
lovingprince58
 

Similar a From Mainframe to Microservice: An Introduction to Distributed Systems (20)

Distributed computing for new bloods
Distributed computing for new bloodsDistributed computing for new bloods
Distributed computing for new bloods
 
Everything you always wanted to know about Distributed databases, at devoxx l...
Everything you always wanted to know about Distributed databases, at devoxx l...Everything you always wanted to know about Distributed databases, at devoxx l...
Everything you always wanted to know about Distributed databases, at devoxx l...
 
Intro to distributed systems
Intro to distributed systemsIntro to distributed systems
Intro to distributed systems
 
Introduction
IntroductionIntroduction
Introduction
 
Distributed systems and scalability rules
Distributed systems and scalability rulesDistributed systems and scalability rules
Distributed systems and scalability rules
 
Scylla Summit 2018: Consensus in Eventually Consistent Databases
Scylla Summit 2018: Consensus in Eventually Consistent DatabasesScylla Summit 2018: Consensus in Eventually Consistent Databases
Scylla Summit 2018: Consensus in Eventually Consistent Databases
 
Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...
Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...
Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...
 
Database Expert Q&A from 2600hz and Cloudant
Database Expert Q&A from 2600hz and CloudantDatabase Expert Q&A from 2600hz and Cloudant
Database Expert Q&A from 2600hz and Cloudant
 
Design Patterns for Distributed Non-Relational Databases
Design Patterns for Distributed Non-Relational DatabasesDesign Patterns for Distributed Non-Relational Databases
Design Patterns for Distributed Non-Relational Databases
 
Design Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databasesDesign Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databases
 
Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...
Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...
Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...
 
Module: Mutable Content in IPFS
Module: Mutable Content in IPFSModule: Mutable Content in IPFS
Module: Mutable Content in IPFS
 
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedInJay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn
 
Data Consitency Patterns in Cloud Native Applications
Data Consitency Patterns in Cloud Native ApplicationsData Consitency Patterns in Cloud Native Applications
Data Consitency Patterns in Cloud Native Applications
 
Cassandra
CassandraCassandra
Cassandra
 
The Power of Determinism in Database Systems
The Power of Determinism in Database SystemsThe Power of Determinism in Database Systems
The Power of Determinism in Database Systems
 
OSI reference model
OSI reference modelOSI reference model
OSI reference model
 
The Reactive Principles: Eight Tenets For Building Cloud Native Applications
The Reactive Principles: Eight Tenets For Building Cloud Native ApplicationsThe Reactive Principles: Eight Tenets For Building Cloud Native Applications
The Reactive Principles: Eight Tenets For Building Cloud Native Applications
 
Cassandra - A Decentralized Structured Storage System
Cassandra - A Decentralized Structured Storage SystemCassandra - A Decentralized Structured Storage System
Cassandra - A Decentralized Structured Storage System
 
Highly available distributed databases, how they work, javier ramirez at teowaki
Highly available distributed databases, how they work, javier ramirez at teowakiHighly available distributed databases, how they work, javier ramirez at teowaki
Highly available distributed databases, how they work, javier ramirez at teowaki
 

Más de Tyler Treat

The Observability Pipeline
The Observability PipelineThe Observability Pipeline
The Observability Pipeline
Tyler Treat
 
Distributed Systems Are a UX Problem
Distributed Systems Are a UX ProblemDistributed Systems Are a UX Problem
Distributed Systems Are a UX Problem
Tyler Treat
 

Más de Tyler Treat (10)

Cloud-Native Observability
Cloud-Native ObservabilityCloud-Native Observability
Cloud-Native Observability
 
The Observability Pipeline
The Observability PipelineThe Observability Pipeline
The Observability Pipeline
 
Distributed Systems Are a UX Problem
Distributed Systems Are a UX ProblemDistributed Systems Are a UX Problem
Distributed Systems Are a UX Problem
 
The Future of Ops
The Future of OpsThe Future of Ops
The Future of Ops
 
Building a Distributed Message Log from Scratch - SCaLE 16x
Building a Distributed Message Log from Scratch - SCaLE 16xBuilding a Distributed Message Log from Scratch - SCaLE 16x
Building a Distributed Message Log from Scratch - SCaLE 16x
 
Building a Distributed Message Log from Scratch
Building a Distributed Message Log from ScratchBuilding a Distributed Message Log from Scratch
Building a Distributed Message Log from Scratch
 
So You Wanna Go Fast?
So You Wanna Go Fast?So You Wanna Go Fast?
So You Wanna Go Fast?
 
Simple Solutions for Complex Problems
Simple Solutions for Complex ProblemsSimple Solutions for Complex Problems
Simple Solutions for Complex Problems
 
Probabilistic algorithms for fun and pseudorandom profit
Probabilistic algorithms for fun and pseudorandom profitProbabilistic algorithms for fun and pseudorandom profit
Probabilistic algorithms for fun and pseudorandom profit
 
The Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going DistributedThe Economics of Scale: Promises and Perils of Going Distributed
The Economics of Scale: Promises and Perils of Going Distributed
 

Último

The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
shinachiaurasa2
 

Último (20)

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
 
The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...
The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...
The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...
 
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfLearn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
 
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112  Starting From 5K to 15K High Profile Escorts ...Chinsurah Escorts ☎️8617697112  Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
 
The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
A Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docxA Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docx
 
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verifiedSector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 18, Noida Call girls :8448380779 Model Escorts | 100% verified
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
ManageIQ - Sprint 236 Review - Slide Deck
ManageIQ - Sprint 236 Review - Slide DeckManageIQ - Sprint 236 Review - Slide Deck
ManageIQ - Sprint 236 Review - Slide Deck
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
 
BUS PASS MANGEMENT SYSTEM USING PHP.pptx
BUS PASS MANGEMENT SYSTEM USING PHP.pptxBUS PASS MANGEMENT SYSTEM USING PHP.pptx
BUS PASS MANGEMENT SYSTEM USING PHP.pptx
 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
 
10 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 202410 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 2024
 
Define the academic and professional writing..pdf
Define the academic and professional writing..pdfDefine the academic and professional writing..pdf
Define the academic and professional writing..pdf
 
8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students
 
Exploring the Best Video Editing App.pdf
Exploring the Best Video Editing App.pdfExploring the Best Video Editing App.pdf
Exploring the Best Video Editing App.pdf
 

From Mainframe to Microservice: An Introduction to Distributed Systems

  • 1. From Mainframe to Microservice An Introduction to Distributed Systems @tyler_treat Workiva
  • 2. An Introduction to Distributed Systems ❖ Building a foundation of understanding ❖ Why distributed systems? ❖ Universal fallacies ❖ Characteristics and the CAP theorem ❖ Common pitfalls ❖ Digging deeper ❖ Byzantine Generals Problem and consensus ❖ Split-brain ❖ Hybrid consistency models ❖ Scaling shared data and CRDTs
  • 3. “A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.” –Leslie Lamport
  • 4. Scale Up vs. Scale Out Vertical Scaling ❖ Add resources to a node ❖ Increases node capacity, load is unaffected ❖ System complexity unaffected Horizontal Scaling ❖ Add nodes to a cluster ❖ Decreases load, capacity is unaffected ❖ Availability and throughput w/ increased complexity
  • 5. A distributed system is a collection of independent computers that behave as a single coherent system.
  • 6. Why Distributed Systems? Availability Fault Tolerance Throughput Architecture Economics serve every request resilient to failures parallel computation decoupled, focused services scale-out becoming manageable/ cost-effective
  • 8. “You have to design distributed systems with the expectation of failure.” –Ken Arnold
  • 9. Distributed systems engineers are the world’s biggest pessimists.
  • 10. Universal Fallacy #1 The network is reliable. ❖ Message delivery is never guaranteed ❖ Best effort ❖ Is it worth it? ❖ Resiliency/redundancy/failover
  • 11. Universal Fallacy #2 Latency is zero. ❖ We cannot defy the laws of physics ❖ LAN to WAN deteriorates quickly ❖ Minimize network calls (batch) ❖ Design asynchronous systems
  • 12. Universal Fallacy #3 Bandwidth is infinite. ❖ Out of our control ❖ Limit message sizes ❖ Use message queueing
  • 13. Universal Fallacy #4 The network is secure. ❖ Everyone is out to get you ❖ Build in security from day 1 ❖ Multi-layered ❖ Encrypt, pentest, train developers
  • 14. Universal Fallacy #5 Topology doesn’t change. ❖ Network topology is dynamic ❖ Don’t statically address hosts ❖ Collection of services, not nodes ❖ Service discovery
  • 15. Universal Fallacy #6 There is one administrator. ❖ May integrate with third-party systems ❖ “Is it our problem or theirs?” ❖ Conflicting policies/priorities ❖ Third parties constrain; weigh the risk
  • 16. Universal Fallacy #7 Transport cost is zero. ❖ Monetary and practical costs ❖ Building/maintaining a network is not trivial ❖ The “perfect” system might be too costly
  • 17. Universal Fallacy #8 The network is homogenous. ❖ Networks are almost never homogenous ❖ Third-party integration? ❖ Consider interoperability ❖ Avoid proprietary protocols
  • 18. These problems apply to LAN and WAN systems (single-data-center and cross-data-center) No one is safe.
  • 19. “Anything that can go wrong will go wrong.” –Murphy’s Law
  • 20.
  • 21. Characteristics of a Reliable Distributed System Fault-tolerant Available Scalable Consistent Secure Performant nodes can fail serve all the requests, all the time behave correctly with changing topologies state is coordinated across nodes access is authenticated it’s fast!
  • 22.
  • 23. Distributed systems are all about trade-offs.
  • 24. CAP Theorem ❖ Presented in 1998 by Eric Brewer ❖ Impossible to guarantee all three: ❖ Consistency ❖ Availability ❖ Partition tolerance
  • 26. Consistency ❖ Linearizable - there exists a total order of all state updates and each update appears atomic ❖ E.g. mutexes make operations appear atomic ❖ When operations are linearizable, we can assign a unique “timestamp” to each one (total order) ❖ A system is consistent if every node shares the same total order ❖ Consistency which is both global and instantaneous is impossible
  • 27. Consistency Eventual consistency replicas allowed to diverge, eventually converge Strong consistency replicas can’t diverge; requires linearizability
  • 28. Availability ❖ Every request received by a non-failing node must be served ❖ If a piece of data required for a request is unavailable, the system is unavailable ❖ 100% availability is a myth
  • 29. Partition Tolerance ❖ A partition is a split in the network—many causes ❖ Partition tolerance means partitions can happen ❖ CA is easy when your network is perfectly reliable ❖ Your network is not perfectly reliable
  • 31. Common Pitfalls ❖ Halting failure - machine stops ❖ Network failure - network connection breaks ❖ Omission failure - messages are lost ❖ Timing failure - clock skew ❖ Byzantine failure - arbitrary failure
  • 32. Digging Deeper Exploring some higher-level concepts
  • 33. Byzantine Generals Problem ❖ Consider a city under siege by two allied armies ❖ Each army has a general ❖ One general is the leader ❖ Armies must agree when to attack ❖ Must use messengers to communicate ❖ Messengers can be captured by defenders
  • 35. Byzantine Generals Problem ❖ Send 100 messages, attack no matter what ❖ A might attack without B ❖ Send 100 messages, wait for acks, attack if confident ❖ B might attack without A ❖ Messages have overhead ❖ Can’t reliably make decision (provenly impossible)
  • 36. Distributed Consensus ❖ Replace 2 generals with N generals ❖ Nodes must agree on data value ❖ Solutions: ❖ Multi-phase commit ❖ State replication
  • 37. Two-Phase Commit ❖ Blocking protocol ❖ Coordinator waits for cohorts ❖ Cohorts wait for commit/rollback ❖ Can deadlock
  • 38. Three-Phase Commit ❖ Non-blocking protocol ❖ Abort on timeouts ❖ Susceptible to network partitions
  • 39. State Replication ❖ E.g. Paxos, Raft protocols ❖ Elect a leader (coordinator) ❖ All changes go through leader ❖ Each change appends log entry ❖ Each node has log replica
  • 40. State Replication ❖ Must have quorum (majority) to proceed ❖ Commit once quorum acks ❖ Quorums mitigate partitions ❖ Logs allow state to be rebuilt
  • 44. Split-Brain ❖ Optimistic (AP) - let partitions work as usual ❖ Pessimistic (CP) - quorum partition works, fence others
  • 45. Hybrid Consistency Models ❖ Weak == available, low latency, stale reads ❖ Strong == fresh reads, less available, high latency ❖ How do you choose a consistency model? ❖ Hybrid models ❖ Weaker models when possible (likes, followers, votes) ❖ Stronger models when necessary ❖ Tunable consistency models (Cassandra, Riak, etc.)
  • 46. Scaling Shared Data ❖ Sharing mutable data at large scale is difficult ❖ Solutions: ❖ Immutable data ❖ Last write wins ❖ Application-level conflict resolution ❖ Causal ordering (e.g. vector clocks) ❖ Distributed data types (CRDTs)
  • 47. Scaling Shared Data Imagine a shared, global counter… “Get, add 1, and put” transaction will not scale
  • 48. CRDT ❖ Conflict-free Replicated Data Type ❖ Convergent: state-based ❖ Commutative: operations-based ❖ E.g. distributed sets, lists, maps, counters ❖ Update concurrently w/o writer coordination
  • 49. CRDT ❖ CRDTs always converge (provably) ❖ Operations commute (order doesn’t matter) ❖ Highly available, eventually consistent ❖ Always reach consistent state ❖ Drawbacks: ❖ Requires knowledge of all clients ❖ Must be associative, commutative, and idempotent
  • 51. CRDT ❖ Add to set is associative, commutative, idempotent ❖ add(“a”), add(“b”), add(“a”) => {“a”, “b”} ❖ Adding and removing items is not ❖ add(“a”), remove(“a”) => {} ❖ remove(“a”), add(“a”) => {“a”} ❖ CRDTs require interpretation of common data structures w/ limitations
  • 52. Two-Phase Set ❖ Use two sets, one for adding, one for removing ❖ Elements can be added once and removed once ❖ { “a”: [“a”, “b”, “c”], “r”: [“a”] } ❖ => {“b”, “c”} ❖ add(“a”), remove(“a”) => {“a”: [“a”], “r”: [“a”]} ❖ remove(“a”), add(“a”) => {“a”: [“a”], “r”: [“a”]}
  • 53.
  • 55. Distributed architectures allow us to build highly available, fault-tolerant systems.
  • 56. We can't live in this fantasy land where everything works perfectly all of the time.
  • 57.
  • 58.
  • 59. Shit happens — network partitions, hardware failure, GC pauses, latency, dropped packets…
  • 63. Consider the trade-off between consistency and availability.
  • 64. Partition tolerance is not an option, it’s required. (if you’re building a distributed system)
  • 65. Use weak consistency when possible, strong when necessary.
  • 66. Sharing data at scale is hard, let’s go shopping. (or consider your options)
  • 68. Further Readings ❖ Jepsen series Kyle Kingsbury (aphyr) ❖ A Comprehensive Study of Convergent and Commutative Replicated Data Types Shapiro et al. ❖ In Search of an Understandable Consensus Algorithm Ongaro et al. ❖ CAP Twelve Years Later Eric Brewer ❖ Many, many more…