SlideShare una empresa de Scribd logo
1 de 41
Descargar para leer sin conexión
consistency without borders
Peter Alvaro, Peter Bailis,
Neil Conway, Joseph M. Hellerstein
UC Berkeley
The transaction concept
	
  DEBIT_CREDIT:	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  BEGIN_TRANSACTION;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  GET	
  MESSAGE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  EXTRACT	
  ACCOUT_NUMBER,	
  DELTA,	
  TELLER,	
  BRANCH	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  FROM	
  MESSAGE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  FIND	
  ACCOUNT(ACCOUT_NUMBER)	
  IN	
  DATA	
  BASE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  IF	
  NOT_FOUND	
  	
  |	
  ACCOUNT_BALANCE	
  +	
  DELTA	
  <	
  0	
  THEN	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  PUT	
  NEGATIVE	
  RESPONSE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  ELSE	
  DO;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  ACCOUNT_BALANCE	
  =	
  ACCOUNT_BALANCE	
  +	
  DELTA;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  POST	
  HISTORY	
  RECORD	
  ON	
  ACCOUNT	
  (DELTA);	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  CASH_DRAWER(TELLER)	
  =	
  CASH_DRAWER(TELLER)	
  +	
  DELTA;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  BRANCH_BALANCE(BRANCH)	
  =	
  BRANCH_BALANCE(BRANCH)	
  +	
  DELTA;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  PUT	
  MESSAGE	
  ('NEW	
  BALANCE	
  ='	
  ACCOUNT_BALANCE);	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  END;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  COMMIT;	
  	
  
The transaction concept
	
  DEBIT_CREDIT:	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  BEGIN_TRANSACTION;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  GET	
  MESSAGE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  EXTRACT	
  ACCOUT_NUMBER,	
  DELTA,	
  TELLER,	
  BRANCH	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  FROM	
  MESSAGE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  FIND	
  ACCOUNT(ACCOUT_NUMBER)	
  IN	
  DATA	
  BASE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  IF	
  NOT_FOUND	
  	
  |	
  ACCOUNT_BALANCE	
  +	
  DELTA	
  <	
  0	
  THEN	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  PUT	
  NEGATIVE	
  RESPONSE;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  ELSE	
  DO;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  ACCOUNT_BALANCE	
  =	
  ACCOUNT_BALANCE	
  +	
  DELTA;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  POST	
  HISTORY	
  RECORD	
  ON	
  ACCOUNT	
  (DELTA);	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  CASH_DRAWER(TELLER)	
  =	
  CASH_DRAWER(TELLER)	
  +	
  DELTA;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  BRANCH_BALANCE(BRANCH)	
  =	
  BRANCH_BALANCE(BRANCH)	
  +	
  DELTA;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  PUT	
  MESSAGE	
  ('NEW	
  BALANCE	
  ='	
  ACCOUNT_BALANCE);	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  END;	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  COMMIT;	
  	
  
An application-level contract
Write	
   Read	
  
Application
Opaque
store
Transactions
Pervasive distribution
Pervasive distribution
partial failure
Pervasive distribution
partial failure
asynchrony
Pervasive distribution
CAP
partial failure
asynchrony
Research on consistency
Write	
   Read	
  
Application
Opaque
store
Consistency
models
R1(X=1)	
  R2(X=1)	
  W1(X=2)	
  W2(X=0)	
  
W1(X=1)	
  W1(Y=2)	
  R2(Y=2)	
  R2(X=0)	
  
Research on consistency
Write	
   Read	
  
Application
Opaque
store
Consistency
models
Assert:
balance > 0
causal?
PRAM?
delta?
fork/join?
red/blue?
release?
SC?
(translation)
R1(X=1)	
  R2(X=1)	
  W1(X=2)	
  W2(X=0)	
  
W1(X=1)	
  W1(Y=2)	
  R2(Y=2)	
  R2(X=0)	
  
Meanwhile, in industry…
Application
Opaque
store
Custom solutions
Write	
   Read	
  
Meanwhile, in industry…
Application
Opaque
store
Custom solutions
Assert:
balance > 0
(prayer)
Write	
   Read	
  
Distributed consistency:
staying relevant
•  Is this an important problem?
•  Is academia disconnected from reality?
•  OK, what now?
Goal: help programmers write correct applications.
Today: some promising approaches
Case study: a graph
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
Partitioned, for scalability
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
Replicated, for availability
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
Problem: deadlock detection
Task: Identify strongly-connected
components
Waits-for graph
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
Problem: garbage collection
Task: Identify nodes not reachable
from Root. Root	
  
Refers-to graph
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
Correctness
Deadlock detection
•  Safety: No false positives-
•  Liveness: Identify all deadlocks
Garbage collection
•  Safety: Never GC live memory!
•  Liveness: GC all orphaned memory
T1
T2
T4
T10
T3
T6
T5
T9
T7
T8
T11
T12
T13
T14
ParLLon	
  
Root	
  
Consistency at the extremes
Storage
Object
Flow
Language
Application
Linearizable
key-value store?
	
  
	
  Custom solutions?
Consistency at the extremes
Storage
Object
Flow
Language
Application
Linearizable
key-value store?
	
  
	
  Custom solutions?
Efficient Correct
Consistency across the stack
Object-level consistency
Capture semantics of data structures that
•  allow greater concurrency
•  maintain guarantees (e.g. convergence)
Storage
Object
Flow
Language
Application
Insert	
   Read	
  
Convergent
data structure
(e.g., Set CRDT)
Object-level consistency
Insert	
   Read	
  
Commutativity
Associativity
Idempotence
Reordering
Batching
Retry/duplication
Tolerant to
Application
Convergent
data structures
Object-level consistency
?	
   ?	
  
GC Assert:
No live nodes are reclaimed
Assert:
Graph replicas
converge
Flow-level consistency	
  
Storage
Object
Flow
Language
Application
Flow-level consistency	
  
Capture semantics of data in motion
•  Asynchronous dataflow model
•  component properties à system-wide guarantees
Graph
store
Transaction
manager
Transitive
closure
Deadlock
detector
Confluent ConfluentConfluent
Flow-level consistency
Order-insensitivity (confluence)
output	
  set	
  =	
  f(input	
  set)	
  	
  	
  
{	
  	
  	
  	
  	
  	
  	
  	
  }	
  
{	
  	
  	
  	
  	
  	
  	
  	
  }	
  
=	
  
Flow-level consistency
Confluence is compositional
output	
  set	
  =	
  f	
  Ÿ	
  g(input	
  set)	
  	
  	
  
Flow-level consistency
Confluence is compositional
output	
  set	
  =	
  f	
  Ÿ	
  g(input	
  set)	
  	
  	
  
Graph
store
Memory
allocator
Transitive
closure
Garbage
collector
Confluent Not
Confluent
Confluent
Graph
store
Transaction
manager
Transitive
closure
Deadlock
detector
Confluent ConfluentConfluent
Graph queries as dataflow
Graph
store
Memory
allocator
Transitive
closure
Garbage
collector
Confluent Not
Confluent
Confluent
Graph
store
Transaction
manager
Transitive
closure
Deadlock
detector
Confluent ConfluentConfluent
Graph queries as dataflow
Confluent
Coordinate	
  here	
  
Language-level consistency	
  
DSLs for distributed programming?
•  Capture consistency concerns in the
type system
	
  
	
  
Storage
Object
Flow
Language
Application
Language-level consistency	
  
CALM Theorem:
Monotonic à confluent
Conservative, syntactic test for confluence
	
  
Language-level consistency
Deadlock detector
Garbage collector
Language-level consistency
Deadlock detector
Garbage collector
nonmonotonic	
  
Storage
Object
Flow
Language
Application
correct
reusable
efficient
intuitive
Where we’ve been;
where we’re headed
Storage
Object
Flow
Language
Application
correct
reusable
efficient
intuitive
Where we’ve been;
where we’re headed
Storage
Object
Flow
Language
Application
correct
reusable
efficient
intuitive
Where we’ve been;
where we’re headed
Remember
•  Consistency is an application-level property
•  Correctness and performance are compatible
•  Meet programmers on their home turf
•  Build bridges!
Queries?

Más contenido relacionado

Similar a consistency without borders

Scalability: Rdbms Vs Other Data Stores
Scalability: Rdbms Vs Other Data StoresScalability: Rdbms Vs Other Data Stores
Scalability: Rdbms Vs Other Data Stores
Ramki Gaddipati
 
Kudler has plenty of room to increase sales while controlling cost.docx
Kudler has plenty of room to increase sales while controlling cost.docxKudler has plenty of room to increase sales while controlling cost.docx
Kudler has plenty of room to increase sales while controlling cost.docx
DIPESH30
 

Similar a consistency without borders (20)

Improving application design with a rich domain model (springone 2007)
Improving application design with a rich domain model (springone 2007)Improving application design with a rich domain model (springone 2007)
Improving application design with a rich domain model (springone 2007)
 
Unit 5
Unit 5Unit 5
Unit 5
 
How to Send a Receipt, Topics in Concurrency and Distributed Systems
How to Send a Receipt, Topics in Concurrency and Distributed SystemsHow to Send a Receipt, Topics in Concurrency and Distributed Systems
How to Send a Receipt, Topics in Concurrency and Distributed Systems
 
Spring Transaction
Spring TransactionSpring Transaction
Spring Transaction
 
Transaction
TransactionTransaction
Transaction
 
Forward Chaining in HALO
Forward Chaining in HALOForward Chaining in HALO
Forward Chaining in HALO
 
Ch 9
Ch 9Ch 9
Ch 9
 
Dynamic Itemset Counting
Dynamic Itemset CountingDynamic Itemset Counting
Dynamic Itemset Counting
 
Dynamic Itemset Counting
Dynamic Itemset CountingDynamic Itemset Counting
Dynamic Itemset Counting
 
Using R to Characterize Hedge Fund Clustering
Using R to Characterize Hedge Fund ClusteringUsing R to Characterize Hedge Fund Clustering
Using R to Characterize Hedge Fund Clustering
 
Practical approach for testing your software with php unit
Practical approach for testing your software with php unitPractical approach for testing your software with php unit
Practical approach for testing your software with php unit
 
Droid on Chain - Berry Ventura Lev, Kik
Droid on Chain - Berry Ventura Lev, KikDroid on Chain - Berry Ventura Lev, Kik
Droid on Chain - Berry Ventura Lev, Kik
 
Dependency injection - the right way
Dependency injection - the right wayDependency injection - the right way
Dependency injection - the right way
 
Building high productivity applications
Building high productivity applicationsBuilding high productivity applications
Building high productivity applications
 
"An introduction to object-oriented programming for those who have never done...
"An introduction to object-oriented programming for those who have never done..."An introduction to object-oriented programming for those who have never done...
"An introduction to object-oriented programming for those who have never done...
 
Scalability: Rdbms Vs Other Data Stores
Scalability: Rdbms Vs Other Data StoresScalability: Rdbms Vs Other Data Stores
Scalability: Rdbms Vs Other Data Stores
 
0273685988 ch13
0273685988 ch130273685988 ch13
0273685988 ch13
 
Kudler has plenty of room to increase sales while controlling cost.docx
Kudler has plenty of room to increase sales while controlling cost.docxKudler has plenty of room to increase sales while controlling cost.docx
Kudler has plenty of room to increase sales while controlling cost.docx
 
Event Sourcing with Cassandra (from Cassandra Japan Meetup in Tokyo March 2016)
Event Sourcing with Cassandra (from Cassandra Japan Meetup in Tokyo March 2016)Event Sourcing with Cassandra (from Cassandra Japan Meetup in Tokyo March 2016)
Event Sourcing with Cassandra (from Cassandra Japan Meetup in Tokyo March 2016)
 
DataWeave 2.0 - MuleSoft CONNECT 2019
DataWeave 2.0 - MuleSoft CONNECT 2019DataWeave 2.0 - MuleSoft CONNECT 2019
DataWeave 2.0 - MuleSoft CONNECT 2019
 

Último

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Último (20)

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 

consistency without borders

Notas del editor

  1. 2 things happened: academia ran down the stack, creating new I/O-level models with no accompanying intuitive contract for apps.Industry ran up the stack, developing best practices for app-level consistency without support (or guarantees!) from infrastructure
  2. Recover common ground with programmers that was lost with the abandonment of xacts
  3. To guard against failures, we replicate.NB: asynchrony =&gt; replicas might not agree
  4. Very similar looking criteria (1 safe 1 live). Takes some work, even on a single site. But hard in our scenario: disorder =&gt; replica disagreement, partial failure =&gt; missing partitions
  5. recall “the developer’s conundrum.” essentially 3 choices: 1) SC, meaningful at app level,
  6. Ie, reorderability, batchability, tolerance to duplication / retryNow programmer must map from application invariants to object API (with richer semantics than read/write).
  7. All API calls to add() commute; no CC necessaryApproach: adjacency lists as setsHowever, not sufficient to synchronize GC.
  8. However, not sufficient to synchronize GC.Perhaps more importantly, not *compositional* -- what guarantees does my app – pieced together from many convergent objects – give?To reason compositionally, need guarantees about what comes OUT of my objects, and how it transits the app.
  9. We are interested in the properties of component *outputs* rather than just internal state. Hence we are interested in a different property: confluence.A confluent module behaves like a function from sets (of inputs) to sets (of outputs)
  10. Confluence is compositional: Composing confluent components yields a confluent dataflow
  11. All of these components are confluent! Composing confluent components yields a confluent dataflowBut annotations are burdensome
  12. M – a semantic property of code – implies confluenceAn appropriately constrained language provides a conservative syntactic test for M.
  13. M – a semantic property of code – implies confluenceAn appropriately constrained language provides a conservative syntactic test for M.
  14. Also note that a data-centric language give us the dataflow graph automatically, via dependencies (across LOC, modules, processes, nodes, etc)
  15. Consistency models and mechanisms are only useful if they help programmers write correct applications.