Presentation given at DevLink 2010

1. Dropping
Building Scalable Systems That Work
Chris Patterson

2. Chris Patterson
@P h atB o yG

5. Agenda
Scalability 2010
Availability 2010
Consistency (circa 1983)
Consistency (eventually 2010)

6. Scalable

7. Scalable
A system that
Can handle growth without failure
Can be expanded to handle even greater growth

8. Scale Up

9. Scale Up
Replace existing system with larger, faster system
At some point, growth exceeds capacity
Leads to vendor lock-in

10. Scale Out

11. Scale Out
Functional Partition
Increase overall capacity by separating bounded
contexts
Data Partitioning
Increase capacity within a functional partition

12. Scale Out, Industry Trends
Cloud
Elastic Expansion
Rich Internet Applications
HTML5/JS, as well as Flex/Silverlight
So-ware as a Service

13. That Work

14. Available
A system that
Is in an operable state
Can provide expected functionality when needed

15. Living in the Clouds
Available
24/7, online, anywhere
Response time is a dimension of availability
95th percentile is key latency metric

16. Partial Availability
Data Partitioning
A single database failure only impacts data on that
server
System can be partially available for a subset of
customers

17. 2
PICK

18. Project Triangle
Cost Scope
Schedule

19. Dating Triangle
Intelligent Attractive
Emotionally
Stable

20. Brewer’s Theorem
Consistency Availability
Partition
Tolerance
Credited to Professor Eric A. Brewer

21. Scalable Systems That
A system that is
Available
Scalable
Partition Tolerant

22. Consistency
Changes to values are consistent in a system
An updated address
A changed telephone number
An additional family member
The inventory level of an item

23. Atomic
ACID

24. Atomic

25. m ic
A to

26. Atomic

27. Atomic
All or Nothing

28. Atomic
Maintain referential
integrity
Rollback incomplete
transactions
Ensure consistent state

29. Atomic
Modifications are not
accessible to concurrent
transactions
Lack of isolation can
lead to inconsistent state

30. Atomic
Completed transactions
survive system failures

31. ACID Defined, circa 1983
State of the art system
Single machine
Vertically scaled up
DB/2 (or even ISAM)

32. Distributed Transactional
2PC
Two-phase commit protocol
Averages availability of involved systems
System A (99.9%, 43m) x System B (99.9%, 43m)
Results in combined availability of 99.8% (86m)

33. Temporal Inconsistency
If I pick up a book at the bookstore, and walk around
with it for an hour, another customer is unable to
purchase that book.
Once I decide to buy in on Amazon instead, a
bookstore employee will gladly replace the book on
the shelf
I go on enjoying my coffee knowing I saved $4.20

34. Eventual Consistency
Relax temporal consistency
We already do this
IsolationLevel.ReadUncommited
memcached
CDN

35. Eventual Consistency
Make temporal consistency windows explicit
Command-Query Responsibility Segregation
(it’s just command query separation)
www.msteched.com/2010/NorthAmerica/ARC302
Persistent View Model

36. Keep Transactions Internal
Transactions should not cross service boundaries
Avoid adding remote resources to the DTC
Avoid cross-database transactions

37. Protocols
Create protocols for cross-functional transactions
Maintain the state of each exchange in each
participating functional partition
Example in action
Magnum.StateMachine
Magnum.Channels

38. Simple Transaction
Created Completed

39. Created Do Work Completed

40. Created Do Work Completed
FAIL! Failed

41. Pick Up Book Buy Book Sale
Replace Book No Sale

42. Pick Up Book Buy Book Sale
Replace Book No Sale
Damage Book Loss

43. Protocols
Long-lived transaction (saga, workflow)
Orchestration of correlated
Commands
Events
Handling of compensations
Some things can’t be compensated

44. Sagas
When(RequestReceived)
.Then((saga, request) =>
{
saga.OrderId = request.OrderId;
saga.CustomerId = request.CustomerId;
})
.Publish(saga => new SendOrderDetailsRequest
{
RequestId = saga.CorrelationId,
CustomerId = saga.CustomerId,
OrderId = saga.OrderId,
})
.TransitionTo(WaitingForResponse));

45. Sagas
During(WaitingForResponse,
When(ResponseReceived)
.Then((saga, response) =>
{
saga.OrderCreated = response.Created;
saga.OrderStatus = response.Status;
})
.Publish(saga => new OrderDetails
{
CustomerId = saga.CustomerId,
OrderId = saga.OrderId,
Created = saga.OrderCreated.Value,
Status = saga.OrderStatus,
})
.TransitionTo(Completed));

46. Sagas
State Initial { get; set; }
State WaitingForResponse { get; set; }
State Completed { get; set; }

47. Sagas
Event<RetrieveOrderDetails> RequestReceived { get; set; }
Event<OrderDetailsResponse> ResponseReceived { get; set; }
Event<OrderDetailsRequestFailed> RequestFailed { get; set; }

48. Sagas
public virtual string CustomerId { get; set; }
public virtual string OrderId { get; set; }
public virtual OrderStatus OrderStatus { get; set; }
public virtual DateTime? OrderCreated { get; set; }

49. Durable Messaging
Asynchronous
Temporal decoupling increases availability
Supports eventual consistency
Given no further changes in state, the system will
become consistent

50. Idempotent
Uniquely identify every command
Allows once-and-only-once completion

51. Search Keyword: BASE
Basically Available
So--state
Eventually Consistent

52. Questions?

53. Contact
chris [@phatboyg] .com
http://blog.phatboyg.com
http://phatboyg.lostechies.com