2. Performance Testing & Instrumenting
the KGB SMS Service
for the Super Bowl
J. Nathan Matias
Cambridge Geek Night, 3 August 2010
Cambridge Union Society, Cambridge UK

6. Acquisition

7. Launching in
Acquisition
New Markets

8. Launching in Super Bowl
Acquisition
New Markets TV Ad

15. Emberlight

18. KGB SMS Answers

19. To: 118118
Text:
Wot is the
wether 4cast
4 london
tonite

20. To: 118118 118118:
Text: Weather
Wot is the forecast for
wether 4cast London …
4 london
tonite

26. MX Rails
Telecom Server
Customers

27. MX Rails Rails Rails
Telecom Server Server Server
Customers Databases
Specialized
Servers

28. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
http: Aethernet, EngineYard
MX Rails Rails Rails
Telecom Server Server Server
Customers Databases
Specialized
Servers

29. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
http: Aethernet, EngineYard
MX Rails Rails Rails
Telecom Server Server Server
Customers Databases
Specialized
Servers

30. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails
Telecom Server Server Server
Firefox
Customers Databases
Specialized
Business
Servers Reporting

31. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails
Telecom 7 web applications
Server Server Server
Firefox
Customers Databases
Specialized
Business
Servers Reporting

32. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
over a thousand
controller actions
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails
Telecom 7 web applications
Server Server Server
Firefox
Customers Databases
Specialized
Business
Servers Reporting

33. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
over a thousand
controller actions
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails
Telecom 7 web applications
Server Server Server
Firefox
Customers over a thousand
Databases
Specialized
Business
models Servers Reporting

34. Agents
Firefox Firefox Firefox Firefox Firefox
Javascript
over a thousand
controller actions
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails
Telecom 7 web applications
Server Server Server
SMS network Firefox
Scalability
Customers over a thousand
Databases
Specialized
Business
models Servers Reporting

35. Agents
Internet + Browser
Firefox Scalability
Firefox Firefox Firefox Firefox
Javascript
over a thousand
controller actions
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails
Telecom 7 web applications
Server Server Server
SMS network Firefox
Scalability
Customers over a thousand
Databases
Specialized
Business
models Servers Reporting

36. Agents
Internet + Browser
Firefox Scalability
Firefox Firefox Firefox Firefox
Javascript
over a thousand
controller actions
http: Aethernet, EngineYard
Firefox
MX Rails Rails Rails Complex
Telecom 7 web applications
Server Server Server
SMS network Business
Firefox
Scalability Reports
Customers over a thousand
Databases
Specialized
Business
models Servers Reporting

40. Testing Software

41. Performance Testing Services

42. GUI Based Programmable
screenshot via
www.lynchconsulting.com.au/blog/

43. GUI Based Programmable
es
Conte nt Websit
screenshot via
www.lynchconsulting.com.au/blog/

44. GUI Based Programmable
es s
Conte nt Websit Web A pplication
screenshot via
www.lynchconsulting.com.au/blog/

45. GUI Based
Web HTTP
serve web
Browser Request
content
A arrives
Web HTTP
serve web bs ites
Browser Request ntent We
arrives
content Co
B
screenshot via
www.lynchconsulting.com.au/blog/

46. Customer question
A arrives

47. agent
Customer question
takes
A arrives
question

48. agent agent
Customer question
takes checks
A arrives
question sources

49. agent agent agent
Customer question
takes checks answers
A arrives
question sources question

50. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
Customer followup
A question

51. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2
Customer followup
takes
A question
question

52. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2
Customer followup
takes checks
A question
question history

53. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation

54. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation
second
Customer
half of
A
question

55. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation
second agent 3
Customer
half of takes
A
question question

56. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation
agent 4
takes
conversation
second agent 3
Customer
half of takes
A
question question

57. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation
agent 4
takes
conversation
second agent 3 agent 3
Customer
half of takes attaches
A
question question question

58. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation
agent 4 agent 4
takes receives
conversation 2nd half
second agent 3 agent 3
Customer
half of takes attaches
A
question question question

59. agent agent agent
Customer question
takes checks answers
A arrives
question sources question
agent 2 agent 2 agent 2
Customer followup
takes checks attaches
A question
question history conversation
agent 4 agent 4
takes receives
conversation 2nd half
agent 4
second agent 3 agent 3 answers
Customer conversation
half of takes attaches
A
question question question

60. Collaborative systems require
programmable performance testing
ns
Web Applicatio

61. Amazon
EC2
Grinder Grinder Grinder Grinder
Replica Cluster

62. Amazon
EC2
Agents Agents Agents SMS!
Replica Cluster

63. coordinated simulation
Amazon
EC2
Agents Agents Agents SMS!
Replica Cluster

64. coordinated simulation
e
Unreliabl
Amazon reports!
EC2
Agents Agents Agents SMS!
Replica Cluster

65. coordinated simulation
e
Unreliabl
Amazon reports!
EC2
Agents Agents Agents SMS!
Replica Cluster
Real Pe
rforma
Metrics nce

66. coordinated simulation
e
Unreliabl
Amazon reports!
EC2
Agents Agents Agents SMS!
Replica Cluster
Real Pe
rforma
Business Metrics nce
Reports!

67. Launching in
Acquisition
New Markets

68. Engine Yard
kgb Clusters
multiple
virtual slices
multiple
application servers
Database per slice

69. Engine Yard
kgb Clusters
Load Balancer
multiple
virtual slices
App multiple
application servers
Database per slice

70. Engine Yard
kgb Clusters
Load Balancer
physical
cluster node
App App App
Database Database Database

71. Engine Yard
kgb Clusters
Load Balancer
Caching physical
cluster node
URL App App App
Rewriting
(XSS)
Database Database Database

72. Load Balancer
App App App App App App
Database Database Database

73. View
Queue
Business
Reports
receive take send
question question answer
unexpected behaviour

74. !
Pa inful
View
Queue
Business
Reports!
receive take send
question question answer
unexpected behaviour

75. Launching in Super Bowl
Acquisition
New Markets TV Ad

80. Should we run the ad?

81. Should we run the ad?
How should we try
to survive the ad?

82. Should we run the ad?
How should we try
to survive the ad?
Optimize!

84. Instrumentation

85. Rails:

86. Rails:
Database:

93. The kgb Wizards
Super Bowl Team
Paul Butcher Matt Nodder
Craig Emery Frederick Cheung
Matthew Rudy Jacobs Henry Addison
Gary Swanson Ben Symons
Robert Heaney Tim Parish
J. Nathan Matias

94. Partners
in Optimization

95. Lessons
Launching in Super Bowl
Acquisition
New Markets TV Ad

96. NewRelic Up Front

97. Performance Testing
in Agile Teams

98. when to shield the tester
when to expose the tester

99. Plugs

100. paul@touchtype-online.com

102. Emberlight

105. matt.nodder@kgb.com

106. Stuff to read
• my RailsInside Article
• Performance Testing for Web Applications (MSDN)
(Meier, Farre et al)(http://perftestingguide.codeplex.com/)
• Improving .NET Application Performance &
Scalability (Meier, Vasireddy, et al)
• Statistical Principles for the Performance Tester (Scott Barber)
• Lessons from Giant Scale Services (Eric Brewer)
• MySQL Performance Blog (Percona)
• NewRelic Blog (New Relic)
• Using the MySQL Query Profiler (mysql)