Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/20SN0dP. Tammer Saleh talks about the mistakes people make when building a microservices architecture. He also talks about: when microservices are appropriate, and where to draw the lines between services, dealing with performance issues, testing and debugging techniques, managing a polyglot landscape and the explosion of platforms, managing failure and graceful degradation. Filmed at qconlondon.com. Tammer Saleh is a long time developer, leader, and author of the acclaimed book *Rails AntiPatterns*. Saleh is currently building the Cloud Foundry platform at Pivotal.

1. Quick poll…

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/cloud-anti-patterns

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. Welcome to sunny London!

5. Tammer Saleh
Geek: Unix, Ruby, Golang, etc
Cloud Foundry @ Pivotal
http://tammersaleh.com | tsaleh@pivotal.io

6. Microservice
Anti-patterns
How not to go down in flames.

7. Why microservices?
What is a microservice, and why do I care?

8. Monolithic
Entire application in a single codebase,
deployed and scaled as a single unit.

9. Monolithic
Hard to scale the application.
Impossible to scale the team.

10. It’s not about code… It’s about teams.

11. But it can go wrong.
Here are the most common problems we
see in the wild, and how to fix them.

12. Overzealous Services
The most common
mistake is to
start with
microservices

13. Boring is Beautiful™

14. Solution: Start monolithic and extract
Microservices are complex and add a constant tax to development.
Build a boring application and extract services as needed.

15. Twitter
/tweet
/dashboard
/…
/…
Heavy and Expensive

16. /tweet
Twitter
/dashboard
/…
/…
Lite and Cheap

17. Congratulations
You’re now a microservice architect.

18. Schemas everywhere
/dashboard
/…
/…
/tweet v1
v1
V1 Schema
Deploy v2

19. Schemas everywhere
/dashboard
/…
/…
/tweet v1
v2
V1 Schema
Deploy v2
Migrate v2

20. /dashboard
/…
/…
/tweet v1
v2
V2 Schema
Deploy v2
Schemas everywhere
Migrate v2

21. /dashboard
/…
/…
/tweet v2
v2
V2 Schema
Schemas everywhere

22. Solution: Gatekeeper
/dashboard
/…
/…
/tweet v1
v2
Tweets
/bulk_add
Tweet Service
GET /tweets
GET /tweets/ID
Owns database and migrations

23. Lock-step deployment
Service B
Service A
Service C — v1
v1
v1

24. Lock-step deployment
Service B
Service A
Service C — v2
v1
v1
Deploy v2

25. Lock-step deployment
Service B
Service A
Service C — v2
v1
v1
Deploy v2

26. Lock-step deployment
Service B
Service A
Service C — v2
v2
v2
Deploy v2

27. Solution: Semantic Versioning
Service B
Service A
Service C — v1
v1
v1
vMajor.Minor.Patch
MYBAD.SHINY.OOPS

28. Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1
v1
Deploy v1.2
(extra stuff)
:)
:)

29. Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1.2
v1.2
Deploy v1.2

30. Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1.2
v1.2
Deploy v2
(breaking change)
Service C — v2

31. Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1.2
v2
Service C — v2
Deploy v2

32. Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v2
v2
Service C — v2
Deploy v2

33. OMG ALL THE STEPS!!!
See Rule #1

34. Spiky load between services
Must maintain enough servers to handle peak load.

35. /tweet
/dashboard
/…
/…
Spiky load between services

36. /tweet
/dashboard
/…
/…
Spiky load between services

37. Queues in between services provide buffers that smooth traffic.
Solution: Amortize via queues
Peak load is now much lower.

38. /tweet
/dashboard
/…
/…
Solution: Amortize via queues

39. Solution: Amortize via queues
/tweet
/dashboard
/…
/…
Worker
Complexity: Now clients must deal with asynchronous responses.

40. Hardcoded IPs and Ports
Simple to get started, but immediately leads to deployment issues.

41. Solution 1: Discovery Service
etcdconsul

42. Solution 1: Discovery Service
Service A
Service B
(192.168.0.2:1234)
Where is B?
192.168.0.2:1234

43. Solution 1: Discovery Service
Service A
Service B
(192.168.0.2:1234)
Hey there!

44. Solution 1: Discovery Service
Complexity: Your code must understand the service lookup system.

45. Solution 2: Centralised router
Service AService B
Router
Service B Service A Service AService B
A A BB A A BBA A BB

46. Solution 2: Centralised router
Simplicity: “It’s just DNS.”

47. Router vs Discovery Service
Both require service registration.
Discovery Service is simpler to build
and scale, since it doesn’t need to
route all data.
Discovery service does fewer
network hops.
Router is transparent.
Router can be exposed externally.
Router can cache transparently.
Router can round-robin.
Both require HA and scalability.

48. Dogpiles
Service BService A
You there?
No.
How about now?
I’m busy.
But I wanna chat!
Go away
Whachya doin?
Please leave me alone.
Wanna be friends?
I hate you.

49. Dogpiles
Service B
Service A Service A
Service A Service A
x 100

50. Solution: Circuit Breaker
Service B
Service A Service A
Where’s B?
B’s sick. Wait 10 min.
Hello!
Go to Hell!
Hey, B’s sick :(

51. Debugging hell
Turns out, distributed systems are hard.

52. Debugging Hell
Service 3
Service 1 Service 2
time:1428374783 source:service3 msg:”Error with payment!”
Where?? Why?? Who??

53. Solution: Correlation IDs
Service
Service Service
CID: 1234 CID: 1234
CID:1234
time:1427127483 source:service1 id:1234 msg:”Received request…”
time:1427348748 source:service2 id:1234 msg:”Processing payment"
time:1428374783 source:service3 id:1234 msg:”Error with payment!”

54. Solution: Correlation IDs
1. Tag all incoming requests with
unique ID
2. Service saves ID for all incoming
requests
3. Include that ID in all log lines,
etc.
4. Tag new requests with that ID
ServiceLogs
ID
ID
ID
ID
Complexity: Must be done manually.

55. Missing Mock Servers
Each consuming team has to create
their own mocks and stubs.

56. Missing Mock Servers
Service A
Team A

57. Missing Mock Servers
Service A Service B
HTTP
Team A Team B
Product Surface Area

58. Missing Mock Servers
Service A Service B
HTTP
Mock of Service A
HTTP
Team A Team B
For each service…
For each team…

59. Service A Service B
HTTP
Mock of Service A
HTTP
Team A Team B
Better…Solution: Service Team Provides the Mock
Team B still needs to know
how to run Mock Service A

60. Solution: Service Team Owns the Client
Service A Service B
HTTP
H
TTP
Team A Team B
Client A
MOCK=“true”
Best…
Mock of Service A
Product Surface Area
Both modes are tested in CI
Team A can change
the protocol as they see fit

61. Flying Blind

62. Solution: Graphs, alerts, pages.

63. Solution: Graphs, alerts, pages.
Alerting Rules EnginePlatform Events
OMGWAT???AppsAppsAppsAppsAppsAppsAppsAppsAppsAppsApps

64. Snowflakes

65. Snowflakes
Ruby XSS exploit!
BASH remote exploit!

66. Snowflakes
Containers
Virtual Machines

67. Solution: Golden Image

68. Solution: Golden Image
Golden OS Image
Common Runtimes
and Frameworks
+
Base Platform
=

69. Doomsday Deployments

70. Solution: Predictable Pipelines

71. Solution: Predictable Pipelines
http://concourse.ci

72. Solution: Predictable Pipelines
Need to trust your tests, your
platform, and your automation.

73. Operational Explosion!

74. Operational Explosion!
Operations block Development

75. Solution: AUTOMATE ALL THE THINGS!!!!!
Form a team to build tools that enable developers to
manage the system in an entirely automated way.

76. OMG ALLTHETHINGS????
App deployment, infrastructure provisioning, OS installation,
configuration management, database provisioning, disaster
recovery, application monitoring, HA, blue-green deployments,
self-healing, scaling, runtime installation, log rotation, backups, security
updates, database upgrades, application logs, system logs, database
logs, continuous integration, continuous deployment, service discovery,
monitor queue usage, security monitoring, hotspot detection, error monitoring,
issue notification and escalation, virtual machine migration, shard rebalancing,
circuit breaker monitoring, resiliency testing, database snapshots, flux capacitors,
ion overdrive maintenance, change the oil, dog feeding, cat shooting, pig eating…

77. Time and Money
Solution: AUTOMATE ALL THE THINGS!!!!!

79. In summary…
Start boring and extract to services.
Understand the hidden schemas.
Amortize traffic with queues.
Decouple through discovery tools.
Contain failures with circuit breakers.
Enable other teams through mockable
clients.
Kill your snowflakes.
Automate your deployments.
Build in operations tools from the beginning.
Make use of a platform like Cloud Foundry.

80. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/cloud-
anti-patterns