Many presentations on Microservices offer a high level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some DevOps work and start learning about deployment pipelines, metrics, and logging. Don’t panic. In this presentation we’ll discuss what we learned over the past three years. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent on various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about team communication and organization... and how they are likely the most important tool for surviving a Microservices development team.

1. Surviving In A Microservices
Environment
Steve Pember
CTO, ThirdChannel
Gr8Conf US, 2017
@svpember

2. @svpember
Microservice Blog Posts & Presentations
• Microservices: Yay!
• Microservices: Boo!
• Smashing The Monolith (and here’s how we did it)
• Microservices + Technology X
• Microservices + Methodology Y

3. Microservices have many lessons to offer

4. What even are Microservices?

6. @svpember
Why Choose Microservices?
• Reduce Coupling!
• Right Tool for the Job
• Continuous Delivery
• Smaller codebases are easier to reason about
• Easy to replace old services
• Efficient Scaling
• Can move quickly (once you’re up and running)

9. Help?

10. @svpember
Microservice Topics
• Infrastructure
• Architecture
• Team Communication
• Miscellaneous Advice

11. @svpember
Microservice Topics
• Infrastructure

12. @svpember
Infrastructure
• How do we manage the logs?

13. Centralized Logs are your
#1 Priority

14. @svpember

15. @svpember

16. @svpember

17. @svpember

18. @svpember

19. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?

20. You’d best be monitoring your platform.

22. –Johnny Appleseed
“Type a quote here.”

24. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?

26. @svpember

27. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?
• How/where are our builds done?

28. @svpember

29. @svpember

30. @svpember

31. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?
• How/where are our builds done?
• Do we have any coding conventions?

32. Check Style? Coverage?

33. @svpember

34. @svpember

35. @svpember

36. @svpember

37. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?
• How/where are our builds done?
• Do we have any coding conventions?
• Can I generate a Service template?

38. wget https://github.com/thirdchannel/base-template/
archive/release.zip
(This is not a real url, but you get the idea)

39. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?
• How/where are our builds done?
• Do we have any coding conventions?
• Can I generate a Service template?
• How do we share code?

40. Whoops! Gotcha!

42. @svpember
Sharing Code
• It’s OK to reimplement functionality within each service
• Setup your own internal Artifactory!
• DO share infrastructure libraries (e.g. communications)
• NEVER share Domain or business logic libraries

43. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?
• How/where are our builds done?
• Do we have any coding conventions?
• Can I generate a Service template?
• How do we share code?
• How do we manage our (multiple) environments?

44. @svpember

45. Test & Develop in Isolation

46. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How do we deploy our code?
• How/where are our builds done?
• Do we have any coding conventions?
• Can I generate a Service template?
• How do we share code?
• How do we manage our (multiple) environments?
• Do our Devs get time to work on Infrastructure?

47. @svpember
Microservice Topics
• Infrastructure
• Architecture

48. @svpember
Architecture
• Do we have an overall design vision?

49. Does Anyone Know What We’re Doing?!

52. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?

53. @svpember

54. @svpember

55. Be Careful When Choosing
New Tech

56. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?

57. Integration Tests are the Best Tests

58. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?

59. @svpember
Test Environment
• Run periodically (e.g. nightly)
• Each service generates fixture data
• Service data reset after EACH test

60. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?

61. HTTP vs Async Events

62. @svpember
HTTP
• Well Established
• Built In libraries
• Existing structure for response codes (2**, 4**, 5**, etc)
• Synchronous
• Increases coupling
• Requires services to know which others require their data
• Has dependency on Service Discovery mechanism
• Susceptible to Data Loss

63. @svpember
Events
• Asynchronous
• Pub-sub / Fire and forget
• Loose Coupling
• Prevents Data Loss
• Allows for Reactive systems
• No existing structure for response error handling
• Dependency on Message Broker technology
• Can be difficult for Junior folks to understand

64. Ensure you have Circuit Breaker or a Dead
Letter Mechanism

65. @svpember

66. @svpember
<3 RabbitMq
• Intelligent broker; dumb consumers
• Highly nuanced & robust routing
• Backpressure
• Dead letters /retries
• Message Ordering
• Multiple deliveries
• Ack / Nack/ Reject … re-enqueue

67. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?
• How/where is our data persisted?

68. @svpember
Locality of Reference
• Spatial: “How close together is our data?”

69. @svpember
Locality of Reference
• Spatial: “How close together is our data?”
• Temporal: “How often is our data accessed?”

70. @svpember

71. @svpember

72. Beware the ‘Mini-Monolith’

73. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?
• How/where is our data persisted?
• Do we follow an overall architectural style?

74. Make Sure Everyone Is Aware

75. Applies to Both Intra- and Inter-Service

76. It all
Started
With =>

77. @svpember
Domain Driven Design
• Ubiquitous Language
• Entities
• Aggregates
• Bounded Contexts
• No Direct communications across Boundaries

78. @svpember

79. @svpember

80. … And CQRS

81. @svpember

82. @svpember

83. Allows for interesting
Architectures

85. While we’re at it, go Reactive

86. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?
• How/where is our data persisted?
• Do we follow an overall architectural style?
• When do we create new services?

87. DDD to the rescue

89. @svpember
New Service?
• Initially: design platform around Bounded Contexts, create services from
inner Contexts
• Don’t create services ‘just because’
• Do make an effort to identify boundaries
• Ensure a service has/will have proper context boundaries before creating
• If two services need to communicate synchronously or frequently, good
candidates for MERGING

90. # of services < # of developers

91. @svpember
Microservice Topics
• Infrastructure
• Architecture
• Team Communication

92. @svpember
Team Communication
• How are our teams structured?

93. Conway’s Law is REAL

94. –Johnny Appleseed
“Type a quote here.”
DBA
Engineers
QA
UX

95. @svpember

96. @svpember
Team Communication
• How are our teams structured?
• Who owns which Service?

97. Teams should be Owners

98. @svpember
Team Communication
• How are our teams structured?
• Who owns which Service?
• What’s our process for merging code?

100. @svpember
Team Communication
• How are our teams structured?
• Who owns which Service?
• What’s our process for merging code?
• What’s our process for releasing code?

101. @svpember

102. @svpember
Team Communication
• How are our teams structured?
• Who owns which Service?
• What’s our process for merging code?
• What’s our process for releasing code?
• What’s our process for monitoring code in production? Dealing with bugs?

103. @svpember

104. –Johnny Appleseed
“Type a quote here.”

105. @svpember
Team Communication
• How are our teams structured?
• Who owns which Service?
• What’s our process for merging code?
• What’s our process for releasing code?
• What’s our process for monitoring code in production? Dealing with bugs?
• Do we add more process if something goes wrong?

106. Be Reluctant to Add
New Process

107. High Freedom / High Responsibility

108. @svpember
Microservice Topics
• Infrastructure
• Architecture
• Team Communication
• Miscellaneous Advice

109. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services

110. Any idea
what these
do?

111. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see

112. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready

113. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready
• If a service A has another service B as a dependency => release B first

114. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready
• If a service A has another service B as a dependency => release B first
• How to bootstrap a new service?

115. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready
• If a service A has another service B as a dependency => release B first
• How to bootstrap a new service?
• Don’t release Friday afternoons

116. In Summary

117. Most of the Challenge is In infrastructure

118. <3
Microservices

119. Please share your survival tips!

120. Any Questions?
Anything I missed?

121. Thank you!
@svpember

122. @svpember
Images
• Bad Odds (Jon Snow): https://www.reddit.com/r/gameofthrones/comments/4owm95/s6e9_megathread_gifwallpaperscreenshot_requests/
• Say Anything / John Cusak: http://www.glamour.com/story/happy-birthday-john-cusack-and
• Wilderness Survival Guide: https://www.kobo.com/us/en/ebook/the-wilderness-survival-guide-1
• XKCD: Code Review: https://xkcd.com/1513/
• Monolithic vs Microservices: @alvaro_sanchez
• Picard FacePalm: https://www.flickr.com/photos/30418788@N03/8381426895
• Council of Ricks: https://www.reddit.com/r/rickandmorty/comments/3exy00/it_looks_like_there_might_some_sort_of_council_of/
• Drago (break you): https://giphy.com/gifs/GWD5nSpiHxs3K
• Wrecking Ball: https://www.flickr.com/photos/rhysasplundh/5202454842/in/photostream/
• Rocky Stairs: http://pixgood.com/rocky-stairs.html
• Technical Difficulties: https://www.youtube.com/watch?v=EC15BmzsdhM
• Grafana example: https://i.imgur.com/KOqcD6L.png
• Zipkin example: https://blog.buoyant.io/2016/05/17/distributed-tracing-for-polyglot-microservices/
• Assembly Line: http://www.solidsmack.com/culture/humans-need-apply-new-short-film-explores-future-robots-manufacturing-automation/

123. @svpember
Links
• Eric Evans: DDD and Microservices: https://www.youtube.com/watch?
v=yPvef9R3k-M