The lines between Development and Operations people have gotten blurry and lots of skills needs to be held by both sides. In the talk we'll talk about all of the considerations that are needed to be taken when creating a development and production environment, mentioning Continuous Integration, Continuous Deployment and the Buzzword "DevOps", also talking about some real implementations in the industry. Of course how can we leave out the real enabler of the whole deal, "The Cloud", Giving us a tool set that makes life much easier when implementing all of these practices.

1. 1
Demi Ben-Ari
Co-Founder & CTO at
Panorays

2. Demi Ben-Ari, Co-Founder & CTO @ Panorays
● Google Developer Expert
● Co-Founder of Communities:
○ “Big Things” - Big Data, Data Science, DevOps
○ Google Developer Group Cloud
○ Ofek Alumni Association
In the Past:
● Sr. Data Engineer - Windward
● Team Leader & Sr. Java Software Engineer,
Missile defense and Alert System - “Ofek” – IAF
2

3. ● A lot of (NOT) funny Jokes
● DevOps? WTF?
● Covering the world of cloud
● Process and Environment
● Monitoring pipeline solutions
■ Metrics, Datastores, Dashboards, Alerting
● Conclusions
4

5. 6
A breach to even the
smallest supplier may
cause a cyber typhoon
in the industry.

6. 77
~300 Employees
PNI Data Breach – Photo Services Affected – By Thomas George
BREACHED
- Supplier - Evaluator

7. DevOps

8. Development
Software Engineering
Front End
Backend
QA
UX
Operations
System Admin
Database Admin
Support

9. DevOps
Cross-functional teams
Operators automating
systems
Developers operating systems

11. Running environments
● —Development – Testing – Production
○ Don’t you need more?
Be as flexible as you can
○ Ease of creation and Deployment
○ Identical or Not?
● Virtual Machines vs Bare Metal
● Configuration

13. ●
●
●
●
●

14. https://www.coscale.com/hs-fs/hubfs/Blog_Pictures/2016_06/monolithic_vs_microservices.jpg

15. ●
●
●

16. ●
■ What is your application like?
■ What are the distinct pieces?
■ What data and / or logic must remain together?
■ Can some parts be asynchronous?
●

17. ●
○
○
○
●

18. ●
●
●
●
●

20. Docker Environments
● Docker?
● Orchestration?
VS
● Wait, What about local mode?
○ Minikube vs Docker Engine

22. 35
●
○
○
●
●

23. ●
●
●

24. ●
●
●

25. 38

26. 39
● Linux
○ Bash scripts
○ Crontab
● Automation via Jenkins
● —Continuous Deployment – with every GIT push

27. 43
●
●
●
●
●
●
●

28. 44

29. 46

30. Product
Data
Infrastructure
Biz
47

31. OS CPU Memory Disk
Processes Java
Application
Server
Database
Web Server
Load
Balancer
Users - Other Applications
Monitorin
g System
UI
Many times...all of this was on a single physical server! 48

32. Service A
Queue
DB
Service B
DBCache
Cache DBService C
Web
Server
DB
Analytics Cluster
Master
Slave Slave Slave
Monitoring System???
49

33. 50

34. Basic Concepts
● Monitoring
● White-box
○ internals
● Black-box
○ behavior
● Dashboard
● Alert
● Root cause
● Node and machine
● Deploy
○ Any change to a
service’s running
software or its
configuration.
● KPI - Key Performance
Indicator

35. Monitoring Stack - Let’s fill in the blanks
Alerting
Metrics Collection
Datastore
Dashboard
Data Monitoring
Log Monitoring

36. Situations & Problems
https://imgflip.com/i/1ap5kr
http://kingofwallpapers.com/otter/otter-004.jp

37. MongoDB + Spark
Worker 1
Worker 2
….
….
…
…
Worker N
Spark
Cluster
Master
Write
Read
MasterSharded
MongoDB
Replica Set

38. Cassandra + Spark
Worker 1
Worker 2
….
….
…
…
Worker N
Cassandra
Cluster
Spark
Cluster
Write
Read

39. Cassandra + Serving
Cassandra
Cluster
Write
Read
UI Client
UI Client
UI Client
UI Client
Web
ServiceWeb
ServiceWeb
ServiceWeb
Service

40. So...Solutions
Let’s Start!

41. Monitoring
Operation System

42. Monitoring Operation System Metrics
● What to measure:
○ CPU
○ Memory
○ Disk Space
● How to measure:
○
○
■ Nice and easy UI
■ Even the free account gives great tool
■ Alerting of thresholds

43. Some help
from “the Cloud”

44. AWS’s CloudWatch / GCP StackDriver

45. Report to Where?
● We chose:
●
● Can correlate System and
Application metrics in one
place :)

46. Connections
Connections…
http://www.mememaker.net/meme/connections-connections-everywhere2/

47. Drivers to Datastores
● Actions they usually do:
○ Open connection
○ Apply actions
■ Select, Insert, Update, Delete
○ Close connection
● Do you monitor each?
○ Hint:
● Creating a wrapper in any programming language and reporting the metrics
○ Count, execution times, errors…
○ Infrastructure code that will give great visibility

48. Logging?
Monitoring?
https://lh4.googleusercontent.com/DFVcH-E5XKj8cbhEtI0qabmf_wwVqWWvk0pK5H5rnC_kVxY2tXClKfzV-LvAH61YRLJUEvtO9amjWfjcY4Z57VBYCuQ9
5_hdAVEHgLAuepJiArH0wJERWuzzmgnPysCiIA

49. ELK - Elasticsearch + Logstash + Kibana
http://www.digitalgov.gov/2014/05/07/analyzing-search-data-in-real-time-to-drive-decisions/

50. http://www.funpic.hu/_files/pictures/original/86/71/27186.jpg
78

51. 79
● http://redash.io/
● Open Source: https://github.com/getredash/redash
● Came out as one of many Open source tool by Everything.me
● Created and Maintained by Arik Fraimovich (You rock!)
● Written in Python
● Has an on-premise and hosted solution

52. 80

53. 81

54. 82
● Syren - Open source
● Reporting to:
○ Email, Flowdock, HipChat, HTTP,
Hubot, IRCcat, PagerDuty,
Pushover, SLF4J, Slack,
SNMP, Twilio

55. Summary - Monitoring Stack
Alerting
Metrics Collection
Datastore
Dashboard
Data Monitoring
Log Monitoring

56. https://imgflip.com/i/18kvv1 84

57. https://cdn.meme.am/instances/500x/22605665.jpg
85

58. 86
● Correlating Application and System metrics!!!!
● Ask the right monitoring questions -> answer with Dashboards
● KISS - simple is key, what’s hard, we tend not to do at all
● Alert about what you can actually react to
○ (And to the relevant person)
● Measure whatever you can
○ only way to know if you’re improving
● Monitor your business KPIs too

59. 87
● DevOps requires Culture also
● Take responsibility
○ Production is everybody's responsibility
● Leverage the Cloud and the possibilities of being Flexible

62. Thanks
demi@panorays.com