This presentation describes technics that helps Emind DevOps team to keep applications always-on. Enjoy. Lahav

1. DevOps Sensors 360°
High Availability in the Cloud
Lahav Savir, Architect & CEO
Emind systems Ltd.
lahavs@emind.co

2. About
Lahav Savir
• 15+ years’ experience
• Architect and CEO @ Emind Systems
Emind Systems (est. 2006)
• Highly professional system integrator
• Dedicated Cloud Architecture and DevOps teams
• 24x7 SLA by DevOps Specialists
• ~100 AWS customers
• Partnerships with leading cloud vendors

3. Partnerships
Amazon Advanced Consulting Partner
https://aws.amazon.com/solution-providers/si/emind-systems-ltd

4. What is Availability?
The ability to provide quality service
that can support the online service

5. Downtime
The term downtime refers to periods when a
system is unavailable.
Downtime or outage duration refers to a period
of time that a system/service fails to provide or
perform its primary function.

6. Unavailability / Causes of Downtime
• Hardware failure – 55%
• Human error – 22%
• Software failure – 18%
• Natural disasters – 5%
http://www.continuitycentral.com/news06645.html
Reputable studies have concluded that as much as
75% of downtime is the result of some sort
of human error.
http://searchdatacenter.techtarget.com/feature/The-causes-and-costs-of-data-center-system-downtime-Advisory-Board-QA

7. Hardware / Infrastructure
AWS SLA – 99.9 – 100%
• Redundant servers
– Multiple servers of each type
– EBS, Snapshots
• Multi AZ
– ELB, VPC
• Multi Region
• PaaS
– S3, SQS, DynamoDB, RDS, Route53 . . .

8. Architect
• Plan based on experience and best practices
• Continuously review and correct

9. Knowledge Sharing

10. Preinstalled / Preconfigured
Predefined procedures / scripts
Preinstalled instances / AMIs

11. Automate
Automate repeating tasks

12. Scheduler

13. Orchestration Tools

14. Orchestration Tools

15. Track Changes
• Diff your changes
# List all application folders
# Iterate through this list and
# cd <folder>
# svn ci . -m "[<timestamp>] Checkin <folder>"
• Ready for rollback

16. Security Group Changes

17. Servers Monitoring

18. Log monitoring
Graylog => Nagios checks
http://www.emind.co/open-source/
• check_graylog2_stream

19. Applications Counters / Metrics
net-snmp sub-agent
http://www.emind.co/open-source/
• net-snmp_shell_subagent
# Syntax
# < oid > ; < type > ; < script path >
.1.3.6.1.4.1.39731.2100.1:string:/usr/local/emind/sync_manager/sync_manager.sh status status
.1.3.6.1.4.1.39731.2100.2:string:/usr/local/emind/sync_manager/sync_manager.sh status state
.1.3.6.1.4.1.39731.2100.3:integer:/usr/local/emind/sync_manager/sync_manager.sh status sync_duration_min
.1.3.6.1.4.1.39731.2100.4:integer:/usr/local/emind/sync_manager/sync_manager.sh status idle_duration_h
.1.3.6.1.4.1.39731.2100.5:string:/usr/local/emind/sync_manager/sync_manager.sh status last

20. Application performance

21. CloudWatch
CloudWatch MetricsCloudWatch Alarms
http://www.emind.co/open-source
• check_elb_instances_health
• check_cloudwatch_alarm

22. Robots
goCloud DevOps Central

23. NewVem
• Volume backup
• Security groups
• Instance
utilization
• IO utilization
• Reserved
Instances
• Planned
maintenance

24. Security Sensors
Snort, IPTables, Dome9

25. Human Feeds
Mail Notifications Phone Calls

26. Support Router API
Turning events to ACTIONS

27. lahavs@emind.co
+972 (54) 4321688
http://www.emind.co