A presentation on how we came to find Stackdriver for system and service monitoring, how we implemented it and how we use it on a day to day basis.

1. Monitoring with
Stackdriver
Or: How I Learned How to Stop Worrying and Dig Hosted
Monitoring.
denise stockman - denise@stockmans.org

2. Mad Props
Thanks to Coupa for hosting
and to Stackdriver for sponsoring
and to Advanced AWS for organizing.

3. Hi, I’m denise
●
●
●
●
●
System Architect @ Maverix Biomics
18 years button pushing
Musical Chair Champion
Data center to cloud
Early Stackdriver user &
puppet module maintainer

4. What is Stackdriver?
“Intelligent Monitoring for your AWS
Infrastructure, Systems and App”

5. Agenda
●
●
●
●
●
●
The Environment
Selection Process
Implementation
Day to Day
TODO
Questions

6. A Diverse Stack

7. Further Context
●
●
●
●
●
Green field!
We treat instances like cattle*
Modern and crusty backend services
“Quaint” team
Monitor & report system, service, event and application
statistics in one place.
* As much as possible

8. What to do for monitoring?
Evaluate & Select
● In-house or hosted?
● Modern vs proven?
● Specialized vs general?
● Paid vs free?

9. Let’s Be Honest
●
●
●
●
Not enough time
Small environment
More important work to do
Everyone else is doing it

10. Winner, Winner, Chicken Dinner
●
●
●
●
●
●
●
System & service metrics
Alerting
Custom metrics & annotations
collectd agent
Anomaly detection
AWS integration & cloud friendly
Cost per resource

11. Implementation

12. Consistent Resource Naming
Resource name format
env-function-id.location.domain.com
Tags for resources
env, owner, purpose, version

13. ●
●
●
●
Installs base agent on all hosts
Configures API key
Includes plugins for supported services
Auto-upgrades agent
node /^dev-redis.*.stockmans.org$/ {
class { "stackdriver::redis":
stackdriver_redis_port
=> 6379,
stackdriver_redis_host
=> localhost,
stackdriver_redis_timeout => 2000,
}
}
Agent Deployment

14. ●
●
●
●
Create groups based on filtering criteria
Dashboards auto-created for each group
Groups used in custom dashboarding and alerting policies
Set it once and fuggetaboutit*
Groups

15. Alerting

16. Day to Day Use

17. Custom Dashboards

18. Anomaly Detection

19. Event Correlation

20. Event Logging

21. curl -X POST
-H "x-stackdriver-apikey: DOOOOWEEEEOOOO"
-H "Content-Type: application/json"
-d '{"message":"Mind taking!", "annotated_by":"Mentok"}'
https://event-gateway.stackdriver.com/v1/annotationevent
Annotations

22. Ruby example
customer_id = 31337
api_key = "DOOOOWEEEEOOOO"
StackDriver.init customer_id, api_key
redis = Redis.new
mindsTaken = redis.zcard("minds:taken")
StackDriver.send_metric "mentok.minds.taken", mindsTaken, Time.now.to_i
Custom Metrics

23. TODO.txt
●
●
●
●
●
Expand app metrics with statsd
Automate group creation & management
Annotate orchestrated events
Cloudwatch custom metrics
Stackdriver @ our colo

24. Thanks!
Questions?
denise stockman
denise@stockmans.org
github.com/maverixbio/puppet-stackdriver-agent
forge.puppetlabs.com/dstockman/stackdriver