A successful transition to a modern elastic, containerized, microservice architecture requires automating all things, including your monitoring and alerting infrastructure. In this talk, we share some of the techniques and best practices we learned at New Relic for applying "infrastructure as code" (IaC) techniques to monitoring and alerting during our 10-year journey from a single-region monolithic application to a global multi-region deployment of hundreds of microservices. This session is brought to you by AWS partner, New Relic.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best Practices For
Scalable Monitoring
Kevin Scaldeferri
Lead Software Engineer
New Relic
E N T 3 1 0 - S

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New Relic
2008
2018
Rails FE monolith
Java BE monolith
DB monolith
1 region
10s of FE apps
100s of microservices
100s of databases
Multiple regions

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3. Deploy 1. Code
4. Monitoring
& Alerting 2. Build

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use the power of modern build systems

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
build.js
var publishSourcemap = require(‘@newrelic/publish-sourcemap’).publishSourcemap
publishSourcemap({
sourcemapPath: ‘./dist/bundle.js.map’,
javascriptUrl: ‘https://example.com/assets/bundle.js’,
applicationId: APPLICATION_ID,
nrAdminKey: 'ADMIN_KEY’
}, function (err) { console.log(err || 'Sourcemap upload done')})

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
build.gradle
apply plugin: ‘application’
ext {
newrelicVersion = ‘4.7.0’
}
configurations {
newrelic
}
dependencies {
compile “com.newrelic.agent.java:newrelic-
api:${newrelicVersion}”
newrelic “com.newrelic.agent.java:newrelic-
agent:${newrelicVersion}”
}
task copyAgent(type: Copy) {
from configurations.newrelic
into "$buildDir/lib"
rename ("newrelic-agent-${newrelicVersion}.jar",
'newrelic.jar’)
}
tasks.run {
dependsOn copyAgent
jvmOpts << “-javaagent:$buildDir/lib/newrelic.jar”
}

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
build.gradle
apply plugin: ‘application’
apply plugin: ‘newrelic-agent-plugin’
apply plugin: …
application { … }
newrelic { … }
…

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
build.gradle
apply plugin: ‘company-java-service’
dependencies {
…
}

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. Step 1: Have
a common
deploy system
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. Step 2: Deploy
markers and
version tags
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Inject key monitoring attributes
Cluster / Cell
Region / AZ
Hostname Owning Team

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. Bootstrap
your apps
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Standard dimensional attributes
public void recordEvent(String eventType, Map<String, Object> attributes) {
// Add standard attributes
Map<String, String> envVars = System.getenv();
attributes.put("appName", envVars.getOrDefault("NEW_RELIC_APP_NAME", "unknown"));
attributes.put("version", envVars.getOrDefault("GC_VERSION", "unknown"));
attributes.put("jvmId", ManagementFactory.getRuntimeMXBean().getName());
attributes.put("ipPort", envVars.getOrDefault("CF_PRIMARY_IP_PORT", "unknown"));
attributes.put("hostname", envVars.getOrDefault("CF_FQDN", "unknown"));
attributes.put("region", envVars.getOrDefault("CF_REGION", "unknown"));
attributes.put("zone", envVars.getOrDefault("CF_ZONE", "unknown"));
attributes.put("subnet", envVars.getOrDefault("CF_SUBNET_ID", "unknown"));
NewRelic.recordCustomEvent(eventType, attributes);
}

32. Resource
utilization
tracking
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. Resource
utilization
tracking
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. Monitoring &
Alerting as Code
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Spectrum of DSLs
Full embedded
languages
Collections
of scripts
Template and
config languages

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dashboard Templates
{
"dashboard" : {
"title" : "{{ capitalize(cluster) }} Service Health",
"filter" : {
"event_types" : [ "NodeStatus" ],
"attributes" : [ ]
}
"widgets" : [ {
"visualization" : "faceted_line_chart",
"data" : [ {
"nrql" : "SELECT sum(g1younggenerationCollectionTimeSeconds) FROM NodeStatus
where cluster = ‘{{ cluster }}' TIMESERIES FACET appName"
} ],
"presentation" : {
"title" : "Young Gen Time }
}

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dashboard Templates
{
"dashboard" : {
"title" : "{{ capitalize(cluster) }} Service Health",
"filter" : {
"event_types" : [ "NodeStatus" ],
"attributes" : [ ]
}
"widgets" : [ {
"visualization" : "faceted_line_chart",
"data" : [ {
"nrql" : "SELECT sum(g1younggenerationCollectionTimeSeconds) FROM NodeStatus
where cluster = ‘{{ cluster }}' TIMESERIES FACET appName"
} ],
"presentation" : {
"title" : "Young Gen Time",
"notes" : null
}
}

51. Alerts DSL
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
policy "Unified Data Streams CF Alerts” do
rollup 'condition’
team_low_priority_channels
condition "WARNING Service(s) OOMing" do
type 'nrql’
query "SELECT count(*) FROM cf_docker_event where action = 'oom’”
since 2.minutes
value 'single_value’
critical above: 0, for: 1.minutes
runbook ‘https://…’
end
end

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
clusters.each do |cluster|
policy "#{cluster} MyApp Lag" do
rollup 'condition'
team_alert_channels(cluster)
condition "#{sev_and_cluster('SEV3’, cluster)} MyApp" do
entities [config[cluster][‘lag_monitor_app_name']]
metric 'Custom/ConsumerLag/appendSecondsLag/my_topic/my_app'
value 'max'
critical above: 60, for: 5.minutes
runbook ‘https://…’
end
end
end

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
clusters.each do |cluster|
policy "#{cluster} MyApp Lag" do
rollup 'condition'
team_alert_channels(cluster)
condition "#{sev_and_cluster('SEV3’, cluster)} MyApp" do
entities [config[cluster][‘lag_monitor_app_name']]
metric 'Custom/ConsumerLag/appendSecondsLag/my_topic/my_app'
value 'max'
critical above: 60, for: 5.minutes
runbook ‘https://…’
end
end
end

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
clusters.each do |cluster|
policy "#{cluster} MyApp Lag" do
rollup 'condition'
team_alert_channels(cluster)
condition "#{sev_and_cluster('SEV3’, cluster)} MyApp" do
entities [config[cluster][‘lag_monitor_app_name']]
metric 'Custom/ConsumerLag/appendSecondsLag/my_topic/my_app'
value 'max'
critical above: 60, for: 5.minutes
runbook ‘https://…’
end
end
end

56. Machine
Learning
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Make a plan
for monitoring
Automate
everything
Make it easy to
follow the plan
Get back to
building products

62. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kevin Scaldeferri
@kscaldef

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.