Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1 @JJUG CCC 2018 Spring / 2018-05-26

1. !1
Spring Boot Actuator 2.0 &
Micrometer
2018-05-26
Toshiaki Maki (@making / tmaki@pivotal.io)
#jjug_ccc #ccc_a1

2. Who am I ?
2
Toshiaki Maki (@making) https://blog.ik.am
Sr. Solutions Architect @Pivotal Japan
Spring / Cloud Foundry / Concourse / Kubernetes
" Spring Boot 2": https://note.ik.am (not free!)

3. Trouble in containerized applications / micro services!!
!3
💥 Out of memory error!
💥 Performance issue!
🤔
GC log
Heap Dump
Thread
Dump
Method
Profiling

4. 💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4

5. 💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4

6. 💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4

7. 💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4
😎

8. Agenda
!5
• Spring Boot Actuator 2
• Micrometer
• Micrometer
• Histograms and percentiles
• Micrometer API
• Monitoring Spring Boot apps
on Cloud Foundry and
Kubernetes

9. Spring Boot Actuator 2

10. Spring Boot Actuator
!7
Additional features to help you monitor and manage
your application in production.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

11. Supported Endpoints
!8
• auditevents
• beans
• conditions
• configprops
• env
• flyway
• health
• httptrace
https://demo-micrometer.cfapps.io/actuator
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html
• info
• loggers
• liquibase
• metrics
• mappings
• scheduledtasks
• sessions
• shutdown
• threaddump
• heapdump
• jolokia
• logfile
• prometheus
/actuator/*
•renamed
•added

12. Technology Support
!9
Spring Boot Actuator 1.x Spring Boot Actuator 2.x
•Spring MVC •Spring MVC
•Spring WebFlux
•Jersey

13. How to enable endpoints
!10
Default exposed endpoints are /info and /health.
management.endpoints.web.exposure.include=*
management.endpoints.web.exposure.include=info,health,env
All endpoints are not secured by default.

14. Secure endpoints (Spring MVC)
!11
http.authorizeRequests()
.mvcMatchers("/actuator/health", "/actuator/info")
.permitAll()
.mvcMatchers("/actuator/**")
.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...

15. Secure endpoints (Spring MVC)
!11
http.authorizeRequests()
.mvcMatchers("/actuator/health", "/actuator/info")
.permitAll()
.mvcMatchers("/actuator/**")
.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...
spring.security.user.name=hoge
spring.security.user.password=foo
spring.security.user.roles=ACTUATOR

16. Secure endpoints (Spring MVC)
!12
http.authorizeRequests()
.requestMatchers(EndpointRequest.to("health", "info"))
.permitAll()
.requestMatchers(EndpointRequest.toAnyEndpoint())
.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...

17. Secure endpoints (Spring WebFlux)
!13
http.authorizeExchange()
.matchers(EndpointRequest.to("health", "info"))
.permitAll()
.matchers(EndpointRequest.toAnyEndpoint())
.hasRole("ACTUATOR")
.anyExchange()
.permitAll()
.and()
.httpBasic()
.and()...

18. Health Check
!14
management.endpoint.health.show-details=when_authorized
when authorized

19. Cloud Foundry Support
!15 https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-cloudfoundry.html
https://run.pivotal.io/

20. Micrometer

21. Micrometer
!17
Move to Micrometer
Think SLF4J, but for Metrics
Instrument without vendor lock-in:
• Prometheus
• Netflix Atlas
• Datadog
• InfluxDB
• ...
Multi-dementional metrics
https://micrometer.io/

22. Multi-dimensional VS Hierarchical
!18
{
"counter.200.foo": 100,
"counter.400.foo": 3,
"counter.200.bar": 25,
"counter.200.baz": 50
}
Status URI
Hierarchical (Spring Boot 1.x)

23. Multi-dimensional VS Hierarchical
!18
{
"counter.200.foo": 100,
"counter.400.foo": 3,
"counter.200.bar": 25,
"counter.200.baz": 50
}
Status URI Method...? 😨
Hierarchical (Spring Boot 1.x)

24. Multi-dimensional VS Hierarchical
!19
{
"names": "counter",
"tags": [
{"tag": "uri",
"values":
["foo", "bar", "baz"]},
{"tag": "status",
"values": [200, 400]},
{"tag": "method",
"values": ["GET"]}
]
}
Multi-dimensional (Spring Boot 2.0)

25. Multi-dimensional VS Hierarchical
!20
{
"names": "counter",
"tags": [
{"tag": "uri",
"values":
["foo", "bar", "baz"]},
{"tag": "status",
"values": [200, 400]},
{"tag": "method",
"values": ["GET"]}
]
}
Multi-dimensional (Spring Boot 2.0)
counter?tag=uri:foo&tag=status:200
{
"names": "counter",
"measurements":[
{
"statistic": "COUNT",
"value": 100
}
}
}

26. Multi-dimensional VS Hierarchical
!21
{
"names": "counter",
"tags": [
{"tag": "uri",
"values":
["foo", "bar", "baz"]},
{"tag": "status",
"values": [200, 400]},
{"tag": "method",
"values": ["GET", "POST"]}
]
}
Multi-dimensional (Spring Boot 2.0)
{
"names": "counter",
"measurements":[
{
"statistic": "COUNT",
"value": 100
},
"availableTags":[
{"tag":"method",
"values":["GET", "POST"]}
]]}
counter?tag=uri:foo&tag=status:200

27. Supported monitoring systems
!22
Server polls Client pushes
Prometheus
Atlas, Datadog, Datadog StatsD,
Influx, SignalFx, Telegraf StatsD,
Wavefront, New Relic
Dimensional
Graphite, Ganglia, JMX, Etsy StatsD,
PCF Metrics 1.4
Hierarchical
push
poll

28. Prometheus
!23
monitoring system and time series database.
https://prometheus.io/

29. Prometheus
!24
Prometheus ServerExporter
pull metrics
scrape_configs:
- job_name: foo
static_configs:
- targets:
- 192.168.1.2:8080
metrics_path: /metrics

30. Prometheus
!25
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

31. /actuator/prometheus
!26
system_cpu_count 4.0
system_load_average_1m 1.64
jvm_memory_max_bytes{area="heap",id="PS Eden Space",}
1.05906176E8
jvm_memory_max_bytes{area="heap",id="PS Survivor Space",}
2.1495808E7
jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 3.00941312E8
http_server_requests_seconds_count{exception="None",method="GET"
,status="200",uri="/hello",} 4469.0
http_server_requests_seconds_sum{exception="None",method="GET",s
tatus="200",uri="/hello",} 297.942920787
http_server_requests_seconds_max{exception="None",method="GET",s
tatus="200",uri="/hello",} 0.401581731
https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-metrics.html#production-ready-
metrics-meter

32. Prometheus
!27
Prometheus ServerExporter
pull metrics
scrape_configs:
- job_name: my-boot-app
static_configs:
- targets:
- localhost:8080
metrics_path: /actuator/prometheus

33. Supported Metrics
!28
• JVM Metrics
• Memory, Buffer pools
• GC
• Thread
• Classes
• CPU
• File Descriptors
• Logback
• Uptime
• Tomcat
• Spring MVC
• Spring WebFlux
• RestTemplate
• Cache
• DataSource
• RabbitMQ

34. Prometheus
!29

35. Grafana
!30

36. Datadog
!31
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-datadog</artifactId>
</dependency>
management.metrics.export.datadog.api-key=YOUR-API-KEY

37. Datadog
!32

38. PCF Metrics
!33
Bind Metrics Forwarder Service
https://github.com/cloudfoundry/java-buildpack-metric-writer Use Metric Writer 2.4.0+ which comes with JBP 4.10+

39. Also work with Spring Boot 1
!34
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-spring-legacy</artifactId>
<version>${micrometer.version}</version>
</dependency>

40. Histograms and percentiles

41. Histogram
!36
management.metrics.distribution.percentiles-
histogram.http.server.requests=true

42. Histogram in Prometheus
!37
http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0
The number of requests that took less than 0.447392426 sec.

43. Query percentiles in Prometheus
!38
histogram_quantile(0.9,
sum(rate(http_server_requests_seconds_bucket{status="200"
}[5m])) by (app, uri, le))
https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile()

44. SLA (Service Level Agreement)
!39
management.metrics.distribution.sla.http.server.requests=
100ms, 200ms, 400ms

45. SLA in Prometheus
!40
http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.1",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.2",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.4",} 15687.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0

46. Client-side Percentile
!41
management.metrics.distribution.percentiles.http.server.r
equests=0.5, 0.9, 0.95, 0.99, 0.999
http_server_requests_seconds{...,uri="/hello",quantile="0.5",} 0.050855935
http_server_requests_seconds{...,uri="/hello",quantile="0.9",} 0.051380223
http_server_requests_seconds{...,uri="/hello",quantile="0.95",} 0.40265318
http_server_requests_seconds{...,uri="/hello",quantile="0.99",} 0.40265318
http_server_requests_seconds{...,uri="/hello",quantile="0.999",} 0.40265318

47. Micrometer API

48. Meter (Timer, Counter, Gauge, ...)
!43
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte
r, FunctionTimer, and TimeGauge
@Component
public class FooService {
final Counter counter;
public FooService(MeterRegistry registry) {
this.counter = Counter.builder("received.messages")
.register(registry);
}
public void handleMessage() {
this.counter.increment();
}
}

49. Meter (Timer, Counter, Gauge, ...)
!44
@Component
public class FooService {
final Timer timer;
public FooService(MeterRegistry registry) {
this.timer = Timer.builder("foo").register(registry);
}
public String foo() {
return this.timer.record(() -> {
// ...
});
}
}
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte
r, FunctionTimer, and TimeGauge

50. @Timed
!45
@Component
public class FooService {
@Timed("foo")
public String foo() {
// ...
}
}
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(meterRegistry);
} // Not auto-configured in Spring Boot 2.0
https://github.com/micrometer-metrics/micrometer/issues/361

51. Tag
!46
@Component
public class TweetCounter {
final MeterRegistry registry;
public FooService(MeterRegistry registry) {
this.counter = registry;
}
public void countTweet(String name, String hashTag) {
this.registry.counter("tweet",
"screen_name", name,
"hash_tag", hashTag)
.increment();
}
}

52. MeterFilter
!47
@Bean
public MeterFilter meterFilter() {
return MeterFilter.deny(id -> {
String uri = id.getTag("uri");
return id != null && uri.startsWith("/actuator");
}
);
}
Exclude metrics against Actuator endpoints

53. Add MeterBinders
!48
@Bean
public HystrixMetricsBinder hystrixMetricsBinder() {
return new HystrixMetricsBinder();
}
@Bean
public HibernateMetrics hibernateMetrics() {
return new HibernateMetrics(...);
}
Check io.micrometer.core.instrument.binder package
Some binders are auto-configured https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-
metrics.html#production-ready-metrics-meter

54. Monitoring Spring Boot Apps
on Cloud Foundry and
Kubernetes

55. On Cloud Foundry
!50
App (instance 0)
App (instance 1)
GoRouter
Load Balanced... 🤔
http://example.com/actuator/prometheus
scrape_configs:
- job_name: example
static_configs:
- targets:
- example.com:80
metrics_path: /actuator/prometheus
prometheus.yml
scrape

56. On Cloud Foundry
!51
@Bean
public MeterRegistryCustomizer meterRegistryCustomizer(
@Value("${vcap.application.instance_id:}") String id) {
return registry -> registry.config()
.commonTags("cf_app_instance_id", id);
}

57. On Cloud Foundry
!52
system_cpu_count{cf_app_instance_id="31388366-...",}
} 4.0
system_load_average_1m{cf_app_instance_id="31388366-...",} 1.64
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea
p",id="PS Eden Space",} 1.05906176E8
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea
p",id="PS Survivor Space",} 2.1495808E7
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea
p",id="PS Old Gen",} 3.00941312E8

58. On Cloud Foundry
!53
App (instance 0)
App (instance 1)
GoRouter
Prometheus
scrape_configs:
- job_name: example
dns_configs:
- names:
- example.apps.internal
type: A
port: 8080
metrics_path: /actuator/prometheus
http://demo.apps.inetrnal:8080/actuator/prometheus
https://github.com/making/demo-prom-grafana-on-cf

59. Another solution - Promregater
!54
https://github.com/promregator/promregator

60. On Kubernetes
!55
Pod (instance 0)
Pod (instance 1)
Prometheus
http://xxx.yyy.zzz:8080/actuator/prometheus

61. !56
scrape_configs:
- job_name: k8s-pods
kubernetes_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __address__
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
On Kubernetes
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

62. On Kubernetes
!57
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: /actuator/prometheus
# ...

63. Related Links
!58
Thank you for your attention!
• https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-
ready.html
• https://micrometer.io/docs
• https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-
metrics.html
• https://blog.ik.am/entries/448
• https://github.com/making/demo-micrometer
• https://github.com/making/demo-prom-grafana-on-cf (CF env)
• https://github.com/categolj/blog-prometheus/tree/master/local (Local &k8s env)

64. Related Talks
!59
• More Spring Boot 2? => 13:30-14:15 @Room E+F
Pivotal Reactive
Spring 5 & Spring Boot 2
• More Observability? => 14:30-15:15 @Room I
How to Properly Blame Things for Causing Latency:
An Introduction to Distributed Tracing and Zipkin