We all know not to poke at alien life forms in another planet, right? But what about metrics, do you know how to pick, measure and draw conclusions from them? In this talk we will cover various Site Reliability Engineering topics, such as SLIs and SLOs while we explore real life examples of defining and implementing metrics in a system with examples using Prometheus, an open-source system monitoring and alert platform, to demonstrate implementation. Let's get back to some real science.

1. Application Metrics
with Prometheus examples
Rafael Dohms @rdohms

2. How do you do
metrics?

3. “The Prometheus
Scientist Method”

4. I hope not.

5. jobs.usabilla.com
Rafael Dohms
Staff Engineer
rdohmsdoh.ms

6. FeedbackFeedback
jobs.usabilla.com
Rafael Dohms
Staff Engineer
rdohmsdoh.ms
We are hiring!
jobs.usabilla.com

7. Let’s talk about metrics.
But let’s do it with a
concrete example.

8. Kafka / DDD / Autonomous Microservices / Monitoring

9. Kafka / DDD / Autonomous Microservices / Monitoring

10. Kafka / DDD / Autonomous Microservices / Monitoring

11. Metrics are insights into
the current state of your
application.

12. Metrics tell you if your
service is healthy.

13. Canary Deploys
OksanaLatysheva

14. Metrics tell you what
is wrong.

15. Metrics tell you what
is right.

16. Metrics tell you what
will soon be wrong.

17. Metrics tell you where
to start looking.

18. Site Reliability Engineering

19. SLIs SLOs
◎
SLAs

20. SLIs
Service Level Indicators
“A quantitative measure of some
aspect of your application”
The response time of a request was 150ms
Source: Site Reliability Engineering - O’Reilly

21. SLOs
◎
Service Level Objectives
“A target value or a range of values
for something measured by an SLI”
Request response times should be below 200ms
Source: Site Reliability Engineering - O’Reilly

22. Help you drive architectural
decisions, like optimisation
SLOs
◎
Response time SLO: 150 ms
95th Percentile of Processing time (PHP time): 5ms
As a result we decided to invest more time in exploring the
problem domain and not optimising our stack.

23. SLAs
Service Level Agreements
“An explicit or implicit contract with
your customer,that includes
consequences of missing their SLOs”
The 99th percentile of requests response times should meet our SLO,or
we will refund users
Source: Site Reliability Engineering - O’Reilly

24. Measuring

25. –Etsy Engineering
“If it moves, we track it.”
https://codeascraft.com/2011/02/15/measure-anything-measure-everything/

26. Metrics
Statistics
What is happening right
now?
How often does this
happen?
Telemetry

27. Telemetry
“the process of recording and transmitting the readings of an instrument”

28. Statistics / Analytics
“the practice of collecting and analysing numerical data in large quantities”

29. Statistics / Analytics
“the practice of collecting and analysing numerical data in large quantities”

30. I really miss Ayrton Senna
Statistics / Analytics
“the practice of collecting and analysing numerical data in large quantities”

31. Statistics
Incoming feedback items
with origin information
Telemetry
response time of public
endpoints

32. “If it moves, we track it.”

33. Request Latency
System Throughput
Error Rate
Availability
Resource Usage
“If it moves, we track it.”

34. Request Latency
System Throughput
Error Rate
Availability
Resource Usage
“If it moves, we track it.”
Incoming Data
Peak frequency
CPU
Memory
Disk Space
Bandwith
node
PHP
NginX
Database

35. Request Latency
System Throughput
Error Rate
Availability
Resource Usage
“If it moves, we track it.”
Incoming Data
Peak frequency
CPU
Memory
Disk Space
Bandwith
node
PHP
NginX
Database
Measure Monitoring
Measure measurements

36. Metrics,Everywhere.

38. SLIs

39. Picking good SLIs

40. SLIs may change
according to who is
looking at the data.

41. Understanding the
nature of your system

42. User-Facing
serving system?
availability,throughput,latency

43. Storage System?
availability,durability,latency

44. Big Data Systems?
throughput,end-to-end latency

45. User-Facing and Big Data Systems

46. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
User-Facing and Big Data Systems

47. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
User-Facing and Big Data Systems
More relevant to
development team

48. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
๏Other Metrics
- node,nginx,php-fpm,java metrics
- server metrics: cpu,memory,disk space
- Size of cluster
- Kafka health
User-Facing and Big Data Systems
More relevant to
development team

49. ๏SLIs
- Response time in the“receive”endpoint
- Turn around time,from“receive” to“show”.
- Individual processing time per step
- Data counting: how many,what nature
๏Other Metrics
- node,nginx,php-fpm,java metrics
- server metrics: cpu,memory,disk space
- Size of cluster
- Kafka health
User-Facing and Big Data Systems
More relevant to
development team
More relevant to
Infrastructure team

50. Picking Targets

51. Target value
SLI value >= target
Target Range
lower bound <= SLI value <= upper bound

52. Don’t pick a target based
on current performance
What is the business need?
What are users trying to achieve?
How much impact does it have on the user experience?

53. How long can it take between
the user clicking submit and a confirmation
that our servers received the data?

54. How long can it take between
the user clicking submit and a confirmation
that our servers received the data?
“Immediate"
“We sell as
real time”
“500ms,too
much HTML“
“I don’t know”

55. How long can it take between
the user clicking submit and a confirmation
that our servers received the data?
“Immediate"
“We sell as
real time”
“500ms,too
much HTML“
“I don’t know”
What is human perception of
immediate? 100ms
Collection API should respond within 150ms

56. Some, but not too many.
can you settle an argument or priority based on it?

57. Don’t over achieve.
The Chubby example.

58. Adapt. Evolve.
re-define SLO’s as your product evolves.

59. Meeting Expectations.

60. Attach consequences
to your Objectives.

61. The night is dark and
full of loopholes.
take a friend from legal with you.

62. Safety Margins.
like setting the alarm 5 minutes before the meeting.

63. Metrics in Practice.

64. prometheus.io

65. Push Model
scale this!

66. Pull Model
scale this!

67. Prometheus
Telemetry Statistics
Prometheus
StatsD,InfluxDB,etc…
+
Long Term Storage

68. GaugeHistogramCounter Summary
Cumulative
metric the
represents a
single number
that only
increases
Samples and
count of
observations
over time
A counter,that
can go up or
down
Same as a
histogram but
with stream of
quantiles over a
sliding window.

69. jimdo/prometheus_client_php

70. reads from /metrics
reads from local storage
writes to local storage
your code
/metrics

71. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

72. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

73. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
APC / APCu
Redis

74. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
namespace
metric name
help
label names
buckets

75. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
measurement
label values

76. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);
namespace
metric name
help
labels

77. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

78. <?php
use PrometheusCounter;
use PrometheusHistogram;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$histogram = new Histogram(
$adapter,
'my_app',
'response_time_ms',
'This measures ....',
['status', 'url'],
[0, 10, 50, 100]
);
$histogram->observe(15, ['200', '/url']);
$counter = new Counter($adapter, 'my_app', 'count_total',
'How many...', ['status', 'url']);
$counter->inc(['200', '/url']);
$counter->incBy(5, ['200', '/url']);

79. <?php
use PrometheusRenderTextFormat;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$renderer = new RenderTextFormat();
$result = $renderer->render($adapter->collect());
echo $result;

80. <?php
use PrometheusRenderTextFormat;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
$renderer = new RenderTextFormat();
$result = $renderer->render($adapter->collect());
echo $result;

81. <?php
use PrometheusRenderTextFormat;
use PrometheusStorageAPC;
require_once 'vendor/autoload.php';
$adapter = new APC();
# HELP my_app_count_total How many...
# TYPE my_app_count_total counter
my_app_count_total{status="200",url="/url"} 6
# HELP my_app_response_time_ms This measures ....
# TYPE my_app_response_time_ms histogram
my_app_response_time_ms_bucket{status="200",url="/url",le="0"} 0
my_app_response_time_ms_bucket{status="200",url="/url",le="10"} 0
my_app_response_time_ms_bucket{status="200",url="/url",le="50"} 1
my_app_response_time_ms_bucket{status="200",url="/url",le="100"} 1
my_app_response_time_ms_bucket{status="200",url="/url",le="+Inf"} 1
my_app_response_time_ms_count{status="200",url="/url"} 1
my_app_response_time_ms_sum{status="200",url="/url"} 16
$renderer = new RenderTextFormat();
$result = $renderer->render($adapter->collect());
echo $result;

82. –Also Rafael (today)
“I’ll just try this live demo
again.”
http://localhost:9090/graph http://localhost:8180/metrics
–Rafael (yesterday)
“Demos always fail.”
http://localhost:8180/index
https://github.com/rdohms/talk-app-metrics

83. You can’t act on what
you can’t see.

86. Metrics without
actionability are just
numbers on a screen.

87. Act as soon as an
SLO is threatened .

88. Thank you.
Drop me some
feedback at Usabilla
and make this talk
better.
@rdohms
http://slides.doh.ms
https://joind.in/talk/56e55