This presentation describes how to configure Zabbix (https://zabbix.com/) to configure Google Cloud Platform events through its Monitoring API, using gcpmetrics (https://github.com/odin-public/gcpmetrics/) command line tool.
2. 2
Zabbix – Terminology
Host = any monitored device, where Agent (or SNMP, JMX,
IPMI interfaces) are installed
Examples: Linux/Windows Server, Router, etc.
Item = particular metric configured for a particular Host
Examples: system.cpu.load[all,avg5], system.cpu.num[online], script[echo,hello,world],
net.tcp.service[http] …
Key = type of Item that can be gathered from the Host
Examples: system.cpu.load[<cpu>,<mode>], system.cpu.num[<type>],
script[<parameter1>,<parameter2>,...], net.tcp.service[<service>,<ip>,<port>] …
9. 9
Google Monitoring API v3
Try yourself: https://developers.google.com/apis-explorer/#p/monitoring.googleapis.com/v3/
# Method Description
1 collectdTimeSeries.create
Stackdriver Monitoring Agent only: Creates a new time series. This method is only for use by the Google
Monitoring Agent. Use projects.timeSeries.create instead.
2 groups.create Creates a new group.
3 groups.delete Deletes an existing group.
4 groups.get Gets a single group.
5 groups.list Lists the existing groups.
6 groups.members.list Lists the monitored resources that are members of a group.
7 groups.update Updates an existing group. You can change any group attributes except name.
8 metricDescriptors.create
Creates a new metric descriptor. User-created metric descriptors define custom metrics
(/monitoring/custom-metrics).
9 metricDescriptors.delete Deletes a metric descriptor. Only user-created custom metrics (/monitoring/custom-metrics) can be deleted.
10 metricDescriptors.get Gets a single metric descriptor.
11 metricDescriptors.list Lists metric descriptors that match a filter.
12 monitoredResourceDescriptors.get Gets a single monitored resource descriptor.
13 monitoredResourceDescriptors.list Lists monitored resource descriptors that match a filter. This method does not require a Stackdriver account.
14 timeSeries.create Creates or adds data to one or more time series.
15 timeSeries.list Lists time series that match a filter.
10. 10
Some of App Engine Counters
# Counter ID Value Description
1 http/server/response_count DELTA
INT64 Delta HTTP response count.
2 http/server/response_latencies DELTA,
DISTRIBUTION HTTP response latency.
3 system/network/received_bytes_count DELTA
INT64
Delta count of incoming network
bandwidth.
4 system/network/sent_bytes_count DELTA
INT64
Delta count of outgoing network
bandwidth.
5 system/cpu/usage GAUGE
INT64 CPU usage in megacycles.
Much more: https://cloud.google.com/monitoring/api/metrics
Services can also define their own custom counters.
11. 11
timeSeries.list parameters
# Parameter Example
1 Project Name projects/odin-ap
2 Period of aggregation 3600s
3 Overall aggregation rule REDUCE_SUM
4 Fields for grouping metric.label.response_code
5 Series aggregation rule ALIGN_SUM
6 Filter metric.type = "appengine.googleapis.com/http/server/response_count"
AND resource.label.module_id = "service"
AND metric.label.response_code >= 500
AND metric.label.response_code <= 599
7 Start Time 2016-08-27T23:00:00.000Z
8 End Time 2016-08-28T00:00:00.000Z
More on filters: https://cloud.google.com/monitoring/api/v3/filters#comparisons
12. 12
timeSeries.list example call
# Parameter Example
1 Project Name projects/odin-ap
2 Period of aggregation 3600s
3 Overall aggregation rule REDUCE_SUM
4 Fields for grouping metric.label.response_code
5 Series aggregation rule ALIGN_SUM
6 Filter metric.type =
"appengine.googleapis.com/http/server/response_count"
AND resource.label.module_id = "service"
AND metric.label.response_code >= 500
AND metric.label.response_code <= 599
7 Start Time 2016-08-27T23:00:00.000Z
8 End Time 2016-08-28T00:00:00.000Z
16. 16
Web Scenario
Zabbix: Web Monitoring
Zabbix Host
gcpmetrics
External Check
Agent
Detailed
Analysis
DBs
Cloud SQL
Google Cloud Platform
Monitoring
Monitoring
Services
App Engine
Monitoring API v3
DB
Operations
17. 17
Configure Timeouts of Zabbix
•Default installation of Zabbix is configured to terminate all
scripts in 3 seconds. Google Monitoring API may take 5-10+
seconds to respond, depends on the actual query.
•We suggest to set Zabbix Timeout to 30 seconds
Note: Applies to both server
and agent configurations
22. 22
Create 2 scripts in
$ cd /externalscripts/
$ vim tm-http5xx-absolute.sh
$ vim tm-http5xx-relative.sh
2 scripts are needed to demo both Absolute and Delta behavior of Zabbix
because it doesn’t allow 2 items to refer to the same script.