1. Open APIs for Open Minds
Monitoring Federation OpenStack infrastructure
(http://tinyurl.com/MonitoringFIWARE)
Fernando López Aguilar
(fla@tid.es)
Pablo Rodríguez Archilla
(pra@tid.es)
4. Need to monitor different processes:
• Need to know if they are running correctly.
• Notify administrator of the cloud about problems detected.
• Easy mechanism to provide new metrics to the system.
Need to give a more exhaustive information about the performance of the
servers.
• Need to know network bandwidth and latency (between regions).
Introduction
What are the requirements?
3
5. Need to give a more exhaustive information about the performance of the
servers.
• Memory/Disk/CPU consumption during time.
• User defined performance.
• Provide visual representation of data in form of temporal graphics.
• Provide solution to study data using CEP or Map/Reduce mechanisms.
Introduction
What are the requirements?
4
7. 6
Use the same Bus for OS and monitoring (wrong idea!!!).
• Community, in fact, wants to change the use of Queue message by Pub/Sub
mechanism.
Cannot provide tools to analyse data, only information without management.
Difficult to extend in multi-region environment.
Starting project (first version was with MySQL…).
Does not provide data information inside servers and networks traffic.
Introduction
Monitoring infrastructure using OpenStack Telemetry (Ceilometer)
8. Two different approaches were provided:
• Nagios to provide alerts to monitor of some servers parameters.
• Highly Scalable and distributed solution in order to provide mgmt. and federation
capabilities of the data.
Integration of Big Data + IoT + Cloud solutions.
• Provide tools to allow forensic analysis of monitoring data.
• Provide real time analysis of data in order to evaluate possible actions on time.
Separate the gathering of data from management and analysis.
Introduction
And the solutions …
7
9. Provide a standard solution based on IoT standards (NGSI interface).
Provide a solution to monitor services, servers, networks and whatever we could
need.
Separate the OpenStack Notification Bus from the Monitoring Bus.
Highly scalable solution:
• Asynchronous Node.js server to adapt data gathered and Orion Context Broker for
publish/subscribe.
• MongoDB for storing current data, Hadoop for historical data both highly scalable.
Introduction
And the solutions …
8
10. Fully integrated with OS:
• Listener to the RabbitMQ to know the creation of server to configure the monitoring
process.
Heterogeneity
• Support pre-existing monitoring infrastructure zero install effort.
Easy to extend and federate with other systems.
Data accesses must be controlled with authentication procedures.
Introduction
And the solutions …
9
13. Monitoring solution
Nagios
Nagios monitors the infrastructure to ensure systems, applications, services, and
business processes are functioning properly.
Versions in FI-WARE/XiFi:
• Nagios Core 3.4.1
• nrpe 2.15
• nagios-plugins 1.4.16
More information
• www.nagios.org
• www.nagios.org/download/plugins/
12
14. Monitoring solution
Nagios Remote Plugin Executor
Nagios Remote Plugin Executor (NRPE) to monitoring remote hosts, servers and
services.
Allows to remotely execute Nagios plugins on other Linux/Unix machines.
Allows to monitor remote machine metrics (disk usage, CPU load, etc.).
13
15. Monitoring solution
NAM & DEM
14
NGSI Adapter
Context Broker
Adapter
server
NAM Adapter
Monitoring
probes
Measurement
collectors
DEM Adapter
Monitoring
probes
Measurement
collectors
Parser
R▲
R▲
Network Active Monitoring.
Datacenter and Enablers
Monitoring.
Parser to get data from
measurement collector.
Adapter server to transfer to
NGSI format.
16. 15
Monitoring solution
Deployment diagram
An example of deployment.
Depends of the requirements.
Adopt Hadoop solution for historical
store data.
Adopt Orion as Pub/Sub broker.
18. Per OpenStack node layer
• Managed by administrator nodes staff.
• Not directly accessible for normal users (e.g. FI-LAB user).
• Not strong scalability requirements.
• CBs in this layer send ALL their context information to the aggregation layer CB
(federation), except the information we want to get filtered out.
Federated monitoring solution
Requirement
17
19. Aggregation layer
• Managed by federation nodes staff.
• Globally accessible and shared for federation users, i.e. each federation user
can see and modify other users entities.
• Strong scalability and high availability requirements.
• Single point of integration for COSMOS (Big Data solution of FI-WARE).
• Federation user configure federation to get the subset of information they
want in the per user layer CBs.
Federated monitoring solution
Requirement
18
20. Per User layer (optional)
• Managed by federation users, i.e. CB dedicated instances in the federation.
• Integration with federation portal and/or third parties applications/services.
• Authentication required to access to the Aggregation layer.
• Not strong scalability requirements.
Federated monitoring solution
Requirement
19
25. Future works
Automatic deployment of components based on listener in the OpenStack
Queue System (RabbitMQ).
Evaluate the integration with new version of Ceilometer in the local node.
Introduce Storm solution to offer real time analysis.
Implement a REST service to access to the data.
Whatever requirement we receive from federation users.
24
27. Reference Information
More information and manuals on the Monitoring GEi page at FI-Ware
Catalogue.
• http://tinyurl.com/monitoring-service
More information and manuals on the Orion Context Broker GEi page at
FI-Ware Catalogue.
• http://tinyurl.com/orion-cb-doc
More information and manuals on the COSMOS BigData GEi page ar FI-
Ware Catalogue.
• http://tinyurl.com/cosmos-doc
More FI-LAB Cloud Hosting components.
• http://tinyurl.com/cloud-hosting-ges
26