PCP

1. Red Hat Enterprise Linux 7
Performance Co-Pilot : Overview

2. Agenda :
➢ Objective of performance monitoring
➢ Traditional methods of performance monitoring
➢ Need of a new performance monitoring tool kit
➢ Introduction to Performance Co-Pilot
➢ Component and Architecture of Performance Co-Pilot
➢ Basic performance monitoring tools
➢ Demo
➢ Knowledge sources

3. Objective of performance monitoring :
➢ Early detection of a ( potential ) problem.
➢ Rapidly drill down & pinpoint issue in specific program.
➢ Reduce downtime of mission critical services/systems.

4. Traditional methods of performance
monitoring :
➢ System log files ( rsyslog/syslog-ng/journald )
➢ Native performance monitoring tools
( top/iostat/vmstat/ps etc. )
➢ Combination of scripting languages
( bash/perl/python )
➢ Specific tools vary per platform

5. Need of a new performance monitoring tool kit :
➢ Centralized monitoring of local or distributed systems in a complex
network.
➢ Automated performance monitoring of system and services.
➢ System-level analysis of performance statistics.
➢ To record and replay performance statistics.
➢ Performance regression detection.
➢ Evaluation of effects of an operating system upgrade etc.

6. Introduction to Performance Co-Pilot :
➢ PCP is a system level performance monitoring and performance
management toolkit.
➢ PCP provides a range of services that may be used for collection,
monitoring and analysis of system metrics.
➢ PCP includes many facilities for creating and replaying archive logs
that capture performance information.
➢ Cross platform support : Linux, Mac OS and Windows.
➢ End-to-end support : Hardware, Core OS, Services and
applications.

7. Introduction to Performance Co-Pilot :
➢ Distributed architecture
Monitoring of local and remote nodes.
➢ Real-time or retrospective
Live system or archive.
➢ Pluggable
New agents system metrics within PCP.

8. Performance Co-Pilot – Components :
Collectors
● Collect and export performance metrics
● Performance Metric Domain Agents (PMDA)
● Performance Metric Collection Daemon (PMCD)
Consumers
● Record, visualise, monitor and analyse performance data
● Consume data either in realtime or replay historical data
from archive logs
Note: Hosts may operate as collectors, consumers or both.
● Multiple consumers may connect with one or more collectors

9. Performance Co-Pilot – Collectors :
Performance Metric Domain Agents (PMDA)
● Extracts & exports metric data from a system component
● Communicates with pmcd on local system
Performance Metric Collection Daemon (PMCD)
● One pmcd process per host
● Coordinates handling of fetch requests between consumer
applications and agents
● Listens for connections from localhost & remote clients
● Authenticated & encrypted connection options

10. Performance Co-Pilot – Consumers :
● pmlogger
● Utility to capture and store metrics exported by PMCD
● Concurrent logging of data from local and remote hosts
● Archive playback by other consumer tools (eg pmchart, pmval)
● Tools for log archival, log rotation etc
● Operates on data in realtime only
● pminfo - display PCP metrics available on a host or in an archive
● pmchart - GUI utility providing graphical display of PCP data
● pmstat - vmstat-like utility
● pmatop - top-like utility
● pmie - inference engine and alerting utility
● pmval - display a metric

11. Performance Co-Pilot - Architecture
Webserver
DBMS
Network
Kernel
PMDAs
PMCD
pmchart
pmstat
pmlogger
Consumers
Archive ClientsConsumers

12. Performance Co-Pilot – Distributed Architecture
Agents
PMCD
PMDA's
PMCD
Host 2
PMDA's
PMCD
Host 3
PMDA's
PMCD
Host 1
ClientsConsumers
Clients
Host 4
Consumers

13. Installation:
To install pcp and pcp-gui package, type the following
command:
# yum install pcp pcp­gui
“pcp” package provides a framework and services to support
system-level performance monitoring and performance
management.
“pcp-gui” package provides a visualization tools for the
Performance Co-Pilot toolkit.

14. Services:
To start pmcd and pmlogger services, type the following command:
# chkconfig pmcd on
# chkconfig pmlogger on
# service pmcd start
# service pmlogger start
To check the status of pcp services, type the following command:
# service pcp status
Note: Bug: 1044682 - pcp should use systemd

15. Verification:
pcp - Summary of PCP installation
# pcp
Performance Co­Pilot configuration on localhost.localdomain:
platform: Linux localhost.localdomain 3.10.0­121.el7.x86_64 #1 SMP Tue
Apr 8 10:48:19 EDT 2014 x86_64
hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM
timezone: IST­5:30
pmcd: Version 3.8.10­1, 6 agents
pmda: pmcd proc xfs linux mmv jbd2

16. Configuration files:
Main configuration file for pcp:
/etc/pcp.conf
Main configuration file for pmcd:
/etc/pcp/pmcd/pmcd.conf
Command line options for the pmcd:
/etc/pcp/pmcd/pmcd.options
Default pmlogger config file:
/etc/pcp/pmlogger/config.default
PCP archive logging configuration/control:
/etc/pcp/pmlogger/control
PCP log control mechanisms:
/var/lib/pcp/config/pmlogconf
Note: pmlogger is utility to capture and store metrics exported by PMCD

17. Log files:
Log file directories for pcp components:
/var/log/pcp/pmcd
/var/log/pcp/pmie
/var/log/pcp/pmlogger
/var/log/pcp/pmmgr
/var/log/pcp/pmproxy
/var/log/pcp/pmwebd

18. Directory Organization for Archive Log
Files

19. Performance monitoring tools:
➢ For all PCP monitoring tools, metrics values may come from a real-
time feed (i.e. from pmcd on some host), or from an archive log.
➢ Performance monitoring tools available in Performance Co-Pilot
(PCP).
pmstat
pmatop
pmcollectl
pmval
pmchart
pminfo

20. Performance monitoring tools:
pmstat - vmstat-like utility, intended to monitor system
performance at the highest level.
# pmstat
# pmstat ­t 1 ­T 3
# pmstat ­a
/var/log/pcp/pmlogger/localhost.localdomain/20140607.20.20.0

21. Performance monitoring tools:
pmatop - top-like utility
#pmatop
Record Mode:
pmatop ­w rawfile [ interval [ samples ]]
Playback Mode:
pmatop ­r [ rawfile ] [­g|­m] [­L linelen] [­h host]
Eg:
# pmatop ­w atop.out 1 4
# pmatop ­r atop.out

22. Performance monitoring tools:
pmcollectl - System-level performance monitoring utility that
records or displays specific operating system data for one or more
sets of subsystems.
Basic System­level performance monitoring:
# pmcollectl
Subsystem Specific performance monitoring:
# pmcollectl ­sm
# pmcollectl ­sm –verbose
# pmcollectl ­smcdn –verbose
Record Mode:
# pmcollectl ­f perf­data ­c 3
Playback Mode:
# pmcollectl ­p perf­data

23. Performance monitoring tools:
pminfo - Display PCP metrics available on a host or in an archive.
Full list of all available metrics:
# pminfo ­F
Fetch and print values for all or specific metrics:
# pminfo ­f
# pminfo ­f <metric>
# pminfo ­F ­a
/var/log/pcp/pmlogger/localhost.localdomain/20140607.20.56.0
Eg:
# pminfo ­f proc.nprocs disk.dev.read filesys.free
Descriptions of all or specific metrics:
# pminfo ­T ­d
# pminfo ­T ­d <metric>
Eg:
# pminfo ­T ­d mem.util.cached
# pminfo ­T ­d proc.nprocs disk.dev.read filesys.free

24. Performance monitoring tools:
pmval - Dumps the current values for the named performance
metrics.
# pmval <metric>
Eg:
# pmval proc.nprocs
# pmval kernel.all.load
# pmval ­t 2sec ­s 4 kernel.percpu.cpu.idle

25. Performance monitoring tools:
pmchart - GUI utility providing graphical display of PCP data

26. Knowledge sources:
● What are all the Performance Co-Pilot (PCP) RPM packages in RHEL?
https://access.redhat.com/articles/1146003
● How do I install Performance Co-Pilot (PCP) on my RHEL server to capture
performance logs
https://access.redhat.com/solutions/1137023
● How do I configure a firewall on a RHEL server to allow remote monitoring with
Performance Co-Pilot (PCP)?
https://access.redhat.com/solutions/1145963
● How can I customize the Performance Co-Pilot logging configuration
https://access.redhat.com/articles/1146283

27. Knowledge sources:
● What are the typical Performance Co-Pilot (PCP) deployment strategies ?
https://access.redhat.com/articles/1147393
● PCP extensibility through custom agents
https://access.redhat.com/solutions/749813
● Index of Performance Co-Pilot (PCP) articles, solutions, tutorials and white papers
https://access.redhat.com/articles/1145953
● How does Performance Co-Pilot (PCP) compare with sysstat
https://access.redhat.com/articles/1148133
● Overview of Additional Performance Tuning Utilities in Red Hat Enterprise Linux 7
https://access.redhat.com/articles/785283

28. Questions ?