Chapter 1. Programming Performance Co-Pilot
Table of Contents
Performance Co-Pilot (PCP) provides a systems-level suite of tools that cooperate to deliver distributed, integrated performance management services. PCP is designed for the in-depth analysis and sophisticated control that are needed to understand and manage the hardest performance problems in the most complex systems.
PCP provides unparalleled power to quickly isolate and understand performance behavior, resource utilization, activity levels and performance bottlenecks.
Performance data may be collected and exported from multiple sources, most notably the hardware platform, the operating system kernel, layered services, and end-user applications.
- By writing a Performance Metrics Domain Agent (PMDA) to collect performance metrics from an uncharted performance domain (Chapter 2, Writing a PMDA)
- By creating new analysis or visualization tools using documented functions from the Performance Metrics Application Programming Interface (PMAPI) (Chapter 3, PMAPI--The Performance Metrics API)
Finally, the topic of customizing an installation is covered in the chapter on customizing and extending PCP service in the Performance Co-Pilot User's and Administrator's Guide.
1.1. PCP Architecture
This section gives a brief overview of PCP architecture. For an explanation of terms and acronyms, refer to Appendix A, Acronyms.
PCP consists of numerous monitoring and collecting tools.
Monitoring tools such as pmval and pminfo report on metrics, but have minimal interaction with target systems. Collection tools, called PMDAs, extract performance values from target systems, but do not provide user interfaces.
Systems supporting PCP services are broadly classified into two categories:
- Collector
- Hosts that have the PMCD and one or more PMDAs running to collect and export performance metrics
- Monitor
- Hosts that import performance metrics from one or more collector hosts to be consumed by tools to monitor, manage, or record the performance of the collector hosts
Each PCP enabled host can operate as a collector, or a monitor, or both.
Figure 1.1, “PCP Global Process Architecture” shows the architecture of PCP. The monitoring tools consume and process performance data using a public interface, the Performance Metrics Application Programming Interface (PMAPI).
Below the PMAPI level is the PMCD process, which acts in a coordinating role, accepting requests from clients, routing requests to one or more PMDAs, aggregating responses from the PMDAs, and responding to the requesting client.
Each performance metric domain (such as the operating system kernel or a database management system) has a well-defined name space for referring to the specific performance metrics it knows how to collect.
1.1.1. Distributed Collection
The performance metrics collection architecture is distributed, in the sense that any monitoring tool may be executing remotely. However, a PMDA is expected to be running on the operating system for which it is collecting performance measurements; there are some notable PMDAs such as Cisco and Cluster that are exceptions, and collect performance data from remote systems.
As shown in Figure 1.2, “Process Structure for Distributed Operation”, monitoring tools communicate only with PMCD. The PMDAs are controlled by PMCD and respond to requests from the monitoring tools that are forwarded by PMCD to the relevant PMDAs on the collector host.
The host running the monitoring tools does not require any collection tools, including PMCD, since all requests for metrics are sent to the PMCD process on the collector host.
The connections between monitoring tools and PMCD processes are managed in
libpcp, below the PMAPI level; see the PMAPI(3) man page. Connections between PMDAs and PMCD are managed by the PMDA functions; see the PMDA(3) and pmcd(1) man pages. There can be multiple monitor clients and multiple PMDAs on the one host, but there may be only one PMCD process.
1.1.2. Name Space
Each PMDA provides a domain of metrics, whether they be for the operating system, a database manager, a layered service, or an application module. These metrics are referred to by name inside the user interface, and with a numeric Performance Metric Identifier (PMID) within the underlying PMAPI.
The PMID consists of three fields: the domain, the cluster, and the item number of the metric. The domain is a unique number assigned to each PMDA. For example, two metrics with the same domain number must be from the same PMDA. The cluster and item numbers allow metrics to be easily organized into groups within the PMDA, and provide a hierarchical taxonomy to guarantee uniqueness within each PMDA.
1.1.3. Distributed PMNS
Performance metric namespace (PMNS) operations are directed by default to the host or set of archives that is the source of the desired performance metrics.
In Figure 1.2, “Process Structure for Distributed Operation”, both Performance Metrics Collection Daemon (PMCD) processes would respond to PMNS queries from monitoring tools by referring to their local PMNS. If different PMDAs were installed on the two hosts, then the PMNS used by each PMCD would be different, to reflect variations in available metrics on the two hosts.
Although extremely rarely used, the
-n pmnsfile command line option may be used with many PCP monitoring tools to force use of a local PMNS file in preference to the PMNS at the source of the metrics.
1.1.4. Retrospective Sources of Performance Metrics
The distributed collection architecture described in the previous section is used when PMAPI clients are requesting performance metrics from a real-time or live source.
The PMAPI also supports delivery of performance metrics from a historical source in the form of a PCP archive log. Archive logs are created using the
pmlogger utility, and are replayed in an architecture as shown in Figure 1.3, “Architecture for Retrospective Analysis”.