Product SiteDocumentation Site

Chapter 4. Instrumenting Applications

This chapter provides an introduction to ways of instrumenting applications using PCP.
The first section covers the use of the Memory Mapped Value (MMV) Performance Metrics Domain Agent (PMDA) to generate customized metrics from an application. This provides a robust, extremely efficient mechanism for transferring custom instrumentation into the PCP infrastructure. It has been successfully deployed in production environments for many years, has proven immensely valuable in these situations, and can be used to instrument applications written in a number of programming languages.
The Memory Mapped Value library and PMDA is supported on every PCP platform, and is enabled by default.


A particularly expansive Java API is available from the separate Parfait project. It supports both the existing JVM instrumentation, and custom application metric extensions.
The chapter also includes information on how to use the MMV library (libpcp_mmv) for instrumenting an application. The example programs are installed in ${PCP_DEMOS_DIR}/mmv.
The second section covers the design of the Trace PMDA, in an effort to explain how to configure the agent optimally for a particular problem domain. This information supplements the functional coverage which the man pages provide to both the agent and the library interfaces.
This part of the chapter also includes information on how to use the Trace PMDA and its associated library (libpcp_trace) for instrumenting applications. The example programs are installed in ${PCP_DEMOS_DIR}/trace.


The current PCP trace library is a relatively heavy-weight solution, issuing multiple system calls per trace point, runs over a TCP/IP socket even locally and performs no event batching. As such it is not appropriate for production application instrumentation at this stage.
A revised application tracing library and PMDA are planned which will be light-weight, suitable for production system tracing, and support event metrics and other advances in end-to-end distributed application tracing.
The application instrumentation libraries are designed to encourage application developers to embed calls in their code that enable application performance data to be exported. When combined with system-level performance data, this feature allows total performance and resource demands of an application to be correlated with application activity.
For example, developers can provide the following application performance metrics:
  • Computation state (especially for codes with major shifts in resource demands between phases of their execution)
  • Problem size and parameters, that is, degree of parallelism throughput in terms of sub-problems solved, iteration count, transactions, data sets inspected, and so on
  • Service time by operation type

4.1. Application and Performance Co-Pilot Relationship

The relationship between an application, the pcp_mmv and pcp_trace instrumentation libraries, the MMV and Trace PMDAs, and the rest of the PCP infrastructure is shown in Figure 4.1, “Application and PCP Relationship”:
Application and PCP Relationship

Figure 4.1. Application and PCP Relationship

Once the application performance metrics are exported into the PCP framework, all of the PCP tools may be leveraged to provide performance monitoring and management, including:
  • Two- and three-dimensional visualization of resource demands and performance, showing concurrent system activity and application activity.
  • Transport of performance data over the network for distributed performance management.
  • Archive logging for historical records of performance, most useful for problem diagnosis, postmortem analysis, performance regression testing, capacity planning, and benchmarking.
  • Automated alarms when bad performance is observed. These apply both in real-time or when scanning archives of historical application performance.