Enhancements for PCP archives

One of the key features of PCP is that real-time and historical sources of performance metrics are equivalent from the perspective of a client application consuming and processing performance data. The PCP archive format is the corner-stone of the historical data support. This project aims to improve the usefulness and performance of PCP archives through a number of independent sub-projects.

  • Over time, we would like to reduce the volume of archived data by increasing the effective sampling interval, e.g. going from sub-minute sampling for the most recent archives to hourly sampling for weekly or monthly archives. This process needs to account for counters that might wrap, counters that might be reset at system or service restarts, small amounts of missing data, etc. while maintaining the semantics of the original data. pmlogreduce already does some of this, but there are a number of issues that can best be addressed by a complete re-implementation.
  • There is considerable redundancy in the on-disk metadata that could be reduced by a new incremental format for instance domains.
  • When an archive is opened, not all the metrics are of interest to a particular applications, so run-time efficiency could be enhanced by lazy (just in time) loading of metadata.
These projects will require the student to gain a deep understanding of the PCP archive format created by pmlogger and to apply basic statistical concepts to the transformation of timeseries data.

Expected results: The student will extend their C language programming skills with both green fields and maintenance development within a big and mature software project. They can also expect to gain deep insight into the semantics of performance data stored within a PCP archive.

Prerequisite knowledge: C programming.

Skill level: Intermediate-Advanced

Mentors: Ken McDonell , Mark Goodwin

Interested students so far: 1


PCP version of collectl

The Performance Co-Pilot provides a cross-platform API for accessing live and historical performance data from the local or remote hosts. An initial PCP implementation (pcp-collectl) of the Collectl utility provides some initial support for the same style of reports. This project will extend that initial code to complete the reporting coverage provided by the initial collectl utility. The pcp-collectl utility will be updated to use the more modern pmrep utility modules under the covers. Additionally, the test coverage and documentation will be improved.

Expected results: The student will extend their Python programming skills, and can also expect to gain insight into the meaning of the kernel data reported by this tool, and the sampling techniques it uses.

Prerequisite knowledge: Python programming.

Skill level: Introductory-Intermediate

Mentors: Mark Goodwin , Nathan Scott

Interested students so far: 3


PCP version of nmon

The Performance Co-Pilot provides a cross-platform API for accessing live and historical performance data from the local or remote hosts. The nmon utility provides a curses-based interface for reporting on a variety of system and per-process metrics to a console, as well as a CSV (Comma Separated Value) output format. This project will implement a version of the nmon utility using the PCP metrics API (PMAPI), based on the current nmon source code. Additionally, regression tests will be implemented and documentation will be improved.

Expected results: The student will extend their C language programming skills, and can also expect to gain insight into the meaning of the kernel data reported by this tool, and the sampling techniques it uses.

Prerequisite knowledge: C programming.

Skill level: Introductory-Intermediate

Mentors: Nathan Scott , Mark Goodwin

Interested students so far: 5


Using metrics from statsd with PCP

Performance Co-Pilot is a system performance analysis toolkit which currently does not provide an agent for statsd protocol, a text-based UDP protocol for performance data aggregation. The goal is to design and write a PCP agent as a multi-threaded C process performing aggregation of statsd packets as pluggable modules with two implementations: exact computation and HDR histogram with parsing code as pluggable modules: custom parser or ragel-based parser. Additionally, regression tests will be implemented and documentation provided.

Expected results: The student will extend their C language programming skills, and learn about the inner workings of performance analysis tools like PCP and statsd.

Prerequisite knowledge: C programming.

Skill level: Introductory-Intermediate

Mentors: Lukas Zapletal , Nathan Scott

Interested students so far: 3


Using metrics from bpftrace with PCP and Vector

bpftrace is a high-level tracing language for efficient kernel tracing using the Linux kernel eBPF (extended Berkeley Packet Filters) feature. The goal is to design and write a PCP agent which runs arbitrary bpftrace scripts and stores the output as PCP metrics, and a new Vector widget which visualizes these collected metrics in a live heat map or table.

The Vector widget will include a bpftrace query builder for the rapid creation of bpftrace scripts.

Regression tests will be implemented and documentation provided.

Expected results: The student will extend their C language programming skills, and learn about the inner workings of performance analysis tools like PCP, Vector and eBPF.

Prerequisite knowledge: C and Javascript programming.

Skill level: Advanced

Mentors: Martin Spiers , Marko Myllynen

Interested students so far: 1


Scaling timeseries injest and querying

The Performance Co-Pilot approach to scalable, multi-host performance analysis builds on the Redis distributed data store and its native timeseries support. The pmseries utility and pmproxy daemon provide the tooling and APIs to support this. This project will explore scalability limits of each these programs, with the aim of scaling up PCP timeseries injest and querying to the level of many-thousands-of-nodes.

Expected results: The student will extend their C language programming skills, learn about the Redis distributed data store (and Redis clustering in particular), gain deep familiarity with performance tuning tools such as perf as well as learning to apply PCP tools to analyse complex distributed system performance problems.

Prerequisite knowledge: C programming.

Skill level: Intermediate-Advanced

Mentors: Nathan Scott , Mark Goodwin

Interested students so far: 2


Grafana dashboards for PCP metrics

This project will involve developing Grafana dashboards and panels for displaying high level performance data across the common system level performance analysis domains - CPU, memory, network, disk, applications (and many more). In Grafana, each high level panel can be linked (via a URL) to more detailed performance data, and thus offer an intuitive drill-down model for rapidly solving common, yet complex, performance issues.

The Grafana PCP data source accesses all PCP metric data via Redis and pmproxy REST APIs. Resulting Grafana charts and panels can be exported in JSON format with template variables and included as part of Grafana itself, or shipped separately as part of PCP and installed/imported using the Grafana provisioning mechanisms.

Additionally, the Grafana world map panel displays time series data or geohash data from a metric source overlaid on a world map. The PCP pmseries tool and APIs can injest and respond to queries about performance metrics labeled with geographical location information, with the help of the geohash support in the Redis distributed data store. This project will implement tooling and APIs to interface Grafana and Redis for PCP metrics.

Expected results: The student will extend their C and Javascript programming skills, will gain insight into the semantics of various forms of performance data available from systems and applications, and visualization techniques appropriate to their analysis. The student will also learn about the Redis distributed data store and geolocation services.

Prerequisite knowledge: C, Javascript programming.

Skill level: Intermediate

Mentors: Mark Goodwin , Nathan Scott

Interested students so far: 6