Product SiteDocumentation Site

3.6. Performance Event Metrics

In addition to performance metric values which are sampled by monitor tools, PCP supports the notion of performance event metrics which occur independently to any sampling frequency. These event metrics (PM_TYPE_EVENT) are delivered using a novel approach which allows both sampled and event trace data to be delivered via the same live wire protocol, the same on-disk archive format, and fundamentally using the same PMAPI services. In other words, a monitor tool may be sample and trace, simultaneously, using the PMAPI services discussed here.
Event metrics are characterised by certain key properties, distinguishing them from the other metric types (counters, instantaneous, and discrete):
  • Occur at times outside of any monitor tools control, and often have a fine-grained timestamp associated with each event.
  • Often have parameters associated with the event, which further describe each individual event, as shown in Figure 3.2, “Sample write(2) syscall entry point encoding”.
  • May occur in very rapid succession, at rates such that both the collector and monitor sides may not be able to track all events. This property requires the PCP protocol to support the notion of "dropped" or "missed" events.
  • There may be inherent relationships between events, for example the start and commit (or rollback) of a database transaction could be separate events, linked by a common transaction identifier (which would likely also be one of the parameters to each event). Begin-end and parent-child relationships are relatively common, and these properties require the PCP protocol to support the notion of "flags" that can be associated with events.
These differences aside, the representation of event metrics within PCP shares many aspects of the other metric types - event metrics appear in the Name Space (as do each of the event parameters), each has an associated Performance Metric Identifier and Descriptor, may have an instance domain associated with them, and may be recorded by pmlogger for subsequent replay.
Sample syscall entry point encoding

Figure 3.2. Sample write(2) syscall entry point encoding

Event metrics and their associated information (parameters, timestamps, flags, and so on) are delivered to monitoring tools alongside sampled metrics as part of the pmResult structure seen previously in Example 3.9, “ pmResult Structure”.
The semantics of pmFetch(3) specifying an event metric PMID are such that all events observed on the collector since the previous fetch (by this specific monitor client) are to transfered to the monitor. Each event will have the metadata described earlier encoded with it (timestamps, flags, and so on) for each event. The encoding of the series of events involves a compound data structure within the pmValueSet associated with the event metric PMID, as illustrated in Figure 3.3, “Result Format for Event Performance Metrics from pmFetch.
Result Format for Event Performance Metrics from pmFetch

Figure 3.3. Result Format for Event Performance Metrics from pmFetch

At the highest level, the "series of events" is encapsulated within a pmEventArray structure, as in Example 3.10, “ pmEventArray and pmEventRecord Structures”:

Example 3.10.  pmEventArray and pmEventRecord Structures

typedef struct {
    pmTimeval      er_timestamp;  /* 2 x 32-bit timestamp format */
    unsigned int     er_flags;      /* event record characteristics */
    int              er_nparams;    /* number of ea_param[] entries */
    pmEventParameter er_param[1];
} pmEventRecord;

typedef struct {
    unsigned int     ea_len :  24;  /* bytes for type/len + records */
    unsigned int     ea_type : 8;   /* value type */
    int              ea_nrecords;   /* number of ea_record entries */
    pmEventRecord    ea_record[1];
} pmEventArray;
Note that in the case of dropped events, the pmEventRecord structure is used to convey the number of events dropped - er_flags is used to indicate the presence of dropped events, and er_nparams is used to hold a count. Unsurprisingly, the parameters (er_param) will be empty in this situation.
The pmEventParameter structure is as follows:

Example 3.11.  pmEventParameter Structure

typedef struct {
    pmID             ep_pmid;       /* parameter identifier */
    unsigned int     ep_type;       /* value type */
    int              ep_len;        /* bytes for type/len + vbuf */
    /* actual value (vbuf) here */
} pmEventParameter;

3.6.1. Event Monitor Considerations

In order to simplify the decoding of event record arrays, the PMAPI provides the pmUnpackEventRecords function for monitor tools. This function is passed a pointer to a pmValueSet associated with an event metric (within a pmResult) from a pmFetch(3). For a given instance of that event metric, it returns an array of "unpacked" pmResult structures for each event.
The control information (flags and optionally dropped events) is included as derived metrics within each result structure. As such, these values can be queried similarly to other metrics, using their names - event.flags and event.missed. Note that these metrics will only exist after the first call to pmUnpackEventRecords.
An example of decoding event metrics in this way is presented in Example 3.12, “Unpacking Event Records from an Event Metric pmValueSet:

Example 3.12. Unpacking Event Records from an Event Metric pmValueSet

enum { event_flags = 0, event_missed = 1 };
static char *metadata[] = { "event.flags", "event.missed" };
static pmID metapmid[2];

void dump_event(pmValueSet *vsp, int idx)
    pmResult    **res;
    int	        r, sts, nrecords;

    nrecords = pmUnpackEventRecords(vsp, idx, &res);
    if (nrecords < 0)
        fprintf(stderr, " pmUnpackEventRecords: %s\n", pmErrStr(nrecords));
        printf(" %d event records\n", nrecords);

    if ((sts = pmLookupName(2, &metadata, &metapmid)) < 0) {
        fprintf(stderr, "Event metadata error: %s\n", pmErrStr(sts));

    for (r = 0; r < nrecords; r++)
        dump_event_record(res, r);

    if (nrecords >= 0)

void dump_event_record(pmResult *res, int r)
    int         p;

    pmPrintStamp(stdout, &res[r]->timestamp);
    if (res[r]->numpmid == 0)
       	printf(" ==> No parameters\n");
    for (p = 0; p < res[r]->numpmid; p++) {
        pmValueSet  *vsp = res[r]->vset[p];

        if (vsp->numval < 0) {
            int error = vsp->numval;
            printf("%s: %s\n", pmIDStr(vsp->pmid), pmErrStr(error));
        } else if (vsp->pmid == metapmid[event_flags]) {
            int flags = vsp->vlist[0].value.lval;
            printf(" flags 0x%x (%s)\n", flags, pmEventFlagsStr(flags));
        } else if (vsp->pmid == metapmid[event_missed]) {
            int count = vsp->vlist[0].value.lval;
            printf(" ==> %d missed event records\n", count);
        } else {

void dump_event_record_parameters(pmValueSet *vsp)
    pmDesc      desc;
    char        *name;
    int         sts, j;

    if ((sts = pmLookupDesc(vsp->pmid, &desc)) < 0) {
        fprintf(stderr, "pmLookupDesc: %s\n", pmErrStr(sts));
    } else
    if ((sts = pmNameID(vsp->pmid, &name)) < 0) {
        fprintf(stderr, "pmNameID: %s\n", pmErrStr(sts));
    } else {
        printf("parameter %s", name);
        for (j = 0; j < vsp->numval; j++) {
            pmValue *vp = &vsp->vlist[j];
            if (vsp->numval > 1) {
                printf("[%d]", vp->inst);
                pmPrintValue(stdout, vsp->valfmt, desc.type, vp, 1);