Product SiteDocumentation Site

3.6.2. Event Collector Considerations

There is a feedback mechanism that is inherent in the design of the PCP monitor-collector event metric value exchange, which protects both monitor and collector components from becoming overrun by high frequency event arrivals. It is important that PMDA developers are aware of this mechanism and all of its implications.
Monitor tools can query new event arrival on whatever schedule they choose. There are no guarantees that this is a fixed interval, and no way for the PMDA to attempt to dictate this interval (nor should there be).
As a result, a PMDA that provides event metrics must:
  • Track individual client connections using the per-client PMDA extensions (PMDA_INTERFACE_5 or later).
  • Queue events, preferably in a memory-efficient manner, such that each interested monitor tool (there may be more than one) is informed of those events that arrived since their last request.
  • Control the memory allocated to in-memory event storage. If monitors are requesting new events too slowly, compared to event arrival on the collector, the "missed events" feedback mechanism must be used to inform the monitor. This mechanism is also part of the model by which a PMDA can fix the amount of memory it uses. Once a fixed space is consumed, events can be dropped from the tail of the queue for each client, provided a counter is incremented and the client is subsequently informed.


It is important that PMDAs are part of the performance solution, and not part of the performance problem! With event metrics, this is much more difficult to achieve than with counters or other sampled values.
There is certainly elegance to this approach for event metrics, and the way they dovetail with other, sampled performance metrics is unique to PCP. Notice also how the scheme naturally allows multiple monitor tools to consume the same events, no matter what the source of events is. The downside to this flexibility is increased complexity in the PMDA when event metrics are used.
This complexity comes in the form of event queueing and memory management, as well as per-client state tracking. Routines are available as part of the pcp_pmda library to assist, however - refer to the man page entries for pmdaEventNewQueue(3) and pmdaEventNewClient(3) for further details.
One final set of helper APIs is available to PMDA developers who incorporate event metrics. There is a need to build the pmEventArray structure, introduced in Example 3.10, “ pmEventArray and pmEventRecord Structures”. This can be done directly, or using the helper routine pmdaEventNewArray(3). If the latter, simpler model is chosen, the closely related routines pmdaEventAddRecord, pmdaEventAddParam and pmdaEventAddMissedRecord would also usually be used.
Depending on the nature of the events being exported by a PMDA, it can be desirable to perform filtering of events on the collector system. This reduces the amount of event traffic between monitor and collector systems (which may be filtered further on the monitor system, before presenting results). Some PMDAs have had success using the pmStore(3) mechanism to allow monitor tools to send a filter to the PMDA - using either a special control metric for the store operation, or the event metric itself. The filter sent will depend on the event metric, but it might be a regular expression, or a tracing script, or something else.
This technique has also been used to enable and disable event tracing entirely. It is often appropriate to make use of authentication and user credentials when providing such a facility (PMDA_INTERFACE_6 or later).