PMDAPROMETHEUS
Section: User Commands (1)
Index
Return to Main Contents
NAME
pmdaprometheus
- Prometheus PMDA
SYNOPSIS
$PCP_PMDAS_DIR/prometheus/pmdaprometheus
[
-D
]
[
-n
]
[
-c
config
]
[
-t
timeout
]
[
-u
user
]
DESCRIPTION
pmdaprometheus
is a Performance Metrics Domain Agent (PMDA) which
creates PCP metrics from Prometheus endpoints, which provide HTTP based
access to application metrics.
The default
config
directory is
$PCP_PMDAS_DIR/prometheus/config.d/
,
see ``CONFIGURATION SOURCES'' below.
The default URL fetch
timeout
is
2
seconds.
The default
user
, if not specified with the
-u
option,
is the current user.
If the
-n
option is given, the list of configuration files will not be sorted prior to processing.
This list is sorted by default but that can be expensive if there are a large number of
configuration files (URLs and/or scripts).
If the
-D
option is given, additional diagnostic messages will be written to the PMDA log file,
which is
$PCP_LOG_DIR/pmcd/prometheus.log
by default.
Further details on the Prometheus exposition format can be found at
https://prometheus.io/docs/instrumenting/exposition_formats
CONFIGURATION SOURCES
As it runs,
pmdaprometheus
periodically recursively scans the
$PCP_PMDAS_DIR/prometheus/config.d
directory (or the directory specified with the
-c
option), looking for source URL files (
*.url
)
and executable scripts or binaries.
Any files that do not have the
.url
suffix or are not executable, are ignored - this allows documentation files
such as "README" and non-executable "common" script function definitions to
be present without being considered as config files.
A remote server does not have to be up or stay running - the PMDA tolerates
remote URLs that may come and go over time.
The PMDA will relay data and metadata when/if they are available,
and will return errors when/if they are down.
PCP metric IDs, internal and external instance domain identifiers are
persisted and will be restored when individual metric sources become
available and/or when the PMDA is restarted.
In addition, the PMDA checks directory modification times and will rescan
for new or changed configuration files dynamically.
It is
not
necessary to restart the PMDA when adding, removing or changing configuration files.
URL SOURCES
Each file with the
.url
suffix found in the config directory or sub-directory contains
one complete HTTP or HTTPS URL at which
pmdaprometheus
can reach a Prometheus endpoint.
Local file access is also supported with a conventional
file://somepath/somefile
URL, in which case
somepath/somefile
should contain prometheus formatted metric data.
The first line of a
.url
config file should be the URL, as described above.
Subsequent lines, if any, are prefixed with a keyword that can be
used to alter the http GET request.
A keyword must end with
':'
(colon) and the text extends to the end of the line.
Comment lines that start with
#
and blank lines are ignored.
The only currently supported keyword is
HEADER:
.
HEADER:
headername
:
value ... to end of line
Adds
headername
and its value
to the headers passed in the http GET request for the configured URL.
An example configuration file that provides 3 commonly used headers
and an authentication token might be :
http://somehost/path/endpoint.html
# this is a comment
HEADER: Accept: text/html
HEADER: Keep-Alive: 300
HEADER: Connection: keep-alive
HEADER: Authorization: token ABCDEF1234567890
As mentioned above, header values extend to the end of the line.
They may contain any valid characters, including colons.
Multiple spaces will be collapsed to a single space, and leading
and trailing spaces are trimmed.
A common use for headers is to configure a proxy agent
and the assorted parameters it may require.
Additional keywords may be added in future versions of
pmdaprometheus
e.g. to allow returned result filtering, label manipulation and so forth.
Unrecognized keywords are reported in the PMDA log file but otherwise ignored.
SCRIPTED SOURCES
Executable scripts present in the
$PCP_PMDAS_DIR/prometheus/config.d
directory or sub-directories will be executed and the
stdout
stream containing prometheus formatted metric data will be parsed as though it had come from a URL or file.
The
stderr
stream from a script will be sent to the PMDA log file, which by default can be found in
$(PCP_LOG_DIR)/pmcd/prometheus.log
.
A simple example of a scripted config entry follows:
#! /bin/sh
awk '{
print("# HELP loadavg local load average")
print("# Type loadavg gauge")
printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
}' /proc/loadavg
This script produces the following Prometheus-formatted metric
data when run:
# HELP loadavg local load average
# Type loadavg gauge
loadavg {interval="1-minute"} 0.12
loadavg {interval="5-minute"} 0.27
loadavg {interval="15-minute"} 0.54
If the above script was saved and made executable in a file named
$PCP_PMDAS_DIR/prometheus/config.d/local/system.sh
then this would result in a new PCP metric named
prometheus.local.system.loadavg
which would have three instances for the current load average values:
1-minute
,
5-minute
and
15-minute
.
Scripted config entries may produce more than one PCP leaf metric name.
For example, the above "system.sh" script could also export other metrics
such as CPU statistics, by reading
/proc/stat
on the local system.
Such additional metrics would appear as peer metrics in the
same PCP metric subtree.
In the case of CPU counters, the metric type definition should be
counter
,
not
gauge
.
For full details of the prometheus exposition formats, see
https://prometheus.io/docs/instrumenting/exposition_formats
.
METRIC NAMING
All metrics from a file named
JOB
.*
will be exported as PCP metrics with the
prometheus.JOB
metric name prefix.
Therefore, the JOB name must be a valid non-leaf name for PCP PMNS
metric names.
If the
JOB
name has multiple dot-separated components, the resulting
PMNS names will include those components and care is needed to ensure
there are no overlapping definitions, e.g. metrics returned by
JOB.response
may overlap or conflict with metrics returned by
JOB.response.time
.
Config file entries (URLs or scripts) found in subdirectories of the
config directory will also result in hierarchical metric names.
For example, a config file named
$PCP_PMDAS_DIR/prometheus/config.d/mysource/latency/get.url
will result in metrics being created (by fetching that source URL) below
prometheus.mysource.latency.get
in the PCP namespace.
Scripts found in subdirectories of the config directory similarly result
in hierarchical PCP metric names.
DYNAMIC METRIC NAMES
As described above, changes and new additions can be made to files in
the configuration directory without having to restart the PMDA.
These changes are detected automatically and the PCP metric names below
prometheus
in the PMNS will be updated accordingly, i.e. new metrics will be
dynamically added and/or existing metrics removed.
In addition,
pmdaprometheus
honors the PMCD_NAMES_CHANGE
pmFetch
(3)
protocol that was introduced in PCP version 4.0.
In particular, if
prometheus
metrics are being logged by a PCP version 4.0 or later
pmlogger
(1),
new metrics that appear as a result of changes in the PMDA configuration
directory will automatically start to be logged, provided the root of the
prometheus
PMDA namespace is configured for logging in the
pmlogger
configuration file.
See
pmlogger
(1)
for details.
An example of such a
pmlogger
configuration file is :
log mandatory on 2 second {
# log all metrics below the root of the prometheus namespace
prometheus
}
CONTROL METRICS
The PMDA maintains three special control metrics, as described below.
Each of these metrics is a counter and has one instance for each configured metric source.
The instance domain is adjusted dynamically as new sources are discovered.
If there are no sources configured, the metric names are still defined
but the instance domain will be empty and a fetch will return no values.
-
prometheus.control.calls
-
total number of times each configured metric source has been fetched (if it's a URL)
or executed (if it's a script), since the PMDA started.
-
prometheus.control.fetch_time
-
Total time in milliseconds that each configured metric source has taken to return a document,
excluding the time to parse the document.
-
prometheus.control.parse_time
-
Total time in milliseconds that each configured metric source has taken to parse each document,
excluding the time to fetch the document.
When converted to a rate, the
calls
metric represents the average fetch rate of each source
over the sampling interval (time delta between samples).
The
fetch_time
and
parse_time
counters, when converted to a rate, represent the
average fetch and parsing latency (respectfully), during the sampling interval.
LIMITATIONS
pmdaprometheus
and
libpcp
internals impose some numerical constraints about the number of sources (4095),
metrics (1024) within each source, and instances for each metric (4194304).
INSTALLATION
Install the Prometheus PMDA by using the Install script as root:
# cd $PCP_PMDAS_DIR/prometheus
# ./Install
To uninstall, do the following as root:
# cd $PCP_PMDAS_DIR/prometheus
# ./Remove
pmdaprometheus
is launched by
pmcd
(1)
and should never be executed directly.
The Install and Remove scripts notify
pmcd
when the agent is installed or removed.
When scripts and
.url
files are added, removed or changed in the configuration directory,
it is usually not necessary to restart the PMDA - the changes will
be detected and managed on subsequent requests to the PMDA.
FILES
-
$PCP_PMDAS_DIR/prometheus/Install
-
installation script for the
pmdaprometheus
agent
-
$PCP_PMDAS_DIR/prometheus/Remove
-
undo installation script for the
pmdaprometheus
agent
-
$PCP_PMDAS_DIR/prometheus/config.d/
-
contains URLs and scripts used by the
pmdaprometheus
agent as sources of prometheus metric data.
-
$PCP_LOG_DIR/pmcd/prometheus.log
-
default log file for error messages from
pmdaprometheus
-
$PCP_VAR_DIR/config/144.*
-
files containing internal tables for metric and instance ID number persistence (domain 144).
PCP ENVIRONMENT
Environment variables with the prefix
PCP_
are used to
parameterize the file and directory names used by
PCP
.
On each installation, the file
/etc/pcp.conf
contains the local values for these variables.
The
$PCP_CONF
variable may be used to specify an alternative
configuration file, as described in
pcp.conf
(5).
SEE ALSO
pmcd
(1),
pminfo
(1),
pmlogger
(1),
PMWEBAPI
(3),
pmFetch
(3)
and
https://prometheus.io/docs/instrumenting/exposition_formats
.
Index
-
NAME
-
-
SYNOPSIS
-
-
DESCRIPTION
-
-
CONFIGURATION SOURCES
-
-
URL SOURCES
-
-
SCRIPTED SOURCES
-
-
METRIC NAMING
-
-
DYNAMIC METRIC NAMES
-
-
CONTROL METRICS
-
-
LIMITATIONS
-
-
INSTALLATION
-
-
FILES
-
-
PCP ENVIRONMENT
-
-
SEE ALSO
-