Authenticated Connections


Tools
              pmcd
              pminfo
              pmchart
              pmconfig
              pmdaproc
              pluginviewer
              saslauthd
              saslpasswd2
              sasldblistusers2
              kadmin.local
              kinit
              klist

This chapter of the Performance Co-Pilot tutorial covers setting up authenticated connections between PCP collector and monitor components. An authenticated connection is one where the PCP collector ( pmcd and PMDAs) is made aware of the credentials of the user running the monitor tools. This allows the PCP collector software to perform two important functions:

  1. Grant additional access, allowing potentially sensitive information to be accessed by the authenticated user;
  2. Make access control decisions for users and groups in order to prevent inappropriate access to metrics or over-subscription of server resources.

For an explanation of Performance Co-Pilot terms and acronyms, consult the PCP glossary .

Note: remote host authentication will often warrant the configuration of secure connections between PCP monitors and collectors - this is covered by a separate tutorial: Secure Connections .


Check local PCP collector installation (requires the pcp-verify utility):
$ pcp verify --secure


Overview

All connections made to the PCP metrics collector daemon ( pmcd ) are made using the PCP protocol, which is TCP/IP based. Traditionally, no functionality was available to identify the user making a connection between a PCP monitor and collector. However, as PCP evolved to be able to export sensitive information (event trace parameters and detailed per-process statistics, for example), it became necessary to provide such mechanisms.

The Performance Co-Pilot has two facilities for transfering credentials between the monitoring and collecting components - a local-host-only facility using Unix domain sockets, and SASL-based authentication which can be over either IPv4 or IPv6 sockets (local or remote).


Local Credentials

For local connections, there is a fast local transport mechanism using Unix domain sockets. These sockets provide a facility whereby the userid and groupid of the monitor process is automatically made available to the collector process, with no user intervention being required.

This will become the default localhost transport mechanism over time. However, its use can be requested via the unix: host specification ( -h option) with all monitor tools.

In the following exercise, we make use of per-process metrics from the pmdaproc PCP Collector agent. This PMDA is enabled by default, and makes use of authenticated credentials from the monitor tools to gate access to sensitive information - process identifiers, names, arguments, and so forth. In modern versions of PCP, this information is unavailable (PM_ERR_PERMISSION is returned) unless the PMDA has been given user credentials.


Check for support, then establish a Unix domain socket connection with automatic credentials transfer:
$ $PCP_BINADM_DIR/pmconfig -L unix_domain_sockets
              unix_domain_sockets=true
              
              $ pmprobe -f -h localhost proc.fd.count
              proc.fd.count -12387 No permission to perform requested operation
              
              $ pmprobe -f -h unix: proc.fd.count
              proc.fd.count 118
              
              $ pminfo -dmt -h unix: proc.fd.count
              
              proc.fd.count PMID: 3.51.0 [open file descriptors]
                  Data Type: 32-bit unsigned int  InDom: 3.9 0xc00009
                  Semantics: instant  Units: count
              

A local: specification is also available, which indicates that the monitor tool should first attempt to establish a Unix domain socket connection (with automatic credentials transfer), but failing that fall back to the traditional socket connection style (with no credentials transfer, unless mandated by collector or explicitly requested by the monitor user - described later in this tutorial).

Note: this facility is completely independent and separate to the remote access facility (described later). Therefore, it can still be used even when support for the remote authentication facilities is unavailable or not configured. This local facility is always enabled on platforms that support it - these include Linux, Mac OS X and Solaris.


Remote Access

The alternative authentication facility is the "Simple Authentication and Security Layer" (SASL) which provides for remote authentication. Usually this would be used with a secure connection to ensure sensitive information is not transmitted in the clear.

SASL can be configured with many different authentication mechanisms, and this configuration is done transparently (outside of PCP, as described shortly). This list of mechanisms can be changed via the mech_list entry in the /etc/sasl2/pmcd.conf file, along with other critical security parameters defining how pmcd should handle authentication requests. The mech_list can be set to any one of the installed SASL mechanisms, or a space-separated list of mechanisms from which the monitor tools can select. There are also situations where it makes sense to not set mech_list at all, which we will explore shortly.

To make use of SASL-based authentication, both monitor and collector systems need to be capable of running the SASL code (PCP support, in particular, must be present on each end of the connection). As demonstrated in the following examples, we can use both the pmconfig PCP command and the pluginviewer SASL command to interogate aspects of the installations on each end of a connection. The pluginviewer command has separate options for reporting on client ( -c ) and server ( -s ) components of an installation.

Using sasldb

The SASL library provides a custom authentication technique, independent to the set of users and groups on a system, called sasldb . This involves the creation of a new authentication database separate to the native login mechanisms that a host provides. One advantage of using it is that it does not require any special privileges on the part of the daemon performing authentication ( pmcd in our case - which does not run as a privileged process and thus typically cannot be used with the common authentication databases, such as the /etc/shadow mechanism).

The sasldb approach can be used with several SASL mechanisms, and it is commonly used to verify a SASL setup. Here, we will configure PCP to use sasldb by allowing the "plain" SASL mechanism to authenticate against it.


Check for support:
$ $PCP_BINADM_DIR/pmconfig -L unix_domain_sockets authentication
              secure_sockets=true
              authentication=true
              
              $ pluginviewer -s -m PLAIN
              [...]
              Plugin "plain" [loaded], 	API version: 4
              	SASL mechanism: PLAIN, best SSF: 0, supports setpass: no
              	security flags: NO_ANONYMOUS|PASS_CREDENTIALS
              	features: WANT_CLIENT_FIRST|PROXY_AUTHENTICATION
              

Modify the /etc/sasl2/pmcd.conf configuration file, so that it makes "plain" authentication available to PCP monitor tools, and also to specify the location of the SASL credentials database file.

By default, this has been specified as /etc/pcp/passwd.db . SASL commands allow this database to be manipulated - adding, listing and deleting users, setting their passwords, and so on.


Configure the SASL library:
# grep PCP_SASLCONF_DIR /etc/pcp.conf
              PCP_SASLCONF_DIR=/etc/sasl2
              
              # cat $PCP_SASLCONF_DIR/sasl2/pmcd.conf
              mech_list: plain digest-md5 gssapi
              sasldb_path: /etc/pcp/passwd.db
              

Next we create the database and add some users to it. Note that requires the previous step to have been performed, as that informs the tools about the location of the database.


Administer the SASL database (add, list and disable users):
# saslpasswd2 -a pmcd jack
              Password: xxxxxx
              Again (for verification): xxxxxx
              
              # saslpasswd2 -a pmcd jill
              Password: xxxxxx
              Again (for verification): xxxxxx
              
              # sasldblistusers2 -f /etc/pcp/passwd.db
              jack@server.demo.net: userPassword
              jill@server.demo.net: userPassword
              
              # saslpasswd2 -a pmcd -d jill
              
              # $PCP_RC_DIR/pmcd restart
              

Finally, we're ready to try it out. As we have just restarted pmcd (above), its worth checking its log file - $PCP_LOG_DIR/pmcd/pmcd.log - and if that is free of errors, we can attempt to authenticate.


Establish an authenticated connection:
$ pminfo -fmdt -h pcps://server.demo.net?method=plain proc.fd.count
              Username: jack
              Password: xxxxxx
              
              proc.fd.count PMID: 3.51.0 [open file descriptors]
                  Data Type: 32-bit unsigned int  InDom: 3.9 0xc00009
                  Semantics: instant  Units: count
                  inst [1281 or "001281 bash"] value 4
                  inst [1287 or "001287 make -j 8 default_pcp"] value 5
                  inst [1802 or "001802 bash"] value 4
                  [...]
              
Using saslauthd

The SASL mechanism configured by default on a PCP collector system is sasldb which provides a basic username and password style authentication mechanism with no reliance on external daemons, package dependencies, and so on. Without customisation for individual users, as described above, it allows no access and can thus be considered secure-by-default.

However, it is often more convenient to use the same authentication mechanism that is used when logging in to a host (e.g. PAM on Linux and Solaris). Because the PCP daemons run without superuser privileges, they cannot perform this authentication themselves, and so it must be achieved using a separate, privileged process. SASL provides saslauthd for this purpose.


Check for support:
$ pmconfig -L secure_sockets authentication
              secure_sockets=true
              authentication=true
              
              $ pluginviewer -s -m PLAIN
              [...]
              Plugin "plain" [loaded], 	API version: 4
              	SASL mechanism: PLAIN, best SSF: 0, supports setpass: no
              	security flags: NO_ANONYMOUS|PASS_CREDENTIALS
              	features: WANT_CLIENT_FIRST|PROXY_AUTHENTICATION
              
              $ ps ax | grep saslauthd
               2857 ?        Ss     0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
               2858 ?        S      0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
              [...]
              

In this case, the SASL mechanism configuration is entirely handled by saslauthd so we simply need to ensure we pass authentication through to the daemon - no configuration beyond that should be needed.


Setup use of saslauthd on the PCP Collector:
# grep PCP_SASLCONF_DIR /etc/pcp.conf
              PCP_SASLCONF_DIR=/etc/sasl2
              
              # cat $PCP_SASLCONF_DIR/sasl2/pmcd.conf
              pwcheck_method: saslauthd
              
              # $PCP_RC_DIR/pmcd restart
              

We are now ready to establish an authenticated connection. saslauthd will log into the system log, so any authorisation failures can be further diagnosed using information captured there.


Setup use of saslauthd on the PCP Collector:
$ pminfo -h pcps://server.demo.net?method=plain -dfmt proc.fd.count
              Username: jack
              Password: xxxxxx
              
              proc.fd.count PMID: 3.51.0 [open file descriptors]
                  Data Type: 32-bit unsigned int  InDom: 3.9 0xc00009
                  Semantics: instant  Units: count
                  inst [1281 or "001281 bash"] value 4
                  inst [1287 or "001287 make -j 8 default_pcp"] value 5
                  inst [1802 or "001802 bash"] value 4
                  [...]
              
Kerberos Authentication

The PCP collector system can be configured to authenticate using Kerberos single-sign-on using the GSSAPI authentication mechanism. This mechanism is enabled via the mech_list option in /etc/sasl2/pmcd.conf , and the keytab should be set to /etc/pcp/pmcd/krb5.tab .


Check for support:
$ pmconfig -L secure_sockets authentication
              secure_sockets=true
              authentication=true
              
              $ pluginviewer -s -m GSSAPI
              [...]
              Plugin "gssapiv2" [loaded], 	API version: 4
              	SASL mechanism: GSSAPI, best SSF: 56, supports setpass: no
              	security flags: NO_ANONYMOUS|NO_PLAINTEXT|NO_ACTIVE|PASS_CREDENTIALS|MUTUAL_AUTH
              	features: WANT_CLIENT_FIRST|PROXY_AUTHENTICATION|DONTUSE_USERPASSWD
              

Setup Kerberos authentication on the PCP Collector:
# kadmin.local
              kadmin.local: add_principal pmcd/server.demo.net
              Enter password for principal "pmcd/server.demo.net@DEMO.NET":
              Re-enter password for principal "pmcd/server.demo.net@DEMO.NET":
              Principal "pmcd/server.demo.net@DEMO.NET" created.
              
              kadmin.local:  ktadd -k /root/pmcd-server-demo.tab pmcd/server.demo.net@DEMO.NET
              Entry for principal pmcd/server.demo.net@DEMO.NET with kvno 4, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/root/pmcd-server-demo.tab.
              Entry for principal pmcd/server.demo.net@DEMO.NET with kvno 4, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/root/pmcd-server-demo.tab.
              Entry for principal pmcd/server.demo.net@DEMO.NET with kvno 4, encryption type DES with HMAC/sha1 added to keytab WRFILE:/root/pmcd-server-demo.tab.
              Entry for principal pmcd/server.demo.net@DEMO.NET with kvno 4, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/root/pmcd-server-demo.tab.
              
              kadmin.local: quit
              
              # scp /root/pmcd-server-demo.tab root@server.demo.net:/etc/pcp/pmcd/krb5.tab
              # rm /root/pmcd-server-demo.tab
              
              $ kinit jack@DEMO.NET
              Password for jack@DEMO.NET: xxxxxx
              
              $ klist
              Ticket cache: FILE:/tmp/krb5cc_500
              Default principal: jack@DEMO.NET
              
              Valid starting     Expires            Service principal
              02/07/13 20:46:40  02/08/13 06:46:40  krbtgt/DEMO.NET@DEMO.NET
              	renew until 02/07/13 20:46:40
              
              $ pminfo -h 'pcps://server.demo.net?method=gssapi' -dfmt proc.fd.count
              
              proc.fd.count PMID: 3.51.0 [open file descriptors]
                  Data Type: 32-bit unsigned int  InDom: 3.9 0xc00009
                  Semantics: instant  Units: count
                  inst [1281 or "001281 bash"] value 4
                  inst [1287 or "001287 make -j 8 default_pcp"] value 5
                  inst [1802 or "001802 bash"] value 4
                  [...]