A Complete Guide to Securely Connecting Prometheus (node_exporter) and Prometheus Using Mutual TLS
How to use TLS, client authentication, and CA certificates in Prometheus (node_exporter) and Prometheus
Create a private key and request a certificate for your Prometheus (node_exporter) server
Before you can teach your server to speak TLS, you will need a certificate issued by a trusted certificate authority (CA). If your organization already runs its own CA and you have a private key and certificate for your Prometheus (node_exporter) server, along with your CA's root certificate, you can skip to the next step.
To request a certificate from your CA using the step CLI, bootstrap your CA with step ca bootstrap and run the following command (sub the server name for the actual name / DNS name of your Prometheus (node_exporter) server).
$ step ca certificate "myserver.internal.net" server.crt server.keyYour certificate and private key will be saved in server.crt and server.key respectively.
Request a copy of your CA root certificate, which will be used to make sure each application can trust certificates presented by other applications.
$ step ca root ca.crtYour certificate will be saved in ca.crt.
Configure Prometheus (node_exporter) to authenticate itself with its TLS certificate
We now want to instruct our Prometheus (node_exporter) server to identify itself using the certificate issued in the last step and to force clients to connect over TLS.
Copy the server.crt and server.key files to a node_exporter configuration directory. You may need to make a directory for this, eg. /etc/node_exporter.
$ sudo cp server.crt /etc/node_exporter/server.crt
$ sudo cp server.key /etc/node_exporter/server.keyMake sure these files are owned and readable only by the user that node_exporter runs as.
Now create a file called /etc/node_exporter/web-config.yml and configure your tls_server_config block to use the server certificate and key:
tls_server_config:
# This is the server certificate for your `node_exporter` server.
cert_file: "/etc/node_exporter/server.crt"
key_file: "/etc/node_exporter/server.key"
...Read more
Configure Prometheus (node_exporter) to require clients to authenticate with a certificate issued by your CA
To tell Prometheus (node_exporter) to use mutual TLS and not just one-way TLS, we must instruct it to require client authentication to ensure clients present a certificate from our CA when they connect.
Copy the ca.crt file to a node_exporter configuration directory. You may need to make a directory for this, eg. /etc/node_exporter.
$ sudo cp ca.crt /etc/node_exporter/root_ca.crtMake sure these files are owned and readable only by the user that node_exporter runs as.
Now modify /etc/node_exporter/web-config.yml to require client authentication (in your tls_server_config block):
tls_server_config:
...
# RequireAndVerifyClientCert is the most secure option; clients
# must present a valid client certificate signed by your CA.
client_auth_type: "RequireAndVerifyClientCert"
# This is the CA the client certificate must be signed by.
client_ca_file: "/etc/node_exporter/root_ca.crt"
...That's it! Prometheus (node_exporter) should now be able to receive TLS connections from clients who authenticate themselves using a certificate issued by your trusted CA.
Create a private key and request a certificate for your Prometheus client
Request a new certificate from your CA to represent your Prometheus client.
$ step ca certificate "myuser" client.crt client.keyYour certificate and private key will be saved in client.crt and client.key respectively.
Open a connection from Prometheus using mutual TLS
Now, we need only to configure our Prometheus client to make authenticated requests using our certificate and private key. The CA root certificate will be used to verify that the client can trust the certificate presented by the server.
Add the following job configuration block to your prometheus.yml to authenticate as a client to your targets:
#...
scrape_configs:
- job_name: 'node'
scheme: https
tls_config:
# Prometheus will check that the node_exporter presents a certificate
# signed by this ca.
ca_file: 'ca.crt'
# The cert and key are presented to node_exporter to authenticate
# Prometheus as a client.
cert_file: 'client.crt'
key_file: 'client.key'
static_configs:
- targets: ['myserver.internal.net:9100']
#...Reload Prometheus, and confirm that the Prometheus dashboard shows your target endpoints as "UP"—and using the https:// scheme.
Automate certificate renewal
By default, step-ca issues certificates with a 24 hour expiration. Short-lived certificates have many benefits but also require that you renew your certificates each day before they expire. How you renew certificates is often dependent on how you deploy your application. See the step-ca certificate lifecycle management docs for more information.
All documentation content from the Hello mTLS project is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).
Subscribe to updates
Unsubscribe anytime. See our privacy policy.
© 2024 Smallstep Labs, Inc. All rights reserved.