Introduction

solana-exporter (Solana Rust Exporter) is an advanced, modular monitoring solution for Solana validator nodes. The executable part is implemented in Rust and was initially based on the Golang original by CertusOne but now provides additional functionality. It comprises

a Prometheus exporter executable written in Rust,
a Docker container suitable for docker-compose,
sample Grafana dashboards.

Typical uses include

monitoring Solana validator nodes by the nodes' operator,
managing a stake pool,
monitoring the cluster health.

This guide explains how to set the monitoring stack up and how to stay on top of things using Grafana alerts.

Basics

This chapter covers the setup, configuration and installation of solana-exporter, as well as integration with other tools.

Installation

To install the latest version of solana-exporter, run:

cargo install solana-exporter

Note that the libraries libssl and libudev should be installed on the target machine as well. Refer to your distro's package manager and documentation for guidance on how to install them.

After installation, run

solana-exporter generate

to set up a default configuration file. By default, the generate command will place a config file inside ~/.solana-exporter; this directory will be automatically created, unless the -o flag is present to override the output location.

Running as a service

Run this as a systemd service by a non-root user with a script like this one:

[Unit]
Description=Solana Exporter
After=solana.service
Requires=solana.service

[Service]
User=solana
Restart=always
RestartSec=20
ExecStart=/home/solana/.cargo/bin/solana-exporter

[Install]
WantedBy=multi-user.target

Running as a Docker container

solana-exporter is also available as a container which can be pulled by

docker pull rustiq/solana-exporter:0.4.1

Then run it with

docker run -d \
-v /path/to/your/config.toml:/etc/solana-exporter/config.toml \
-v solana-exporter-data:/exporter rustiq/solana-exporter:0.4.1

We recommend that the config.toml file be bind-mounted to the container, so you have easy access to it on the host machine. However, the persistent database should be stored in a named volume.

Configuration

After running solana-exporter generate, a template config file will be created in either the specified location or the default directory (~/.solana-exporter). This page explains the individual variables and how they affect the exporter.

Sample configuration

rpc = 'http://localhost:8899'
target = '0.0.0.0:9179'
vote_account_whitelist = []
staking_account_whitelist = []

[maxmind]
username = 'username'
password = 'password'

rpc - the location of the JSON-RPC node. This can be a local RPC node, or a public one.
- Remark: Public nodes usually have a rate-limiting policy in place that makes usage with solana-exporter difficult (e.g., delayed response times).
target - the target address/port to export Prometheus gauges to.
vote_account_whitelist - an array that instructs the exporter to only export statistics related to the specified vote pubkeys, and their corresponding node pubkeys if found.
staking_account_whitelist - an array that instructs the exporter to only export APY statistics related to the specified staking pubkeys.
[maxmind] - The exporter can optionally use MaxMind's GeoIP2 Precision City Service to export decentralisation-related metrics. However, this requires you to sign up for a MaxMind account and regularly top-up your account with credits.
- username - the username of the API key.
- password - the password of the API key.

Important note on `vote_account_whitelist` and `staking_account_whitelist`

As explained above, whitelists can be used to restrict the amount APY statistics exported. However, care should be taken if you modify the whitelists and then reload solana-exporter. In particular, for the following gauges:

solana_current_staking_apy
solana_average_staking_apy

if any newly-appearing (e.g., formerly excluded) pubkeys were not fetched for a particular epoch, then solana-exporter will not attempt to "back-fill" data for either the current or past epochs. Therefore, those pubkeys will be missing from the gauges until the next epoch begins. This is due to the fact that the exporter only scrapes the ledger for rewards data once at the beginning of every epoch.

To force a re-fresh of the current epoch's rewards, delete the database.

Overriding the config file location

Standalone program: The default location is ~/.solana-exporter/config.toml. Override this with the -c flag.
Docker container: Change the bind-mount location.

Overriding the database location

To speed up processing and reduce unnecessary network traffic, solana-exporter uses a persistent database to cache some requests.

Standalone program: The default location is ~/.solana-exporter/persistent.db. Override this with the -d flag.
Docker container: The location cannot be overridden; the exporter expects a database to be mounted in /exporter/.

Setting up Prometheus

solana-exporter uses Prometheus to export, monitor and aggregate data for use in other utilities. Refer to Prometheus' documentation on how to get it running on your machine.

If querying a public RPC port, solana-exporter can be run from anywhere, not necessarily from the validator machine.

If querying a private RPC port, install Prometheus on the validator machine. Add the following snippet to the scrape_configs section of the prometheus.yml config file:

  - job_name: solana
    static_configs:
      - targets: ['localhost:9179']

Restart Prometheus. Now the solana-exporter metrics should be available to view at http://localhost:9179/metrics. If running on the validator machine, it is highly advisable to only open the Prometheus datasource port to the Grafana machine. This can be achieved with nftables. Here is an example /etc/nftables.conf:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0;
        # allow connection to Prometheus datasource only locally and from Grafana
        ip saddr { 127.0.0.1, <Grafana IP> } tcp dport 9090 accept
        tcp dport 9090 drop
        # allow connection to Prometheus exporter endpoints only internally
        ip saddr != 127.0.0.1 tcp dport 9100 drop
        ip saddr != 127.0.0.1 tcp dport 9179 drop
    }
    chain forward {
        type filter hook forward priority 0;
    }
    chain output {
        type filter hook output priority 0;
    }
}

Note the order of commands. An accept clause should appear before the corresponding drop clause.

When solana-exporter is used on a mainnet validator node, Grafana must always run on a different machine to circumvent potential DDoS attacks on the validator. In the Grafana dashboard, add the Prometheus data source http://<Validator IP>:9090. Then import the rustiq.json using that data source.

Setting up Grafana

After solana-exporter exports gauges and metrics to Prometheus, it may be useful to visualise these metrics. Grafana allows you to create custom dashboards using Prometheus as a data source.

The repository includes a basic dashboard (rustiq.json) that shows off a range of gauges that solana-exporter can export using graphs. Some included visualisations include the monitoring of basic cluster statistics such as transactions per second, skipped slots percentage. Decentralisation metrics such as the distribution of staked SOL across datacenters can also be tracked (MaxMind account required).

To use the provided dashboard, first add the Prometheus data source. Then, import the dashboard.

Grafana Dashboard

For an in-depth explanation of each exported gauge, see then next chapter.

Sample screenshots

Pie chart of activated stake by datacenter (Gauge used: solana_active_validators_dc_stake)

Heatmap of average skipped slot of validators (Gauge used: solana_skipped_slot_percent)

Heatmap of average staking APY over 5 epochs (Gauge used: solana_average_staking_apy)

Exported feeds

This chapter explains in detail all the feeds that solana-exporter exports to Prometheus.

`solana_active_validators`

Description

Total number of validators in the cluster, grouped by the current status (either current or delinquent).