Metrics for Prometheus and Grafana
TPS, tick time, players, chunks per dimension, entities, memory and the cost of each mod, as an endpoint Prometheus scrapes - behind a key that reads metrics and nothing else, off until you turn it on, with a Grafana dashboard that ships in the jar. New in 0.4.
On this page 7 parts
Why
#The editor's Performance screen shows how the server is doing while you look at it. A monitoring stack shows it when you are not: last night's stutter, the memory that climbs for a week before a crash, the dimension that quietly took half the tick. This module puts Keystone's numbers where Prometheus, Grafana, Victoria Metrics, Grafana Cloud and anything that reads the Prometheus text format already look.
Turn it on
#It is off until you say so, because it opens a door on the local API.
- In
openmetrics.toml(or Settings,openmetrics.enabled) setenabled = true, then restart the server: a module'senabledis read once, when the server starts (the editor marks it, and/keystone reloadsays it is waiting for a restart). - Make a key that can read metrics and nothing else (its secret is shown once):
/keystone key new prometheus metrics.read - Point a scrape job at the local API (loopback port 25586 unless you changed it - a scraper on another machine needs a tunnel or a reverse proxy you trust, never the port opened to the world):
scrape_configs:
- job_name: keystone
metrics_path: /v1/metrics/prometheus
scrape_interval: 15s
authorization:
credentials: <the key you made>
static_configs:
- targets: ['127.0.0.1:25586']
GET /v1/metrics/prometheus answers in Prometheus' text format; add ?format=openmetrics for OpenMetrics (counters without _total in the family name, and a closing # EOF). A caller with no key, or a key without metrics.read, is refused like for every other route.
What it writes
#| Metric | Type | Labels | What |
|---|---|---|---|
keystone_info |
gauge | minecraft, loader |
always 1 |
keystone_tps |
gauge | ticks per second | |
keystone_mspt |
gauge | stat = median, p95, max |
milliseconds of work in a tick, over the last hundred ticks. The average hides the stutter; these do not |
keystone_last_tick_milliseconds |
gauge | the tick that just ended | |
keystone_players_online, keystone_players_max |
gauge | online (not counting the vanished) and the limit | |
keystone_chunks_loaded |
gauge | world |
chunks held loaded per dimension (labels.worlds off: one total, world="all") |
keystone_dimension_tick_seconds_total |
counter | world |
time spent ticking each dimension; rate(...) is the share of the server's time a world takes |
keystone_entities, keystone_block_entities |
gauge | in the loaded world (census.enabled) |
|
keystone_things_by_mod |
gauge | mod |
entities and block entities each mod has standing, the largest labels.mods-most |
keystone_mod_milliseconds_per_tick |
gauge | mod |
only while the profiler has a fresh run (profile.fresh, 30 minutes): what each mod took of an average tick. After that the series disappears rather than show old numbers as current |
keystone_profile_age_seconds |
gauge | how long ago that run finished | |
keystone_player_ping_milliseconds |
gauge | player |
only with labels.players: see below |
keystone_jvm_memory_bytes |
gauge | area, kind |
heap and non-heap memory (jvm) |
keystone_jvm_gc_collections_total, keystone_jvm_gc_seconds_total |
counter | collector |
garbage collections and the time spent in them |
keystone_jvm_threads, keystone_uptime_seconds, keystone_process_cpu_ratio |
gauge | threads, seconds since the JVM started, the process' share of the CPU |
The numbers are read on the server thread, on a timer, and a scrape reads the finished picture: scraping every second costs the server nothing it was not already paying. The census (entities) walks the loaded world once per census.every seconds, not once per scrape.
Privacy: names stay on the server
#No player name is in the metrics unless you set labels.players = true - then each online player's ping appears under their name. Turn it on only if everybody who can read your Prometheus may see who is online. Label values are escaped, so a name cannot add a line of its own. Mods' names are in (they are the point); positions, addresses, ids and chat never are.
Settings
#openmetrics.toml, or Settings.
| Setting | Default | What it does |
|---|---|---|
enabled |
false |
the whole module |
refresh |
10 |
seconds between two readings of the game |
census.enabled, census.every |
true, 30 |
count entities and block entities, and how often (seconds). Turn off on a very large world |
labels.worlds |
true |
per-dimension series rather than one total |
labels.mods, labels.mods-most |
true, 50 |
per-mod series, and how many |
labels.players |
false |
each online player's ping under their name |
profile.fresh |
30 |
minutes a profiler run is shown |
jvm |
true |
the Java series |
The Grafana dashboard
#The dashboard that matches this very version is in the jar: Metrics in the editor has a download button, or fetch GET /v1/metrics/dashboard with the same key. In Grafana: Dashboards, New, Import, choose your Prometheus. It has stat tiles (TPS, p95 tick, players, entities, block entities, heap), TPS and tick-time graphs, chunks and tick share per dimension, what each mod has standing, the profiler's cost per mod, and Java memory and garbage collection. The shape of it, to write your own panels:
{ "title": "Milliseconds per tick (last 100 ticks)", "type": "timeseries",
"targets": [ { "expr": "keystone_mspt{job=~\"$job\"}", "legendFormat": "{{stat}}" } ],
"fieldConfig": { "defaults": { "unit": "ms", "min": 0 } } }
A few queries worth having:
| Question | Query |
|---|---|
| Is the server keeping up? | keystone_mspt{stat="p95"} > 45 |
| Which world takes the time? | rate(keystone_dimension_tick_seconds_total[5m]) |
| Is memory leaking? | keystone_jvm_memory_bytes{area="heap",kind="used"} over a week |
| Time lost to garbage collection | rate(keystone_jvm_gc_seconds_total[1m]) |
| Alert: TPS under 15 for 5 minutes | avg_over_time(keystone_tps[5m]) < 15 |
The API
#| Route | Scope | |
|---|---|---|
GET /v1/metrics/prometheus |
metrics.read |
the metrics (?format=openmetrics) |
GET /v1/metrics/dashboard |
metrics.read |
the Grafana dashboard, as JSON |
metrics.read opens these two routes and nothing else, so the key sitting in your Prometheus configuration can read numbers and cannot change a thing.
Did this page help?
Opens the feedback panel with this page attached. It lands in the same queue as everything else.