STYGION Documentation Search the docs

Metrics for Prometheus and Grafana

TPS, tick time, players, chunks per dimension, entities, memory and the cost of each mod, as an endpoint Prometheus scrapes - behind a key that reads metrics and nothing else, off until you turn it on, with a Grafana dashboard that ships in the jar. New in 0.4.

Keystone 0.5.0 Updated 4 min read
On this page 7 parts
  1. Why
  2. Turn it on
  3. What it writes
  4. Privacy: names stay on the server
  5. Settings
  6. The Grafana dashboard
  7. The API

Why

#

The editor's Performance screen shows how the server is doing while you look at it. A monitoring stack shows it when you are not: last night's stutter, the memory that climbs for a week before a crash, the dimension that quietly took half the tick. This module puts Keystone's numbers where Prometheus, Grafana, Victoria Metrics, Grafana Cloud and anything that reads the Prometheus text format already look.

Turn it on

#

It is off until you say so, because it opens a door on the local API.

  1. In openmetrics.toml (or Settings, openmetrics.enabled) set enabled = true, then restart the server: a module's enabled is read once, when the server starts (the editor marks it, and /keystone reload says it is waiting for a restart).
  2. Make a key that can read metrics and nothing else (its secret is shown once): /keystone key new prometheus metrics.read
  3. Point a scrape job at the local API (loopback port 25586 unless you changed it - a scraper on another machine needs a tunnel or a reverse proxy you trust, never the port opened to the world):
Code
scrape_configs:
  - job_name: keystone
    metrics_path: /v1/metrics/prometheus
    scrape_interval: 15s
    authorization:
      credentials: <the key you made>
    static_configs:
      - targets: ['127.0.0.1:25586']

GET /v1/metrics/prometheus answers in Prometheus' text format; add ?format=openmetrics for OpenMetrics (counters without _total in the family name, and a closing # EOF). A caller with no key, or a key without metrics.read, is refused like for every other route.

What it writes

#
Metric Type Labels What
keystone_info gauge minecraft, loader always 1
keystone_tps gauge ticks per second
keystone_mspt gauge stat = median, p95, max milliseconds of work in a tick, over the last hundred ticks. The average hides the stutter; these do not
keystone_last_tick_milliseconds gauge the tick that just ended
keystone_players_online, keystone_players_max gauge online (not counting the vanished) and the limit
keystone_chunks_loaded gauge world chunks held loaded per dimension (labels.worlds off: one total, world="all")
keystone_dimension_tick_seconds_total counter world time spent ticking each dimension; rate(...) is the share of the server's time a world takes
keystone_entities, keystone_block_entities gauge in the loaded world (census.enabled)
keystone_things_by_mod gauge mod entities and block entities each mod has standing, the largest labels.mods-most
keystone_mod_milliseconds_per_tick gauge mod only while the profiler has a fresh run (profile.fresh, 30 minutes): what each mod took of an average tick. After that the series disappears rather than show old numbers as current
keystone_profile_age_seconds gauge how long ago that run finished
keystone_player_ping_milliseconds gauge player only with labels.players: see below
keystone_jvm_memory_bytes gauge area, kind heap and non-heap memory (jvm)
keystone_jvm_gc_collections_total, keystone_jvm_gc_seconds_total counter collector garbage collections and the time spent in them
keystone_jvm_threads, keystone_uptime_seconds, keystone_process_cpu_ratio gauge threads, seconds since the JVM started, the process' share of the CPU

The numbers are read on the server thread, on a timer, and a scrape reads the finished picture: scraping every second costs the server nothing it was not already paying. The census (entities) walks the loaded world once per census.every seconds, not once per scrape.

Privacy: names stay on the server

#

No player name is in the metrics unless you set labels.players = true - then each online player's ping appears under their name. Turn it on only if everybody who can read your Prometheus may see who is online. Label values are escaped, so a name cannot add a line of its own. Mods' names are in (they are the point); positions, addresses, ids and chat never are.

Settings

#

openmetrics.toml, or Settings.

Setting Default What it does
enabled false the whole module
refresh 10 seconds between two readings of the game
census.enabled, census.every true, 30 count entities and block entities, and how often (seconds). Turn off on a very large world
labels.worlds true per-dimension series rather than one total
labels.mods, labels.mods-most true, 50 per-mod series, and how many
labels.players false each online player's ping under their name
profile.fresh 30 minutes a profiler run is shown
jvm true the Java series

The Grafana dashboard

#

The dashboard that matches this very version is in the jar: Metrics in the editor has a download button, or fetch GET /v1/metrics/dashboard with the same key. In Grafana: Dashboards, New, Import, choose your Prometheus. It has stat tiles (TPS, p95 tick, players, entities, block entities, heap), TPS and tick-time graphs, chunks and tick share per dimension, what each mod has standing, the profiler's cost per mod, and Java memory and garbage collection. The shape of it, to write your own panels:

Code
{ "title": "Milliseconds per tick (last 100 ticks)", "type": "timeseries",
  "targets": [ { "expr": "keystone_mspt{job=~\"$job\"}", "legendFormat": "{{stat}}" } ],
  "fieldConfig": { "defaults": { "unit": "ms", "min": 0 } } }

A few queries worth having:

Question Query
Is the server keeping up? keystone_mspt{stat="p95"} > 45
Which world takes the time? rate(keystone_dimension_tick_seconds_total[5m])
Is memory leaking? keystone_jvm_memory_bytes{area="heap",kind="used"} over a week
Time lost to garbage collection rate(keystone_jvm_gc_seconds_total[1m])
Alert: TPS under 15 for 5 minutes avg_over_time(keystone_tps[5m]) < 15

The API

#
Route Scope
GET /v1/metrics/prometheus metrics.read the metrics (?format=openmetrics)
GET /v1/metrics/dashboard metrics.read the Grafana dashboard, as JSON

metrics.read opens these two routes and nothing else, so the key sitting in your Prometheus configuration can read numbers and cannot change a thing.

Did this page help?

Opens the feedback panel with this page attached. It lands in the same queue as everything else.