Updates and crashes: the guard

New in 0.3, module guard, on by default: it only watches and keeps the last good start. What changes the server on its own - the modpack update (update.*) and backups off the machine (offsite.*) - is off until you turn it on. Update without fear.

Every start that runs good-after (10 minutes) is kept as the last good one: its mod jars as hard links, its config files copied. The next start that differs says what changed: mods updated, added and removed, what a removed mod takes out of the world (its blocks, items and entities), the config files that differ, and - worth reading twice - a jar whose bytes changed while its version did not, or one neither CurseForge nor Modrinth publishes (check-portals; only the jars' hashes leave the machine; a CurseForge key in curseforge-key). A flagged jar is never removed.

The crash doctor reads the game's crash reports and says the cause and the mods it points at, in the editor (Updates & crashes) and on Discord.

• /keystone guard rollback confirm (or one button in the editor) puts the last good start back when the server next stops; what it replaces is kept aside, never deleted. /keystone guard cancel calls a pending rollback or update off (keystone.guard.rollback).
• After crash-limit (3) boots in a row that crash, rollback decides: updates-only (default; only an update Keystone applied is rolled back before the next boot opens a world), always (any change since the last good start) or never. When nothing is rolled back, the server starts closed for maintenance so the same crash does not repeat; /keystone guard release opens it.

update.enabled with update.project (the CurseForge project id): every update.check-every (1 h) CurseForge is asked; a new server pack is downloaded off the server thread from CurseForge's own hosts (up to update.max-mib), checked against its SHA-1, prepared with your extra mods and your config changes kept (update.overwrite names pack files that replace yours, default config/ftbquests/quests/**), backed up and swapped in at a restart - update.apply: empty (the first time nobody has been online for update.empty-for, 3 min), daily (with the next planned restart) or manual (/keystone guard update apply). /keystone guard update check asks now.

offsite.after-backup sends every new backup, sealed with offsite.passphrase (AES-256-GCM, key by PBKDF2, at least 12 characters - keep it somewhere other than this server), to an S3-compatible bucket (offsite.s3.*: AWS, Cloudflare R2, Backblaze B2, Wasabi, MinIO, Hetzner) and/or over SFTP (offsite.sftp.*, with the host key fingerprint). offsite.keep (7) per place. The restore drill (offsite.drill = verify, every offsite.drill-every, 7 days) fetches the newest off-site backup, opens it and reads its level.dat and every region file; /keystone guard offsite drill runs it now.

GET /v1/guard (status.read); GET /v1/guard/detail, POST /v1/guard/rollback, /update and /drill (backup.restore, owner only).