Skip to main content

Vault Unseal Runbook

MOD stores its infrastructure secrets in Vault. This install runs Vault in Shamir seal mode with manual unseal: after any restart of the Vault container (a host reboot, a docker compose cycle, an out-of-disk eviction), Vault comes back sealed and stays sealed until a human unseals it. Nothing in the stack auto-unseals Vault from a file, on purpose — the key material lives only on the key holders' machines.

This runbook is what the Dashboard's red "Vault is sealed" banner and the VAULT_SEALED admin alert point at.

What "sealed" means​

"Sealed" means Vault's key material is locked in memory and every secret is unreadable. The Vault server process is up and answering, but:

  • Postgres credentials from database/creds/* cannot be read,
  • S3/versitygw, SMTP (Stripe, Postmark), API keys and every other KV secret cannot be read,
  • anything that reads a secret at startup will fail or fall back to cached credentials until they expire.

The rest of the platform keeps running on whatever it already holds; it does not crash-loop. The API's seal monitor (services/vault_seal_monitor.py) writes a VAULT_SEALED audit event and alerts the platform admin exactly once per seal, and a VAULT_UNSEALED audit event when it recovers. The boot-reconcile watchdog detects the sealed state and deliberately does not restart Vault — restarting a sealed Vault never unseals it.

You are sealed if any of these is true:

  • the Dashboard shows the red "Vault is sealed" banner (platform admins),
  • GET /health?extended=true returns "vault_sealed": true,
  • from a machine that can reach Vault: vault status prints Seal Status: sealed, or docker exec my_vault vault operator seal-status prints "sealed": true.

Who holds the shares​

Vault was initialized with Shamir key shares: the unseal key is split into N shares, of which M are required (a threshold). The shares were printed exactly once at initialization (or when the key was rekeyed). The operators should have split them by the organization's standard ceremony — for example 1-of-3, 2-of-3, or 3-of-5 — so that M different holders each hold a different share and no single person can unseal alone.

  • If you do not know how many shares exist or what the threshold is, check docker exec my_vault vault operator unseal — after the first share it tells you how many more are needed.
  • If fewer than M shares can be recovered, Vault cannot be unsealed. The data is still on the vault_data volume; recovery then means the rekey/restore path in Backup and Restore, which also needs the shares. Do not delete the vault_data volume in this state.

How to unseal​

From one of the key holders' machines (a laptop or admin box that can reach the install's Vault), not from the server:

  1. Point vault at the install. The server's API address is usually http://<server>:8200 on the internal network (or the mesh/tailnet address of this install):

    export VAULT_ADDR=http://<server>:8200

  2. Run vault operator unseal M times, each time with a different share, until the prompt prints Sealed: false. Each share may only be used once per unseal — reusing the same share M times does nothing:

    $ vault operator unseal
    Key (enter without a newline): <share 1>
    Keys Required: 3
    Shares Required: 3
    Progress: 1/3

    repeat with share 2, then share 3. When done:

    $ vault operator unseal
    Key (enter without a newline): <share 3>
    Sealed: false

  3. Verify:

    $ vault status
    Seal Status:         unsealed

    Then check the platform: the Dashboard banner disappears, and GET /health?extended=true shows "vault_sealed": false. The API logs a VAULT_UNSEALED audit event within a minute.

Restoring a backup also needs the shares​

A Vault backup (vault_data snapshot) is encrypted with the same key material. Restoring a backup into a fresh Vault does not bypass unseal — the fresh Vault comes up sealed and you must run the same vault operator unseal dance with M different shares before the restored secrets are readable. Keep this in mind when planning disaster recovery: backup + shares = restorable; either one alone is not.

Notes​

  • Unsealing puts the key material in Vault's memory only; a restart seals it again. That is the point of the manual-unseal design.
  • Never paste shares into chat, tickets, or logs. The API's audit trail records that Vault sealed/unsealed, but never a share.
  • To change custody or rotate the keys later, run a rekey (vault operator rekey) as a ceremony — the old shares stop working and new shares are printed once.