Skip to main content

Backup and Restore

MOD supports backup and restore controls: CORE/startup/backup.sh produces a timestamped backup set of the live data stores, and a restore runbook exists for each artifact. MOD does not make a customer compliant with any framework, and a backup script alone does not satisfy any continuity control — retention, off-box copy, encryption key custody and restore testing remain the operator's responsibility.

What backup.sh covers​

Run from CORE/startup/, the script writes one timestamped directory per run (default ./backups/<timestamp>/) containing:

  1. PostgreSQL (app database) — pg_dump of the app database, gzipped to postgres.sql.gz.
  2. Keycloak DB — pg_dump of the Keycloak database, gzipped to keycloak.sql.gz.
  3. Object storage — the versitygw posix data volumes plus the IAM volume, tarred per volume. Mounts are resolved from docker-compose.yml, not hardcoded. The legacy MinIO server is only touched when STORAGE_BACKEND=minio; in that case a recursive bucket listing is also captured (minio-listing.txt.gz).
  4. Vault storage — vault operator raft snapshot save when the active vault-config*.hcl uses a raft backend; otherwise a tar of the Vault data volume, with a logged caveat that a file-backend tar reflects on-disk state only (unflushed in-memory writes on an unsealed Vault are not included).
  5. Qdrant, both instances — full storage snapshots via the snapshot API for the platform instance (mod_qdrant, prediction cache) and the user instance (mod_qdrant_user, user RAG). API-key environment variables are referenced, never printed.
  6. Internal-TLS keys — a tarball of the 0700 secure directory that holds the per-service private keys and the local-CA root/intermediate keys (path from CORE/startup/internal-tls/KEYS_LOCATION / MOD_INTERNAL_TLS_KEYS_DIR), with 0600/0700 modes preserved. Key contents are never printed or listed.
  7. Finalize — a MANIFEST recording the run (timestamp, storage backend, Vault config, whether the internal-TLS keys were captured), permissions applied to every artifact, optional encryption, retention cleanup, and the off-box shipping step (see "Off-box backups" below).

What is deliberately excluded​

Vault unseal and root material is never copied into the backup set: the tar excludes unseal-key, root-token and agent-secret-id. This is intentional — a backup set that contained the unseal key would encrypt nothing that matters, because the encryption keys and the secrets to decrypt would live together. Unseal keys must be kept in a separate, offline-safe location (see below).

Permissions and encryption defaults​

  • The script runs with umask 077; the backup directory is 0700 and every artifact is 0600.
  • age encryption (GSEC R-148): every artifact is age-encrypted (X25519 + XChaCha20-Poly1305 — authenticated, offline, no certificate management) to the recipient in the backup.encryption_recipient platform setting (public material only, safe in the DB — the private identity key is never on the box; the operator holds it and supplies it at restore time). The legacy BACKUP_ENCRYPT_RECIPIENT env var still works as a fallback. With backup.encryption_required=true and no recipient available, the run is fail-closed: nothing is written, the failure is recorded in backup_status.json, and the script exits non-zero. Installs that never set the flag and have prior backup sets keep producing backups (explicit UNENCRYPTED posture WARN + the backup.encrypted conformance check fails) until a recipient is configured; fresh installs (no prior sets) are fail-closed out of the box.
  • Integrity manifests: SHASUMS256 (sha256 of every artifact + MANIFEST) travels inside the encrypted set; SHASUMS256.ct is a detached plaintext sha256 of the ciphertext files for transport checks. Unencrypted sets carry SHASUMS256 as well.

Air-gapped tar image selection​

Named volumes are tarred by running tar inside a container with --pull=never — the script never pulls an image. It selects, in order:

  1. BACKUP_TAR_IMAGE (if set),
  2. alpine:3.20 if that image exists locally,
  3. the image of a running MOD container (the API image).

If none of these exists locally, the affected named volume is skipped with a warning rather than triggering a network pull.

Retention​

Backup sets older than 30 days are pruned from the backup directory at the end of each run. Retention pruning is local to the backup directory only — an off-box copy is outside the script's reach and is where long-term retention belongs.

How to run and schedule it​

cd CORE/startup
./backup.sh                 # defaults to ./backups/
./backup.sh /srv/mod-backups  # explicit backup directory

Environment variables the script reads (from the shell or the .env files): BACKUP_ENCRYPT_RECIPIENT, BACKUP_TAR_IMAGE, STORAGE_BACKEND, VAULT_CONFIG_SUFFIX, MOD_INTERNAL_TLS_KEYS_DIR (internal-TLS keys), MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (legacy MinIO listing only), QDRANT_API_KEY and QDRANT_USER_API_KEY, and the off-box set (BACKUP_OFFBOX_TARGET, BACKUP_S3_*, BACKUP_SSH_*, BACKUP_OFFBOX_RETENTION_DAYS — see "Off-box backups").

A typical cron schedule (daily at 02:30, recipient set for encryption):

30 2 * * * BACKUP_ENCRYPT_RECIPIENT=age1... \
  /path/to/CORE/startup/backup.sh /srv/mod-backups >> /var/log/mod-backup.log 2>&1

Copy each completed set off-box after every run; the 30-day retention and the backup directory itself live on the same machine the backup is protecting.

Where to keep unseal keys​

Because unseal material is excluded from the backup set by design, keep the Vault unseal keys and root token separately: offline (not on the same host or volume as the backups), in a location with its own access control, and with the age passphrase/recipient private key stored apart from the encrypted backup sets. If the data volume is lost, only the off-box copy of the unseal key can reopen a restored Vault — the in-tree auto-unseal watcher reads a file that lives inside the data volume itself.

Restore​

The restore runbook lives in docs/security-audit/DR_PLAN_DRAFT.md §4. Order matters because the API depends on Vault, Postgres, Keycloak and storage in that chain:

  1. Vault first — restore the data volume or raft snapshot to a fresh container, then unseal using the off-box key. If Vault data is lost and only the unseal key survives, re-initialisation is not possible with a Shamir key alone; accept Vault recreation and re-inject secrets.
  2. Postgres — fresh volume, then load postgres.sql.gz with psql; recreate the PgBouncer auth role, which a bare restore leaves without a userlist. Verify against a sentinel-table count recorded in the manifest.
  3. Keycloak — restore the DB dump, then verify a known service user and a known human user can log in (proves realm, clients and MFA state).
  4. Object storage — restore the versitygw data and IAM volumes before the service starts; without the IAM volume the API re-mints tenant accounts, but per-task credentials issued before the failure are invalid until workers refetch.
  5. Qdrant — restore the user instance first (user RAG is not rebuildable for free), then the platform prediction cache; the platform instance may legitimately start empty because the scheduler falls back to heuristics.
  6. API and workers last — migrations run against the restored DB; run one small workflow end-to-end and confirm ApiTaskQueue drains and audit-log continuity against the manifest.

Re-applying erasures after a restore (privacy)​

A backup taken before someone was erased does not remember that they were erased: restoring it puts that person's data back, because the only record of the erasure lived in the same database that the restore overwrote. To fix this, every completed privacy erasure is written to an append-only erasure ledger — one line per erased person or tenant, containing only identifiers and timestamps (no personal data). The ledger has two copies: a table in the app database, and a plain log file on the install's persistent data directory that the backup and the restore both leave untouched.

After the restore finishes (the update-system.sh --rollback path does this automatically), the platform runs an erasure replay: it reads the ledger and re-erases anyone it finds whose erased state the restored database no longer shows. Already-erased entries are skipped, and anyone currently under a legal hold is left alone, exactly as during a normal erasure. Each re- erasure is written to the audit log. If the replay does not run as part of the restore, run it by hand from the API container: python3 scripts/replay_erasures.py (checks and reports without changing anything), then python3 scripts/replay_erasures.py --commit (re-applies the erasures).

Honest status: every restore path above is a runbook, not a verified mechanism. An automated restore test is not yet built — no drill has exercised these steps end-to-end, and the recovery-time targets in the DR plan are proposed, not measured. Until a restore drill is run against a real backup set, treat the runbook as unpractised.

Restore test​

CORE/startup/restore-test.sh (GY.C13a) exercises the Postgres step of the runbook against a real backup set: newest set (or --backup <path>), a throwaway mod-restoretest-<ts> postgres container on the same image tag as the live one (docker inspect, random in-process password, no host port, no volume), postgres.sql.gz loaded with psql, then checks on alembic_version, users/tenants counts and every table in the dump. One line is appended to the erasure ledger with RTO (restore duration) and RPO (now − backup timestamp), so the DR-plan recovery targets become measured, not proposed. The scratch container is always removed (trap EXIT); the live DB is never touched. --dry-run prints the plan. Encrypted sets are decrypted with the operator-supplied identity file — --identity <age-keyfile> or MOD_BACKUP_DECRYPT_KEYFILE — after first verifying the ciphertext against SHASUMS256.ct; the decrypted dump is then checked against the in-set SHASUMS256, and a wrong key fails with a clear "wrong or missing identity key" error. The API conformance check that reads these ledger lines is GY.C13b (separate follow-up); the encryption posture itself is the backup.encrypted check (GSEC R-148).

Off-box backups​

The local backup set above lives on the same machine it protects, so backup.sh can ship each finished set to one or both off-box targets (chosen at install time with --backup-target, switchable later in update.sh). This is the step that makes the down -v guard's "verified backup ≤ 24h" check meaningful, and it is where long-term retention belongs.

What gets shipped off-box​

Every run ships:

  • the whole timestamped backup set (all the artifacts listed at the top, plus the internal-TLS private keys — the per-service keys and the local-CA root/intermediate keys, tarred from the 0700 secure dir named in CORE/startup/internal-tls/KEYS_LOCATION, with their 0600/0700 modes preserved so a restore reproduces them),
  • an OFFBOX_MANIFEST.sha256 — the sha256 of every file in the set — uploaded next to it and verified after the upload (the S3 target re-downloads each file and re-hashes it; the ssh target runs sha256sum on the remote),
  • the erasure ledger (see below) as a LATEST copy at a fixed key/path and a dated copy inside the set.

Choosing a target​

Set BACKUP_OFFBOX_TARGET (or the --backup-target install flag) to one of:

  • s3 — upload to an S3 bucket (AWS, or any S3-compatible endpoint such as the stack's own versitygw). Server-side encryption is mandatory and cannot be turned off: by default the upload requests SSE-S3 (AES256), and if you set BACKUP_S3_KMS_KEY it requests SSE-KMS with that key. If BACKUP_S3_SSE=off the script refuses to upload rather than silently ship plaintext. Credentials come from a root-only 0600 file (BACKUP_S3_CREDENTIALS_FILE) or a Vault kv path (BACKUP_S3_CREDENTIALS_VAULT_PATH) — never from the command line or an env dump.
  • ssh — rsync -a --partial over ssh with a dedicated key (BACKUP_SSH_KEY, default ~/.ssh/id_ed25519_mod_backup) to BACKUP_SSH_TARGET (user@host:/path, e.g. a second machine for offline sites). Each run lands in its own dated directory. --delete is never used, so a failed or repeated run can never remove a previous off-box copy.
  • both — do both.

After each run the success/failure of every target and a timestamp are written to CORE/Metadata/backup_status.json (mode 0600), which the API/Dashboard read and the down -v guard checks.

What Object Lock means​

If the S3 bucket has Object Lock enabled, each uploaded object is given a retention period (BACKUP_S3_RETENTION_MODE — GOVERNANCE (default) or COMPLIANCE — and BACKUP_OFFBOX_RETENTION_DAYS, default 30). While in retention the object cannot be deleted or overwritten, even by the account owner:

  • GOVERNANCE — only a specially-privileged user can override it.
  • COMPLIANCE — nobody can override it until the period expires (and it must be the longest retention ever set on that object).

If the bucket has no Object Lock configured, the script logs a warning and the uploads are still encrypted — they just aren't immutable. For a compliance retention guarantee, enable Object Lock on the bucket before pointing the backup at it.

The erasure-ledger latest-copy rule​

The erasure ledger (the append-only record of every privacy erasure) is shipped on every run in two places:

  1. a LATEST copy at a fixed key/path — …/ledger/erasure_ledger.jsonl on S3 and <remote>/ledger/erasure_ledger.jsonl on the ssh target — and
  2. a dated copy inside each timestamped set.

The fixed key/path is deliberately separate from the dated DB dumps, so if you restore an old database backup, the newest ledger still exists off-box to replay (an old restore would otherwise resurrect erased people). When you restore, replay the LATEST ledger, not the one from the same date as the DB dump.

Restoring from each target​

  • S3: pull the set back with any S3 client using the bucket creds (e.g. mc mirror bo/<bucket>/<prefix>/<timestamp>/ ./restore/, or the minio python client), then follow the per-artifact restore runbook above. Verify with OFFBOX_MANIFEST.sha256 (sha256sum -c) before you trust it. Objects still in retention will not let you overwrite them; to re-upload to a locked bucket, use a fresh dated prefix (the script always does).
  • ssh: copy the dated directory back (rsync -a -e "ssh -i $BACKUP_SSH_KEY" user@host:/path/<timestamp>/ ./restore/), verify with OFFBOX_MANIFEST.sha256, then restore as above.

Deleting data volumes (down -v guard)​

Some MOD install scripts can destroy every Docker data volume (docker compose down -v deletes the database, object storage and all other named volumes). To stop an accidental or unattended run from wiping live data, those scripts run the step through a guard (CORE/startup/lib/volume-guard.sh) that refuses the volume-deleting shutdown unless one of these is true:

  1. A verified off-box backup is fresh. The off-box backup run writes CORE/Metadata/backup_status.json; the guard allows the wipe only when that file says verified: true and last_success_utc is within 24 hours. A missing or unreadable file counts as "no verified backup" — so the guard fails safe.
  2. The operator explicitly overrides. Pass --i-understand-this-deletes-data and type the install name exactly at the prompt. A flag with no typed name (for example a non-interactive run that forgot to supply it) is refused.

Every attempt is audited — allowed or refused. Each one appends a single line to CORE/Metadata/volume_guard_audit.jsonl (timestamp, action, decision, reason, backup age, user, host). When the API is up, the attempt is also recorded as a VOLUME_DELETE_GUARD row in the security audit log. If the audit file cannot be written, the guard refuses — a volume-deleting step is never run without a recorded attempt.

This guard only covers MOD's own scripts. A docker compose down -v typed by hand in a shell is not blocked by MOD — it goes straight to Docker. If you type it yourself you are on your own; back up and verify off-box first.