Skip to main content

Customer-Managed Tenant Keys (Vault Transit)

Every tenant has a data-encryption key (DEK) — Tenant.encryption_key and a handful of sibling columns — that wraps secrets, refresh tokens, and other tenant-scoped sensitive values. By default that DEK is itself wrapped ("enveloped") by a single install-wide master key, SECRET_ENCRYPTION_KEY, a Fernet key read from the environment. That master key never changes without an env edit and a restart, and it never leaves the API container.

The platform setting crypto.key_provider selects which key-encryption-key (KEK) backend performs that wrap/unwrap. This page covers the second backend it supports — Vault's transit secrets engine — which gives each tenant its own named key inside Vault instead of one shared master key, and keeps the KEK itself inside Vault at all times.

What the setting does​

  • crypto.key_provider (local default | vault_transit) — which backend wraps per-tenant DEKs.
    • local — unchanged behaviour: wrap/unwrap with SECRET_ENCRYPTION_KEY in-process, exactly as before this feature existed.
    • vault_transit — wrap/unwrap via Vault's transit engine (datakey/plaintext, decrypt, rewrap). The API only ever sees the plaintext DEK for the instant it is generated or unwrapped, or Vault's own ciphertext envelope — the per-tenant transit key itself never leaves Vault. This is what "customer-managed key" means here: the operator (or, in an on-prem install, the customer's own Vault administrators) controls the transit key's lifecycle — creation, rotation, access policy — independently of the API.
  • crypto.vault_transit_mount (default transit) — the Vault secrets-engine mount path used for all transit calls.
  • crypto.vault_transit_key_prefix (default tenant-) — each tenant's transit key is named <prefix><tenant_id>, e.g. tenant-7.

Flipping crypto.key_provider to vault_transit is gated: set_setting calls services.key_provider.check_vault_transit_ready() first, which refuses the switch (HTTP 409, no partial state — the setting row is never written) unless Vault is reachable, unsealed, and the configured mount exists. It deliberately does not check that every tenant's own transit key exists yet, because tenants can be created after the switch — a tenant's first wrap under a freshly-flipped setting fails loudly if its key is missing, the same failure shape as decrypting with the wrong key.

Existing tenants keep working (mixed envelopes)​

Switching the setting is not a migration. Which backend wrapped a given stored value is read from the value's own format, not from the live setting:

  • Vault transit ciphertext is self-describing — it always looks like vault:v<N>:<base64>.
    • anything else is treated as the legacy local-Fernet format.

That means a fleet with some tenants wrapped under local and others under vault_transit reads correctly all at once, in either direction, forever — there is no one-time cutover window where old rows become unreadable, and switching the setting back to local does not orphan tenants that were wrapped while it was vault_transit. The only thing that proactively moves a tenant from one Vault key version to the next is the rewrap job below; nothing else needs to run as a migration step.

Operator runbook​

1. Enable a transit mount​

From a Vault operator token (not the API's own AppRole):

vault secrets enable -path=transit transit

Use whatever mount path you intend to put in crypto.vault_transit_mount (default transit). This is a one-time, install-wide step — Vault's own vault:v<N>:... ciphertext never records which mount produced it, so do not change crypto.vault_transit_mount (or crypto.vault_transit_key_prefix) after any tenant has already been wrapped under the old value — their existing ciphertext would start resolving against the wrong mount/key name and every unwrap for that tenant would fail loudly. Treat both as fixed at first use.

2. Create a transit key per tenant​

The API never creates transit keys itself — only verifies the mount exists before allowing the setting to flip. For each tenant that will use vault_transit, create its named key once:

vault write -f transit/keys/tenant-<tenant_id>

(adjust the path prefix if crypto.vault_transit_key_prefix is not the default tenant-). A tenant created after the switch gets its first wrap attempt fail with a clear Vault error until its key exists — create it before (or immediately after) provisioning that tenant.

3. Give MOD's AppRole a transit policy​

The API authenticates to Vault the same way it does for every other secret read — the Vault Agent sidecar token first, falling back to the service AppRole login (VAULT_APPROLE_ROLE_ID / VAULT_APPROLE_SECRET_ID). Attach a policy scoped to only the transit operations this feature uses — no sys/policies, no key deletion, no arbitrary mount administration:

# transit-tenant-keys.hcl
path "sys/mounts"                      { capabilities = ["read"] }
path "transit/datakey/plaintext/tenant-*" { capabilities = ["update"] }
path "transit/decrypt/tenant-*"           { capabilities = ["update"] }
path "transit/rewrap/tenant-*"            { capabilities = ["update"] }
vault policy write transit-tenant-keys transit-tenant-keys.hcl
vault write auth/approle/role/<mod-approle-role> policies+=transit-tenant-keys

(sys/mounts read is what check_vault_transit_ready() uses to confirm the mount exists before letting the setting change land.) If crypto.vault_transit_key_prefix is not the default tenant-, adjust the three transit/* paths above to match.

4. Flip the setting​

Via the platform settings API/admin UI, set crypto.key_provider to vault_transit. This call fails closed (409) with a specific remediation message if Vault is sealed, unreachable, or the mount from step 1 isn't there — fix whichever it names and retry. On success, every new tenant-key wrap (new tenant creation, or an existing tenant's first secret if it never had a key yet) uses Vault transit from that point on; existing wrapped keys are untouched until you explicitly rewrap them.

5. Rotate — rewrap evidence​

Rotating the KEK is two Vault-side/API-side steps:

  1. Advance the transit key's version in Vault (operator action, outside the API): 
    vault write -f transit/keys/tenant-<tenant_id>/rotate
  2. Run the rewrap job so every vault_transit-wrapped tenant's stored ciphertext moves to the new version, without the plaintext DEK ever being exposed: 
    from services.key_rotation_status import vault_transit_rewrap_evidence
    report = vault_transit_rewrap_evidence(db)
    Pass tenant_ids=[...] to scope it to specific tenants, or omit it to sweep every tenant in the install. The report is shaped like the other key-rotation evidence sources (source / as_of / summary), with a per-tenant breakdown: rewrapped, unchanged (already current), skipped (tenant is still local-wrapped — nothing to rewrap; local rotation is the existing SECRET_ENCRYPTION_KEY_PREVIOUS path), or error (collected per-tenant so one bad key never blocks the rest of the fleet — check the error field and retry that tenant once fixed).

What this does not claim​

vault_transit supports a customer-managed-key model — the KEK lifecycle lives in Vault, under policy the operator controls, separately from the API's own master key. It is not, by itself, a compliance claim: it does not make the install FIPS-aligned or FedRAMP-aligned. See FIPS mode and Compliance profiles for what those actually require.