Internal per-service TLS certificates

The services inside a MOD install — Vault, Keycloak, Qdrant, Valkey, Postgres, PgBouncer, and the API behind nginx — talk to each other across the internal Docker network. This page covers the certificates that encrypt those hops.

In plain words

Think of it like a company's internal post office. Every internal service gets its own ID badge (a certificate) signed by the install's own security office (the local CA). When one service calls another over TLS, it checks the badge: "was this signed by our office, is it in date, and does it name the service I meant to call?" That check is what makes the internal traffic confidential and pinned to the right service.

Where the trust anchor (the local CA) comes from

The install uses one local CA for the internal hops:

• When the front Caddy runs in internal mode (MOD_FRONT_TLS_MODE=internal, i.e. tls internal), the internal CA is the front Caddy's own local CA — the same one LAN browsers already downloaded and trust (served at /mod-local-ca.crt). Nothing new to install: the same trust anchor covers both the public site and the internal services.
• When the front is public (Let's Encrypt) or off, Caddy has no local CA, so the install creates its own dedicated local CA once (a root and an intermediate certificate, valid 10 years). That dedicated CA becomes the trust anchor for the internal hops.

Which one it is, the install records for itself. If the front Caddy's local CA is ever regenerated (its root changes), the install notices and re-issues the internal certificates against the new anchor automatically.

What gets issued

One certificate per service, under CORE/startup/internal-tls/:

Service	Certificate valid for (host names)
Vault	vault, my_vault, localhost, 127.0.0.1
Keycloak	keycloak, my_keycloak_server, localhost, 127.0.0.1
Qdrant	qdrant, mod_qdrant, localhost, 127.0.0.1
Valkey	redis, my_redis, localhost, 127.0.0.1
Postgres	postgres-app, my_postgres_db, localhost, 127.0.0.1
PgBouncer	pgbouncer, mod_pgbouncer, localhost, 127.0.0.1
API	api, my_fastapi_app, localhost, 127.0.0.1 (nginx→API)

Each service directory holds:

• cert.pem — the service's certificate
• key.pem — its private key (mode 600; never leave this box)
• fullchain.pem — certificate + intermediate, what a server presents
• ca.pem — intermediate + root, the trust chain

Plus a single shared bundle internal-tls/ca-bundle.pem — the file clients verify against.

Each certificate is an ECDSA P-384 key with a 90-day validity.

Where the private keys live

The public files — every cert.pem, fullchain.pem, ca.pem, the CA certificates, and ca-bundle.pem — always stay in internal-tls/ under the install directory.

The private keys (the CA's root.key / intermediate.key and each service's key.pem) go where the filesystem can actually protect them:

• Native Linux host (any real disk): the keys stay in internal-tls/, mode 600, exactly as above. Nothing changes.
• A Windows-mounted install (WSL /mnt/c, or a FAT/9p mount) — where chmod 600 is a silent no-op and the keys would sit readable by any user on the box: the keys are moved out of the install directory into a private state directory on a native Linux filesystem — /var/lib/mod/internal-tls by default, or ~/.local/state/mod/internal-tls when the system path isn't writable — with mode 700 on the directory and 600 on each key. The move is automatic and one-time: on the first run after an install on such a mount the existing keys are copied over, checksummed, and the insecure copies deleted.

So a box never ends up with its private keys sitting mode-777 on a Windows drive. A small pointer file, internal-tls/KEYS_LOCATION (it contains only a path, no key material), records where the private keys live so the later service switch-over and the container mounts can find them without guessing. Once a box has migrated its keys, it keeps using that directory on every later run — even if the install later ends up on a native Linux filesystem — because "already protected" is never undone automatically.

How it stays current (no per-box manual work)

The install re-checks the certificates in three places, all in the system:

1. At install (start-main-server.sh) — right after the services come up.
2. On every update (update.sh) — after the update is healthy.
3. On the reboot/watchdog path (boot-reconcile.sh, driven by the systemd/WSL watchdog the install sets up) — at boot and on a repeating timer.

A certificate is re-issued when less than 30 days are left, when it no longer chains to the current local CA, or when a file is missing. The private key is kept across renewals (only the certificate is re-signed), so a renewal never changes the service's identity. Everything is idempotent: re-running never touches a certificate that is still healthy.

Is it on yet? (inert by design)

Issuing the certificates does not switch anything to TLS. Until the services are pointed at them (an owner-scheduled restart window, together with Postgres/PgBouncer TLS + SCRAM), the internal traffic behaves exactly as before. The enforcement dial for that switch-over is the platform setting gsec.transport.internal_tls_enforce (off → report → require); the certificates are ready and waiting regardless.

Verifying on a box

cd CORE/startup
# re-issue anything due (idempotent; safe to run any time)
bash lib/internal-certs.sh "$PWD" renew

# check one service (prints subject / names / dates only — never the key)
openssl verify -CAfile internal-tls/ca-bundle.pem internal-tls/vault/cert.pem
openssl x509 -in internal-tls/vault/cert.pem -noout -subject -dates -ext subjectAltName

Note (WSL): on a Windows-mounted install (/mnt/c) the POSIX file modes may not stick (the mount ignores chmod). The tool detects this and automatically moves the private keys to a protected native-Linux state directory (/var/lib/mod/internal-tls, or ~/.local/state/mod/internal-tls when the system path isn't writable) — see Where the private keys live. The public certificates stay in internal-tls/, where mode bits do not matter.