Secrets

docs/secrets.md

← All docs

Managing secrets in kilter projects

Short, practical guide for any kilter deploy project. The golden rule: deploy-time configuration is a secret, wired per-environment — there is no .env shortcut.

Why

At the current kilter CLI version, a plain env: block in kilter.yaml is not shipped to deployed pods. The only deploy-time env mechanism is the appSecrets → per-environment secrets: path. Two failure modes follow:

  • Silent fallback — a var that isn't wired is just absent, so the app uses its in-code default (e.g. an API client quietly falling back to localhost and reporting the upstream as unreachable).
  • Hard 400 — if an environment's secret object doesn't exist in its namespace, kilter deploy refuses: missing secrets in <ns>: <name>.

Both mean the same thing: a secret wasn't wired for that environment.

How it fits together

kilter.yaml                          secrets/<env>.env        in-cluster
─────────────────────────            ─────────────────        ──────────────
appSecrets:                          MY_API_URL=...       →    Secret object
  - <app>-secrets                    (plaintext, gitignored)   (SOPS-encrypted
environments:                                                   by kilter deploy,
  prod:                                                         decrypted by Flux)
    secrets:                    ┌───────────────────────┐
      <app>-secrets: ──────────┤ points at the file    │
        secrets/prod.env        └───────────────────────┘
  staging:
    secrets:
      <app>-secrets: secrets/staging.env
  • appSecrets: declares the secret name(s) the app mounts. Mounted optional: true in dev, so local kilter up works without the files.
  • Each environment maps that name to a local plaintext KEY=VALUE file.
  • kilter deploy SOPS-encrypts the file to the cluster's age recipient (sopsRecipient in kilter.yaml); Flux decrypts it into a real k8s Secret in the environment's namespace.

Rules of thumb

  1. One secret file per environment. secrets/prod.env, secrets/staging.env, … Each is independent; a value set in one is not inherited by another.
  2. secrets/ is gitignored — never commit plaintext. Only the SOPS-encrypted form belongs anywhere shared, and kilter deploy produces that. Verify before every commit: 
    git check-ignore secrets/<env>.env   # must print the path
git ls-files secrets/                # must be empty
  3. Adding a new environment = add an environments.<env>.secrets block and create secrets/<env>.env. Skipping either reproduces the 400.
  4. Adding a new variable = add it to every environment's file, not just the one you're testing. A var present in one env file and missing from another is the silent-fallback bug waiting to happen.
  5. Back up the source files out-of-band. The plaintext lives only on your machine; the cluster keeps only the encrypted copy. A lost laptop = reconstruct-from-memory. Use a password manager / shared vault for the canonical values.
  6. Treat every value as sensitive. Even "harmless" ones like service URLs leak topology; rotate any real credential the moment a plaintext file lands somewhere it shouldn't.

Recipes

Add a variable to an existing environment

echo 'NEW_VAR=value' >> secrets/prod.env   # repeat for each env file
kilter deploy --env prod                    # re-ship the secret

Add a new environment (e.g. staging)

# kilter.yaml
environments:
  staging:
    secrets:
      <app>-secrets: secrets/staging.env

cp secrets/prod.env secrets/staging.env     # then edit values for staging
kilter deploy --env staging

Emergency: create the secret directly in-cluster (bypasses SOPS/Flux — last resort; the next kilter deploy overwrites it from the file)

kubectl create secret generic <app>-secrets \
  --from-env-file=secrets/staging.env \
  -n <app>-staging --context <cluster-context>

Mini-cluster gotchas

  • Cross-namespace egress is firewalled. Pointing one app's *_API_URL at another namespace's service requires the target namespace labeled kilter.dev/role=infra; the kilter-app egress NetworkPolicy otherwise allows only same-namespace + port 443. The external hostname is no workaround — the gateway 403s hairpinned in-cluster traffic.
  • kubectl needs an explicit context. ~/.kube/config defaults to netshire prod. Use --context kind-kilter-<project> for a project's local KIND cluster or --context admin@mini-cluster for netshire — never rely on the default.