5 Security and Secrets
kiyi edited this page 2026-09-10 19:54:46 -05:00

Security and Secrets

This repository and wiki are public. Operational repositories live in the Homelab organization, whose repositories currently display Internal visibility. Repository visibility is not a substitute for secret handling.

Never commit or publish

  • Passwords, API keys, access tokens, RKE2 node tokens, or Kubernetes service-account credentials
  • Kubeconfigs, including rke2.yaml
  • TrueNAS, Cloudflare, AIStor/S3, PIA, Forgejo runner, database, or OAuth credentials
  • TLS/SSH private keys, OpenBao unseal/recovery material, bootstrap root tokens, or credential-bearing environment files
  • Real Kubernetes Secret data/stringData, even when base64 encoded

Base64 is encoding, not encryption. These restrictions also cover issue comments, screenshots, workflow logs, build contexts, packages and release artifacts.

Selected ownership model

Material Authoritative location and delivery
Application and platform runtime secret values OpenBao; ESO reconciles narrowly scoped values into Kubernetes Secrets
Application-specific CI authentication Owning application repository's Forgejo Actions secrets
Platform deployment authentication Homelab/platform Actions secrets unless deliberately shared
CI credentials intentionally shared across repositories Organization Actions secrets with appropriately limited access
Secret names, key names, policies and references Owning operational repository
Reusable public examples Sanitized placeholders only

OpenBao + External Secrets Operator is the chosen runtime architecture, not a future optional replacement for ordinary Forgejo-to-cluster injection. Forgejo remains appropriate for CI credentials. OpenBao/ESO integration is scaffolded but not yet confirmed deployed.

Prefer namespace-scoped SecretStores, dedicated service accounts, and policies restricted to each consumer's backend path. Use Kubernetes authentication and verified TLS. Do not share an OpenBao root token with workloads or runners. Verify supported CRD/provider/authentication fields against the selected ESO release before enabling the scaffold.

OpenBao bootstrap and recovery gates

Before activation, record the hosting location, release/image digest, durable data path, initial TLS trust, unseal procedure, audit logging, normal administrator identities and backup/restore procedure.

  • OpenBao must be recoverable without obtaining the TrueNAS CSI key from the same unavailable OpenBao instance. Independent static/local persistence or external placement needs a tested recovery design; placement remains undecided.
  • Initial TLS must not depend solely on cert-manager/ESO that cannot start until OpenBao works.
  • Preserve unseal/recovery material and required bootstrap access in independently protected storage outside the backend it unlocks. Do not use dev mode as the deployed service.
  • Back up storage, policies/authentication configuration and TLS trust, and test restoration across the intended failure boundary.
  • Use normal restricted administrator identities after setup and revoke the initial root token after bootstrap is complete.
  • Prove delivery of a harmless test value before migrating live credentials.

Existing retained Kubernetes Secrets can keep established consumers running during a backend outage, but reconciliation, rotation and clean-cluster recovery can fail. Monitor OpenBao availability/seal status and ESO readiness/staleness. Retention is not a backup.

TrueNAS CSI secret contract

The platform scaffold expects the following contract; confirm it matches the installed controller and node definitions before activation:

Field Expected value
Kubernetes namespace truenas-csi
Kubernetes Secret truenas-api-credentials
Secret data key api-key
Proposed OpenBao KV v2 mount secret
Proposed relative backend path infrastructure/truenas/csi
Backend property api-key
Proposed auth role/service account eso-truenas-csi

The locally saved truenas-csi-secret.yaml is a bootstrap artifact only. Do not add it to a repository or upload it into chat. There is no need to provide its plaintext contents now or later. If contract reconciliation is necessary, provide only metadata.name, metadata.namespace and key names, or a fully redacted copy.

Keep the current CSI Secret and working workloads intact while establishing the backend. Enter the real API key directly into OpenBao through a secure administrative session without exposing it to logs or shell history. Activate the ExternalSecret only after checking the intended value, ownership and rollback behavior. The scaffold uses creationPolicy Orphan and deletionPolicy Retain to avoid automatic loss of the runtime Secret; those settings alone do not prove safe adoption.

Verify SecretStore and ExternalSecret readiness, inspect only Secret metadata/key names, and validate provisioning, attachment, mounts and existing workloads. Retire the local manifest from routine use only after delivery and recovery work. The installed driver's credential-consumption behavior governs rotation; the recorded environment-variable contract requires a controlled controller/node rollout. Keep the prior credential valid through verification, then revoke it.

An inactive reference-only ExternalSecret can be committed before backend activation. It must not embed or import the local manifest and must remain outside active deployment roots until the secrets manager is functional.

Cloudflare and certificate automation

Cloudflare is the selected authoritative DNS provider for the new automation. The proposed KV v2 path is infrastructure/cloudflare/cert-manager, property api-token. ESO materializes cert-manager/cloudflare-api-token with key api-token through the dedicated eso-cert-manager identity. Scope access to the required DNS zone and ensure the issuer's cluster resource namespace matches the Secret location.

Use cert-manager DNS-01 issuance: test with ACME staging, then trusted production issuance and renewal. A staging certificate is not publicly trusted. The proposed dev certificate Secret is istio-tenant-gateway/dev-gateway-tls and is owned by cert-manager, not a second ESO writer. The Cloudflare token Secret is owned by ESO. Reconcile these proposed names with the working dev route before adoption and preserve existing UDS gateway-tls ownership.

The ingress issue already reports valid dev TLS. That does not establish automated issuance/renewal, or prove that the proposed Secret name is currently in use.

CI and temporary credentials

Keep environment-specific non-secret configuration in Git. Pipelines receive only credentials needed for their build, publishing or deployment job. If a tool requires a credential-bearing file, create it with restrictive permissions, use it once, and remove it in cleanup. Do not retain operational kubeconfigs or application secret files as routine runner configuration.

Existing administrator bootstrap material, including exported UDS certificate files, is transitional. Preserve required working access while migrating; do not delete the only recovery copy before OpenBao and independent recovery are validated. Runtime values rotate in OpenBao, CI credentials in Forgejo; trigger reconciliation and consumer-specific reloads afterward.

Secret scanning and infrastructure boundaries

Homelab/workflows#1 tracks secret scanning. Cover history and proposed changes with narrow documented suppressions. If a credential is committed, revoke/rotate it, assess exposure, and remediate history; removing the latest copy is insufficient.

DNS, AIStor, external PostgreSQL and Forgejo remain outside Kubernetes; Forgejo may share control-plane hardware, so outside Kubernetes does not imply hardware independence. Windows execution remains on Windows Hyper-V. Linux runner cutover completed on owner acceptance on 2026-09-10: global ci/deploy pools now depend on the cluster. Preserve an operator recovery path and local UDS artifacts for outages. Pool credentials live in OpenBao/ESO; global queue polling uses a separate instance-admin read:admin token. Deleted runner registrations require replacement credentials for rollback. See Linux Runner Migration.

See repository and UDS delivery architecture and the platform repository's docs/secrets.md for implementation references. No secret values are needed to maintain this documentation.