Homelab Operations Status

Production deployed — 2026-09-16

The owner confirmed GitchAlerts is deployed to production. Production run 64 selected prod / namespace gitchalerts and passed the gateway check, signed artifact verification, rescan (0 blocking, 2 mitigated, 0 scanner errors), Redis/application rollout and HTTPS application health. The managed OpenBao session was revoked afterward.

GitchAlerts 0.12.4 / bundle 0.1.6; release ci-100e14d72cdc9da1b80c067f507d279f6e9e0c3a-1789, also deployed successfully to dev in run 63. Archive SHA256: 2a0ae31cb20008800666a9f0bb370321b333b4a5500122b54c61dc8f1b82a2e5.

Production uses external PostgreSQL, S3 assets and OpenBao runtime values at secret/apps/gitchalerts/prod/runtime. Hostnames are alerts.gitchegumi.com and alerts-webhook.gitchegumi.com. The configured Keycloak groups are /GitchAlerts/Users and /GitchAlerts/Admins.

This supersedes earlier preparation-only status. Detailed provider/webhook, group-revocation, asset-write, restart/restore and certificate-renewal tests are not asserted by the deployment result. Dev deletion and TrueNAS retirement are not claimed. The exact applied gateway/certificate YAML and reusable first-production routing automation remain platform follow-up; successful app deployment alone does not prove those changes were committed to IaC.

Migration record.

The older dated sections retain their historical evidence; GitchAlerts production deployment is no longer pending.

Scope: this environment only. Last updated 2026-09-10 (runner cutover complete; other sections retain their recorded evidence). Owner-reported results are distinguished from automated validation. This page records operational facts; it is not a generic installation guide.

Linux runner migration: complete

Owner-confirmed cutover completed 2026-09-10 after global queue polling permissions were corrected. Merged platform PR36 supplies worker bundle 0.1.2. Global ci/deploy pools share four CI slots plus one deployment slot across Sager, Optiplex and Media; Windows Hyper-V is unchanged. Normal delivery uses runner_test=off; old canary labels no longer have matching pools.

Workflows runs 196/197/206 cover queueing and lifecycle cleanup; Actual runs 128/137/144 cover build/scan/transfer, credentials and signed dev deployment. The owner confirmed dev login, cleanup and final cutover. Migration issues platform#32 and workflows#2 are closed. Exhaustive fault injection remains unmeasured, not asserted passed. See Linux Runner Migration for evidence, accepted scope and rollback limitations.

Actual Budget: accepted in production

The owner confirmed RKE2 startup, Keycloak login, intact existing budgets, successful sync, persistence after a deployment restart, and successful use of https://actual.gitchegumi.com. Dev validation used https://actual.dev.gitchegumi.com. No fresh external-network or unauthorized-user test was reported in this cutover conversation.

• Repository: Homelab/actual-budget.
• Namespace and Deployment: actual-budget; one replica, Recreate strategy.
• Actual 26.9.0; pinned amd64 image docker.io/actualbudget/actual-server@sha256:59cab86c77e764855119e2d5ddd96b9c9cefd90913f4feed7f4ee1433a6482c5.
• Retained NFS source: 10.0.0.116:/mnt/homelab/apps/actual, mounted at /data; PV actual-budget-data, PVC actual-data; UID 3000, GID 3001, supplementary group 568. This data is still on TrueNAS; the application runtime moved to RKE2.
• Production cutover configuration: istio-tenant-gateway/actual-budget-prod Gateway and Certificate; secret actual-budget-prod-tls; issuer letsencrypt-production.
• Existing Keycloak client ID remains actual-budget-dev to preserve identity. Production callback was added; app server URL changed to production. The client name is a historical identifier, not evidence that traffic is still on dev.
• Only Actual's hostname was removed from the shared npm-legacy Istio route and Gateway. Other legacy applications continue using their existing route.
• The owner handles retirement of the stopped TrueNAS container and NPM host. Completion of that cleanup is not claimed here.

Delivery findings

1. The archive-filename fix shipped in v1.3.6; the currently adopted delivery workflow release is v1.5.0 (source-security callers remain on v1.4.3). UDS 0.35.1 rejects the generic bundle.tar.zst filename with a misleading not-found error. New releases preserve the generated UDS filename; legacy archives are verified then copied unchanged to a compatible local name.
2. The original migration release was ci-4519315e4e5d761ee917c04ca529de98eeeafb12-270. Existing release assets were reused through workflow fixes. Scanning and explicit, expiring image-finding mitigations remain enforced.
3. Private Forgejo release attachments require the scoped RELEASE_READ_TOKEN (Homelab organization secret); the automatic Actions token did not authorize the attachment route on this instance.
4. The initial app deployer token expired after hours. Manual replacement is a transitional mechanism, not the desired operating model.
5. App deployment initially stalled because its static PV was absent. Provisioning was a separate action and was ultimately applied by the owner. Future app Actions must orchestrate this prerequisite.
6. The original migration archive was dev-only. New v1.5.0 builds include both reviewed environment profiles and production routing. Use a new dual-profile release for promotion; do not reuse the old dev-only migration archive. Production promotion acceptance is still pending.
7. Existing sessions were intentionally allowed to survive migration. Restart/persistence acceptance is manual; session revocation is optional in this environment.

Certificates and credentials

cert-manager manages issuance and renewal for Certificate resources. The production script requested a managed certificate and displayed its renewal schedule. A future renewal event has not yet been observed; verify renewal health and that the Istio listener serves the renewed certificate, rather than assuming configuration alone proves it.

The requested delivery experience is environment selection (dev or prod) and a manual Action run, with no routine expiring kubeconfig replacement. Workflows #19 and platform #24 track Forgejo OIDC/OpenBao credential issuance. Workflows #5 tracks exact-artifact promotion. These remain open until live acceptance.

The intended solution issues temporary credentials automatically for each run; the owner should not rotate those credentials. Any remaining manually managed integration credential should have a maintenance interval of at least three months where supported. Do not substitute one cluster-admin kubeconfig shared across all repositories.

Documentation ownership

Operational manifests and scripts belong in Homelab/platform, reusable jobs in Homelab/workflows, and app-specific configuration/tests in the app repository. This wiki is the documentation repository and links to implementation and evidence. Application Delivery Automation covers the reusable design without homelab-specific addresses.

Automatic credentials: verified 2026-09-08

Actual run 95 and run 96 passed on main at 15fcde4f2fbe4fbe181fc622c2f488cf01ed8526. Each obtained Forgejo OIDC identity, authenticated to OpenBao, requested a two-hour Kubernetes credential, passed 12 required and 7 forbidden permission probes and revoked its OpenBao session. These were credential-validation runs, not deployments. No manual token replacement was needed.

OpenBao JWT trust and its Kubernetes secrets-engine role are configured. The runner initially resolved openbao.admin.app.gitchegumi.com to the public address, which served the app.gitchegumi.com certificate. The owner reports fixing the runner path through a hosts-file mapping to 10.0.0.240; both subsequent Actions passed with TLS verification enabled. Update 2026-09-10: cluster CoreDNS now forwards through Pi-hole and both tested replicas resolve the private VIPs. The operator browser works without hosts entries when Secure DNS does not bypass Pi-hole. Technitium migration remains separate.

Environment promotion: dev and prod deployed successfully

Workflows #30, platform #30, Actual #18 and bundles #7 are merged. Workflows v1.5.0 is released; 105 library tests and hosted integration/security checks passed. Actual's validator now checks separate strict dev/prod custom-resource profiles, unknown fields/GVKs and unclassified files in both directories.

The owner confirmed the new dev deployment worked correctly. Dev uses namespace/package actual-budget-dev, client actual-budget-test and disposable emptyDir data. Production retains namespace/package actual-budget, PV actual-budget-data, PVC actual-data and historical SSO client actual-budget-dev. Build once, select dev, then select prod with the same new signed release tag.

Production retry findings

• Run 105 passed credential issuance, verification and rescan, then hit Helm server-side ownership conflicts from the manual production URL cutover: kubectl-patch on Package spec.sso and VirtualService hosts/gateways; kubectl-set on the Deployment server-hostname environment value. Rollback also failed. Owner output confirmed the original Deployment and pod remained 1/1 Running; Helm revision 1 was deployed and latest revision 3 failed, not pending.
• A one-time targeted ownership-metadata repair was prepared. It preserves application specs, data and other field managers; it is not a normal upgrade requirement. The subsequent production retry completed successfully in run 107.
• Run 106 stopped before deployment at permission 14/20: create deployments.apps in actual-budget-dev. Owner confirmed deleting the dev namespace after testing, which also deleted its Role and RoleBinding. The current shared permission contract checks both environments even for prod. Reapply the merged platform RBAC before retrying. This recreates the namespace/permissions, not the dev application. OpenBao session revocation succeeded.

Run from the owner's WSL checkout of merged platform main:

kubectl apply -f /mnt/e/forgejo/homelab/platform/infrastructure/deployment/actual-budget/rbac.yaml

For released v1.5.0, keep the dev namespace/RBAC when cleaning up a test app, or reapply this prerequisite after deleting it. Production deployment is now verified: run 107 selected prod/actual-budget, deployed signed release ci-a20d05f741c712161c17d2fed8f9312adc2b69bf-422, passed deployment and rollout checks, and revoked the OpenBao session. The owner also confirmed prod worked. Remaining live negative credential cases, observed certificate renewal and full future-app storage/bootstrap orchestration are not claimed complete. Static retained PV provisioning remains a platform prerequisite.

Upgrading Actual

Update both app image digests and environment version values through a reviewed app PR, along with the approved-image assertion and version documentation. Build/publish once, test the release in dev, back up production data, then deploy prod with that same release tag. Credentials are issued automatically. Review scan findings against the new digest rather than copying old risk acceptance blindly. Empty dev validates app behavior, not migration of existing budget data; application rollback and database restore are separate procedures.

Disposable namespace automation: in review

Workflows #31 separates selected-environment permission checks and adds an optional reviewed namespace/binding bootstrap. Platform #31 supplies durable ClusterRoles plus an identity-specific Fail/Deny admission policy. The deployer ServiceAccount remains in platform-provisioning. Prod will not require dev RBAC; a dev deployment can restore its deleted namespace and binding without manual credential renewal.

These changes are not yet released or installed. Local workflow tests (112) and platform manifest/bootstrap tests pass. Platform policy installation and live admission denial/recreation acceptance remain required, followed by a library release and caller pin updates. The running prod application is unchanged. The installer checks policy type checking before granting bootstrap permissions; do not apply the grants file without its admission policy.