UDS Application Delivery and Repository Architecture

This page defines repository ownership, organization-wide planning, UDS delivery, CI/CD, and runtime secrets. Architecture lives in this public wiki; operational definitions and implementation issues live with their owning Homelab repositories.

ComfyUI automatic candidate decision — 2026-09-23

The owner approved automatic weekly candidate publication and manual deployment. Both the Monday 08:23 UTC schedule and the manual Update ComfyUI candidate → Run workflow → main action check declared upstream revisions first. Unchanged or already published candidates skip image and bundle builds. Changed candidates receive immutable kiyi source/lock and image-selection commits, scanned and signed image publication, and a signed UDS bundle. Existing Forgejo issue watches and account email settings deliver ready/failure notices on Comfy-UI #1. Routine deployment remains a separate owner action to prod, following the existing decision.

Implementation is proposed in workflows PR #53 and dependent Comfy-UI PR #26. The shared publisher opt-in is limited to ComfyUI and verifies the original protected-main run, bot commit identities, permitted source revisions and image-only manifest changes. Existing security gates, signing and exact expiring mitigations remain required. Dependency resolution and credentialed snapshot publication use separate jobs.

Local Linux validation passed 28 ComfyUI tests and 188 shared-library tests with one existing skip; the shared PR also passed hosted acceptance, library and security checks. Activation requires owner review/merge and a shared-library release (proposed v1.16.0), then the ComfyUI caller merge. Live update-only dispatch, automatic publication and email receipt still require validation; this records an accepted decision and tested proposal, not an enabled updater. The operator runbook covers activation and recovery. Earlier dated status sections below remain historical.

Image mitigation identity proposal — 2026-09-23

Comfy-UI update run 176 passed source scanning but blocked its locally built image on 232 vulnerability occurrences and 22 image-secret detections. The 224 previously reviewed Debian CVE/package-version cases appeared with identical package versions and severities in retained scan evidence; their build-input fingerprints changed. Shared workflows PR #54 proposes a selector that binds a temporary CVE review to the exact CVE, package, installed version, OS component, image repository and severity. Signed image receipts still bind the concrete digest and archive. Comfy-UI draft PR #27 applies the selectors without extending their 2026-10-18 expiry. Eight new vulnerability occurrences and 22 changed secret fingerprints remain blocking until separately reviewed. Image-secret false positives remain exact because a line/rule match alone cannot establish that a new value is still public. The run's failure notification also received Forgejo HTTP 403, so the kiyi update token's issue-comment permission needs correction before watch/email delivery can be verified. No candidate image, bundle or deployment resulted from run 176. Merge and release the shared library before the dependent Comfy-UI caller pin; both changes are proposed, not deployed.

Production hostname policy — 2026-09-21

Apps migrated from TrueNAS retain their existing production URLs. Their hostname/certificate handoff follows the legacy migration procedure below; do not rename them to the standard domain merely because they move into Kubernetes. New apps use the existing standard UDS gateways and domain (normally <app>.app.gitchegumi.com on istio-tenant-gateway/tenant-gateway, covered by the existing wildcard certificate). No per-app gateway or certificate is needed for those standard hostnames. Existing legacy gateway support remains.

ComfyUI will use comfyui.app.gitchegumi.com. platform PR #54 permits standard gateway reads during managed onboarding, and Comfy-UI PR #25 updates routing, SSO and deployment metadata. Dedicated ComfyUI edge/certificate proposals were removed. These changes are prepared, not evidence of production deployment. Dev is owner-confirmed running; subsequent ComfyUI updates go directly to prod, with the previous release and separate data backups retained for recovery.

Production deployment update — 2026-09-16

GitchAlerts is deployed to production (owner confirmed). Run 64 deployed the same signed bundle 0.1.6 tested in dev run 63, with managed credentials, successful rescan, rollout and HTTPS health. See GitchAlerts Migration for the exact release and evidence limits. The first-production gateway/certificate handoff is distinct from routine application deployment: remaining IaC reconciliation and reusable edge provisioning are tracked in platform#45. Earlier dated implementation-status paragraphs below retain historical context.

ComfyUI runtime decision — 2026-09-18

The owner clarified that distroless was a size-reduction idea, not a requirement. The priority is a simple ComfyUI build within the existing 24 GiB runner workspace, preserving current worker placement. Comfy-UI PR #9 replaces the earlier PR #8 scratch assembly with the pinned slim Python base and installs the locked dependencies once in the final stage. Temporary build tools and caches are removed in that same layer; the large virtual environment is no longer copied between stages. All 17 declared custom nodes, read-only host models, authentication and security gates remain. Local unit, parser and native package-cleanup checks passed. Run 72 subsequently completed build/export within the unchanged runner limit, producing a 4.46 GB archive; its security gate blocked publication. Scanned publication, GPU acceptance and deployment remain unverified. No runner storage increase or single-node restriction is being adopted. Comfy-UI #1 owns acceptance.

Dependency exception decision — 2026-09-18

The owner authorized exact, expiring mitigations for dependency findings without available fixes and separately approved reviewed image-secret false-positive exceptions. workflows PR #43 adds an explicit false-positive classification and upstream evidence link for an exact immutable-image/build-input finding. Gitleaks source/history secrets, malware and scanner errors remain non-waivable; this is not permission to accept real credential leaks. Comfy-UI PR #10 updates fixable dependencies and records temporary residual-risk controls and upstream-constant provenance, expiring 2026-10-18. Both changes are proposed, not yet adopted: release the shared capability first (proposed v1.13.0), then merge its paired application pins and run the normal scanned image build. Full scans, signing and dev acceptance remain required.

ComfyUI simplification — 2026-09-19

The owner requested a normal upstream installation instead of custom source rewrites. Comfy-UI PR #12 proposes one Docker stage, shallow Git fetches of the same 17 declared nodes, one installation of the existing hashed requirements with uv, and a shell entrypoint that executes upstream main.py. Five Python build/runtime helpers and all ComfyUI/RVC source patches are removed. Group authorization moves to a small custom-node adapter; the existing models, writable-state layout and access restrictions remain. Ordinary node import failures follow upstream behavior and require dev acceptance. Local authentication/entrypoint tests, a real pinned Git fetch and an installer dry run passed; this is a proposed change, not a published or deployed image. Run 84 was left untouched. Shared ClamAV scanning remains a separate cost; no end-to-end speedup has yet been verified.

Container publication decision — 2026-09-19

The owner approved building, scanning and publishing a container in one trusted disposable CI job. workflows PR #44 removes the large inter-job image artifact transfer. Registry/signing credentials are supplied only to the final publishing step after passing scans; step scoping is not separate-job isolation. Protected-branch owner dispatch, archive/digest verification, signed evidence before registry writes, and small scan-report artifacts remain. ComfyUI run 91 built/exported in 10m53s and passed its image gate, including 25m27s of malware staging/scanning, but the image artifact finalization timed out after a 37m52s upload step. The fix has 19 focused local tests passing; release adoption and live publication remain pending. No runner storage/placement change or scanner bypass is included.

ComfyUI saved workflows from /mnt/assets/ComfyUI/user/default/workflows belong under the already-persistent /data/user/default/workflows. The media-node backing directory is /mnt/assets/comfyui-cluster/dev/state/user/default/workflows for dev and /mnt/assets/comfyui-cluster/prod/state/user/default/workflows for production. Existing files still need migration and a save/restart persistence check; configuration is not evidence that the files have been copied.

ClamAV evidence reuse decision — 2026-09-19

The owner approved full scanning at container publication with reuse of signed results during bundle creation and deployment. workflows PR #45 implements a seven-day window from the original scan for the exact image digest, pinned engine and full-scan profile. Signatures and report hashes are verified before reuse; expired, missing or incompatible evidence triggers a fresh scan of the existing image, without a rebuild. The window explicitly retains the original signature database rather than claiming today's malware coverage. Trivy checks remain current. Security artifacts include ClamAV findings, per-image status, original scan time and reuse provenance. PR source scans remain unchanged and PR-provided reports cannot authorize skips. The implementation has 62 focused local tests passing and is proposed, not released or adopted by ComfyUI; Linux CI and live reuse validation remain pending.

Bundle storage simplification — 2026-09-20

ComfyUI run 129 passed security and SBOM generation, then its CI worker was evicted because docker-data exceeded the 24 GiB EmptyDir limit. The owner requested simplification before increasing runner resources. workflows PR #49 proposes one shared image-only Zarf package plus small dev/prod manifest packages in the same signed bundle, temporary caches and assembly scratch, removal of intermediate archives, and signing/publication in the build job without an inter-job bundle transfer. Deployment selects images before the chosen environment; older signed bundles remain supported. Final-step credential scoping is not separate-job isolation. No runner limits or placement change is included. All 167 local tests and a small real UDS 0.35.1/Zarf 0.83.0 packaging exercise passed. A subsequent packaging-only test used the real published ComfyUI image and pinned UDS binary on a hard 24 GiB ext4 filesystem, including a preallocated 6 GiB allowance for other runner files. The old path failed with no space left on device; the simplified path completed and verified its bundle hashes, peaking at 22.33 GiB with 1.11 GiB usable headroom. Final usage was 10.05 GiB including the allowance; the final bundle was 4.34 GB. The old run retried a registry transfer, so timings and the exact disk difference are not a controlled comparison. Security scans were represented by synthetic receipts and the fixed allowance, not rerun; signing/upload, actual complete CI overhead, release adoption and live deployment remain unverified. No production capacity change follows from this local test. Detailed test evidence is recorded in PR #49.

Repository organization

The Homelab organization and its repository shell are established. As observed on 2026-09-02, the organization is marked Private and its eight repositories are marked Internal in Forgejo. Record those actual settings rather than describing every repository as Private. Visibility never makes it acceptable to commit secrets.

Repository	Responsibility
gitchegumi/homelab	Public wiki, architecture decisions, reusable runbooks, diagrams, sanitized boilerplate YAML, and documentation issues. Historical issues remain useful references; active operational work belongs in Homelab.
Homelab/platform	RKE2/UDS platform configuration, MetalLB, CSI, shared Istio edge, cert-manager, OpenBao/ESO, platform packages and bundles, and platform deployment ownership.
Homelab/workflows	Reusable Forgejo Actions workflows, common CI/CD scripts, secret scanning, and runner work. This is the established name; the earlier proposed pipelines repository is superseded.
Homelab/bundles	Composition of versioned packages across applications and reusable coordinated migration/rollback procedures. It consumes packages owned by platform/application repositories rather than duplicating their source.
Application repositories	Each owns its application source or upstream integration, chart/manifests, Zarf package, application bundle, tests, release workflow, and issues.

Application repositories currently established: actual, comfy-ui, satisfactory-game-server, tradegumi, and foundryvtt. A repository's existence does not mean its application is deployed.

Platform-specific bundles remain in platform. Application-specific bundles remain with their applications. The bundles repository owns compositions that span those release boundaries. Each deployed resource must have one configuration owner; do not create parallel copies across repositories.

Organization roadmap and issue ownership

The Homelab Roadmap is the shared board across repositories. Its columns are Backlog, Ready, In Progress, Review, Done, and Blocked. Issues stay in the repository responsible for implementation and are added to the board without creating duplicate tracking issues.

Work	Current tracker
Istio primary edge and NPM passthrough, formerly public issue 20	Homelab/platform#1
Storage/data-protection conventions, formerly public issue 6	Homelab/platform#2
NVIDIA GPU enablement	Homelab/platform#3
Secret scanning	Homelab/workflows#1
Linux runner migration — completed on owner acceptance 2026-09-10	Homelab/workflows#2 and Homelab/platform#32, both closed; global ci/deploy pools are live
Repeatable application migration and rollback	Homelab/bundles#1

Use fully qualified issue links across repositories. Preserve old references as historical context and update active documentation to the new destination. The board is a planning view; completion evidence and acceptance criteria remain in the owning issue. Links from this public wiki to operational repositories require appropriate access.

Source-of-truth boundaries

• This wiki defines architecture and records decisions. Specific completed runbooks and current issue evidence take precedence over older conceptual diagrams.
• Each operational repository owns how its resources are built, configured, packaged, tested, and released.
• Public YAML is reusable boilerplate or explicitly labeled historical/transitional material, not a second operational deployment source.
• The live cluster is a deployment target. Reconcile emergency live changes into the owning repository.
• Git contains non-secret configuration and artifact references. An OCI registry contains immutable images and published packages; the tested bundle archive is published to the selected Forgejo artifact/release or supported OCI destination. Artifact publishing is not yet demonstrated by the repository shell.

Platform repository baseline

The initial platform structure and deployed-file reconciliation were committed as kiyi, most recently in bd5dc8b6c6.

Path	Contents and readiness
bootstrap/rke2	Sanitized host/RKE2 configuration examples and platform notes
infrastructure/uds-core	Core ownership and prerequisite notes
infrastructure/metallb and infrastructure/truenas-csi	Configuration structure for established services; environment capture and exact installed artifact pins remain necessary
infrastructure/istio/npm-legacy	Confirmed deployed Gateway, VirtualService, ServiceEntry, Kustomization, 29-host allowlist, and validation script
infrastructure/istio/dev	Deployed dev gateway; production dev certificate issuance and trusted TLS verified
infrastructure/cert-manager, infrastructure/external-secrets, infrastructure/openbao	Deployed OpenBao/ESO and cert-manager; CSI adoption passed, production certificates issued, persistent OpenBao HTTPS confirmed
environments/homelab/transitional	Only the confirmed legacy edge resources, for controlled migration/testing
packages/legacy-edge and bundles/legacy-edge	Configuration-only Zarf package and UDS bundle definitions
bundles/platform	Inactive full-platform example with unresolved deployment prerequisites
docs and scripts	Current-state, deployment, recovery and validation guidance; local validation tooling

The four supplied deployed YAML files matched the platform configuration semantically. The imported legacy validation script rejects failed requests instead of counting matching HTTP 000 failures as success. It compares behavior using insecure TLS requests, so its success alone does not establish certificate trust. Repository initialization performed local validation, not a new live-cluster audit, UDS artifact build, or deployment.

UDS package and bundle model

UDS bundle-style deployment is the target. A zarf.yaml packages charts/manifests and required images; an uds-bundle.yaml composes versioned packages. A UDS Operator Package resource is a separate integration API, not the bundle archive or Zarf artifact.

Raw kubectl manifests remain only for clearly marked current-state migration, diagnostics, and controlled testing. Do not recursively apply infrastructure or activate unresolved example files. Controllers require pinned charts/images, CRDs, and reviewed UDS network/policy integration before their packages are enabled.

Before adopting existing kubectl-managed resources into a Zarf/Helm release, compare the live resources and inspect ownership. Test an explicit adoption plan for the intended objects. Do not delete the serving Gateway or take broad ownership of UDS Core resources to force installation. Preserve the existing tenant/admin gateway releases and TLS ownership.

A deployment consumes the exact immutable artifact that passed validation. Promotion must not rebuild it. The full-platform example is not yet a fresh-cluster installer; host/RKE2, storage, recoverable secrets, and external-service prerequisites must be satisfied first.

CI/CD and deployment

1. Validate source, manifests/charts, package and bundle definitions; test owned application code.
2. Scan committed content, dependencies and images, and discover/pin every required image.
3. Build the versioned Zarf package and bundle, then exercise them in a designated test environment.
4. Sign/publish from an approved protected branch or release tag and retain source/version/checksum provenance.
5. Deploy that same tested artifact with narrowly scoped CI authentication from a runner with cluster access.
6. Verify health, routes, runtime secret readiness, and rollback; record evidence in the owning issue and clean transient credentials.

Forgejo Actions secrets hold CI-only credentials at the narrowest useful scope: application-specific credentials in the application repository, platform deployment credentials in platform, and intentionally shared credentials at organization scope. Runtime application values are delivered by OpenBao and ESO. CI should not need to extract and package runtime values merely to deploy their references.

Use protected/manual production promotion and isolate privileged deployment execution from untrusted pull-request code. Runner labels, registry namespace, signing identity, exact artifact pins and deployment credential scope still require implementation records. The workflow repository shell does not establish a functioning pipeline.

Secrets architecture

OpenBao is the selected self-hosted runtime secrets backend. External Secrets Operator (ESO) reconciles scoped backend values into Kubernetes Secrets consumed by CSI, cert-manager, and applications. Forgejo Actions secrets remain the CI credential store. This supersedes the older recommendation to inject normal application runtime secrets from Forgejo workflows or postpone the external backend.

OpenBao → External Secrets Operator → Kubernetes Secrets → workloads/controllers
Forgejo Actions secrets → scoped CI authentication → build/publish/deploy jobs

Secret values never belong in Git, build contexts, bundle archives, logs, or public documentation. Rotation occurs at the owning backend followed by reconciliation and the consumer-specific reload/rollout. See Security and Secrets for contracts, bootstrap handling, recovery gates, and the Cloudflare/TrueNAS transition.

Current ingress and delivery status

Reconciled on 2026-09-03. OpenBao/ESO and CSI adoption are complete for the agreed scope in platform#5; the initial root token has been retired. An isolated OpenBao restore environment is not a closure requirement. Production dev, tenant and admin certificates have been issued by cert-manager using ESO-sourced Cloudflare credentials.

The working edge path is Xfinity TCP 80/443 -> Sager relay (10.0.0.107) -> Istio tenant VIP (10.0.0.241). The relay performs TCP forwarding only. Explicit legacy SNI hosts continue to NPM (10.0.0.116:443); native hosts terminate TLS at Istio. The admin VIP (10.0.0.240) remains the intended internal-only endpoint, and must not be added to WAN forwarding.

Fresh read-only checks: OpenBao UI via the admin VIP returned trusted HTTP 200; Keycloak admin and Grafana returned 302; portal returned 302 and SSO 301 through the tenant VIP; Forgejo through the relay returned trusted HTTP 200. The relay Deployment is 1/1 ready. The owner confirmed OpenBao works in a normal browser at https://openbao.admin.app.gitchegumi.com/ui/ without port forwarding. An admin hostname through the tenant VIP failed certificate hostname verification; this is not proof of external isolation.

Core gateways remain owned by their existing uds-istio-config Helm releases. Server-specific references select cert-manager Secrets; public companion CA Secrets preserve Keycloak OPTIONAL_MUTUAL client trust. Public trust copies are operator-managed. The one-time Core cutover uses Helm client-side merge for Helm 4 compatibility with Zarf ownership. Preserve the equivalent overrides in the real Core bundle before the next Core deployment. See the HTTPS, trust and rollback runbook and relay runbook.

Remaining implementation sequence

1. Finish the evidence gaps in platform#1: current router-rule confirmation, complete LAN hostname resolution coverage, external admin isolation, native versus passthrough log correlation, and the agreed direct-NPM rollback check. Prior owner evidence records all 29 legacy hosts matching NPM; it does not establish every current LAN DNS record or public isolation.
2. Establish the controlled deployment pipeline before migrating applications. The assistant writes/publishes code and opens a PR; the owner reviews, merges, updates WSL, builds and deploys. Manual infrastructure deployment remains acceptable.
3. Deploy Actual from Homelab/actual at a dev hostname, verify application behavior and persistence after pod recreation, then perform the production hostname handoff below. Actual supplies the remaining application evidence for #1/#2. Keep both issues open until the agreed evidence is recorded.

Per-application hostname handoff

1. Keep the existing NPM Proxy Host and data intact. Back up application data and record the previous package version and routing state.
2. Deploy the native application through its UDS bundle at an Istio-owned dev hostname. Validate trusted HTTPS, login, application behavior and persistent data before production cutover.
3. Prepare a coordinated change removing the production hostname from every npm-legacy SNI/host allowlist and validator inventory, and adding its native route and certificate coverage. Keep platform edge changes in platform and app resources in the application repository. Do not leave the same hostname claimed by both native TLS termination and legacy passthrough.
4. Apply the reviewed handoff during the agreed window; validate from LAN and externally. Native and passthrough request logs should establish which backend handled the probes.
5. If validation fails, restore the prior route/allowlist configuration. A routing rollback does not undo data writes or schema changes; follow the application's data rollback plan. Retain NPM configuration until acceptance. Retire NPM only when the legacy inventory reaches zero.

Failure and recovery

Retain previous tested artifacts and component-specific rollback instructions. Redeploying an older bundle does not reverse database schema changes, CRD changes, or storage deletion. Preserve independent OpenBao, PostgreSQL, and object-storage recovery paths. A runner failure must not destroy the only copy of credentials or configuration.