This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Linux runner migration — complete

Owner-confirmed cutover complete, 2026-09-10. Platform PR36 is merged. Global Kubernetes ci/deploy pools now provide four CI slots plus one deployment slot, scale to zero, and can run on Sager, Optiplex or Media. Worker bundle 0.1.2 retains KEDA 2.20.2 and the existing controller. Windows Hyper-V remains unchanged.

The final blocker was global queue HTTP 403: the separate queue-reader token needs instance-administrator Admin Read (read:admin), in addition to valid global pool UUID/token pairs. The owner reported successful cutover after this correction. This is owner acceptance, not a fresh independent cluster inspection. platform#32 and workflows#2 are closed by request.

Acceptance includes workflows runs 196, 197 and 206 (concurrency/queueing, cancellation, deliberate failure and cleanup), Actual run 128 (build/scan/transfer), run 137 (credentials), and run 144 (signed dev deployment), with owner-confirmed dev login and post-run zero nonterminal deployment pods. Exhaustive fault injection and continuous total-pod timelines were not measured; closure does not claim those tests passed. The owner accepts fixing overlooked legacy consumers later.

Use standard ci/deploy labels and runner_test=off. Historical ci-k8s/deploy-k8s probes have no matching pools. Credentials remain in OpenBao/ESO; deleted registrations require replacement credentials for rollback. The operator builds/deploys UDS manually; platform runbook owns current maintenance. Pi-hole forwarding resolved cluster split DNS; browser Secure DNS must not bypass local records. DNS replacement and GPU/application migration issues remain separate.

Prepared onboarding connectivity correction (2026-09-12)

Platform PR41 prepares worker bundle 0.1.3. Run 269 discovered GitchAlerts dev/prod and obtained OpenBao and Kubernetes credentials, then reset on its first API request. The CI Package currently allows HTTPS on 443 but omits the deploy pool’s UDS-generated KubeAPI rule. The proposed correction adds that rule to CI and keeps onboarding on a separate CI slot so a waiting deployment cannot block registration. All CI workers gain API reachability; they still receive no mounted Kubernetes token, and privileged issuance remains bound to the protected platform workflow in OpenBao.

Repository, pinned schema/native Istio checks and 13 runner regressions passed locally. The network diagnosis is consistent with configured policy, not a live policy/proxy inspection. No bundle was built or deployed by Codex; owner rollout and successful reconciliation remain pending. Follow the worker runbook for build, verification and rollback.

Prepared container build-network correction (2026-09-18)

ComfyUI CI demonstrated a nested Docker bridge DNS failure while the same resolver worked through the private worker pod network. Its CI build progressed through source downloads and native dependency compilation after selecting the pod network and HTTPS Debian mirrors. This verifies the alternate build path, not a root-cause repair of bridge/CNI behavior. The worker pod remains the existing per-job network boundary; no Kubernetes host networking or cluster policy change was applied.

Workflows PR39, kiyi commit bd19963dae, adds an explicit build_network: worker-pod opt-in to the signed-image builder with a private-loopback-daemon/context guard and receipt recording. All hosted acceptance, security and library integration checks passed (runs 240–242). It is prepared, not released or adopted: owner review/merge, a new immutable minor workflow release, caller pin updates and a successful signed-image run remain required. Existing v1.9.0 callers keep their default network. See adoption and rollback. ComfyUI application/runtime acceptance remains tracked in Comfy-UI#1.

Historical record

Earlier pending states below are superseded by the completion statement above.

Earlier update: global cutover prepared (2026-09-10)

The owner selected instance-wide runner pools and accepts repairing any overlooked legacy consumers later. Platform PR35 merged standard ci/deploy labels. Platform PR36 prepares global queue polling and worker bundle 0.1.2, preserving four CI slots plus one deployment slot. Global registrations replace the organization-scoped UUID/token pairs in the existing OpenBao paths; the separate polling token requires an instance administrator with read:admin permission. Application deployment permissions remain unchanged.

The owner reported deleting the old runner registration and will build/deploy from the PR36 branch to restore hosted checks before merging, without bypassing branch protection. Local checks and all 12 regression tests passed; global deployment, cross-organization execution and final Compose shutdown are not yet verified. Deleted registrations require replacement credentials for rollback. Follow the branch runbook. Codex has not built or deployed these bundles.

Real canary acceptance: Actual Budget run 137 passed credential validation and run 144 deployed the signed dev release successfully; the owner confirmed actual.dev login and no nonterminal deployment pods. Both CoreDNS replicas returned private VIPs after Pi-hole forwarding. For the operator browser, disabling Secure DNS restored OpenBao access without hosts entries; the private VIP serves a certificate covering *.admin.app.gitchegumi.com. This does not establish household DNS migration.

The older entries below are historical and their pending items or initial organization scope are superseded where explicitly updated above.

Linux Runner Migration

Status: KEDA controller readiness confirmed by operator-provided live output on 2026-09-09 after recovery from the initial UDS admission failure. Both worker pools are deployed and their secrets reconciled; the first concurrency probe passed all seven jobs and both pools returned to zero nonterminal pods. Migration acceptance remains pending. The owner selected Linux runner work as the next environment priority. workflows#2 tracks architecture and the concurrency probe, merged in planning PR #32 at e1d7f23. platform#32 tracks infrastructure implementation and live acceptance; platform PR #33 merged at 85d000c.

The target is on-demand Kubernetes worker pods: zero idle workers, one Actions job per disposable pod, and a five-worker ceiling, initially four CI slots plus one deployment slot. Sager, OptiPlex and Media are all eligible pod hosts, explicitly selected by the owner. The control-plane node is excluded. Kubernetes resource requests may reduce actual concurrency below the ceiling; the earlier Media memory snapshot does not exclude it from scheduling.

Owner-provided configuration and SSH inspection confirmed one Docker Compose runner on Media under /srv/forgejo-runner, with voa, homelab and forgejo connections, shared voa/ci/deploy labels and capacity one. Homelab registration ID 4 and global registration ID 6 belong to this installation. Its private privileged DinD engine and nested jobs have 3 CPU / 16 GiB limits; the runner has 0.5 CPU / 512 MiB limits. Persistent Docker data and cache are under /mnt/scratch-ssd/forgejo-runner/. No host Docker socket or published Docker API port is configured. No connection secrets were copied into source or artifacts.

The canary pins KEDA chart/images 2.20.2, runner 13.0.0, Docker 29.7.2 and the observed Ubuntu action-image digest. Each pod has a non-root one-job runner and a private privileged Docker native sidecar, loopback Docker API and disposable storage. Canary labels are ci-k8s and deploy-k8s. Each pod requests 1.1 CPU / 8,320 MiB; limits are 3.5 CPU / 8,704 MiB with a 7 GiB nested job ceiling. Validate real build/scanner peaks against the previous 16 GiB budget.

Review corrections in platform 1aca5af scope KEDA's two Istio TLS exceptions to their actual rendered workloads and individual API ports, with STRICT defaults. The hosted runner-validation workflow and local bundle build now run the same worker semantic checks, checksum-locked KEDA/UDS/ESO source-schema checks, native Istio validation and eight negative regressions. The lock records exact source versions and hashes; no installed-cluster schema snapshot is needed. The suite passed locally in the pinned CI image. These checks establish source validation, not live admission or migration acceptance; current hosted results and rebuilt artifact hashes belong in the PR and implementation issue.

The initial implementation uses distinct persistent pool registrations in OpenBao/ESO Secrets mounted only by the runner. These are disposable pods, not server-enforced single-use registrations. The earlier mandatory broker proposal is deferred because per-job ephemeral registration and cleanup require additional control-plane automation. KEDA is trusted with namespaced Secret access. Privileged DinD retains node compromise risk; scoped UDS exemptions and namespaces do not provide VM isolation. Canary HTTPS egress also permits reachable internal HTTPS; preserve existing workflow authorization and OpenBao trust gates.

The operator-built controller 0.1.0 bundle failed in keda-prerequisites before the KEDA chart: four ingress allow rules generated the same NetworkPolicy name because their descriptions matched. Platform PR #34, commit 3432b71, gives each source/destination pair a unique description, versions the controller package/bundle as 0.1.1, and adds a UDS Core 1.11.1 name-collision check plus three regressions. Source validation and all eleven regression tests passed locally, and all hosted checks passed for that commit. The operator subsequently reported a successful KEDA rollout; Codex did not build the corrected bundle or perform this deployment. The recovery instructions target controller 0.1.1; the supplied readiness output does not independently identify the deployed artifact checksum. The worker bundle remains 0.1.0.

Operator evidence on 2026-09-09: kubectl -n keda rollout status deployment/keda-operator reported successful rollout. kubectl -n keda get deployments showed keda-admission-webhooks, keda-operator and keda-operator-metrics-apiserver each READY 1/1, UP-TO-DATE 1 and AVAILABLE 1. This establishes controller readiness and clears the earlier controller admission blocker; it does not prove worker admission, queue access or job execution. Evidence is recorded in platform#32.

The owner confirmed completing both persistent pool registrations, OpenBao secrets, policies and Kubernetes auth roles. Queue polling uses a dedicated token belonging to gitchegumi, with organization Read access, following the owner’s explicit selection. The earlier kiyi token received HTTP 403 with Must be an organization owner; kiyi was not elevated. Both ci-k8s and deploy-k8s organization queue checks returned HTTP 200 with the owner-account token. Codex commits and browser mutations continue as kiyi.

After worker deployment, owner-provided output showed both openbao SecretStores Valid/Ready, all four ExternalSecrets SecretSynced/Ready, and both ScaledJobs Ready, Active=False, Paused=False (CI min/max 0/4, deployment 0/1). Both namespaces had zero pods before dispatch. In probe run 196, four CI pods reached 2/2 Running with zero restarts: one on Sager, two on OptiPlex and one on Media. Forgejo showed four CI jobs and the deployment probe running, with the remaining two CI jobs waiting. The supplied watch output duplicated the CI namespace; deployment-pod placement and the total Kubernetes pod ceiling still need direct confirmation.

Probe completion: browser verification of run 196 showed all six CI jobs and the deployment probe succeeded. The owner then queried both namespaces with --field-selector=status.phase!=Succeeded,status.phase!=Failed; both returned No resources found. This verifies the first successful job/queue-drain lifecycle and return to zero nonterminal worker pods. It does not establish continuous total pod-count evidence or failure/cancellation behavior.

Still required: full pod-count/timeline evidence, failure/empty-queue race tests, actual nested-job DNS/TLS to required services, real builds/scanners and disposable deployment. Account for every global/VOA consumer before stopping Compose. After acceptance and deliberate label transfer, stop both Compose services on Media while preserving configuration/storage for rollback.

Platform owns the controller/worker packages and operator runbook. Workflows owns compatibility and acceptance. Windows-native execution remains on Windows Hyper-V. Retain an external Linux recovery path and operator UDS artifacts for cluster outages. The legacy runner configuration, credentials and labels are unchanged. KEDA and both worker-pool configurations are ready; the first live probe completed successfully as recorded above. See operations status for accepted application delivery evidence.

Operator workflow: the owner builds and deploys on their own machine to gain hands-on experience with Defense Unicorns tooling. Until explicitly requested again, provide individual UDS, Zarf, Helm and supporting commands with explanations of inputs, outputs and verification; do not introduce build/deployment wrapper scripts or direct the operator to build.sh. Existing helpers are not authorization to automate future operator steps.

Cancellation evidence: run 197 and all seven jobs were Canceled. The owner’s subsequent nonterminal-pod queries returned no resources in either pool. This verifies cancellation of startup/queued work and subsequent cleanup. The inspected deployment probe and CI job 1 show 48s in Set up job and 0s in their hold step; cancellation while a shell command is executing was not established by attempt 1 and was checked separately in attempt 2 below. Continuous post-cancel churn observation and other failure tests remain outstanding.

Executing-step cancellation verified: run 197 attempt 2 shows all seven jobs Canceled. The inspected deployment and CI slot 1 hold steps both ran for 9s and logged started=2026-09-09T21:58:06Z, on workers forgejo-deploy-lc4vw-qg4kp and forgejo-ci-s66wk-f9n2q. Owner post-cancellation queries excluding Succeeded and Failed pods returned no resources in either pool. Normal completion, startup cancellation and executing-step cancellation now each have cleanup evidence. Exact cleanup latency and continuous absence of post-cancel churn were not measured; other failure tests and full migration acceptance remain pending.

Deliberate shell-failure reporting and cleanup verified: run 206, commit 93fa5899f0eb88dfb703cde349e736ac4a76bfd8, reached the Deliberately fail step in both pools. CI worker forgejo-ci-kx7bt-mbbzk and deployment worker forgejo-deploy-qtnz8-qml4b printed the intended failure message and reported exit code 42. The owner subsequently queried both namespaces excluding Succeeded/Failed pods; neither contained nonterminal pods. This verifies intentional job failure and cleanup, not image-pull/task-fetch failures or controller-restart recovery. CI setup spent about 2m34s pulling the action image, while the deployment job finished in about 25s; retain these observations when evaluating disposable-engine startup costs. Real build/scanner/OIDC/disposable-deployment acceptance and consumer cutover remain pending.

Prepared real-workload acceptance (2026-09-09): workflows PR34 adds a static deploy-k8s reusable route restricted to read-only credential validation or OpenBao-backed dev deployment. Actual PR21 adds the manual build/scan/archive-transfer probe and canary selections under the existing deploy-bundle.yml caller, retaining the OpenBao-bound workflow identity. Default off preserves normal delivery; do not widen the OpenBao role. Merge the workflows PR, manually publish workflow release v1.7.0, then merge Actual and follow its operator runbook: compatibility, credential validation, then disposable dev deployment using the existing signed dual-profile release. No new runner or workload UDS bundle is needed. Local unit/contract and shell checks pass. All hosted checks pass for workflows head 5977bd1143 (runs 207–209) and Actual head 2e0f64f4f1 (runs 123–124). All three live acceptance runs remain pending. These PRs do not transfer production labels or authorize Compose shutdown.

Compatibility passed; dispatch correction required (2026-09-09): Actual run 128 passed build (12m47s) and verify (4m18s), including source/upstream-image scans, Docker build, archive checksum/source-label verification and isolated execution. Post-run zero-pod confirmation is pending. The owner selected validate in run 129, but the normal pipeline.deploy job ran against dev and succeeded in 4m49s. This is not Kubernetes credential-validation acceptance. Forgejo copies caller conditions into reusable jobs and rebinds inputs; the old empty-value fallback incorrectly selected normal deployment. Actual PR22 uses forgejo.event.inputs.runner_test and removes the fallback. Parent reusable calls intentionally have no runner labels; concrete canary jobs already specify deploy-k8s. Merge the correction after checks, then start a new main dispatch with validate. No new workflow-library release or UDS bundle is needed. The owner chose to retain per-job tool installation without adding caching.