Plan the Linux Forgejo runner migration to Kubernetes #2
Labels
No labels
app-migration
bug
ci-cd
cluster
Documentation
duplicate
enhancement
gpu
invalid
networking
question
security
Storage
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Homelab/workflows#2
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
Define how the existing Linux Forgejo Actions runner could move into Kubernetes after the cluster is dependable. The Windows runner must remain on its Windows Hyper-V VM to preserve a true Windows execution environment.
Scope
Acceptance criteria
Dependencies / blockers
Planning can begin after cluster bootstrap; implementation depends on stable networking, storage if required, and a proven Forgejo Actions path.
Planning is now In Progress on the Homelab Roadmap. PR #32 records the read-only inventory, proposed KEDA/on-demand worker model, provisional four CI plus one deploy capacity split, canary concurrency probe and outage/rollback plan. Wiki: https://git.gitchegumi.com/gitchegumi/homelab/wiki/Linux-Runner-Migration.
Verified Forgejo 16.0.3, runner v13.0.0, two persistent matching runner registrations, four Ready RKE2 nodes, and no installed KEDA/runner pods. Local YAML/Bash validation and 16 mocked control-flow cases passed. The new acceptance workflow is manual and uses ci-k8s/deploy-k8s canary labels; it has not been dispatched.
Checked the documented architecture, bootstrap/outage and retained Windows boundary items. Keep this issue open: exact host/backend inventory, runtime credential supply, build isolation, pinned UDS packaging and live scaling/isolation/acceptance remain incomplete. Existing runners, labels, credentials and workloads are unchanged. The owner has been asked for current Linux runner host/startup details and their preferred five-worker interpretation.
The Compose host/configuration inventory is now recorded in PR #32 and the wiki. Both existing Homelab/global registrations are connections from the same Media runner process, with capacity one and shared DinD. Per the owner, Sager, OptiPlex and Media are all eligible pod hosts.
Platform implementation issue: Homelab/platform#32. Canary packages/runbook: Homelab/platform#33. Both staged UDS bundles built locally; Helm lint/render, repository checks and 15 custom-resource schema checks passed. The first design uses separate persistent pool identities in ESO runtime Secrets and private per-job DinD; it does not claim server-enforced single-use registration. Dedicated credentials and live scaling/build acceptance remain outstanding. Existing Compose services, labels and workloads are unchanged.
Planning PR #32 is merged at
e1d7f23d81, including the current Compose inventory and manual concurrency probe. Marked the documented inventory criterion complete. Infrastructure PR Homelab/platform#33 remains under review; its review fixes now have a passing hosted runner-validation job. Runtime credentials, live canary acceptance and deliberate Compose handoff remain tracked in Homelab/platform#32. No live runner change is implied by the planning merge.Live progress on 2026-09-09: the operator confirmed successful keda-operator rollout, with keda-admission-webhooks, keda-operator and keda-operator-metrics-apiserver each 1/1 Ready and Available. This clears the controller deployment blocker encountered during platform’s initial 0.1.0 attempt; the source fix is platform PR #34. Detailed evidence and remaining work are tracked in Homelab/platform#32 and the updated https://git.gitchegumi.com/gitchegumi/homelab/wiki/Linux-Runner-Migration .
Credentials and OpenBao setup instructions have been provided, but completion is not yet confirmed. Queue-reader identity/permissions, worker deployment, canary job execution, concurrency/failure tests and delivery compatibility remain pending. Keep this planning/acceptance issue open; controller readiness alone does not complete migration. Legacy Compose and Windows Hyper-V remain the established execution paths pending acceptance and deliberate consumer handoff.
First live canary passed: https://git.gitchegumi.com/Homelab/workflows/actions/runs/196/jobs/0/attempt/1 at
e1d7f23d81shows all six CI jobs and deploy-probe successful. Browser observation showed four CI jobs plus deploy-probe running with two CI jobs waiting, followed by queue advancement. Owner output established CI pod placement on Sager, OptiPlex and Media and, after completion, zero nonterminal pods in both worker namespaces.Credential setup, queue access and ESO reconciliation are verified; polling uses a dedicated gitchegumi organization-Read token because the installed organization queue endpoint requires an organization owner. Codex publishing remains kiyi.
Evidence and remaining acceptance are in Homelab/platform#32 and the migration/operations wiki. Continuous total pod-count evidence, cancellation/failure cases, real build/scanner/OIDC/deployment compatibility and global/VOA handoff are still pending. Keep this issue and the migration open; legacy Compose remains preserved.
Next compatibility work is prepared in #34 and Homelab/actual-budget#21. The owner should merge workflows#34, publish workflow release v1.7.0, then merge Actual#21 and run its three manual stages: build/scan/artifact transfer, read-only credential validation, and an existing signed release to disposable dev. The credential stages retain the OpenBao-bound deploy-bundle.yml caller; the static canary reusable route rejects production deployment.
Local tests pass and hosted checks are queued/running. This is prepared source, not live compatibility acceptance. No new UDS bundle or runner deployment is required. Homelab/platform#32 and the Linux Runner Migration wiki record the details; migration stays In Progress and the legacy runner remains in place.
Migration completed by owner confirmation on 2026-09-10. Global Kubernetes ci/deploy pools now replace the legacy Linux runner, with four CI slots plus one deployment slot across Sager, Optiplex and Media. Worker bundle 0.1.2 uses instance-wide KEDA polling; the final HTTP 403 blocker was the separate queue-token Admin Read permission. OpenBao/ESO provides the two global registrations and polling token. Windows Hyper-V remains unchanged.
Evidence: workflows run 196 passed concurrency/queue-drain; 197 attempts 1 and 2 verified cancellation cleanup; 206 verified deliberate failure cleanup. Actual Budget run 128 passed build/scanning/archive transfer, 137 passed credential validation, and 144 deployed signed dev release successfully; the owner confirmed dev login and zero nonterminal deployment pods. Final cutover completion is owner-reported, not an independent cluster inspection by Codex.
The owner accepts addressing overlooked VOA/other consumers later. Unmeasured continuous pod-count timelines, controller-restart recovery and exhaustive failure injection are not being claimed as tested or retained as migration blockers. Deleted registrations require replacement credentials for rollback. Historical comments/checklists are superseded by this acceptance decision. Closing the migration issue as requested. Current record: https://git.gitchegumi.com/gitchegumi/homelab/wiki/Linux-Runner-Migration