Add a shared workflow release channel with centralized upgrades and rollback

kiyi commented

2026-10-13 09:29:58 -05:00

Member

Priority: highest — owner direction, 2026-10-13. Complete this shared-channel foundation before continuing the bundle-by-bundle workflow upgrades.

Problem

Application repositories currently pin Homelab/workflows to numbered releases in both uses and library_ref. Each compatible workflow fix therefore requires updating every bundle repository. The owner requested tracking a shared channel so bundles can adopt the latest approved workflow centrally and roll back centrally when needed.

Proposed direction

Introduce a protected release channel such as stable-v1, promoted only to reviewed, published, tested releases. Keep numbered vMAJOR.MINOR.PATCH release tags immutable. Consumers migrate to the channel once; subsequent compatible promotions and workflow rollbacks happen in Homelab/workflows. Breaking caller or policy changes require an explicit migration, potentially to stable-v2.

Required operator procedure

Provide a protected, manual workflow in Homelab/workflows that takes one published vX.Y.Z release and promotes it to stable-v1. The operator updates this shared channel once; compatible consumers adopt it through a one-time migration and need no per-bundle version edits for later promotions. Use the same audited manual procedure to roll back to an earlier compatible published release. Keep numbered release tags immutable. A Git tag is advanced to the selected release/snapshot; it is not a branch merge. Preserve exact run provenance and safe behavior if the channel moves while a run is queued, executing or rerun.

Acceptance criteria

Define the channel, protection, promotion, and audited rollback contract in the owning repository. Preserve immutable numbered releases and allow individual consumers to retain exact-version pins.
Extend workflow and consumer validation beyond the current exact-semver-only contract, updating uses and library_ref handling together.
Bind expanded workflow YAML and checked-out library scripts/tools to the same exact reviewed release for each run. Independently resolving a moving ref twice is insufficient; cover channel changes while runs are queued or executing, and rerun behavior. Record exact provenance in evidence.
Add reviewed, operator-dispatched promotion and rollback support through the owning Forgejo Actions pipeline; demonstrate centralized upgrade and rollback with a disposable consumer before broad adoption.
Prepare the one-time bundle-template and existing-consumer migration. Preserve workflow names, job IDs/display names, required status checks, scanning, signing, and deployment credential boundaries. Coordinate mitigation-policy schema compatibility.
Update release/adoption runbooks and instance Documentation with the verified procedure and distinguish prepared, tested, and adopted states.

Scope

This issue tracks implementation; no channel or consumer migration is complete yet. Workflow rollback changes future workflow executions. Application rollback continues to use the existing exact signed-release deployment procedure and is not an automatic live cutover.

**Priority: highest — owner direction, 2026-10-13.** Complete this shared-channel foundation before continuing the bundle-by-bundle workflow upgrades. ## Problem Application repositories currently pin `Homelab/workflows` to numbered releases in both `uses` and `library_ref`. Each compatible workflow fix therefore requires updating every bundle repository. The owner requested tracking a shared channel so bundles can adopt the latest approved workflow centrally and roll back centrally when needed. ## Proposed direction Introduce a protected release channel such as `stable-v1`, promoted only to reviewed, published, tested releases. Keep numbered `vMAJOR.MINOR.PATCH` release tags immutable. Consumers migrate to the channel once; subsequent compatible promotions and workflow rollbacks happen in `Homelab/workflows`. Breaking caller or policy changes require an explicit migration, potentially to `stable-v2`. ## Required operator procedure Provide a protected, manual workflow in `Homelab/workflows` that takes one published `vX.Y.Z` release and promotes it to `stable-v1`. The operator updates this shared channel once; compatible consumers adopt it through a one-time migration and need no per-bundle version edits for later promotions. Use the same audited manual procedure to roll back to an earlier compatible published release. Keep numbered release tags immutable. A Git tag is advanced to the selected release/snapshot; it is not a branch merge. Preserve exact run provenance and safe behavior if the channel moves while a run is queued, executing or rerun. ## Acceptance criteria - [ ] Define the channel, protection, promotion, and audited rollback contract in the owning repository. Preserve immutable numbered releases and allow individual consumers to retain exact-version pins. - [ ] Extend workflow and consumer validation beyond the current exact-semver-only contract, updating `uses` and `library_ref` handling together. - [ ] Bind expanded workflow YAML and checked-out library scripts/tools to the same exact reviewed release for each run. Independently resolving a moving ref twice is insufficient; cover channel changes while runs are queued or executing, and rerun behavior. Record exact provenance in evidence. - [ ] Add reviewed, operator-dispatched promotion and rollback support through the owning Forgejo Actions pipeline; demonstrate centralized upgrade and rollback with a disposable consumer before broad adoption. - [ ] Prepare the one-time bundle-template and existing-consumer migration. Preserve workflow names, job IDs/display names, required status checks, scanning, signing, and deployment credential boundaries. Coordinate mitigation-policy schema compatibility. - [ ] Update release/adoption runbooks and instance Documentation with the verified procedure and distinguish prepared, tested, and adopted states. ## Scope This issue tracks implementation; no channel or consumer migration is complete yet. Workflow rollback changes future workflow executions. Application rollback continues to use the existing exact signed-release deployment procedure and is not an automatic live cutover.

kiyi added this to the Homelab Roadmap project

2026-10-13 09:29:59 -05:00

kiyi added the

ci-cd

label

2026-10-13 13:24:18 -05:00

kiyi referenced this issue from a commit

2026-10-13 14:10:50 -05:00

Add protected manual stable-v1 workflow promotion and rollback

kiyi referenced this issue

2026-10-13 14:16:11 -05:00

Add manual stable-v1 workflow promotion and audited rollback #94

kiyi commented

2026-10-13 14:17:03 -05:00

Author

Member

Source implementation is prepared in #94 at e600b0c3b9: protected manual version selection, frozen reusable adapters, exact-commit checks, fresh canary, atomic channel/audit updates and audited rollback, plus one-time caller migration tooling. Existing scan/workflow/job names are preserved.

Local validation: 365 tests passed with one native-Trivy integration skip; final 11 channel tests and 3 migration tests passed, including deterministic snapshots, native Git atomic promotion/rollback and generated shell/provenance checks.

This issue remains open and highest priority/In Progress. No live channel tag, promotion or consumer migration has occurred. Hosted upgrade/rollback, queued/rerun acceptance and template/consumer adoption still need evidence before completion.

Source implementation is prepared in #94 at e600b0c3b94b03191ac154fe4aaf7641701fc78d: protected manual version selection, frozen reusable adapters, exact-commit checks, fresh canary, atomic channel/audit updates and audited rollback, plus one-time caller migration tooling. Existing scan/workflow/job names are preserved. Local validation: 365 tests passed with one native-Trivy integration skip; final 11 channel tests and 3 migration tests passed, including deterministic snapshots, native Git atomic promotion/rollback and generated shell/provenance checks. This issue remains open and highest priority/In Progress. No live channel tag, promotion or consumer migration has occurred. Hosted upgrade/rollback, queued/rerun acceptance and template/consumer adoption still need evidence before completion.

kiyi referenced this issue

2026-10-13 18:08:37 -05:00

Report safe channel Git rejections and retain failure evidence #95

kiyi commented

2026-10-13 18:09:07 -05:00

Author

Member

Hosted first-use evidence: run 618 executed the published v1.36.0/main implementation and failed at the immutable snapshot Git push, before canary or stable-v1 publication. The original log withheld the remote rejection reason, so authentication/tag protection must not be assumed as a proven cause.

PR #95 at 30a67f5360 adds fixed safe rejection categories and an early public failure artifact, preserving an existing audit if a later verification fails. Full source suite: 369 tests passed with one local Trivy integration skip; 14 channel cases, scaffold/mocks, Ruff and staged Gitleaks passed. No tag-rule changes or new promotion dispatch occurred.

Next operator step after review/merge and passing protected-main checks: start a fresh Promote workflow channel dispatch from main, selecting the intended published release (v1.36.0 is available) with empty rollback_reason. Do not use Rerun. Review the actual rejection category/server logs before changing authorization. Empty protected-tag allowlists deny all writes; do not assume the automatic token is exempt or that Git author metadata identifies its principal. Hosted promotion/rollback and consumer acceptance remain pending; this issue stays open.

Hosted first-use evidence: run 618 executed the published v1.36.0/main implementation and failed at the immutable snapshot Git push, before canary or stable-v1 publication. The original log withheld the remote rejection reason, so authentication/tag protection must not be assumed as a proven cause. PR #95 at 30a67f536046959cd46c8a1de9ddb375b8edd5ac adds fixed safe rejection categories and an early public failure artifact, preserving an existing audit if a later verification fails. Full source suite: 369 tests passed with one local Trivy integration skip; 14 channel cases, scaffold/mocks, Ruff and staged Gitleaks passed. No tag-rule changes or new promotion dispatch occurred. Next operator step after review/merge and passing protected-main checks: start a fresh Promote workflow channel dispatch from main, selecting the intended published release (v1.36.0 is available) with empty rollback_reason. Do not use Rerun. Review the actual rejection category/server logs before changing authorization. Empty protected-tag allowlists deny all writes; do not assume the automatic token is exempt or that Git author metadata identifies its principal. Hosted promotion/rollback and consumer acceptance remain pending; this issue stays open.

kiyi referenced this issue

2026-10-13 18:45:48 -05:00

Authorize protected release and channel writes with native OIDC #96

kiyi commented

2026-10-13 18:48:14 -05:00

Author

Member

Run 624 attempt 2 confirms the authorization blocker: release creation returned HTTP 422 because v1.36.1 is protected. The saved Bots/Owners allowlists do not identify the automatic Actions token as an authorized writer.

PR #96 at 080ffdcfee prepares separate native Forgejo Authorized Integrations for release and channel publication. Exact workflow/main/manual-dispatch restrictions, owning-repository write scope, public audience variables, first-attempt/actor/current-source guards and short-lived JWT renewal preserve the existing tag protections and status names. No PAT or new CI secret is needed. The runbook includes the exact one-time integration and tag-owner checks.

Validation: 378 source tests passed with one existing local Trivy skip; scaffold/mocked scanner cases, focused authorization tests, native Git URL matching, Ruff and staged Gitleaks passed. No integration, tag protection, numbered release or stable channel was changed. Operator-approved account setup, hosted publication/promotion/rollback/queued-run acceptance and consumer adoption remain pending. This issue stays open and In Progress. Documentation PR #42 is updated with the confirmed blocker and prepared procedure.

Run 624 attempt 2 confirms the authorization blocker: release creation returned HTTP 422 because v1.36.1 is protected. The saved Bots/Owners allowlists do not identify the automatic Actions token as an authorized writer. PR #96 at 080ffdcfee6bd6b11313d73edba238bf25a56c63 prepares separate native Forgejo Authorized Integrations for release and channel publication. Exact workflow/main/manual-dispatch restrictions, owning-repository write scope, public audience variables, first-attempt/actor/current-source guards and short-lived JWT renewal preserve the existing tag protections and status names. No PAT or new CI secret is needed. The runbook includes the exact one-time integration and tag-owner checks. Validation: 378 source tests passed with one existing local Trivy skip; scaffold/mocked scanner cases, focused authorization tests, native Git URL matching, Ruff and staged Gitleaks passed. No integration, tag protection, numbered release or stable channel was changed. Operator-approved account setup, hosted publication/promotion/rollback/queued-run acceptance and consumer adoption remain pending. This issue stays open and In Progress. [Documentation PR #42](https://git.gitchegumi.com/Homelab/Documentation/pulls/42) is updated with the confirmed blocker and prepared procedure.

kiyi commented

2026-10-13 19:08:13 -05:00

Author

Member

Configuration progress, 2026-10-13: after the owner's explicit approval, workflow-release-v1 and workflow-channel-v1 are saved under kiyi. Saved restrictions match the reviewed exact workflow/main/manual-dispatch contract and write:repository for Homelab/workflows only. The owner-selected Bots Code/Releases/Actions Write permissions were verified; gitchegumi remains the human dispatcher through DEPLOY_ACTORS.

Setup receipt and public audience values are posted on PR #96. The two repository audience variables remain an owner step because Settings are not exposed to kiyi. Documentation PR #42 now records configured grants separately from pending source merge, variable wiring and hosted protected-write/channel acceptance; its local check/build and staged Gitleaks passed. No publisher dispatch, release, tag move or consumer migration occurred. This issue remains open/In Progress.

Configuration progress, 2026-10-13: after the owner's explicit approval, workflow-release-v1 and workflow-channel-v1 are saved under kiyi. Saved restrictions match the reviewed exact workflow/main/manual-dispatch contract and write:repository for Homelab/workflows only. The owner-selected Bots Code/Releases/Actions Write permissions were verified; gitchegumi remains the human dispatcher through DEPLOY_ACTORS. [Setup receipt and public audience values](https://git.gitchegumi.com/Homelab/workflows/pulls/96#issuecomment-6162) are posted on PR #96. The two repository audience variables remain an owner step because Settings are not exposed to kiyi. [Documentation PR #42](https://git.gitchegumi.com/Homelab/Documentation/pulls/42) now records configured grants separately from pending source merge, variable wiring and hosted protected-write/channel acceptance; its local check/build and staged Gitleaks passed. No publisher dispatch, release, tag move or consumer migration occurred. This issue remains open/In Progress.

kiyi commented

2026-10-13 19:34:47 -05:00

Author

Member

Run 637 attempt 1 stopped at passing_checks(main_sha) with Released-commit check is not successful: Library checks / tests. It was dispatched on merged main 32765cfec550baa9114e7ee278846568cee5cc05 while its Library checks were still running. This stopped before token acquisition, canary publication or channel advancement; it does not establish a remaining protected-tag authorization failure.

The main check runs have since passed: Library checks 636 and Repository security 635. Next operator step: start a fresh Promote workflow channel dispatch from main, selecting v1.36.0 and leaving rollback reason empty. Do not rerun attempt 1: the protected publisher requires a fresh first attempt. Hosted protected-write/canary/channel acceptance remains pending and this issue stays open. Terraform PR #3 is refreshed to published v1.36.0 independently of the pending stable-v1 channel.

Run [637 attempt 1](https://git.gitchegumi.com/Homelab/workflows/actions/runs/637/jobs/0/attempt/1) stopped at `passing_checks(main_sha)` with `Released-commit check is not successful: Library checks / tests`. It was dispatched on merged main `32765cfec550baa9114e7ee278846568cee5cc05` while its Library checks were still running. This stopped before token acquisition, canary publication or channel advancement; it does not establish a remaining protected-tag authorization failure. The main check runs have since passed: [Library checks 636](https://git.gitchegumi.com/Homelab/workflows/actions/runs/636) and [Repository security 635](https://git.gitchegumi.com/Homelab/workflows/actions/runs/635). Next operator step: start a fresh **Promote workflow channel** dispatch from `main`, selecting `v1.36.0` and leaving rollback reason empty. Do not rerun attempt 1: the protected publisher requires a fresh first attempt. Hosted protected-write/canary/channel acceptance remains pending and this issue stays open. [Terraform PR #3](https://git.gitchegumi.com/Homelab/terraform-modules/pulls/3) is refreshed to published v1.36.0 independently of the pending stable-v1 channel.

kiyi referenced this issue

2026-10-15 13:49:42 -05:00

Allow the reviewed Terraform runner correction in stable-v1 #104

kiyi referenced this issue

2026-10-16 19:21:18 -05:00

Allow channel CI reruns to adopt promoted workflow fixes #112

kiyi referenced this issue

2026-10-16 20:32:56 -05:00

Automatically release and promote workflows after merges to main #113