Implement build-once artifact publishing and controlled promotion #5
Labels
No labels
app-migration
bug
ci-cd
cluster
Documentation
duplicate
enhancement
gpu
invalid
networking
question
security
Storage
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Homelab/workflows#5
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Build the delivery pipeline before migrating applications. Generic release jobs must support any repository; Zarf/UDS packaging is an optional caller profile. Actual will be the first homelab application consumer.
Acceptance:
Dependencies: #3 for reuse/runner activation, #1 and #4 for scanning. Existing runner may be used before the optional Kubernetes migration in #2. Assistant supplies code and PRs; owner reviews, merges, builds and deploys. No cluster deployment is authorized by creating this issue.
Implementation is proposed in #8, stacked on security foundation #7. Container builds scan the local archive and upload signed scan decisions/SBOM evidence before any registry push; UDS deployment verifies signatures, exact payload hashes, current scans and mitigation expiry.
Validation: 25 unit tests pass, including publication ordering and upload-failure blocking. Real scanner and signature checks passed; a modified signed file was rejected. No application image/bundle was built or deployed.
App-template code is published on Homelab/bundles branch feat/app-bundle-template (555c63a). Its caller contracts and placeholder rejection were validated. That empty repository needs a main branch before its template PR can be opened.
Remaining acceptance after review/merge: configure existing runner labels and scoped credentials, verify cross-repository calls and protected-branch dispatch on this instance, then exercise Actual as the first real image/bundle/deployment consumer and record health/persistence evidence. Caller app tests belong before release in each app workflow. Runner migration remains deferred under #2. Keeping this issue open until live acceptance is recorded.
Checklist review: checked the implemented exact-image scan/SBOM/checksum/provenance and Cosign signing/verification capability, plus deployment authentication/health/retention/rollback documentation. Evidence: #8, pipeline/container.py, pipeline/delivery.py and docs/delivery.md. Real signing tests accepted valid evidence and rejected tampering; unit tests confirm evidence uploads precede registry pushes and failed uploads block publication.
The remaining compound items stay open: application-owned tests must be wired into a real caller; end-to-end build/publish and runtime-secret exclusion require first-app acceptance; protected-ref and credential isolation need live proof; no dev-to-prod acceptance run or Actual migration has occurred.
Current design clarification: dev/prod selection reuses the same container image digest but produces separately signed environment-specific bundles. It does not promote one identical bundle archive across environments, so the exact-artifact promotion item is not checked. Deployment is manual-only. Homelab/bundles#2 is now merged (superseding the earlier empty-main blocker); Homelab/bundles#3 adds ci/deploy labels and dev/prod callers. Production still requires its shared HTTPS gateway/DNS and deliberate legacy-route cutover.
The reusable security caller is now proven in Homelab/bundles, and PR #12 adds the same trusted gate to the workflow library itself. This reduces the remaining delivery work to live release acceptance rather than missing scan implementation.
No additional acceptance box is checked yet: we still need one real application to supply its tests, build and publish an immutable image, prove trusted-ref credential isolation, deploy to dev, and deliberately promote the same digest to production. Actual remains the intended first consumer after the generic validation/rendering gaps in #4 are closed.
The generic source-validation prerequisite is now implemented and released: #13 adds Helm/Kustomize rendering, ShellCheck and Kubernetes schema validation; #15 fixes occurrence-specific IaC mitigation matching in v1.2.1. See the reconciled baseline evidence in #4.
Homelab/bundles#6 is merged with matching v1.2.1
uses/library_refpins in all four security, container-build, bundle-build and deploy callers. Its source-security PR run 16 passed. The owner confirmed required source-security checks are configured on main; workflow-file patterns are intentionally omitted in favor of the branch PR/review/check controls.This clears the previously recorded generic rendering/validation gap. No build, publication, deployment or promotion was exercised by the version update, so the delivery checklist remains 2/7. Next acceptance remains Actual: wire application tests, prove protected-ref/credential isolation, build and publish once, validate dev health/HTTPS/persistence, and verify approved promotion/rollback using the intended immutable image digest.
Actual remains the first app consumer, now in the owner-created Homelab/actual-budget repository. Migration tracker: Homelab/actual-budget#1; implementation: Homelab/actual-budget#2. The owner will stop the TrueNAS instance before RKE2 accesses its retained dataset, then retain the stopped container for rollback until migration/observation is complete.
The PR pins upstream Actual 26.9.0 by amd64 digest, adds retained static-NFS storage prerequisites, single-writer dev manifests, caller validation before packaging and manual shutdown acknowledgement. The initial source scan and app configuration checks passed. No image was rebuilt: this third-party app follows the upstream-image profile of the template.
The delivery acceptance checklist stays open. Production selection is intentionally unavailable in this app PR until exact-bundle promotion is implemented, avoiding a second bundle or second writer. Other remaining live evidence includes scoped credential/protected-ref release, signed bundle build, dev app/persistence checks and approved production cutover.
Progress update — 2026-09-08
Actual Budget has completed the first real application delivery exercise: build, security evidence, signing, publication, deployment to RKE2, and production cutover. The owner confirmed login, intact data, sync, and persistence after restart at https://actual.gitchegumi.com. Homelab/actual-budget#1 and Homelab/platform#1/#2 are closed. The first-consumer acceptance item is now checked. Published artifact: https://git.gitchegumi.com/Homelab/actual-budget/releases/tag/ci-4519315e4e5d761ee917c04ca529de98eeeafb12-270
The v1.3.x fixes made the actual build/publish/deploy path usable, including scan evidence, release downloads, and retaining the native Zarf archive name. Automatic credential support landed in Homelab/workflows#26 (v1.4.0); diagnostic improvements and reusable-workflow OIDC preservation followed in #27/#28 (v1.4.1/v1.4.2). Live credential issuance acceptance remains tracked in Homelab/workflows#19 and Homelab/platform#24.
Remaining here: make environment selection and promotion a complete Actions path for all future apps. The intended interface is choose dev or prod and run the action, with signature/scan checks, credentials, required provisioning, deployment, and rollout verification handled by Actions. Prove that promotion uses the approved artifact without rebuilding; the successful Actual production routing change does not by itself prove that automated promotion contract. Preserve the single-writer constraint for Actual data; a separate dev instance needs separate data.
All workflows should expose meaningful stage names, progress during long operations, elapsed time, actionable redacted failures, and a final result/evidence summary. Manual restart/functional acceptance can remain manual. Keep this issue In Progress until the general promotion and authorization acceptance criteria are demonstrated.
Update — 2026-09-08: build-once promotion is implemented and released in workflows v1.5.0 (workflows#30), with platform#30, bundles#7 and actual-budget#18 merged. The old separately-built-environment-bundles limitation is superseded: one signed UDS archive now contains separate dev/prod packages with identical pinned images. Caller validation precedes packaging; scans, SBOMs, receipt verification and publication remain enforced. The caller-test and build-once acceptance items are checked.
Actual dev was successfully deployed and tested by the owner with disposable data. Production run https://git.gitchegumi.com/Homelab/actual-budget/actions/runs/105/jobs/1/attempt/1 selected the production package from the signed archive, passed verification/rescan and reached Helm, but hit manual-cutover field ownership conflicts. Production stayed healthy. The subsequent run https://git.gitchegumi.com/Homelab/actual-budget/actions/runs/106/jobs/1/attempt/1 stopped at permission 14/20 before deployment. The owner confirms deleting actual-budget-dev after testing; this removed its Role/RoleBinding. Reapply the merged platform RBAC before retrying. Current shared permission validation requires both namespaces even for prod.
Keep In Progress: successful same-release production promotion and live protected-ref/credential-isolation acceptance remain outstanding. Empty dev validates application behavior, not migration of existing production budget data. Preserve production backups for upgrades; do not treat an application rollback as a database rollback.
Same-release dev-to-prod promotion is verified. Run 104 selected dev/actual-budget-dev and run 107 selected prod/actual-budget. Both downloaded ci-a20d05f741c712161c17d2fed8f9312adc2b69bf-422, passed deployment and rollout checks, and revoked OpenBao sessions. The owner confirmed both environments worked and initiated production promotion. Build-once and exact-artifact promotion acceptance are checked.
Follow-up #31 and Homelab/platform#31 remove the dev-RBAC dependency from prod and restore disposable dev namespaces automatically. Keep the outstanding live protected-ref/credential-isolation acceptance open. Namespace bootstrap is implemented in PRs but not released/installed; its delete/recreate acceptance remains pending.
Completed with owner-approved scope on 2026-09-09. Workflows/platform #31 and Actual #19/#20 plus bundles #8 are merged; workflows v1.6.0 is adopted and the owner ran platform bootstrap. Both live namespace and RoleBinding admission denial probes passed.
Evidence: Actual runs 104 and 107 deployed dev/prod from the same signed release without rebuilding. Run 118 exercised production with dev removed: selected prod, passed 12 required and 7 forbidden permission checks, verified signatures/hashes, rescanned, deployed only the production package, passed rollout/HTTPS health and revoked the OpenBao session. Automatic per-run credentials eliminate manual kubeconfig replacement.
Explicitly deferred, NOT passed: remaining live unauthorized repository/ref/audience and credential-expiry cases; comprehensive PR publish/deploy credential-isolation proof; dev namespace delete/recreate acceptance. The owner elected to verify dev recreation naturally on its next use. Existing unchecked compound criteria retain that distinction. These are not blockers for the accepted delivery automation scope. Future apps still require reviewed per-app platform trust/permission onboarding. Closing and moving to Done at the owner’s request.
kiyi referenced this issue2026-09-24 15:11:41 -05:00