This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

GitchAlerts migration

Production deployed — 2026-09-16

The owner confirmed GitchAlerts is deployed to production. Production run 64 selected prod / namespace gitchalerts and passed the gateway check, signed artifact verification, rescan (0 blocking, 2 mitigated, 0 scanner errors), Redis/application rollout and HTTPS application health. The managed OpenBao session was revoked afterward.

GitchAlerts 0.12.4 / bundle 0.1.6; release ci-100e14d72cdc9da1b80c067f507d279f6e9e0c3a-1789, also deployed successfully to dev in run 63. Archive SHA256: 2a0ae31cb20008800666a9f0bb370321b333b4a5500122b54c61dc8f1b82a2e5.

Production uses external PostgreSQL, S3 assets and OpenBao runtime values at secret/apps/gitchalerts/prod/runtime. Hostnames are alerts.gitchegumi.com and alerts-webhook.gitchegumi.com. The configured Keycloak groups are /GitchAlerts/Users and /GitchAlerts/Admins.

This supersedes earlier preparation-only status. Detailed provider/webhook, group-revocation, asset-write, restart/restore and certificate-renewal tests are not asserted by the deployment result. Dev deletion and TrueNAS retirement are not claimed. The exact applied gateway/certificate YAML and reusable first-production routing automation remain platform follow-up; successful app deployment alone does not prove those changes were committed to IaC.

Migration record.

Earlier dated sections below are historical preparation records; their pending-production statements are superseded by the status above.

Current status (2026-09-15)

The owner reports dev deployed successfully and Keycloak login works after joining /GitchAlerts/Users. Earlier run 49 reached the Helm upgrade; subsequent owner logs identified a malformed DATABASE_URL. Full provider/storage acceptance and production cutover remain pending.

Bundle PR #11 is merged with bundle 0.1.5 consuming released GitchAlerts 0.12.4 (source 995d481f6f1d6d275d7cb74330a01e70d6537601, web 0.3.4, ingress/worker 0.2.3). The release includes the upstream group implementation, resolving the prior dependency. Both profiles configure the full-path groups claim, Users/Admins admission, invite-free onboarding and scoped Forgejo update-check HTTPS access. No OpenBao additions are required for these settings. Local schema/startup checks and published image metadata/entrypoints passed; compiled web output includes the group configuration. Hosted CI, signed build and live role/update-check acceptance remain separate gates. No cluster deployment was performed for this update.

Group membership remains operator-managed: /GitchAlerts/Users is for workspace access, /GitchAlerts/Admins for instance administration. The owner must verify Users-only, Admins-only, neither-group and revoked-admin behavior after deploying a compatible release. Production S3 wiring is prepared separately below and is not deployed.

Prepared storage profiles (2026-09-15)

Bundle PR #12, commit 256b72bfe3508e5d88a4f44a6880d5b14ce7ba54, prepares bundle 0.1.6 with the same GitchAlerts 0.12.4 images. Dev retains its local uploads PVC; prod uses the existing S3 bucket with six S3 settings mapped from secret/apps/gitchalerts/prod/runtime to web only. The prepared egress permits the previously supplied 10.0.0.116:30320; the current private endpoint and runtime path still need owner confirmation. No credentials were read.

Both profiles are in one archive: after merge, build once, test the dev environment, then select prod using the same signed release tag. This does not alter the owner’s current 0.1.5 Keycloak groups build. The profile runbook records selection, initial routing cutover, verification and rollback. Local validation passed for 27 resources, 28 negative cases and startup tests for both profiles; hosted CI, a signed 0.1.6 build and live S3 acceptance remain pending.

This supersedes the older proposed production NFS uploads prerequisite. Production objects stay in S3; no bucket creation, copying or deletion is planned. The owner already copied the production database and accepts the gap until cutover, so another import is not required unless that decision changes. Stop TrueNAS before starting Kubernetes prod, coordinate both production routes, and preserve dev until production acceptance. No live cutover or resource deletion was performed.

Current status (2026-09-14)

The owner confirmed that dev SecretStore openbao and ExternalSecret gitchalerts-runtime both reached Ready after platform run 544. Application acceptance remains pending.

Bundle PR #10 prepares bundle/package 0.1.4 for GitchAlerts 0.12.3. Web uses the upstream migration entrypoint, initializes a genuinely empty dev database automatically, and releases ingress/worker only after success. All services perform the upstream schema readiness check. The old manual dev baseline is superseded once this update is merged and built. Existing or inconsistent schemas must follow the upstream migration/recovery guide; do not reset an existing database.

Local validation passed: 28 resources, 14 negative cases, startup success/failure/restart for both profiles, published-image metadata/entrypoints, and the packaged migration acceptance suite against the published web image with disposable PostgreSQL 18 and Redis. These are local results, not cluster acceptance. Hosted CI, merge, a new signed build and dev deployment/acceptance are still required. No UDS archive was built or cluster changes performed for this update.

Production has a separate storage prerequisite: the owner confirmed existing assets use S3, whereas the bundle still configures local storage and a production uploads PVC. Correct and validate that contract before production cutover. Dev remains an empty test environment.

The dated entries below describe historical states.

Current status (2026-09-13)

Run 37 verified signed bundle 0.1.2 and passed the rescan gate, then timed out during the dev Helm upgrade. Owner-provided Kubernetes events identify two blockers: UDS denied pods with proxy.istio.io/config, and ExternalSecret gitchalerts-runtime could not find Kubernetes SecretStore openbao. Redis is running and the 5Gi uploads PVC is bound. No token-authentication error was shown.

Platform PR #42 prepares automatic runtime ESO/OpenBao access for each opted-in app; GitchAlerts PR #8 removes the forbidden annotation in both profiles and prepares bundle 0.1.3. The owner confirmed the existing secret KV v2 mount with secret/apps/gitchalerts/dev/runtime and a separate prod path; these are paths, not OpenBao namespaces. Values remain owner-managed. The generated runtime identities are separate from deployment and registry policies.

Apply the platform runbook’s one-time permission upgrade (including interactive login inside the OpenBao pod), reconcile, populate the runtime keys privately and verify ESO readiness. Then build/deploy the corrected signed app bundle to dev. Local platform, pinned real OpenBao, ESO schema and bundle regression/startup tests pass; live runtime authentication and application acceptance remain pending. Preserve PVCs/database.

Current status (2026-09-12)

Platform run 288 completed reconciliation with both hl-21-dev and hl-21-prod ready. GitchAlerts run 24 passed all 113 required and 11 forbidden dev permission probes, signed artifact verification and the scan gate (zero blocking, two mitigated findings). All four images reached the cluster registry. Helm installation then failed because Istio rejected gitchalerts-web at spec.http[1].timeout: 0s. Partial dev resources may exist; application readiness and production cutover are not verified.

PR #6 prepares bundle 0.1.2: omit the web timeout in both profiles, use Istio’s disabled default for SSE, and correct the local validator with regression cases. Local checks passed for 28 resources, 10 negative cases and startup behavior. After merge, build a new signed release from updated main and deploy it to dev; the old archive still contains the invalid route. Inspect partial resources and preserve PVCs/database before retrying. See the PR runbook for commands.

Earlier entries below are historical preparation/build evidence.

Build publication and deployment onboarding (2026-09-11)

Run 18 built and published bundle 0.1.1 after the owner increased the Forgejo NPM timeout. The signed release contains the 483 MiB archive and signed receipt. Run 19 stopped before OpenBao authentication because workflows v1.6.0 lacked a GitchAlerts permission contract. It deployed nothing.

The owner requires automatic onboarding rather than manual per-app contracts and roles. A shared platform reconciler, managed workflow mode, template opt-in and GitchAlerts caller adoption are prepared for review. See Application Delivery Automation. The existing signed release can be used after the platform installation and workflow publication; no identity-only bundle rebuild is required. Cluster deployment and acceptance are still pending.

Status recorded 2026-09-11: bundle prepared and validated; no cluster deployment or production cutover performed.

Tracking: platform#28, bundle implementation and dev acceptance, initial bundle PR, and release 0.12.2 update.

Confirmed inputs

The owner confirmed production web https://alerts.gitchegumi.com, production callbacks https://alerts-webhook.gitchegumi.com, shared PostgreSQL at 10.0.0.116:5432, and an empty dev environment. These are supplied inputs, not evidence of cluster connectivity or completed database migration.

The bundle prepares alerts.dev.gitchegumi.com and alerts-webhook.dev.gitchegumi.com for dev, separate dev/prod namespaces and secret paths, native UDS Keycloak sign-in, an app-specific ephemeral Redis, fresh dev uploads, and a production claim for existing TrueNAS uploads. Production storage uses a proposed platform-owned, Retain, prebound NFS PV; its actual export and permissions remain to be inventoried.

Prepared and tested

The initial merged bundle pinned release 0.12.0 images (web 0.3.0, ingress/worker 0.2.0) and Redis by amd64 digest. One Recreate app pod groups web, ingress and worker; web runs Prisma migrations before the other processes start. Dev must use its own database, role, keys and test integrations because worker startup can renew provider subscriptions.

Initial local validation passed for all 28 Kubernetes/custom resources, three negative validation cases, startup success/failure behavior, and compatibility with shared workflows v1.6.0. Hosted bundle validation and required source-security checks passed for initial commit 3a0f591f4d1d6c7d41a823314110f2a9f2c59710.

Release 0.12.2 update

The merged update PR provides bundle/package 0.1.1 with release 0.12.2 (web 0.3.2, ingress/worker 0.2.1), pinning all three published amd64 image digests. Redis remains unchanged. Release 0.12.2 removes package managers from the runtime images, so startup now invokes Prisma, Next.js and tsx through Node directly, preserving the single web migrator and the ingress/worker readiness gate. The source release adds no application schema, migration or environment-contract changes relative to 0.12.0.

Hosted bundle validation and required source-security checks passed for commit 46a0c725d3da8cd9ffe6266f903df6cfc75dbde3; the update merged into main as 2150f1d6f925222314cae84ec3dc01e96c367cbf. Local checks passed for all 28 resources and four negative cases, plus startup success/failure for both environment profiles. The actual published images were also tested on an isolated internal Docker network against disposable PostgreSQL and Redis: fresh-database initialization and migration status succeeded, web and ingress returned HTTP 200, and worker subscribed to Redis. Test containers used the manifest startup commands and runtime security settings. Temporary test resources were removed. This verifies local image startup; it does not establish cluster, native SSO or provider-integration acceptance. No UDS bundle archive was built or deployed, and no production system or shared PostgreSQL instance was changed.

A fresh dev database needs an explicit baseline before the first normal startup because historical migrations assume an earlier schema. The updated runbook provides individual commands to confirm the dev database is empty, create the current schema and record the historical migrations. This is exclusively for the new empty dev database; existing databases continue through normal migrations. Startup never automatically baselines an existing database.

The repository runbook contains individual fresh-checkout, validation, UDS/Zarf build and deployment commands, the secret contract, dev acceptance, production sequencing and rollback. Pull the merged main branch before following these commands.

Redis image scan review

The owner-triggered build run 11 failed before bundle creation/publication on two Redis image-configuration findings (DS-0002 and DS-0025). The mitigation PR records exact fingerprints, owner gitchegumi, an expiry of 2026-10-12 00:00 UTC and the verified evidence: configured non-root runtime controls and an empty APK cache in the pinned Redis image. Local validation enforces those controls; re-evaluation of the retained scan decision produces zero blockers and two mitigated findings. This does not replace a fresh image scan. After merge, a new build must run from updated main; rerunning the old commit retains its empty policy. The owner will handle the non-blocking qs dependencies separately. Successful bundle publication, cluster deployment and acceptance remain unverified.

Remaining prerequisites and cutover

• Provision separate databases/roles and verify PostgreSQL TLS requirements and reachability. With the merged 0.12.3 update, use the supported automatic initializer for the empty dev database; no manual baseline is required.
• Establish each namespace's scoped OpenBao/ESO integration, populate runtime secrets privately, and configure intended native SSO users.
• Confirm dev DNS/TLS and storage; establish app-specific deployment permissions before using Actions deployment.
• Inventory the running TrueNAS version, database location, uploads export/capacity/UID/GID/ACLs and backup/restore evidence. Preserve both encryption keys with adopted data.
• Build and scan the bundle, deploy only dev, and record login, uploads, SSE/overlay, provider callbacks, worker renewal, restarts and restore results.

Production follows the existing database-first migration plan: repoint and validate the old app against shared PostgreSQL at its current version first. Before Kubernetes uses production data, stop the old web, ingress and worker. Coordinate both production hostnames with the platform-owned legacy passthrough and native gateway configuration. Preserve the old deployment and backups through an agreed observation period.

Rollback stops Kubernetes writers before restarting TrueNAS and restores both routes and the verified DNS path. A schema change may require a coordinated database/uploads restore with matching keys; do not delete PVCs or restore over active writers. The actual cutover, rollback test and eventual TrueNAS retirement remain pending.