UDS Core Prerequisites and Readiness

This page preserves the completed prerequisite gate for historical issue #9 and deployment record issue #10. Core deployment and gateways are established in the subsequent build record. Ongoing shared-platform work is tracked in Homelab/platform and Homelab Roadmap. These readiness observations are historical, not a fresh health audit.

Environment-specific domains, IP addresses, hostnames, bucket names, access keys, passwords, tokens, and certificate material must remain placeholders in this reusable documentation.

Selected compatibility baseline

Component	Selected version	Status
RKE2 / Kubernetes	Kubernetes 1.35 via RKE2	Ready
UDS Core	1.11.1	Deployment recorded complete; capture installed flavor before packaging/adoption
UDS CLI	0.35.1	Ready
UDS-bundled kubectl	1.36.3	Ready; communicates with the Kubernetes 1.35 API server

Pin these versions in the deployment record and re-check official compatibility before future upgrades.

Capacity and cluster health

• Four nodes report Ready: one control-plane/etcd node and three workers.
• The API server verbose readiness check passes.
• No unexpected pods are Pending, CrashLoopBackOff, Error, or otherwise unready.
• No PVCs are unbound.
• Worker CPU and memory satisfy the agreed UDS Core baseline.
• The control-plane node has 94 GiB available and 58% disk use after cleanup; DiskPressure is false.
• The storage backend has the required capacity and health, and the CSI dataset has no quota.

GPU capacity is not a UDS Core prerequisite. Optional GPU enablement remains separate from this readiness gate.

Networking and ingress

• The RKE2 CNI is healthy across the cluster.
• MetalLB 0.16.1 is installed in Layer 2 mode.
• Disposable LoadBalancer testing passed and was cleaned up.
• The pool can allocate the tenant and admin Istio gateway addresses.

Record deployment-specific values only in private runtime configuration:

Purpose	Placeholder
Tenant gateway address	TENANT_GATEWAY_ADDRESS
Admin gateway address	ADMIN_GATEWAY_ADDRESS
Tenant domain	TENANT_DOMAIN
Admin domain	ADMIN_DOMAIN

UDS Core supplies Istio and separate tenant and admin ingress gateways. Do not install another ingress controller solely for UDS Core.

DNS and TLS

• Authoritative DNS is hosted externally and the tenant and admin domain hierarchy is selected.
• Tenant and admin wildcard DNS records are created or staged.
• Separate tenant and admin certificates cover each apex and wildcard name.
• Both certificates are valid public-CA full chains and their private keys were verified.
• The four certificate/key files have been Base64 encoded outside the repository.

UDS certificate refresh — transitional bootstrap procedure

The following export/file procedure describes existing bootstrap handling. The selected target is OpenBao for runtime secret values, ESO for scoped delivery, and cert-manager for certificate issuance/renewal. Automation is not yet confirmed deployed; preserve the working tenant/admin certificates until a tested ownership transition exists. Do not point multiple writers at the same TLS Secret. Valid dev TLS is separately reported complete by platform#1.

Do not persist these local exports as the routine deployment source. Protect temporary material and independent recovery copies under Security and Secrets.

Renewal by Nginx Proxy Manager or any other certificate source does not update the certificate files exported for UDS. After every renewal:

1. Export the renewed full chain and matching private key for both tenant and admin domains.
2. Validate each certificate date, SAN coverage, full chain, and certificate/private-key match.
3. Replace the corresponding files under ~/.config/uds/secrets/ and preserve restrictive permissions.
4. Recreate the Base64 values without line wrapping.
5. Update the private UDS runtime configuration or Kubernetes secrets that consume those values.
6. Restart or roll the affected ingress workloads if the deployment does not reload secrets automatically.
7. Verify the certificates presented by representative tenant and admin endpoints.

Never place real domains or certificate material in this wiki or the repository.

Persistent storage

• TrueNAS CSI NFS and iSCSI provisioning passed disposable tests.
• iSCSI attach, mount, write, read, reattach, and cleanup passed.
• truenas-iscsi is the default expandable StorageClass.
• truenas-nfs is available when ReadWriteMany storage is explicitly required.
• Disposable test PVCs and workloads were removed.

The default StorageClass uses a Delete reclaim policy. Deleting a PVC can delete its backing storage, so application backups and recovery procedures are required.

External object storage

An S3-compatible AIStor service runs on storage external to the Kubernetes cluster. UDS runtime configuration must use placeholders:

Purpose	Placeholder
S3 endpoint	S3_ENDPOINT
Loki chunks bucket	LOKI_CHUNKS_BUCKET
Loki admin bucket	LOKI_ADMIN_BUCKET
Velero backup bucket	VELERO_BUCKET

• Loki and Velero use separate least-privilege access keys.
• Read, write, and delete operations passed for each authorized bucket.
• Cross-service bucket isolation passed.
• HTTPS connectivity and certificate validation passed from a disposable in-cluster pod.
• Test objects and pods were removed.

Recovery implications

Keeping object storage outside Kubernetes preserves Loki objects and Velero backup data when the cluster is unavailable or rebuilt. It also avoids a circular recovery dependency in which the cluster must be restored before its backups can be read. The external service is still part of the recovery chain: protect its datasets, configuration, credentials, DNS, and TLS material with independent snapshots or backups and document how a rebuilt cluster reaches it.

External Keycloak PostgreSQL

• PostgreSQL 18 with pgvector runs on storage external to the Kubernetes cluster.
• A dedicated KEYCLOAK_DB_NAME database is owned by a dedicated KEYCLOAK_DB_USER login role.
• The application role is not a superuser.
• Authentication and a version query passed from a disposable in-cluster pod.
• The temporary Kubernetes Secret and test pod were removed.
• The persistent host-path dataset is owned by UID/GID 999:999 as required by the container.

Use KEYCLOAK_DB_HOST, KEYCLOAK_DB_PORT, KEYCLOAK_DB_NAME, KEYCLOAK_DB_USER, and KEYCLOAK_DB_PASSWORD_SECRET in reusable examples.

Because identity data lives outside Kubernetes, a cluster rebuild can reconnect to the preserved database. Recovery still depends on independent PostgreSQL backups, tested restores, protected credentials, and reachable networking. Kubernetes volume backups do not replace database-native backups.

Registry, node, and external-service prerequisites

• Authentication to registry.defenseunicorns.com succeeded.
• Required inotify and file-descriptor settings are persisted on every node:
  • fs.inotify.max_user_instances = 1024
  • fs.inotify.max_user_watches = 1048576
  • fs.file-max = 13181250
  • fs.nr_open = 13181250
• Time synchronization checks passed on every node.
• Registry, object-storage, and PostgreSQL reachability have been validated.

Repeated DNSConfigForming warnings indicate that some hosts advertise more than Kubernetes' three nameserver limit. Kubernetes retained valid resolvers and cluster health checks passed, so this is a documented non-blocking cleanup item.

Secrets and credentials

Follow the Security and Secrets policy.

• Historical bootstrap material was stored outside the repository under ~/.config/uds/secrets/. This is transitional, not the target runtime source of truth. Migrate values/references to OpenBao/ESO with tested recovery and preserve working access until the transition succeeds.
• The directory should be mode 700 and secret files mode 600.
• Commit only sanitized examples such as uds-config.example.yaml.
• Never commit registry tokens, TLS private keys, certificates, object-storage credentials, PostgreSQL credentials, Keycloak bootstrap credentials, kubeconfigs, service-account tokens, or real Kubernetes Secret values.

Readiness checklist

• Kubernetes 1.35 compatibility selected and documented.
• UDS Core 1.11.1 and UDS CLI 0.35.1 selected and verified.
• Worker CPU and memory baseline satisfied.
• Nodes, API server, CNI, pods, and PVCs healthy.
• MetalLB Layer 2 allocation validated.
• TrueNAS CSI, default StorageClass, expansion, and backend headroom validated.
• Public-repository secret policy defined.
• UDS Registry authentication verified.
• Tenant and admin DNS and TLS material prepared.
• Loki and Velero external object-storage buckets provisioned and isolated.
• External Keycloak PostgreSQL provisioned and reachable.
• Node sysctl baseline validated and persisted.
• Time synchronization validated on every node.
• Registry and external-service reachability validated from the cluster.
• Final readiness audit passes with no unexplained unhealthy workloads.

Historical readiness gate passed; subsequent records confirm Core/gateway deployment. Continue through the current platform issues rather than repeating this gate as unfinished work. Revalidate relevant checks before upgrades.

Authoritative references

• UDS Core supported distributions
• UDS Core production external services
• UDS Core production deployment
• UDS open-file and inotify runbook
• UDS Core 1.11.1 release
• UDS CLI releases
• Kubernetes version-skew policy