Kubernetes Test and Production Deployment

Current deployment target

The dedicated k3s cluster hosts an automatically synchronized test environment and a manually promoted production environment. Both use the same Helm chart and immutable images; environment differences live in the private sens-platform-infra repository.

Endpoint	Test	Production
Platform API	https://api.dev.iot-sens.schlossers.at	https://api.iot-sens.schlossers.at
Web console	https://test-frontend.dev.iot-sens.schlossers.at	https://app.iot-sens.schlossers.at (pending GitOps sync)
Documentation	https://docs.dev.iot-sens.schlossers.at	https://docs.iot-sens.schlossers.at
Argo CD	https://argo.iot-sens.schlossers.at	shared control plane
OpenBao	https://secrets.iot-sens.schlossers.at	shared control plane, separate KV-v2 mounts

The Production hostname is approved and the product and infrastructure contracts prepare its release-coupled activation. Its live activation still requires the pinned Production desired state to pass the infrastructure validations, followed by a manual Argo CD sync. Test and production use separate namespaces, Secrets, TLS Secrets, PVCs, Argo CD Applications, and Helm release names.

Repository responsibilities

The product repository owns application source, Dockerfiles, the portable Helm chart, tests, and public documentation. The private infrastructure repository owns cluster resources, Argo CD Applications, static environment values, and immutable release selection. Neither repository contains secret values.

The approved release target includes the bounded raw-ingestion slice and disabled-by-default flow workloads:

Test:        platform-api + web-console + docs + TimescaleDB + NATS + ingestion workers
Production:  platform-api + web-console + docs + TimescaleDB + NATS + ingestion workers
Optional:    flow-worker + script-sandbox-worker + flow-action-worker

There is no mock MQTT broker or legacy decoder worker. Both environments render the raw pipeline. Flow workloads render only when explicitly enabled. Production activation remains gated by the raw-ingestion live checks and the flow security, fixture, action, and outage gates documented in the roadmaps.

Immutable release contracts

Test release metadata contains exactly:

• platformApiTag;
• webConsoleTag;
• docsTag;
• databaseMigrationsTag;
• mqttIngestionWorkerTag;
• rawStorageWorkerTag;
• flowWorkerTag;
• scriptSandboxWorkerTag;
• flowActionWorkerTag;
• rawIngestionBootstrapTag;
• rawIngestionRetentionTag.

Before this product revision is promoted, the private Production release schema and promotion workflow must accept and render the same image-key set. Disabled Flow images remain explicit immutable metadata even while their Deployments are disabled.

The product workflow publishes these images:

ghcr.io/sens-gmbh/sens-platform-platform-api:<version>
ghcr.io/sens-gmbh/sens-platform-web-console:<version>
ghcr.io/sens-gmbh/sens-platform-docs:<version>
ghcr.io/sens-gmbh/sens-platform-database-migrations:<version>
ghcr.io/sens-gmbh/sens-platform-mqtt-ingestion-worker:<version>
ghcr.io/sens-gmbh/sens-platform-raw-storage-worker:<version>
ghcr.io/sens-gmbh/sens-platform-flow-worker:<version>
ghcr.io/sens-gmbh/sens-platform-script-sandbox-worker:<version>
ghcr.io/sens-gmbh/sens-platform-flow-action-worker:<version>

Test builds use immutable MAJOR.MINOR.PATCH-alpha.RUN.ATTEMPT SemVer tags, where RUN is the one-based ci workflow position on main inside the stable root-version line and ATTEMPT is that workflow’s GitHub run attempt. A new patch, minor, or major base starts at alpha.1.1; failed or cancelled runs may leave gaps. Production uses the corresponding stable MAJOR.MINOR.PATCH tag after that release line has passed test. Mutable tags such as latest, test, or branch names are not release inputs. The complete state and bump rules are defined in Platform Release Version.

Web-console runtime contract

The web-console container serves static Vite output through its own non-root Node.js HTTP server on port 8080. It exposes:

• /healthz;
• /readyz;
• /version;
• /metrics;
• /runtime-config.json;
• SPA fallback for frontend routes.

Test runtime configuration sets SENS_WEB_CONSOLE_API_BASE_URL=https://api.dev.iot-sens.schlossers.at and uses TLS Secret sens-platform-web-console-dev-tls. /runtime-config.json must report service: "web-console", the selected release version, test, and the API origin. No credential or secret is part of this response or image.

Production sets SENS_WEB_CONSOLE_API_BASE_URL=https://api.iot-sens.schlossers.at, exposes app.iot-sens.schlossers.at, and uses Kubernetes TLS Secret sens-platform-web-console-production-tls. cert-manager owns that Secret through the existing letsencrypt-prod-sens ClusterIssuer; the web-console has no application Secret or OpenBao group.

The test API CORS allowlist contains the exact API, test web-console, and test documentation origins. Production contains only its exact web-console and documentation origins. The allowlist must never use a wildcard for credentialed browser requests and does not replace API authentication or authorization.

GitOps flow

After the application checks succeed on main, the final CI job validates the open release line, derives the immutable Alpha tag, and dispatches publish-images with that tag and the tested commit. The resulting Actions run is therefore named publish-images / MAJOR.MINOR.PATCH-alpha.RUN.ATTEMPT before any job starts; there is no separate preparation workflow in the Actions list. After every image is available, it dispatches deploy-test-main.yml in the infrastructure repository. Its run name includes the immutable Alpha tag. The workflow updates only the test Application revision and generated release files, validates the resulting release contract, pushes the desired state, and then exits. It does not keep a GitHub-hosted runner waiting for Argo or the public endpoints. SENS Platform Ops correlates the expected commit and Alpha tag with Argo, Kubernetes, and the public API and web-console version/readiness endpoints until the test release is ready. Static runtime configuration is not regenerated.

Before publishing an automatic alpha, the workflow requires the stable root package.json version to be greater than both the desired GitOps Production version and the live Production /version. Once a release base is live, new alphas for that same base are skipped without failing the completed CI checks. The job output points the operator to Prepare next version or pnpm version:next.

The final CI job derives RUN from the main CI history and the root version stored at each workflow commit. Its checkout therefore needs full Git history and its job-scoped GitHub token needs read-only Actions access. An unavailable or malformed Actions history fails closed before any image is published; no mutable counter or additional secret is used. The job receives content-write permission only to send the internal publish-test-images repository dispatch after validation; all other CI jobs remain read-only.

For pull requests, the application checks and nine container validation builds run in parallel. Each image has an isolated BuildKit scope in the GitHub Actions cache. These validation jobs use cache-only output and never publish an image. On main, CI skips that cache-only matrix because the separate image workflow immediately builds and pushes the same nine images for the tested commit. It dispatches the GitOps update only after every image is available. The cache is a performance optimization and is not a release artifact or source of truth.

The database and JetStream integration job builds all packages/* workspaces before it starts its isolated dependencies. Workspace package exports resolve the compiled dist/ files, so the job does not depend on output from another job or a previous checkout.

CI can also be started manually through GitHub Actions for the selected Git ref. A manual run uses the same event-based split: pull requests validate the container definitions, while main leaves the release builds to publish-images. A successful manual CI run on main also satisfies the existing successful-main-CI trigger for image publication.

Superseded CI runs for the same branch or pull request are cancelled. A newer successful automatic image run also cancels its older automatic predecessor; manual releases use independent concurrency groups and are never cancelled by automatic test releases. A cancelled image matrix may leave unselected partial tags in GHCR, but it cannot dispatch a release. Failed or cancelled CI events cannot cancel an already running successful image publication.

The GitHub Actions cache has a 10-GB repository limit and may evict scopes. A cache miss, eviction, export failure, or cache-service rate limit falls back to a complete image build without changing the resulting immutable image. Build record uploads are disabled so the parallel matrix does not consume artifact storage for diagnostic .dockerbuild files.

Argo CD applies the ingestion release in these waves:

1. embedded TimescaleDB and the complete NATS dependency set at wave -20;
2. the versioned database migration Job at wave -10;
3. the compatible JetStream bootstrap Job at wave -9;
4. raw-storage-worker at wave -5;
5. when enabled, script-sandbox-worker at wave -4, flow-worker at wave -3, and flow-action-worker at wave -2;
6. mqtt-ingestion-worker at wave 0.

The NATS ConfigMap, Service, and StatefulSet deliberately share wave -20. The ConfigMap must not remain in the default wave 0, because Argo CD waits for the NATS StatefulSet to become healthy before advancing and the Pod cannot start without that mounted configuration. The StatefulSet Pod template carries a checksum of the complete rendered NATS ConfigMap. Any configuration change, including max_file_store, therefore triggers a controlled Pod restart before the JetStream bootstrap runs; a mounted ConfigMap update alone does not reload the running NATS process.

The managed test environment keeps its original 2Gi StatefulSet volume-claim template even though its bound PVC has subsequently been expanded to 5Gi. Production keeps its existing 5Gi template, while the chart default for a new installation is 4Gi. Kubernetes rejects an in-place template change and does not rewrite it when the bound PVC grows. Do not delete the StatefulSet or its PVC to resolve such a sync error. First compare the rendered and live volumeClaimTemplates and restore the compatible template request in Git; a real storage migration requires a separately reviewed runbook.

The receiver is forced to one replica and Recreate. NATS, worker operation Services, and raw data have no Ingress.

A failed migration blocks application rollout. The deployment smoke check waits for API and web-console /version responses to match the selected tag and for both /readyz endpoints to become ready.

Production promotion is a reviewed, manual GitOps operation. It selects the API, web-console, documentation, migration, ingestion, and disabled Flow image set from the tested alpha release. Flow Worker, Script Sandbox and Action Worker remain disabled until their separate gates pass.

The only supported operator entry point is prod-release in SENS Platform Ops. An Admin or SuperAdmin reviews the frozen commit, alpha tag, stable target, current Production version, health/sync state, and 24-hour warning, confirms the action, and enters a fresh TOTP code. Health, sync, capacity, and observation findings are warnings; contract violations remain blocking.

The durable Ops controller then:

1. revalidates the full commit, common alpha tag, package.json version, and absence of another active release;
2. dispatches this repository’s publish-images.yml with strict Production inputs and waits for all immutable stable images;
3. re-reads the test release and stops non-retryably if it changed;
4. dispatches the infrastructure promotion with the frozen commit and tag;
5. waits for Argo to observe the new target and synchronizes only sens-platform-production;
6. waits for Synced and Healthy, then verifies API, web console, and documentation readiness and version.

Browser and web process may terminate. Valkey persists the job, and the controller resumes claimed commands after restart. Successful steps are never repeated. A retryable failed GitHub run uses Re-run failed jobs. There is no automatic rollback.

After the sync, run pnpm version:status. Only when package.json, GitOps Production, and live platform-api report the same stable version may the next major, minor, or patch version PR be prepared. Use the Prepare next version Actions workflow for a review PR or pnpm version:next <type> locally.

The former pnpm release:prod, pnpm release:prod:password, and SENS_PRODUCTION_RELEASE_PASSWORD_SCRYPT contracts are removed. Do not restore them as a fallback. If the portal path fails, repair the failed controller, credential, workflow, Argo, or live-verification boundary and continue the audited job or create a new tested candidate.

Runtime secrets

Application values are stored in separate OpenBao test and production mounts. Namespace-scoped External Secrets Operator instances synchronize only these active folders:

• /platform-api;
• /database-migrations;
• /raw-storage-worker;
• /netmore-mqtt;
• /nats-auth.

Reloader restarts only Deployments or the NATS StatefulSet that reference a changed synchronized Secret. The NATS auth Secret contains separate bootstrap, publisher, and consumer identities. The web console and documentation have no OpenBao groups. The embedded TimescaleDB password and GHCR pull credential are catalog-bound bootstrap groups synchronized to their existing Secret names.

See Secret Management and Runtime Configuration for the full contracts.

Cutover and cleanup

Apply the repository changes as a coordinated release:

1. prepare the infrastructure workflow and release schema;
2. publish the product revision containing the web-console image and new Helm chart;
3. select that revision in test and wait for Argo CD to become Synced and Healthy;
4. verify the test web console, runtime configuration, API login, and one protected request;
5. promote the same tested web-console image to Production, review the Ingress, TLS Secret and CORS diff, then synchronize Production manually;
6. verify the Production console and certificate before removing obsolete live resources and completing the final infrastructure cleanup.

After a successful sync, verify that no mock MQTT resources remain. The NATS and TimescaleDB PVCs are active state and must not be deleted during application rollback. Do not remove worker runtime Secrets while either ingestion workload is enabled.

These cluster operations are intentionally separate from Git edits because they require an authenticated Kubernetes context and OpenBao break-glass or Ops portal access.

Verification

Repository checks:

pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm docker:build
pnpm docs:check
docker compose config
pnpm helm:lint
pnpm helm:template:test
pnpm helm:template:production
pnpm helm:check-runtime-config

Live checks must confirm:

• test and Production API and web console return their selected immutable versions;
• each /runtime-config.json identifies web-console, its environment, and the expected environment-local API origin;
• login, refresh, and a protected API call work from both exact web-console origins;
• the Production certificate for app.iot-sens.schlossers.at is valid and the test web-console origin is rejected by the Production API;
• test and production contain NATS, bootstrap, retention, and both raw-ingestion workers, but no mock MQTT or legacy decoder workload;
• enabled flow workloads use the selected immutable image tag, non-root and read-only security contexts, and the sandbox egress NetworkPolicy;
• receiver strategy is Recreate with one replica;
• stream and consumer limits match ADR-0004;
• no rendered manifest or image contains a secret value;
• test live input has QoS 1 and the excluded customer creates no stream or raw row;
• TimescaleDB, API, documentation, and existing persistent data remain healthy.

Rollback first disables the MQTT receiver, then the raw consumer, and selects a preceding compatible product revision. Historical GHCR images stay available. NATS and TimescaleDB data remain in place for recovery; JetStream is not a database backup.