Observability Model

Purpose

This document defines initial observability requirements.

Observability must support human operators and future AI-assisted support workflows.

Logs

The shared Pino serializer never copies raw Error.message values or stacks to container logs. It emits only a bounded error name/code plus a redacted message; stack traces belong in the separately sanitized, optional Sentry adapter.

Services must use structured logs.

Logs should include:

• Timestamp.
• Log level.
• Service name.
• Message.
• Correlation ID where available.
• Tenant ID where safe and relevant.
• Organization ID where safe and relevant.
• Device ID where safe and relevant.
• Error code where relevant.

Logs must not include secrets or sensitive raw credentials.

The Phase 4 service baseline uses JSON logs for backend deployables. Every log record includes:

• service name,
• runtime environment,
• log level,
• timestamp,
• message.

Logs include correlationId when request, event, or worker context provides one. The x-correlation-id HTTP header is the initial transport header for operation endpoints and future HTTP APIs.

Metrics

Initial metrics should cover:

• HTTP request count.
• HTTP error count.
• HTTP latency.
• MQTT connection state.
• MQTT messages received.
• MQTT messages rejected.
• Internal stream backlog.
• Telemetry messages processed.
• Flow execution and block failures.
• Unknown device messages.
• Database write failures.
• Export jobs created.
• Export jobs completed.
• Export job failures.

Backend services expose Prometheus-compatible metrics at:

GET /metrics

The Phase 4 baseline exposes:

• sens_service_info{service_name,environment} 1
• sens_service_ready{service_name,environment}
• sens_http_requests_total{service_name,environment,method,route,status_code}
• sens_http_request_duration_seconds
• Node.js process metrics from prom-client

Phase 7A adds sens_auth_events_total{service_name,environment,event} for bounded login, refresh, logout, password, and session outcomes. It deliberately has no user, tenant, organization, raw identifier, or token labels.

Phase 7C adds sens_audit_operations_total{service_name,environment,operation,outcome}. operation is limited to begin, finalize, and entity; outcome is success or failure. Audit failure logs contain only correlation ID, route template, audit stage, and safe error details. Neither metric nor log adds raw route parameters, tenant names, or user identifiers.

Raw-ingestion receiver metrics add bounded counters and gauges for MQTT connection/subscription state, received QoS and retained flags, fixed filtering reasons, parsed/malformed results, JetStream publish/duplicate/failure results, and publish latency.

Raw-storage metrics add durable pending and acknowledgement-pending gauges, redeliveries, committed stored/unknown_device/malformed/duplicate outcomes, database errors, terminal/max-delivery signals, and processing latency. No ingestion metric has topic, customer, tenant, device, DevEUI, event ID, or payload labels.

Flow operations add Raw Stored outbox backlog and age, Raw Stored and Replay consumer lag, execution status totals, bounded block-key latency/errors, sandbox timeout/resource failures, measurement revision/idempotency conflicts, Action pending/paused/failed/dead-lettered totals, SMTP failures, rate-limit deferrals, Replay totals and persistent Dead Letter totals. Tenant, Flow, execution, Action, Raw and recipient IDs are log correlation fields where safe, not metric labels.

The implemented flow-specific metric surface includes:

• sens_flow_stream_pending, sens_flow_replay_stream_pending, and sens_flow_dead_letter_stream_pending,
• sens_flow_executions_total, sens_flow_failures_total, and sens_flow_dead_letters_recorded_total,
• sens_flow_node_duration_seconds and sens_flow_node_failures_total, where tenant scripts share the bounded custom_script label,
• sens_flow_action_stream_pending, sens_flow_action_outcomes_total, and sens_flow_action_failures_total,
• sens_script_sandbox_requests_total and sens_script_sandbox_execution_duration_seconds.

The Helm chart can optionally render a PrometheusRule through monitoring.prometheusRule.enabled. It provides initial Replay-backlog, Dead Letter-recorder-backlog, and sandbox-failure alerts. The option remains off by default because the core chart does not require the Prometheus Operator; a deployment that enables it must already provide that CRD and route the rule to its monitoring stack.

The temporary manual IOTA migration adds sens_iota_snapshot_import_runs_total{service_name,environment,outcome}, sens_iota_snapshot_import_duration_seconds{service_name,environment}, and sens_iota_snapshot_import_entities_total{service_name,environment,entity,outcome}. Its label values are fixed and low-cardinality. Tenant, source URL, username, DevEUI, source record identifiers, and credentials are never metric labels.

Metrics must not use tenant IDs, organization IDs, device IDs, user IDs, tokens, raw payload values, or other high-cardinality or sensitive labels.

External Operations Data Plane

The implemented production monitoring baseline deliberately lives outside the platform server failure domain:

• sens-platform-infra deploys victoria-metrics-k8s-stack chart 0.87.0 into monitoring with vmagent, Node Exporter and kube-state-metrics;
• Grafana, VMSingle, vmalert and Alertmanager remain disabled in the platform cluster;
• vmagent sends Prometheus Remote Write through bearer-protected vmauth to the separate Coolify server;
• the Coolify stack runs VictoriaMetrics Community v1.150.0 with seven-day retention, vmauth, vmalert and Blackbox Exporter;
• sens-platform-ops queries only fixed metric allowlists and aggregates GitHub, Argo CD, Kubernetes and OpenBao health through read-only adapters.

OpenBao adds an internal metrics listener, sealed-state and audit-failure signals, Raft and snapshot alerts, and capacity alerts for its dedicated audit volume. Operational server logs pass through a strict Fluent Bit allowlist. Native audit records remain on the dedicated 180-day audit volume and are not forwarded to VictoriaLogs, Sentry, or PostHog. Secret operations expose only low-cardinality action, environment, outcome, and dependency labels; group IDs, key names, users, values, paths, lengths, and hashes are forbidden.

The public write contract is POST https://ops.iot-sens.schlossers.at/_metrics/api/v1/write. Query paths use a different bearer token. VictoriaMetrics itself is not publicly exposed. The portal, its BFF, and the monitoring time series are operational tooling, not Platform customer APIs and not storage for tenant or telemetry payloads.

Platform services own their bounded /metrics surface. The product Helm chart owns stable Service labels and the HTTP metrics port. The infrastructure repository owns VMServiceScrape, Remote Write, firewall, token rotation and operator lifecycle. The Ops repository owns the seven-day store, query allowlist, charts, external probes and alarm evaluation.

The NATS StatefulSet runs the Apache-2.0 Prometheus NATS Exporter as a bounded sidecar. NATS’ unauthenticated monitoring endpoint listens only on 127.0.0.1:8222 inside the Pod; the Service exposes only the exporter’s /metrics endpoint on port 7777. vmagent therefore receives stream and consumer counters without receiving message bodies, subjects, credentials, or tenant labels. The exported stream names and consumer names are low-cardinality technical contract identifiers.

Current external rules cover public endpoint and TLS failure, Node readiness, CPU, memory and filesystem pressure, unavailable Deployment replicas, CrashLoops and restarts, missing ready Endpoints, HTTP 5xx rate, missing cluster metrics and vmagent Remote Write errors. V1 displays these alarms only in the Ops portal. Notification delivery, log aggregation and tracing remain separate future decisions.

Monitoring time series are not backed up in V1. Their loss does not remove platform, customer or telemetry data. The independent server is intentionally single-node for V1; the controlled platform-outage proof remains an operational release gate.

Local Observability Verification

The repository includes a local dev cockpit for Phase 4 observability checks. It can start the backend service skeletons, capture their stdout and stderr, inspect structured JSON logs, call health and readiness endpoints, validate baseline Prometheus metrics, and verify correlation ID behavior.

The dev cockpit is a local development tool only. It is not a log aggregation stack, not a production dashboard, not a Kubernetes component, and not part of the public Platform API.

Health and Readiness

Every service must expose health and readiness signals.

Health answers whether the process is alive.

Readiness answers whether the service can serve traffic or process work.

Readiness should consider critical dependencies such as database or broker where appropriate.

Backend deployables expose these operation endpoints:

GET /healthz
GET /readyz
GET /metrics

GET /healthz returns HTTP 200 when the process is alive. It does not check external dependencies.

GET /readyz returns HTTP 200 when all readiness checks pass and HTTP 503 when one or more checks fail. platform-api checks both configuration and PostgreSQL, because authentication and authorization require current database state and every /v1 request requires a durable audit start. mqtt-ingestion-worker requires an active MQTT connection, confirmed subscription, NATS connectivity, and the expected stream. raw-storage-worker requires PostgreSQL, NATS, and the expected durable consumer.

flow-worker readiness covers PostgreSQL, Raw Stored, Replay and Dead Letter consumers plus its outbox relays. script-sandbox-worker readiness proves its NATS request subscription. flow-action-worker validates NATS, PostgreSQL and, when enabled, complete SMTP configuration before readiness. A growing Replay or Dead Letter backlog and a missing retained Raw row for queued work are alert conditions.

Responses include x-correlation-id. If the caller supplies a valid x-correlation-id, the service reuses it; otherwise the service generates a new correlation ID.

The platform-api also exposes:

GET /test

GET /test returns { "success": true } and emits a structured JSON log entry with the message test endpoint called. This endpoint exists only as an early deployment smoke test and must not be used as a product API contract.

Tracing

Correlation IDs should flow through HTTP requests, internal events, and worker logs.

Full distributed tracing may be introduced later.

Alerting

Initial alerting should consider:

• Service down.
• Database unavailable.
• Broker unavailable.
• MQTT disconnected.
• Queue backlog growing.
• Flow or sandbox errors above threshold.
• Unknown devices above threshold.
• Export job failures.
• Disk/storage pressure.

Central operations integrations

The external Ops host runs VictoriaLogs with 30-day retention. Kubernetes and Coolify Fluent Bit collectors forward only the versioned structured-log allowlist. Kubernetes collection reads container files and Pod/namespace metadata; it has no Secret, pods/log, exec, attach, or port-forward access. Logs are not backed up. sens-platform-ops exposes bounded filters and exact correlation lookup only to Admin and SuperAdmin.

Production API, web console, and documentation readiness are probed every 15 seconds by the primary EU prober on the external Ops host. The dedicated SLO VictoriaMetrics retains 400 days. Missing or stale evidence is never uptime; the primary prober needs 99.5 percent coverage in the evaluated 30-day or calendar-month window for complete operational history. The public contract distinguishes complete evidence, a continuous ramp-up, actual gaps, a successfully queried period without observations, and an unavailable evidence source. The ramp-up forecast assumes continued stable sampling; it is not an SLA commitment. The persistent victoria-metrics-slo-data volume must survive Ops redeployments because losing it restarts coverage from the first new sample. Because prober, storage, and portal share one host, this is not independent SLA evidence. vmalert sends firing and resolved notifications through Alertmanager and a dedicated STARTTLS relay.

The public component history defaults to 24 rolling one-hour intervals and can switch to seven rolling 24-hour intervals or 13 completed calendar months. Short-term intervals expose raw availability: a fully evidenced 100-percent interval is shown as online, while a lower value states that an outage was observed. Evidence gaps and source failures take precedence. Four bounded MetricsQL range queries per component provide the short-term buckets; arbitrary query windows are not exposed.

Sentry error reporting is an optional adapter using the EU service, no default PII, sanitized exceptions, release tags, and 5 percent production performance sampling. PostHog is an optional Test and Production EU product signal adapter. Its SDK loads only after consent, disables autocapture, pageviews, replay, and person profiles, and sends only the fixed event contract with HMAC pseudonyms. Neither SaaS is required for health, authorization, correctness, alerting, or on-premise operation.

The PostHog event name is the stable event identifier; PostHog does not require a separately provisioned event ID. Every event also receives the fixed environment and appVersion properties in the browser adapter. The current allowlist is:

Event	Allowed event-specific properties	Product question
auth_login_attempt_completed	outcome, durationBucket	Where do login attempts fail or become slow?
dashboard_view_completed	outcome, durationBucket	How often do dashboards load successfully, empty, or with an error?
device_registration_completed	outcome, durationBucket	Can administrators complete the core device-onboarding step?
flow_editor_opened	none	How often does a flow-editing attempt begin?
flow_action_completed	action, outcome, optional errorCategory, durationBucket	Which validate, test, save, or release actions fail?
flow_editor_closed	dirty	How often is the editor left with unsaved work?

Device registration is captured only for creation, never for an ordinary device update. It contains no device, DevEUI, tenant, organization, or user identifier. Failed local validation and failed API creation both produce the same bounded failure outcome without forwarding an error message.

Full distributed tracing remains deferred. Correlation IDs continue across the existing HTTP and NATS contracts.

The planned tracing extension introduces a vendor-neutral OpenTelemetry Collector and optional OTLP export after W3C trace context, sampling, redaction, cardinality, and cost limits are specified. Payloads, credentials, and user, tenant, device, or telemetry data must not become span attributes or events. The collector and every SaaS exporter remain optional for customer-owned installations.

The planned issue-investigation extension can deep-link a sanitized Sentry issue summary to the existing bounded log search. It may pass only an exact correlation ID and narrow time window. The action is visible only to Admin and SuperAdmin with logs.read; viewers retain the safe issue summary without cross-tenant log access. It does not expose raw Sentry request data, arbitrary Sentry queries, or arbitrary LogsQL.