Versioned Telemetry Flows

Decision

Telemetry interpretation is a versioned directed acyclic graph. Every executable Flow belongs to exactly one tenant and Device Type, and a Device Type owns at most one Flow. One released Flow version is active.

MQTT -> raw stream -> raw-storage-worker
  -> Raw row + transactional outbox
  -> RawStoredEventV1 -> flow-worker
  -> one transaction:
       Measurements + Device Current Values
       + Action jobs + Action outbox + successful execution
  -> flow-action-worker

The MQTT receiver never publishes directly to the Flow worker. The worker loads the authoritative Raw row and verifies tenant, Device, Device Type, idempotency key, and SHA-256 against RawStoredEventV1 before execution.

Device-Type field contract

device_field_definitions is the authoritative allowlist of values a Device Type may store. A field has an immutable key, versioned scalar or JSON schema, optional numeric unit, one or both storage targets, optional Timeseries measurement type, lifecycle status, and lock version. JSON is current-only; Gauge and Counter require Number. The schema may change before first use; released bindings or stored data then protect schema, unit, and measurement type. Storage targets may only be extended and fields are deprecated instead of deleted.

flow-graph.v2 carries its Device-Type ID, executable block nodes, and exactly two terminal system nodes: current-output and timeseries-output. These nodes are generated from the field contract, expose only allowed target ports, and cannot be deleted or duplicated. Released versions pin their field snapshots in flow_version_field_bindings.

Compile and runtime rules

Each immutable block version publishes named input and output ports. An edge is valid only when both ports exist, their versioned schemaRef values match exactly, the target is free, and no cycle is created. JSON Schema 2020-12 is authoritative. The editor checks eagerly; the API recompiles on validation and release; runtime validates every actual output.

An unconnected prepared field is a warning. A dual-target field must connect both ports to the same source node and port. Deprecated fields cannot be newly connected. A missing optional decoder output creates no write and does not fail the execution. The former Save-Measurement and Update-Device-Attribute blocks and the device-attribute Action contract are removed.

The execution frame stores immutable JSON values by nodeId.outputPort. Fan-out is allowed and fan-in requires an explicit block. External side effects are represented only by Action intents.

Timestamps, transaction, and replay

FlowExecutionInputV2 separates the normalized Netmore timestamp observedAt from the local MQTT/Raw receipt time rawReceivedAt. Both storage targets use only observedAt as the business timestamp.

Measurements, current values, remaining Action jobs and outbox events, and the successful execution status commit in one database transaction. Any storage error rolls back all of them.

Transactional outbox rows remain authoritative until publication succeeds. Pending rows have no retention deadline. After publication, the shared hourly retention job keeps seven days of delivery history and deletes older rows in bounded batches.

Every valid out-of-order point is stored in Timeseries. The current snapshot advances only for a newer Netmore timestamp. Equal timestamps use Raw receipt and execution identity as deterministic tie-breakers, while replaying the same latest Raw row may correct its value. A Device-Type change removes the prior current snapshot in the Device update transaction; Timeseries and audit history remain.

Measurements retain revision heads for idempotent live retries and complete Replay revisions. Missing former Replay outputs receive tombstones. Replay mode suppresses Actions but may write corrected Measurements and Current Values atomically.

Versions, parameters, scripts, and templates

The first processing attempt pins the active released version and effective parameter snapshot. Parameter resolution is Device, Device Type, Tenant, default. Retries keep both pins. Activation can move to any compatible released version without mutating past executions.

Custom scripts are synchronous pure TypeScript functions without imports and run in the existing QuickJS/WASM sandbox. V2 templates store field requirements by key, schema, unit, measurement type, and target rather than Device-Type UUIDs. Copying validates the chosen target Device Type and creates its single independent Flow. V1 template versions remain deprecated and read-only.

Preview runs use the same engine and sandbox but persist no execution, Measurement, Current Value, or Action job. Raw-derived fixtures require explicit confirmation because their materialized input outlives normal Raw retention.

Flow administration collections use one explicit navigation contract:

Resource	Navigation
Custom blocks, Flow templates, E-mail templates, recipient bindings, fixtures, parameter values	Counted cursor pages with 25, 50, or 100 rows and server search
Custom-block, Flow-template, and E-mail-template versions	Progressive 25-entry history with explicit older-page loading and no exact total
Released Custom blocks in the Studio	Server search and explicit 50-entry continuation
Built-in blocks and schemas	Complete, hard-bounded registry response without pagination
Flow graph nodes, edges, and parameter definitions	Complete arrays bounded by the graph contract
Tenant Action control	One bounded resource
Flow executions and Action Dead Letters	Anchored progressive 25-entry lists with explicit older-page loading and no total

Opaque cursors bind actor, tenant, collection, filters, page size, and parent resource. Continuation failures preserve loaded rows for retry. Synchronous cursor deduplication prevents duplicate requests, and late responses from an old search or tenant generation cannot replace the current result.

Execution and Dead Letter investigations order by creation timestamp and UUID. The cursor fixes the upper time anchor and binds the actor, Tenant, Flow, historical Organization assignment, status, lower time bound, and page size. Organization filtering uses the assignment that was valid when the Execution ran. Refresh replaces the anchor; it never mixes rows from the old investigation with the new one.

Actions and production gate

Send-Email remains asynchronous. Template and recipient resolution, rate limiting, pause/resume, retry, Dead Letter, and manual redrive behavior are unchanged. An ambiguous SMTP acknowledgement may still require explicit redrive confirmation.

Flow and Action workloads remain gated by the production release runbook. Activation requires anonymized EM320-TH payload evidence, successful staging replay/live proof, sandbox and NetworkPolicy evidence, observed stream lag, and verified SMTP configuration.