Device Ingestion Architecture

Status

Implemented for raw persistence. Live rollout evidence remains an operational gate.

External Source

The initial Network Server is Netmore. The final broker, topic, QoS, payload, timestamp, malformed-input, exclusion, and idempotency semantics are defined in the MQTT ingestion contract.

Stage 1: Raw Persistence

Netmore MQTT
  -> mqtt-ingestion-worker
  -> SENS_RAW_INGESTION_V1 / sens.ingestion.raw.v1
  -> raw-storage-worker / raw-storage-v1
  -> TimescaleDB raw row

The receiver:

• validates configuration, TLS, subscription, and the minimal MQTT envelope;
• preserves the original payload and safe receipt metadata;
• publishes exactly once to the internal stream and waits for persistence acknowledgement;
• does not access the database or execute telemetry flows.

The raw-storage consumer:

• uses a durable pull consumer with explicit acknowledgement;
• stores parsed, malformed, and unknown-device events;
• resolves tenant and device only through the globally unique registry DevEUI;
• inserts one row for each distinct received event;
• makes queue redelivery idempotent;
• acknowledges only after the database transaction commits;
• removes committed broker messages through work-queue retention;
• retries temporary dependency failures with bounded backoff;
• exposes queue lag, retry, max-delivery, and database failure signals.

Stage 2: Telemetry Flow

Raw DB commit
  -> transactional outbox
  -> raw-stored stream
  -> RawStoredEventV1 / raw-stored stream
  -> flow-worker loads and verifies raw row
  -> typed DAG
  -> normalized measurements + action outbox

The raw insert and outbox row are written atomically. The outbox publisher is idempotent. The MQTT receiver never dual-publishes to raw storage and flow processing. Malformed and unknown-device rows remain available for diagnosis but produce no raw-stored event. Raw Stored is a work queue: a Flow acknowledgement removes the reference immediately, while unacknowledged backlog remains bounded by seven days and 1 GiB. Replay uses its separate pinned stream. Flow failures cannot make raw persistence fail.

Delivery and Idempotency

The pipeline tolerates duplicates, delayed messages, out-of-order messages, malformed inputs, unknown devices, flow failures, and temporary broker or database outages. Flow failures are recorded independently from raw processing.

Queue redelivery never creates another raw row. Parsed identity is SHA-256 over normalized DevEUI, UTC timestamp, frame counter, port, and device payload. Malformed identity is SHA-256 over topic, a NUL separator, and exact bytes.

Testing

A permanently deployed mock MQTT broker is not used. Adapter and contract tests use versioned, anonymized real payload fixtures and controlled client fakes. Gated integration tests cover real JetStream delivery, the guarded Raw Stored retention migration, work-queue acknowledgement removal, and real TimescaleDB idempotency. No live Netmore credential is required for tests.