raw-storage-worker

Responsibility

services/raw-storage-worker consumes RawIngestionEventV1 from the durable raw-storage-v1 pull consumer and stores the event through RawIngestionMessagesRepository. It is the boundary at which the durable broker event becomes authoritative raw database evidence.

Processing sequence

1. Decode the internal bytes as strict UTF-8 JSON.
2. Validate the complete RawIngestionEventV1 contract.
3. Terminate an invalid internal event because retry cannot repair its schema.
4. Resolve device and tenant only through the globally unique registered DevEUI; the Netmore topic customer is metadata, never tenant authority.
5. Store the parsed, malformed, or unknown-device record idempotently in a PostgreSQL transaction.
6. Send the JetStream acknowledgement, wait up to five seconds for its confirmation, and let work-queue retention remove the committed broker message.

If commit succeeds but acknowledgement fails, redelivery is safe: the repository returns the existing record instead of inserting another raw row. If PostgreSQL is unavailable, the missing acknowledgement keeps the message in JetStream for later processing. PostgreSQL is the authoritative raw archive; JetStream retains only unacknowledged backlog.

Backpressure and retries

The default concurrency is four database transactions per worker and is bounded between 1 and 16. The durable consumer allows at most 16 unacknowledged messages. While a database transaction runs, the worker sends progress signals so the 60-second acknowledgement window is not mistaken for a stalled handler.

Temporary database or acknowledgement failures are negatively acknowledged with backoff at 1 second, 5 seconds, 30 seconds, 2 minutes, 10 minutes, 30 minutes, and then 1 hour. The consumer contract allows 100 deliveries. Invalid internal contracts are terminated immediately and surfaced as operational errors.

Readiness and scaling

Readiness checks both PostgreSQL and the named JetStream consumer. It also updates stream-pending and acknowledgement-pending gauges. The handler is safe for redelivery and multiple instances, but replica and concurrency changes must still be capacity-tested against database connections and stream limits.

Failure diagnosis

Signal	First interpretation
Database errors	PostgreSQL connection, migration compatibility, lock, or capacity problem.
Redeliveries	A previous transaction or acknowledgement did not complete.
Terminal invalid-contract events	Publisher/consumer contract drift or corrupted internal bytes.
Raw Stored outbox failures	Raw Stored stream capacity, NATS storage, authorization, or connectivity.
Stream pending grows	Consumer unavailable or slower than incoming MQTT volume.
Ack pending remains high	Handlers are blocked or database transactions are slow.

The Raw Stored stream reserves 1 GiB. Its limit can be increased in place by a compatible bootstrap after the NATS server file-store ceiling has been raised. The Helm checksum restarts NATS before the bootstrap job updates the stream. Do not reduce the limit or roll back the server ceiling while current usage is above the previous 256-MiB limit; keep the larger capacity as the forward fix until the consumer backlog is empty.

Raw Stored uses work-queue retention with a 7-day maximum age. The one-time rawIngestionBootstrap.migrateRawStoredRetention mode converts only the known legacy limits-retention stream. It requires MQTT ingestion, raw storage, and the Flow worker to be disabled, verifies the Flow consumer is fully drained, rechecks the unchanged state after five seconds, and only then recreates the stream through the normal bootstrap. See the Raw Stored work-queue migration runbook.

See Database Migrations for the raw schema and Netmore Raw Ingestion for incident checks.