Historian Versus Live MQTT: Pick the Path Operations Can Act On

Historian Versus Live MQTT: Pick the Path Operations Can Act On

Most industrial IoT setups push every reading into both live MQTT streams and a historian. The split looks clean on paper. In practice, operators end up staring at two versions of the same data and still miss the window to act.

The real distinction is not storage. It is whether the data supports a decision that must be made now or one that needs to be reviewed later.

MQTT versus historian in operator workflows

Live MQTT feeds current values to screens and control loops. A historian stores the same values with timestamps for later queries. When teams treat both paths as interchangeable, operators receive either stale numbers during a shift or raw streams that cannot be audited after the fact.

We see this pattern on sites where edge gateways forward everything to both systems without routing rules. The MQTT path stays useful only while the decision window stays open. The historian path stays useful only when someone needs to reconstruct what happened.

Live MQTT for decisions that expire

Operators on the floor need readings that reflect the current state of the line. Once a batch moves or a pressure spike passes, the value loses its action value. In one deployment we set a freshness threshold of 15 seconds for any MQTT topic tied to immediate control actions. Topics older than that were dropped at the edge broker instead of queued.

This rule keeps dashboards from showing numbers that no longer match the physical equipment. It also reduces the chance that an operator acts on data that has already changed.

Historian for audit and shift review

Shift reviews and incident reports require stable records that can be queried days later. A historian with proper indexing lets engineers pull exact sequences without rebuilding them from logs. The same data loses value if it sits in MQTT retain buffers that get overwritten or lost on restart.

We keep historian writes on a separate edge process that batches every 30 seconds. This keeps MQTT traffic light while still giving audit teams the granularity they need for root-cause work.

Two ways to route the same sensor

Teams usually choose between two patterns when both paths are required.

• Route every reading through a single edge service that publishes to MQTT and writes to the historian in the same transaction. This keeps timestamps aligned but adds latency to the live path.
• Split at the device or gateway so critical control tags go only to MQTT and all tags go to the historian on a slower schedule. This keeps the live path fast but requires careful tag lists to avoid missing context later.

The first option works when teams can tolerate 200–300 ms extra delay. The second option works when the control loop must stay under 100 ms end-to-end. We have used both on the same site by tagging topics with an “action” or “audit” label at the gateway.

One failure mode and how we fixed it

A common failure is MQTT topics that keep publishing after the historian write path fails. Operators continue to see live values while the audit trail stops. In one case this gap lasted four hours because the edge service retried historian writes silently and never surfaced the error.

We added a simple counter on the gateway that tracks consecutive failed historian writes. When the counter hits five, the service stops publishing the affected topics to MQTT and raises an alert on the existing operations channel. The stop forces the issue into view instead of letting the two systems drift.

One decision that matters

Pick the path based on the lifetime of the decision, not the lifetime of the data. Route time-sensitive control through MQTT with explicit freshness rules. Route everything else through the historian with batch writes that do not interfere with the live path.