An OTA pipeline is infrastructure your product depends on for its entire field life. Built casually, it's also the single most common way to turn a routine update into a mass support incident.
Most OTA disasters share a common root cause: the update mechanism was treated as a feature to ship once, rather than infrastructure that has to survive every possible failure — power loss mid-write, a corrupted download, a bad image that boots into a crash loop, or a device that drops connectivity halfway through. A design that only handles the happy path works fine in testing and fails expensively in the field, at scale, usually on the update that matters most. Much of that reliability comes down to test coverage on the update-handling logic itself — state machine transitions, malformed-image handling, resumable-transfer edge cases — exercised on a host well before an image ever ships to a device.
The single most important design decision is whether the device writes new firmware directly over the running image, or into a separate, inactive partition that only becomes active after it's verified. Direct-overwrite designs are simpler but have no safe fallback — if power is lost mid-write, the device has no valid firmware to boot into. A dual-bank (A/B) scheme writes the new image to the inactive bank, verifies it, and only then switches the boot pointer, leaving the previously-working bank intact as a fallback the entire time. The flash cost is real — roughly double the firmware storage — but it's what makes a failed update recoverable instead of fatal.
A new firmware image that passes signature checks can still be functionally broken — a bug that only manifests after boot. The critical design pattern here is a watchdog-backed boot confirmation: the device boots the new image provisionally, and only marks it permanently active after it confirms successful operation (a known set of self-checks passing, a successful check-in with the backend, or similar). If that confirmation never happens within a timeout, the bootloader automatically reverts to the last known-good bank on the next boot. Without this, a subtly broken update can brick an entire fleet with no remote recovery path.
Even a well-tested update can behave differently across the diversity of real-world field conditions than in a lab. Rolling an update to a small percentage of the fleet first, monitoring check-in health and error rates, and only expanding once that cohort is stable is standard practice for any fleet past a trivial size — and it turns a potential mass incident into a contained one.
OTA infrastructure gets designed alongside the firmware architecture from the start, not bolted on before launch — dual-bank partitioning, signature verification, and watchdog-backed rollback are treated as required, not optional, and power draw during download and write is measured with the same power-profiling approach we apply to the rest of the firmware. See our firmware work and cloud & device platform work for how update infrastructure fits into a full fleet-management build.
Related reading: our automotive OTA update field notes cover how these principles extend to a safety-critical, regulated deployment context.
Our Solar Micro-Inverter Fleet Monitoring case study covers OTA rollout across a distributed fleet.
