Not every edge-AI feature needs an AI accelerator. Here's how to tell whether a quantized model on a plain MCU is enough, or whether the product needs dedicated inference silicon.
"Edge AI" covers a huge range of actual compute requirements — a wake-word detector and a real-time camera-based defect classifier have almost nothing in common except that both run "on the device." The hardware decision follows from the workload, not the other way around.
Small, quantized models — TinyML-scale keyword spotting, simple vibration/anomaly classifiers, low-resolution sensor fusion — often run acceptably on a Cortex-M class MCU's integer pipeline, especially with an optimized inference runtime (e.g. TensorFlow Lite for Microcontrollers) and int8 quantization. The advantage is real: one less part, one less power domain, one less thing to qualify and source. This is the default starting point, and for a large share of edge-AI features it's also the ending point.
Once the workload involves real-time vision (object detection, not just simple thresholding), continuous audio processing beyond wake-word, or model sizes that don't fit in an MCU's SRAM/flash budget, the calculus shifts toward parts with a dedicated neural processing unit (NPU) or GPU-class accelerator — application processors like NVIDIA's Jetson line, Qualcomm's edge AI platforms, or MCU-class parts with an integrated NPU. These bring real inference throughput but also bring a heavier BOM, a Linux-class software stack, and materially higher power draw — which usually rules them out for anything battery-constrained.
We typically start by trying to fit the model on the target MCU, quantizing aggressively, and only escalating to dedicated AI hardware once that's demonstrably insufficient for the accuracy/latency the product needs. MCU-scale inference (TensorFlow Lite for Microcontrollers) is a good fit for basic anomaly detection or wake-word triggers, but we move to NPU-class hardware — architectures like NXP's i.MX8M Plus, paired with runtimes such as TensorRT — the moment a client's requirements involve continuous vision processing, complex object detection, or real-time robotics workloads (for example, ROS2-based systems) where inference has to happen at the edge without cloud round-trip latency. See also edge AI quantization and on-device vs. cloud AI, and our AI & intelligence work.
