Most models are trained at a precision the target edge hardware can't run efficiently. Quantization closes that gap — but it's a real engineering tradeoff, not a free optimization.
Models are typically trained in 32-bit floating point (FP32), because training benefits from the numerical precision and dynamic range that gives. Most edge hardware — MCUs, low-power SoCs, many embedded NPUs — either can't run FP32 efficiently or can't run it at all within a realistic power and memory budget. Quantization is the process of converting a trained model to a lower-precision numeric format so it actually fits and runs efficiently on that hardware, and the choice of target format is a real engineering decision, not a checkbox. It's one half of the broader on-device vs. cloud placement decision — quantization is what makes on-device placement feasible in the first place.
Quantization maps a model's weights (and often its activations) from a high-precision format to a lower-precision one — commonly FP32 down to FP16, INT8, or in more aggressive cases even lower bit-widths. Fewer bits per value means less memory to store the model, less memory bandwidth to move values during inference, and — critically for edge hardware — access to integer arithmetic units that are often faster and far more power-efficient than floating-point units on constrained silicon. The cost is precision: representing values with fewer bits necessarily loses some numerical detail, which can translate into accuracy loss if not managed carefully.
| Format | Size vs. FP32 | Typical accuracy impact | Hardware fit |
|---|---|---|---|
| FP16 | 50% | Usually minimal | Broad support, good middle ground |
| INT8 | 25% | Small to moderate, calibration-dependent | Strong support on modern edge NPUs/DSPs |
| INT4 / lower | 12.5% or less | Can be significant without careful technique | Narrower hardware/tooling support |
FP16 is often close to a free lunch — a meaningful size and speed improvement with accuracy impact that's frequently negligible, because it retains floating-point semantics with just reduced range and precision. INT8 requires more care: converting to integer arithmetic means choosing how the model's floating-point value ranges map onto the available integer range (a process called calibration), and a poorly calibrated conversion can produce a real accuracy drop, particularly for models with layers sensitive to precision loss.
There are two general approaches, and which one is worth the effort depends on how much accuracy loss the use case can tolerate. Post-training quantization converts an already-trained FP32 model directly, using a representative dataset to calibrate the integer ranges — it's fast and requires no retraining, and for many models the accuracy loss is small enough to accept. Quantization-aware training instead simulates the effects of quantization during the training process itself, letting the model adapt its weights to be more robust to the precision reduction — it produces better accuracy at the lower precision, at the cost of needing to retrain (or fine-tune) the model, which requires the original training pipeline and data.
Quantization frameworks can simulate lower-precision behavior on a development machine, but the actual accuracy and performance on target hardware can differ due to how a specific chip's inference runtime implements quantized operations. A model that quantizes cleanly in simulation should still be validated end-to-end on the actual target device before considering the conversion final — this is where framework-specific quirks (TensorFlow Lite Micro, ONNX Runtime, or a vendor SDK) most often surface, and where the MCU chosen for the product starts to matter as much as the model itself.
We treat the precision target as a decision made against the specific hardware and accuracy requirements of the product, validated on real target hardware rather than simulation alone. See our AI & Intelligence work for how this fits into deploying inference on constrained embedded hardware.
Our Driver-Behavior Edge-AI Dash case study covers a real deployment of a quantized model on constrained hardware.
