Quantizing a Model to INT8

Quantizing is the lever with the best ratio of effort to result: activations become a quarter of their size, so the SRAM a model needs falls with them, and so does the flash its weights occupy. On MobileNetV2 at 224x224 it is the difference between a model that needs 250 KiB of SRAM and one that needs 62.50 KiB.

TiGrIS reads ONNX QDQ models directly, so quantizing is something you do to the model before the compiler sees it. This page covers what the compiler requires, the settings that decide whether a quantized model compiles at all, and how to check the result.

What TiGrIS requires

QDQ format. The quantizer must write QuantizeLinear and DequantizeLinear nodes around float operators. That is QuantFormat.QDQ in ONNX Runtime.

Static quantization. Scales and zero points have to be in the file, because the plan is built ahead of time and the arena is sized from them.

A fully quantized graph. A plan executes on one dtype, so an operator the quantizer skipped leaves the graph mixed and the compiler refuses it.

Nothing else. Activations stated as uint8, which the ONNX Runtime quantizer emits by default, are handled: uint8 value v and int8 value v - 128 denote the same real number under zero points that differ by the same 128, so the compiler restates them in the signed domain the kernels work in.

The recipe

import numpy as np
import onnx
from onnxruntime.quantization import (
    CalibrationMethod, QuantFormat, QuantType, quantize_static,
)

model = onnx.load("model.onnx")
graph_input = model.graph.input[0]
shape = [d.dim_value or 1 for d in graph_input.type.tensor_type.shape.dim]


class Calibration:
    """Feeds representative samples. Random data is fine for a mechanical
    check and wrong for anything you intend to deploy."""

    def __init__(self, samples):
        self.samples = iter(samples)

    def get_next(self):
        return next(self.samples, None)


quantize_static(
    "model.onnx",
    "model.int8.onnx",
    Calibration(representative_samples),
    quant_format=QuantFormat.QDQ,
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
    per_channel=True,
    calibrate_method=CalibrationMethod.MinMax,
)

Then compile as usual:

tigris analyze model.int8.onnx -m 64K

warning: input axis 0 (batch_size) is unset; using 1 (--input-shape overrides)
╭────────────────────── TiGrIS - mobilenetv2.qdq-int8 ───────────────────────╮
│ Operators            65                                                    │
│ Tensors              174 (66 activations)                                  │
│ Peak memory (naive)  1.44 MiB                                              │
│ Largest tensor       1x96x112x112 (1.15 MiB)                               │
│ Quantization         INT8 (QDQ)                                            │
│ Input                input 1x3x224x224 float32, stored as int8 at scale    │
│                      0.0413059, zero point -2                              │
│ Output               output_QuantizeLinear_Input 1x1000 float32, stored as │
│                      int8 at scale 0.0587218, zero point 6                 │
╰────────────────────────────────────────────────────────────────────────────╯

Note the Input and Output rows. The model declares float32 at both ends and the plan executes on int8, so the runtime converts at the boundary: an application hands over and reads back float32 either way. Quantizing does not change the interface your code talks to.

Per-channel or per-tensor

per_channel=True gives each output channel its own weight scale. It costs plan size and buys accuracy. Measured on MobileNetV2, calibrated identically both ways:

Weights	Plan	Quantization error against the float model
per-tensor	3.39 MiB	max 49.1 LSB, mean 45.9
per-channel	3.81 MiB	max 17.5 LSB, mean 13.8

Roughly a third of the error for 0.42 MiB. Take per-channel unless flash is the binding constraint. The activation memory does not change either way: both have a 1.44 MiB naive peak and a 62.50 KiB floor, because only the weights are affected.

One catch: axis on DequantizeLinear arrived in opset 13, so per-channel quantization of an opset-12 model produces a file ONNX Runtime refuses to load, with Unrecognized attribute: axis for operator DequantizeLinear. Convert first:

from onnx import version_converter
onnx.save(version_converter.convert_version(model, 13), "model.opset13.onnx")

Settings that do not work, and how they fail

QuantFormat.QOperator writes QLinearConv, QLinearAdd, QGemm and friends instead of QDQ pairs. TiGrIS has none of those operators, and shape inference cannot see through them, so the failure arrives as a shape error rather than a format one:

Error: Tensor 'output' has unknown rank; TiGrIS requires concrete deployment shapes

If you see that on a model you just quantized, check the format before anything else.

quantize_dynamic computes activation scales at run time, so the file carries none. The compiler reports the weight dtype it cannot place:

Error: Cannot compile a plan with unsupported tensor dtypes: unsupported
activation tensor dtype(s): ONNX dtype 2 (317_quantized, 317_zero_point, ...)

A partially quantized graph fails with both dtypes named:

Error: Cannot compile a plan with unsupported tensor dtypes: mixed activation
dtypes cannot use a graph-wide runtime dispatcher: float32 (...), int8 (...)

Read the float32 list. It names the operators the quantizer skipped, which is usually an operator it does not support, and the fix is upstream in the quantizer rather than in the compiler.

Check the result

Quantization changes what the model computes, so measure rather than assume. Compare three things on the same inputs:

1. The float model under ONNX Runtime, which is the reference.
2. The quantized model under ONNX Runtime, which tells you what quantization cost you. This is the number that decides whether the model is still good enough, and it has nothing to do with TiGrIS.
3. The compiled plan, which tells you whether TiGrIS reproduces the quantized model.

On MobileNetV2 the third comparison lands well inside the second:

TiGrIS vs ONNX Runtime int8    : max  6.0 LSB, mean  5.0 LSB
ONNX Runtime int8 vs float     : max 17.5 LSB, mean 13.8 LSB

A few LSB between TiGrIS and ONNX Runtime is expected and is not drift. Both requantize with integer arithmetic but break rounding ties differently, and a one-LSB difference early in a network is amplified by any later layer whose effective scale is large.

If the third comparison is large while the second is small, that is a compiler problem and worth reporting. If the second is large, the model needs better calibration data or per-channel weights, and no amount of compiler work will recover it.

Calibration data

Everything above is mechanical. The one judgement call is the calibration set, which decides the scales and therefore the accuracy. It has to look like what the model will see when deployed: the same preprocessing, the same value range, the same kind of content. A few dozen representative samples is usually enough, and random noise is not representative of anything, however convenient it is for checking that the pipeline runs.