Quantizing a Model to INT8

Quantizing is the lever with the best ratio of effort to result: activations become a quarter of their size, so the SRAM a model needs falls with them, and so does the flash its weights occupy. On MobileNetV2 at 224x224 it is the difference between a model that needs 250 KiB of SRAM and one that needs 62.50 KiB.

TiGrIS reads ONNX QDQ models directly, so quantizing is something you do to the model before the compiler sees it. This page covers what the compiler requires, the settings that decide whether a quantized model compiles at all, and how to check the result.

What TiGrIS requires

QDQ format. The quantizer must write QuantizeLinear and DequantizeLinear nodes around float operators. That is QuantFormat.QDQ in ONNX Runtime.

Static quantization. Scales and zero points have to be in the file, because the plan is built ahead of time and the arena is sized from them.

Explicit float sections. Supported float32 operators after a DequantizeLinear can run inside an int8 plan, ending at a QuantizeLinear or a float model output. Their activations take four bytes per element. A binary operator with a uniform constant can stay int8 when followed by quantization.

Runtime support. Runtime 0.11.x refuses a plan with float sections.

Nothing else. Activations stated as uint8, which the ONNX Runtime quantizer emits by default, are handled: uint8 value v and int8 value v - 128 denote the same real number under zero points that differ by the same 128, so the compiler restates them in the signed domain the kernels work in.

The recipe

import numpy as np
import onnx
from onnxruntime.quantization import (
    CalibrationMethod, QuantFormat, QuantType, quantize_static,
)

model = onnx.load("model.onnx")
graph_input = model.graph.input[0]
shape = [d.dim_value or 1 for d in graph_input.type.tensor_type.shape.dim]


class Calibration:
    """Feeds representative samples. Random data is fine for a mechanical
    check and wrong for anything you intend to deploy."""

    def __init__(self, samples):
        self.samples = iter(samples)

    def get_next(self):
        return next(self.samples, None)


quantize_static(
    "model.onnx",
    "model.int8.onnx",
    Calibration(representative_samples),
    quant_format=QuantFormat.QDQ,
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
    per_channel=True,
    calibrate_method=CalibrationMethod.MinMax,
)

Then compile as usual:

tigris analyze model.int8.onnx -m 64K

warning: input axis 0 (batch_size) is unset; using 1 (--input-shape overrides)
mobilenetv2-int8.onnx   int8, 67 operators
  input    input    1x3x224x224 float32, stored as int8 at scale 0.0186584, zero point -14
  output   output   1x1000 float32
fits 64.00 KiB fast memory, 58 stages, 53 tiled

memory
  unscheduled        1.44 MiB
  largest tensor     1.15 MiB   1x96x112x112
  this plan         63.00 KiB   1.00 KiB headroom
  slow memory        1.29 MiB
  slow traffic       5.07 MiB   per inference: 1.80 MiB written, 3.26 MiB read
  also fits at      32.00 KiB   64 stages, 62 tiled
  also fits at      16.00 KiB   64 stages, 63 tiled
  does not fit at    8.00 KiB

flash
  plan   3.39 MiB   weights 3.38 MiB, overhead 10.25 KiB

Note the input and output rows. The model declares float32 at both ends. The input is stored as int8 and converted at the boundary; the output is a float32 section result. An application hands over and reads back float32 either way. Quantizing does not change the interface your code talks to.

Per-channel or per-tensor

per_channel=True gives each output channel its own weight scale. It costs plan size and buys accuracy. Measured on MobileNetV2, calibrated identically both ways:

Weights	Plan	Quantization error against the float model
per-tensor	3.39 MiB	max 49.1 LSB, mean 45.9
per-channel	3.81 MiB	max 17.5 LSB, mean 13.8

Roughly a third of the error for 0.42 MiB. Take per-channel unless flash is the binding constraint. The activation memory does not change either way: both have a 1.44 MiB unscheduled peak and a 15.00 KiB floor, because only the weights are affected.

One catch: axis on DequantizeLinear arrived in opset 13, so per-channel quantization of an opset-12 model produces a file ONNX Runtime refuses to load, with Unrecognized attribute: axis for operator DequantizeLinear. Convert first:

from onnx import version_converter
onnx.save(version_converter.convert_version(model, 13), "model.opset13.onnx")

Settings that do not work, and how they fail

QuantFormat.QOperator writes integer operators such as QLinearMatMul instead of QDQ pairs. TiGrIS refuses them with the format named:

cannot compile: 1 unsupported operator
  quantized_matmul (QLinearMatMul: ONNX QOperator format; TiGrIS ingests QDQ. Re-export with quant_format=QuantFormat.QDQ)

quantize_dynamic computes activation scales at run time. Its DynamicQuantizeLinear and MatMulInteger operators receive the same QOperator-format refusal. Use static quantization with QDQ instead.

A float operator without a kernel is still refused. For example, a DequantizeLinear followed by InstanceNormalization gives:

cannot compile: 1 unsupported operator
  normalize (InstanceNormalization: InstanceNormalization has no kernel in the runtime)

Float activations outside a section still fail dtype validation. A separate float input feeding a Relu, alongside an int8 path, gives:

cannot compile: float32 activations in an int8 plan must follow a Dequantize, If or While: float_input, float_output

The list names the float32 tensors no Dequantize, If or While reaches. Check those paths rather than requiring the whole model to be int8. compile ends each refusal with Error: no plan written and exits with status 1.

Check the result

Quantization changes what the model computes, so measure rather than assume. Compare three things on the same inputs:

1. The float model under ONNX Runtime, which is the reference.
2. The quantized model under ONNX Runtime, which tells you what quantization cost you. This is the number that decides whether the model is still good enough, and it has nothing to do with TiGrIS.
3. The compiled plan, which tells you whether TiGrIS reproduces the quantized model.

On MobileNetV2 the third comparison lands well inside the second:

TiGrIS vs ONNX Runtime int8    : max  6.0 LSB, mean  5.0 LSB
ONNX Runtime int8 vs float     : max 17.5 LSB, mean 13.8 LSB

Rounding ties can differ between TiGrIS’s fixed-point kernels and ONNX Runtime’s QDQ evaluation. A one-LSB difference early in a network can be amplified by a later layer whose effective scale is large. A float section can carry that upstream difference into a float output.

If the third comparison is large while the second is small, that is a compiler problem and worth reporting. If the second is large, the model needs better calibration data or per-channel weights, and no amount of compiler work will recover it.

Calibration data

Everything above is mechanical. The one judgement call is the calibration set, which decides the scales and therefore the accuracy. It has to look like what the model will see when deployed: the same preprocessing, the same value range, the same kind of content. A few dozen representative samples is usually enough, and random noise is not representative of anything, however convenient it is for checking that the pipeline runs.