Skip to content
develop

Compiling a PyTorch Model

This page quantizes a PyTorch model with PT2E and ExecuTorch’s Cortex-M quantizer. The quantized program then goes one of two ways:

  • Compile it for the TiGrIS runtime, from the ONNX file torch.onnx.export writes or from the .pt2 archive torch.export.save writes.
  • Keep ExecuTorch: lower the program for Cortex-M with ExecuTorch and let tigris export rewrite the .pte into row bands, so the same ExecuTorch build runs it in less RAM.

On a device compares the two on a Cortex-M7. A float model can also be exported with torch.onnx.export and quantized as an ONNX file with ONNX Runtime, as Quantizing a Model to INT8 describes.

The exported ONNX carries int8 weights, per-channel for convolutions and per-tensor for fully connected layers, int32 biases and QuantizeLinear/DequantizeLinear pairs, which the compiler reads like any other QDQ model. No conversion tool and no graph rewrite sits in between.

Terminal window
pip install executorch==1.6.0 tosa-tools==2026.5.0 onnxscript tigris-ml --extra-index-url https://download.pytorch.org/whl/cpu

PyTorch’s index, added as an extra index, gives the CPU-only builds of executorch 1.6.0, the torch 2.15.0 it pins and torchao 0.18.0, without the CUDA runtime the PyPI wheels install on Linux; the other packages come from PyPI. ExecuTorch’s Cortex-M quantizer imports Arm’s TOSA serializer, which the package installs only with its Ethos-U and VGF extras; without tosa-tools the import fails with ModuleNotFoundError: No module named 'tosa_serializer'. onnxscript is what torch.onnx.export(..., dynamo=True) needs. These versions are the ones this page was checked with. The recipe below takes MobileNetV2 from torchvision; add it to the same line to run the recipe as written.

import torch
import torchvision
from torch.export import export
from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e
from executorch.backends.cortex_m.quantizer.quantizer import CortexMQuantizer
model = torchvision.models.mobilenet_v2(weights="IMAGENET1K_V1").eval()
def channels_last(x):
return x.to(memory_format=torch.channels_last)
example = (channels_last(torch.randn(1, 3, 224, 224)),)
graph = export(model, example).module(check_guards=False)
graph = prepare_pt2e(graph, CortexMQuantizer())
for x in representative_samples: # NCHW float tensors, preprocessed as deployed
graph(channels_last(x))
graph = convert_pt2e(graph)
onnx_program = torch.onnx.export(export(graph, example), example, dynamo=True)
onnx_program.save("mobilenet_v2.int8.onnx")

Two settings decide whether this produces an int8 model:

  • Channels-last inputs. The quantizer annotates the convolutions only when the example and calibration inputs are channels-last. With contiguous inputs it leaves them in float, and the compiler refuses the model. For the DS-CNN in the table below:

    cannot compile: 2 unsupported operators
    node_conv2d (Conv: an int8 Conv needs an int32 bias)
    node_conv2d_8 (Conv: an int8 Conv needs an int32 bias)
  • check_guards=False. By default ExportedProgram.module() adds a guard submodule, and the quantizer’s preparation pass stops on it with call_module is not supported.

The ONNX keeps the model’s NCHW input shape, and analyze prints the input in that order, 1x3x224x224. The plan stores it channels-last, 1x224x224x3, as tigris inspect shows, and tigris run and the runtime take it in that order.

Terminal window
tigris analyze mobilenet_v2.int8.onnx --int8-io -m 256K
tigris compile mobilenet_v2.int8.onnx --int8-io -m 256K -o mobilenet_v2.tgrs
mobilenet_v2.int8.onnx int8, 65 operators
input x 1x3x224x224 int8 at scale 0.0311183, zero point -1
output dequantize_per_tensor_78 1x1000 int8 at scale 0.0586708, zero point 6
fits 256.00 KiB fast memory, 17 stages, 15 tiled
memory
unscheduled 1.44 MiB
largest tensor 1.15 MiB 1x96x112x112
this plan 247.62 KiB 8.38 KiB headroom
slow memory 343.00 KiB
slow traffic 1.84 MiB per inference: 784.98 KiB written, 1.08 MiB read
smallest budget 9.38 KiB 9.34 KiB does not fit
flash
plan 3.52 MiB weights 3.38 MiB, overhead 141.69 KiB

The quantized program starts with a quantize and ends with a dequantize, so the export declares float32 at both ends. Compiled without the option, the application hands over float32 and the runtime quantizes it by ONNX’s rule: a division by the scale, rounded with ties to even. ExecuTorch’s scalar Cortex-M kernel multiplies by the reciprocal of the scale instead, and its MVE kernel rounds that product differently again. Over 200 million values at 2,000 scales, ONNX’s rule and the scalar kernel gave a different int8 value 97 times. With --int8-io the application quantizes with the rule its own pipeline uses, and the plan starts from the same int8 values. The output name comes from the exporter, here the last dequantize node.

The converted program can also go to the compiler as it is. Save it, and tigris exports it to ONNX itself:

torch.export.save(export(graph, example), "mobilenet_v2.pt2")
Terminal window
tigris compile mobilenet_v2.pt2 --int8-io -m 256K -o mobilenet_v2.tgrs

For a .pt2, analyze and compile add the ExecuTorch model memory row: the planned buffer of ExecuTorch’s Cortex-M7 lowering of the same program, measured when executorch is installed.

Compare on int8 inputs, so that every reference starts from the same values. The reference is ExecuTorch’s Cortex-M lowering of the same converted program, run on the host implementations of its operators. These perform CMSIS-NN’s integer requantization, as the TiGrIS int8 kernels do. On 20 random int8 inputs per model, each plan compiled untiled and at the smallest budget analyze reports:

Model Weights Smallest budget Plan against the ExecuTorch program, CMSIS-NN pooling Plan against the converted PT2E model
DS-CNN, 1x49x10 random 640 B 0 of 240 values differ 94 of 240, max 2 LSB
ResNet-8, 3x32x32 random 640 B 0 of 200 73 of 200, max 2 LSB
MobileNetV2 0.35, 3x96x96 random 3.31 KiB 0 of 20,000 18,694 of 20,000, max 37 LSB
MobileNetV2, 3x224x224 ImageNet 9.38 KiB 0 of 20,000 12,794 of 20,000, max 5 LSB

The untiled plan and the smallest-budget plan gave the same values on every model. The reference computes each global average pool by CMSIS-NN’s arm_avgpool_s8 rule, the integer window sum divided by the count and rounded half away from zero. ExecuTorch’s host implementation pools in float instead; that changes the reference only for ResNet-8, where 11 of 200 values then differ by 1 LSB.

The converted PT2E model computes the requantization in float, so it differs from both the plan and the ExecuTorch program. That difference belongs to the reference, not to the compiler. What quantization costs against the float model is a separate measurement, as in Quantizing a Model to INT8.

These are host runs; On a device runs ExecuTorch and the TiGrIS runtime on a Cortex-M7.

To stay on ExecuTorch, lower the converted program for the core of your firmware, write the .pte, and rewrite it with tigris export. Continuing the recipe:

from executorch.backends.cortex_m.edge_compile_config import cortex_m_edge_compile_config
from executorch.backends.cortex_m.passes.cortex_m_pass_manager import CortexMPassManager
from executorch.backends.cortex_m.target_config import CortexM, CortexMTargetConfig
from executorch.exir import ExecutorchBackendConfig, to_edge
from executorch.exir.passes.quantize_io_pass import QuantizeInputs, QuantizeOutputs
edge = to_edge(export(graph, example), compile_config=cortex_m_edge_compile_config())
edge = edge.transform(passes=[QuantizeInputs(edge, [0]), QuantizeOutputs(edge, [0])])
edge = edge.transform(CortexMPassManager(target_config=CortexMTargetConfig(cpu=CortexM.M7)))
with open("mobilenet_v2.pte", "wb") as f:
f.write(edge.to_executorch(ExecutorchBackendConfig()).buffer)

QuantizeInputs and QuantizeOutputs make the program take and return int8 tensors, as --int8-io does for a plan; tigris export rewrites programs with int8 inputs and outputs. CortexMTargetConfig names the core, and ExecuTorch sizes each CMSIS-NN scratch buffer for that core’s kernels. Export keeps every band on those kernels.

For the MobileNetV2 0.35 at 96x96 of the table above, this lowering writes a program of 64 kernel calls, here mbv2.pte:

Terminal window
tigris export mbv2.pte --header mbv2_pools.h
mbv2.pte ExecuTorch program, 64 instructions, CMSIS-NN DSP kernels (Cortex-M4, M7, M33, M35P)
rewrite
instructions 0-2 8 row bands, output in place
instructions 3-5 8 row bands, output in place
instructions 6-8 2 row bands
extra MACs +3.67%
instructions after 109
ExecuTorch RAM on Cortex-M, from the program
model memory 37.50 KiB 135.00 KiB before 3.60x less; activations and CMSIS-NN scratch
method allocator 29.28 KiB 23.75 KiB before ExecuTorch's records for the method
temp allocator 448 B 448 B before needed to load the method
total 67.22 KiB 159.19 KiB before 2.37x less
plus the runner's own input and output buffers, stack and statics
model input memory reused once read, as in the original
wrote mbv2_export.pte 1.70 MiB +6.38 KiB

The firmware loads mbv2_export.pte as it loaded the original, with its planned buffer, method allocator and temp allocator sized from mbv2_pools.h. tigris export explains each row, the limits it takes and the header.

tigris run runs a program on the host, on ExecuTorch built from a 1.6 checkout. Run both programs on the same input and compare the outputs byte for byte:

Terminal window
tigris run mbv2.pte --executorch ~/executorch --input x.bin --output a.bin
tigris run mbv2_export.pte --executorch ~/executorch --input x.bin --output b.bin
cmp a.bin b.bin

x.bin holds the int8 input in memory order, 1x96x96x3, as tigris inspect mbv2.pte shows it; a .npy of the 1x3x96x96 tensor the program takes in PyTorch works too. For DS-CNN, ResNet-8, MobileNetV2 0.35 and MobileNetV2 at 224x224 of the table above, lowered for Cortex-M7, both programs write the same bytes on each of 5 random inputs.

DS-CNN, ResNet-8 and MobileNetV2 0.35 at 96x96 from the table above ran on a NUCLEO-H753ZI (Cortex-M7 at 480 MHz) three ways: on ExecuTorch 1.6.0, the program as lowered and the program tigris export wrote, each lowered for Cortex-M7 with int8 input and output; and on the TiGrIS runtime, a plan compiled from the same .pt2 with --int8-io and generated with codegen --backend cmsis-nn. All link CMSIS-NN v8.0.0. All three gave the same int8 outputs on every input, 0 of 240, 0 of 200 and 0 of 1,000 bytes differing, and the same outputs as ExecuTorch’s runtime on the host.

Model memory, the activations and kernel scratch of one inference. For ExecuTorch, the planned buffer: the program figure, which the device writes in full. For TiGrIS, the fast arena the CMSIS-NN backend requires, the plan’s activations plus CMSIS-NN scratch, and the slow arena: the plan figures the firmware’s arenas were sized to.

Model ExecuTorch as lowered ExecuTorch after tigris export TiGrIS runtime
DS-CNN 16,000 B 12,160 B, 1.32x less 16,256 B at -m 16K
ResNet-8 49,728 B 24,192 B, 2.06x less 51,456 B at -m 48K
MobileNetV2 0.35, 96x96 138,240 B 38,400 B, 3.60x less 56,192 B at -m 20K: 23,936 B fast, 32,256 B slow

Runtime bookkeeping, what each runtime keeps besides the model memory. For ExecuTorch, the method allocator (measured on the device, equal to the program figure), the temp allocator pool (the program figure, the capacity the runtime needs to resolve kernel arguments from it) and the runtime’s statics (linker map), which include its kernel registry: 2,000 slots, 24,000 B, by default, or 612 B with ExecuTorch built with MAX_KERNEL_NUM=51, the 51 kernels these firmwares register (28 prim ops and the 23 cortex_m operators). For TiGrIS, the tensor table, the executor workspace and the runtime’s statics (linker map).

Model ExecuTorch registry ExecuTorch as lowered ExecuTorch after tigris export TiGrIS runtime
DS-CNN 2,000 slots 30,112 B 32,952 B 279 B
DS-CNN 51 slots 6,724 B 9,564 B 279 B
ResNet-8 2,000 slots 30,576 B 35,024 B 327 B
ResNet-8 51 slots 7,188 B 11,636 B 327 B
MobileNetV2 0.35, 96x96 2,000 slots 49,576 B 55,236 B 1,867 B
MobileNetV2 0.35, 96x96 51 slots 26,188 B 31,848 B 1,867 B

Firmware RAM total: .data and .bss from the linker map plus the heap used and the stack high-water mark measured on the device. It is the model memory, the runtime bookkeeping and the measuring firmware’s own C library, heap and stack, 3,688 to 3,700 B with ExecuTorch and 3,921 to 4,317 B with TiGrIS.

Model ExecuTorch registry ExecuTorch as lowered ExecuTorch after tigris export TiGrIS runtime
DS-CNN 2,000 slots 49,800 B 48,800 B 20,456 B
DS-CNN 51 slots 26,412 B 25,412 B 20,456 B
ResNet-8 2,000 slots 84,004 B 62,916 B 55,720 B
ResNet-8 51 slots 60,616 B 39,528 B 55,720 B
MobileNetV2 0.35, 96x96 2,000 slots 191,504 B 97,324 B 62,376 B
MobileNetV2 0.35, 96x96 51 slots 168,116 B 73,936 B 62,376 B

ExecuTorch after tigris export holds the least model memory on all three models. The firmware total adds each runtime’s bookkeeping, which is where the two runtimes differ most. With ExecuTorch’s registry sized to its kernels, ExecuTorch after tigris export needs the least RAM in total for ResNet-8, 39,528 B against 55,720 B for TiGrIS, and TiGrIS needs the least for DS-CNN, 20,456 B against 25,412 B, and for MobileNetV2, 62,376 B against 73,936 B. With the default registry, TiGrIS needs the least for all three.

For each model the TiGrIS row is the plan with the smallest firmware total among the budgets measured: DS-CNN and ResNet-8 run as one stage, MobileNetV2 runs tiled. The H753’s SRAM holds both TiGrIS arenas; on a part with a small fast SRAM next to a larger memory, the tiled plan keeps 23,936 B in the fast one. tigris export gives the method allocator and the time per inference for the two ExecuTorch programs.