This content is for v0.15.0. Switch to the latest version for up-to-date documentation.
tigris analyze
Check whether a model fits a memory budget before you compile it, and see what
stops it when it does not. With --trace, run the plan once on the host runtime
and see what it did. For an ExecuTorch program, see how much RAM
tigris export saves it, without writing a file.
tigris analyze MODEL -m FAST [-m SLOW] [-f FLASH] [-v | --trace | --json]tigris analyze MODEL.pte [--cpu CORE] [--allow-input-overwrite] [--max-extra-macs PERCENT] [--json]With -m, analyze exits 1 when compile with the same options would write no plan, and 0
otherwise, so a build can use it as a fit check. Without -m, it exits 1 when compile would
write no plan at any fast budget. It also searches for the smallest fast budget that fits,
to 32 bytes, with the slow memory given or unbounded without one.
Options
Section titled “Options”| Flag | Type | Description |
|---|---|---|
MODEL |
path | ONNX or TFLite model, PyTorch program (.pt2, see PyTorch programs) or ExecuTorch program (.pte, see ExecuTorch programs) |
-m, --mem |
size, repeatable | Fast memory, then slow memory: -m 32K -m 1M or -m 32K+1M. Without -m, the model’s own figures and the smallest budget are reported. |
-f, --flash |
size | Flash for the plan; adds a flash verdict |
-v, --verbose |
flag | Adds the stages table, or every event of a trace |
--json |
flag | The analysis as a versioned JSON document |
--trace |
flag | Runs the plan on the host runtime and reports what it did instead |
--input |
path, repeatable | Input for --trace: .bin or .npy, NAME=FILE for several; zeros when omitted |
--input-shape |
NAME:1x3x224x224, repeatable |
Shape for an input the model leaves open; an open dimension is otherwise bound to 1 |
--int8-io |
flag | Analyze the plan with its quantized inputs and outputs declared int8, as compile --int8-io writes it |
--cpu |
core | ExecuTorch program only: the firmware’s Cortex-M core, as for export |
--allow-input-overwrite |
flag | ExecuTorch program only: let the rewrite reuse the model input’s memory once it has been read |
--max-extra-macs |
percent | ExecuTorch program only: limit on recomputed multiply-accumulates, 5% or 5 |
Sizes and the two memory pools are explained in the CLI overview.
The report
Section titled “The report”tigris analyze vww_96_int8.tflite -m 32K -m 1M -f 1Mvww_96_int8.tflite int8, 31 operators input input_1_int8 1x96x96x3 int8 at scale 0.00392157, zero point -128 output Identity_int8 1x2 int8 at scale 0.00390625, zero point -128fits 32.00 KiB fast memory, 8 stages, 7 tiled
memory TFLite Micro arena 72.00 KiB its planner and reduction scratch; persistent allocations and other kernel scratch excluded TFLite Micro export -24.69 KiB head 72.00 KiB -> 45.00 KiB, persistent +2.31 KiB, +0.25% MACs, reference kernels on arm-none-eabi unscheduled 54.00 KiB largest tensor 36.00 KiB 1x48x48x16 this plan 28.22 KiB 3.78 KiB headroom slow memory 45.00 KiB budget 1.00 MiB slow traffic 109.13 KiB per inference: 18.00 KiB written, 91.12 KiB read smallest budget 2.50 KiB with 1.00 MiB slow memory; 2.47 KiB does not fit
flash plan 239.32 KiB weights 213.93 KiB, overhead 25.39 KiB flash budget 1.00 MiB fitsVerdict
Section titled “Verdict”The first block names the model, its inputs and outputs in the axis order the
model declares, and the verdict. inspect on the compiled
plan shows the order the caller hands over:
| Verdict | Meaning |
|---|---|
fits <budget> fast memory, N stages, K tiled |
A plan can be compiled. A stage counts as tiled when it runs in tiles or as part of a chain. |
fits <budget> fast memory; slow memory X needed, Y given |
The fast memory is enough, the second -m is not. |
does not fit <budget> fast memory: N stages exceed it |
One line per stage follows, largest first, with the fast memory it still needs and why it cannot shrink. |
cannot compile: N unsupported operators |
The operators and reasons follow; --json lists all of them. |
cannot compile: REASON |
An output no operator computes, or an activation dtype the runtime does not run. |
no budget given; memory figures only |
No -m was given. |
A stage that does not fit names its reason:
| Reason | What sets the floor |
|---|---|
smallest tile |
The stage is already tiled as finely as its operators allow. |
smallest 2D tile |
The same, for a stage tiled in height and width. |
smallest tile of a chain of N stages |
A chain of stages whose tiles flow through fast memory together. |
runs untiled: NAME, ... |
The named operators cannot tile, so the stage needs its whole input and output at once. |
When a Model Does Not Fit works through what to change in each case.
Memory
Section titled “Memory”| Row | Meaning |
|---|---|
TFLite Micro arena |
For a single-graph TFLite model that tigris export can read, with its buffers inside the FlatBuffer, and whose tensor types TFLite Micro’s arena model knows: the arena head TFLite Micro’s own memory planner places for the same file, its tensors plus the scratch buffers of MEAN, SUM, REDUCE_MAX, REDUCE_MIN and REDUCE_ALL. Other kernels’ scratch and TFLite Micro’s persistent allocations come on top. |
TFLite Micro export |
How much TFLite Micro’s arena goes down when tigris export rewrites the model: the head before and after, the growth of the persistent allocations, the multiply-accumulates the rewrite adds, and last the kernels and target these figures are for, TFLite Micro’s reference kernels on arm-none-eabi. Shown with the TFLite Micro arena row, as none and the reason when export would write nothing, followed by the kernels and target when the reason is that no smaller arena was found. |
ExecuTorch model memory |
For a PyTorch program (.pt2): the planned buffer of ExecuTorch’s Cortex-M7 lowering of the same program, which holds its activations and CMSIS-NN scratch. ExecuTorch’s figure, read from the lowered program; not measured and the reason without executorch; see PyTorch programs. |
unscheduled |
The activation peak if the whole model ran as one stage. |
largest tensor |
The largest activation and its shape. |
this plan |
The fast memory the plan’s activations use, and the headroom left in the budget. A kernel library’s scratch, such as CMSIS-NN’s, comes on top in the fast arena at run time; Runtime Integration sizes it. |
slow memory |
What the tensors that cross stage boundaries need in the slow pool. |
slow traffic |
Bytes written to and read back from slow memory in one inference, measured by running the plan once on the host runtime. It reads not measured with the reason when there is no host runtime or when a While loop’s iteration count depends on the input values. It is absent when the plan does not fit the slow budget. |
smallest budget |
The smallest fast budget that fits, to 32 bytes, with the slow memory given or unbounded without one, and the budget 32 bytes below it, which does not fit. none when compile writes no plan at any fast budget, with the reason: the slow memory given, the flash budget, or the runtime loader. Each budget tried is a full compile. |
plan is the size compile writes, split into weights and overhead. with -c lz4
estimates the plan with compressed weights; it appears only where compile -c lz4
writes a plan at the given budgets, which also have to hold each stage’s
decompressed weights, and where it is at least 5% smaller than plan. as int8
estimates a float32 model’s plan after quantization. With -f, flash budget says whether the plan fits.
Runtime build
Section titled “Runtime build”A plan that needs more than a default runtime build holds, for example more tensors
or more outputs from one stage than the runtime’s limits allow by default, ends the
report with the -D options its runtime needs, as inspect
states them for a compiled plan:
runtime build -DTIGRIS_MAX_STAGE_OUTPUTS=19A runtime built without them refuses the plan. The row is absent when the defaults suffice. Runtime Integration lists the limits.
Stages
Section titled “Stages”-v adds one row per stage:
stages stage ops untiled peak in out tiling 0 0 45.00 KiB 1 1 chain of 7, tile height 2 1 1 36.00 KiB 1 1 in chain 0 2 2 54.00 KiB 1 1 in chain 0 3 3 45.00 KiB 1 1 in chain 0 4 4 27.00 KiB 1 1 in chain 0 5 5 36.00 KiB 1 1 in chain 0 6 6 36.00 KiB 1 1 in chain 0 7 7-30 22.50 KiB 1 1 untiledops are the operator indices the stage runs, untiled peak is what the stage
would need without tiling, and in and out count the tensors that enter and
leave the stage. tiling says how the stage runs: untiled, N tiles, axis A, halo K for tiles along one axis, N tiles of H x W, halo K for tiles in height
and width, chain of N, tile height H for the first stage of a chain, or
in chain C for the others.
Execution trace
Section titled “Execution trace”--trace compiles the plan in memory, runs it once on the host runtime bundled
with the package, and prints what the runtime did. The numbers are measured, not
predicted: the runtime reports every allocation, load, spill and tile as it
happens. It needs -m and a plan that fits the fast and the slow budget;
otherwise it stops with no plan to trace, and analyze without --trace says
why.
tigris analyze vww_96_int8.tflite -m 32K -m 1M --tracevww_96_int8.tflite traced on runtime 0.13.0, host reference backend, zero input moved 18.00 KiB written, 91.12 KiB read fast peak 28.22 KiB of 28.22 KiB
stage kind tiles read written fast peak slow used 0 chain 7 12 73.12 KiB 18.00 KiB 28.22 KiB 45.00 KiB 7 untiled 18.00 KiB 2 B 22.50 KiB 18.03 KiB
272 events; their bytes equal the runtime's own counters. Host alignment is 32 bytes; a target with less uses at most these bytes.moved is the traffic between fast and slow memory for one inference: bytes
written to slow memory and bytes read back. When the slow pool is external PSRAM,
this traffic costs time on every inference. fast peak is the highest fast memory
use, out of the arena the host runtime gives the plan: the size
tigris_fast_arena_required() returns. Each row of the table is one stage or
one chain of stages: how it ran, how many tiles, what it read and wrote, the
highest fast memory use and the slow memory in use. When weights are compressed,
a weights row gives the bytes decompressed into fast memory.
The last line checks the trace against the runtime’s own counters. If the two
disagree, a mismatch: line names the difference. The host aligns tensors to 32
bytes; a target with smaller alignment needs at most the memory shown.
With -v, every event follows, grouped by stage and by tile:
stage 0 chain 7 alloc model/activation_6/Relu;model... slow+27648 18.00 KiB alloc model/activation/Relu;model/b... fast+0 4.12 KiB ... tile 1 rows 2-24 alloc input_1_int8 fast+22272 6.47 KiB load input_1_int8 slow+576 fast+22272 6.47 KiB rows 2-24 move model/activation/Relu;model/b... fast+384 fast+0 2.62 KiB op Conv_3 ... spill model/activation_6/Relu;model... fast+20736 slow+29184 1.50 KiB rows 2-3 reset fast+22272| Event | Meaning |
|---|---|
alloc |
A tensor gets its place in fast or slow memory. |
load, spill |
A tensor, or the rows of it a tile needs, moves from slow to fast memory or back, with source and destination offsets. |
move |
Fast memory compacts, or a chain keeps the rows the next tile shares with this one. |
op |
One kernel call. |
reset |
A tile frees its fast memory. |
weights, copy |
Decompressed weights, and state or control-flow operands copied between buffers. |
The trace runs on zeros unless --input gives real data, in the formats
tigris run reads. The schedule is fixed at compile time, so
the input changes the trace only where the plan branches on data (If, While).
A While loop whose iteration count depends on the input values does not run on
zeros: --trace then needs --input.
--trace needs the host runtime, which platform wheels bundle; see
Installation.
--json prints one document with report: "tigris-analysis" and a version,
then these sections:
| Section | Holds |
|---|---|
model |
Name, format, dtype, interface, operator and tensor counts, unscheduled peak, largest tensor, unsupported operators |
tflite_micro |
TFLite Micro’s arena head, what it excludes, and export with what tigris export finds for arm-none-eabi; null for an ONNX model and for a TFLite model without the TFLite Micro arena row |
executorch |
For a .pt2: the ExecuTorch model memory row as arena_bytes, the cpu it is for, what it includes, and the reason when it was not measured; null for other models |
fast |
Budget, verdict, scheduled peak, stage counts, every blocking stage with its reason |
slow |
Budget, peak, whether it fits, measured traffic as written and read bytes or null |
flash |
Budget, weight and plan bytes, whether it fits, estimates |
runtime_build |
Each build limit the plan needs above its default, with name, needed and default; empty when the defaults suffice |
smallest_fit |
The smallest budget row as budget_bytes, below_bytes and slow_budget_bytes; budget_bytes is null when no fast budget fits, with reason memory, flash or loader, and the whole field is null when the model cannot compile |
stages |
One entry per stage |
Sizes are in bytes. A field that does not apply is null.
tigris analyze vww_96_int8.tflite -m 32K -m 1M --json "slow": { "budget_bytes": 1048576, "peak_bytes": 46080, "fits": true, "overflow_stages": [], "traffic": { "written": 18434, "read": 93312 } },--json and --trace are two different outputs; a command takes one of them.
PyTorch programs
Section titled “PyTorch programs”A .pt2 archive, as torch.export.save writes it, is read through
torch.onnx.export, so analyze, compile and inspect read it like the ONNX
file that export gives. Reading it needs torch and onnxscript;
Compiling a PyTorch Model installs both. The
memory section adds what ExecuTorch needs for the same program, measured when
executorch is installed as well:
tigris analyze mbv2.pt2 --int8-io -m 20Kmbv2.pt2 int8, 65 operators input x 1x3x96x96 int8 at scale 0.0317549, zero point 0 output dequantize_per_tensor_78 1x1000 int8 at scale 0.00531865, zero point 10fits 20.00 KiB fast memory, 17 stages, 15 tiled
memory ExecuTorch model memory 135.00 KiB planned buffer of its Cortex-M7 lowering: activations and CMSIS-NN scratch unscheduled 135.00 KiB largest tensor 108.00 KiB 1x48x48x48 this plan 18.38 KiB 1.62 KiB headroom slow memory 31.50 KiB slow traffic 219.23 KiB per inference: 68.48 KiB written, 150.75 KiB read smallest budget 3.31 KiB 3.28 KiB does not fit
flash plan 1.67 MiB weights 1.61 MiB, overhead 63.38 KiBanalyze lowers the program for Cortex-M7 with ExecuTorch’s Cortex-M backend
and reads the size of the planned buffer from the
result: the one buffer ExecuTorch runs the model in, activations and CMSIS-NN
scratch together. The TiGrIS rows below it count the plan’s activations: 18.38
KiB in fast memory and 31.50 KiB in slow memory for this plan, with CMSIS-NN’s
scratch, 5.00 KiB for this model, on top in the fast arena. When the row cannot
be measured it reads not measured and the reason, for example
executorch is not installed or its Cortex-M lowering failed: and the first
line of ExecuTorch’s error.
Each command exports the program again, which takes seconds; for a model you analyze often, the ONNX file the guide writes reads faster.
ExecuTorch programs
Section titled “ExecuTorch programs”For an ExecuTorch program lowered for Cortex-M, analyze runs the search of
tigris export and prints what it
finds, without writing a file:
tigris analyze mbv2.ptembv2.pte ExecuTorch program, 64 instructions, CMSIS-NN DSP kernels (Cortex-M4, M7, M33, M35P)
ExecuTorch RAM on Cortex-M, from the program as lowered 159.19 KiB model memory 135.00 KiB, method 23.75 KiB, temp 448 B after tigris export 67.22 KiB 2.37x less; model memory 37.50 KiB, method 29.28 KiB, temp 448 B; +3.67% MACs plus the runner's own input and output buffers, stack and staticsEach row is ExecuTorch’s total for the program, computed from the file: the
model memory in its planned buffer, the method allocator and the temp allocator,
as export defines them. after tigris export
adds the ratio to the program as lowered and the multiply-accumulates the bands
recompute. When export finds nothing smaller, the row repeats the total with
no smaller total found.
--cpu, --allow-input-overwrite and --max-extra-macs work as for export.
The budgets and the plan options apply to models tigris compiles, so -m with
a .pte stops with Error: -m applies to ONNX, TFLite and .pt2 models.
analyze exits with 1 for a program export refuses, with the same error, and
0 otherwise. --json prints report, version, model with format pte,
and executorch_export with the fields of
export --json.
Examples
Section titled “Examples”Will it fit, and how far can the budget go down:
tigris analyze model.tflite -m 64KOn-chip SRAM next to external PSRAM, with a flash check:
tigris analyze model.onnx -m 232K -m 6M -f 4MA stock ONNX export with a free batch dimension, at batch 4:
tigris analyze model.onnx -m 256K --input-shape input:4x3x224x224The per-stage table and a trace on a real input:
tigris analyze model.tflite -m 32K -vtigris analyze model.tflite -m 32K --trace --input sample.npy