Skip to content
v0.15.0

This content is for v0.15.0. Switch to the latest version for up-to-date documentation.

tigris run

Execute a model on your own machine, with no board attached. The file picks the runtime:

File Runs on Needs
.tgrs plan The TiGrIS runtime bundled in tigris-ml, on its reference kernels A platform wheel, or your own host library
TFLite model TFLite Micro’s reference kernels The tflite-micro package
ExecuTorch program (.pte) lowered for Cortex-M ExecuTorch with its cortex_m operators on CMSIS-NN’s portable C kernels --executorch, an ExecuTorch 1.6 checkout

A plan runs the same runtime code a device runs on its reference kernels. A TFLite model and an ExecuTorch program run on the runtime tigris export writes for, so you can run a model and its export on the same input and compare the outputs byte for byte.

Terminal window
tigris run MODEL --input FILE --output FILE [--executorch PATH] [--json]
Flag Type Description
MODEL path Compiled .tgrs plan, TFLite model or ExecuTorch program
--input FILE or NAME=FILE, repeatable, required Input tensor as .bin or .npy; name each file with NAME=FILE when the model has several inputs
--output path, required .bin or .npy for one output tensor, .npz for named outputs; the file must not exist
--executorch PATH path ExecuTorch programs: an ExecuTorch 1.6 checkout with its submodules; the host runner is built from it once and cached
--json flag Execution metadata as JSON

The format is read from the file contents, so the extension does not matter. --executorch with a plan or a TFLite model is a usage error. An ONNX model or a PyTorch .pt2 program runs as a plan: compile it with tigris compile first, as run says:

Error: model.onnx is an ONNX model; compile it with tigris compile and run the plan

Tensors use the axis order and dtype that tigris inspect reports for the file: for a plan, the stored axis order and the declared interface dtype. A .bin file holds the contiguous little-endian elements; a .npy file carries that shape and dtype. An ExecuTorch program also takes a .npy array in its PyTorch sizes, see Input layout. run does no preprocessing: images, audio or CSV data have to be converted to the model’s input tensor first.

A zoo download ships an example input and the matching reference output:

Terminal window
tigris zoo fetch electricity-hourly -o downloaded-model
tigris run downloaded-model/model.tgrs \
--input downloaded-model/example-input.bin --output prediction.bin
model.tgrs runtime 0.13.0, bundled, host reference backend
fast arena peak 768 B measured; plan and workspace excluded
slow arena peak 0 B
wrote prediction.bin 1x24 float32

example-output.bin is the ONNX Runtime output for the example input, so compare numerically, not byte for byte. Float results differ in the last bits:

import numpy as np
prediction = np.fromfile("prediction.bin", dtype="<f4")
reference = np.fromfile("downloaded-model/example-output.bin", dtype="<f4")
print(np.abs(prediction - reference).max()) # 1.2e-07 for electricity-hourly

The tolerance each zoo model is validated against is recorded in its evaluation.json.

When the plan sits next to a zoo manifest.json that lists the plan’s file name, run also checks the plan’s checksum and that the loaded runtime, bundled or selected with TIGRIS_HOST_LIBRARY, is within the model’s runtime range. The range comes from download.json when present, whose artifact ID must match the manifest, and from the manifest otherwise. run refuses to execute if a check fails.

run loads a TFLite model into TFLite Micro’s Python interpreter, from the tflite-micro package, which has wheels for Linux x86-64:

Terminal window
pip install tflite-micro
tigris run vww_96_int8.tflite --input image.bin --output prediction.bin
vww_96_int8.tflite TFLite Micro 0.dev20261023044614, reference kernels
wrote prediction.bin 1x2 int8

The first line names the tflite-micro release. run reports no memory figures: the arena a model takes on a device is what tigris export reports for each target. run sizes TFLite Micro’s arena with the same arena model, for the 64-bit build in the package, plus what TFLite Micro takes while it plans the arena or while a kernel prepares. A model outside that arena model, for example one with several subgraphs, an LSTM or a custom operator, gets an upper bound instead: every tensor and scratch buffer of every subgraph placed at once, plus every persistent allocation. A kernel the package lacks stops TFLite Micro while it loads the model, and its own message follows TFLite Micro cannot run MODEL::

Error: TFLite Micro cannot run float_gelu_approximate.tflite: Didn't find op for builtin opcode 'GELU'

run takes a program lowered by ExecuTorch’s Cortex-M backend and runs its method forward on a host build of ExecuTorch 1.6:

Terminal window
tigris run mbv2.pte --executorch ~/executorch --input image.bin --output prediction.bin
mbv2.pte ExecuTorch 1.6.0 at 4776f71b6f0d, CMSIS-NN v8.0.0 portable kernels
wrote prediction.bin 1x1000 int8

The first line names the ExecuTorch release and commit the runner was built from and the CMSIS-NN release it links. run reports no memory figures: the RAM the program takes on a 32-bit Cortex-M target is what tigris inspect shows.

On the first run against a checkout, run builds a host runner, the ExecuTorch runtime with its cortex_m operators on CMSIS-NN, and says so:

building the ExecuTorch host runner at 4776f71b6f0d, once; it is kept in ~/.cache/tigris/pte-runner

The runner stays in the user’s cache directory, on Linux $XDG_CACHE_HOME/tigris/pte-runner or ~/.cache/tigris/pte-runner, and run builds it again only for another commit or other uncommitted changes in the checkout, or for a tigris-ml release that changes the runner. The build needs CMake, a C++ compiler, network access, since ExecuTorch’s Cortex-M backend fetches CMSIS-NN from GitHub, and torch and PyYAML in the Python that runs tigris, for ExecuTorch’s code generator. A checkout of the release tag, in a directory named executorch, works:

Terminal window
git clone --branch v1.6.0 https://github.com/pytorch/executorch.git ~/executorch
git -C ~/executorch submodule update --init --recursive

The host runs CMSIS-NN’s portable C kernels. run takes programs lowered for scalar kernels (Cortex-M0, M0+, M3, M23) or DSP kernels (Cortex-M4, M7, M33, M35P). CMSIS-NN’s DSP kernels write the same bytes as its portable C kernels, so the host reproduces the outputs of a DSP target: a ResNet-8 program gave the same 200 output bytes for 20 inputs on the host and on a NUCLEO-H753ZI. The DSP code itself runs only on the target. A program lowered for MVE kernels runs on its Cortex-M55 or M85 target:

Error: bands_m55.pte was lowered for MVE kernels (Cortex-M55, M85); the host runs programs lowered for scalar or DSP kernels

A program that calls an operator the runner does not register stops with ExecuTorch’s own message after the host runner stopped on MODEL:.

The method takes one tensor. inspect shows its shape in memory order, with its dim order when that differs from PyTorch’s order:

input 0 1x96x96x3 int8, dim order 0,2,3,1

The Cortex-M backend keeps this 1x3x96x96 PyTorch input channels last, so its bytes in memory run as 1x96x96x3. A .bin file holds the bytes in that order. A .npy array can take either view:

.npy shape Read as
The PyTorch sizes, 1x3x96x96, as torch prints the tensor Laid out in the dim order
The sizes in memory order, 1x96x96x3, or one dimension Taken as it is

Dimensions of size 1 are ignored. An array whose shape reads both ways with different bytes is refused; give that input as a .bin file or a one-dimensional .npy. Messages name a tensor as inspect shows it, its shape in memory order and its dim order:

Error: short.bin holds 100 bytes; the method input takes 27648 (1x96x96x3 int8, dim order 0,2,3,1)

A .bin output holds the output bytes in memory order, a .npy output the array in its PyTorch sizes, and an .npz output one array per method output, named 0, 1 and so on.

Terminal window
tigris run model.tgrs --input input.bin --output prediction.bin --json
{
"runtime_version": "0.13.0",
"runtime_source": "bundled",
"backend": "reference",
"output": "prediction.bin",
"memory": {
"fast_capacity_bytes": 1024,
"slow_capacity_bytes": 1664,
"fast_peak_bytes": 768,
"slow_peak_bytes": 0,
"executor_workspace_bytes": 63
},
"tensors": [
{
"name": "output",
"dtype": "float32",
"shape": [
1,
24
]
}
]
}

For a plan, runtime_source is bundled, or TIGRIS_HOST_LIBRARY= followed by the resolved absolute path of the library that was loaded, for example TIGRIS_HOST_LIBRARY=/path/to/libtigris_host.so. The arena peaks are what the runtime used in this run. They exclude the plan itself, the executor workspace and the Python process.

For a TFLite model and an ExecuTorch program the document has no memory and adds runtime, tflite-micro or executorch. A TFLite model’s document has no runtime_source. An ExecuTorch program’s names the runner’s path in runtime_source, adds runtime_commit, runtime_changes (the hash of uncommitted changes in the checkout, or null) and cmsis_nn_version, has backend cmsis-nn, and each tensor carries its dim_order.

The same runtime is available as tigris.runtime.Session:

import numpy as np
from tigris.runtime import Session
with Session("model.tgrs") as session:
outputs = session.run({"input": np.load("input.npy", allow_pickle=False)})
print(session.runtime_version, session.memory)

run takes a dict of arrays keyed by input name and returns a dict keyed by output name. session.inputs and session.outputs describe the interface. Calls on one session are serialized; separate sessions run independently. Session(model, slow_capacity=N) sets the slow arena size in bytes; the default is a conservative host allocation, not a compiled requirement.

run executes a plan on the portable float32 and int8 reference kernels, a TFLite model on TFLite Micro’s reference kernels and an ExecuTorch program on CMSIS-NN’s portable C kernels. A successful run shows that the file loads and executes on that path. A model and its export that write the same bytes on an input compute the same result for that input on that path. For a plan, correctness takes a numerical comparison against a reference, as above. Parity results recorded for a zoo model apply to the runtime releases it was tested with, not to every compatible release. On-device latency, memory placement and the ESP-NN and CMSIS-NN optimized kernels are measured on the target, with the target backends and the tutorials.

The host library ships in the platform wheels for Linux x86-64 and ARM64 (glibc 2.28 or newer), macOS 11 or newer on x86-64 and ARM64, and Windows x86-64. Elsewhere pip installs the pure Python wheel, where every other command works and run on a plan stops with “Bundled host runtime is unavailable”. run on a TFLite model or an ExecuTorch program does not use the host library, so it also works in the pure Python wheel.

On any platform the runtime compiles on, build the host library yourself and point TIGRIS_HOST_LIBRARY at it. This also works with the pure Python wheel.

On Linux and macOS:

Terminal window
git clone --branch vX.Y.Z https://github.com/raws-labs/tigris-runtime.git
cd tigris-runtime
cmake -S . -B build-host -DTIGRIS_BUILD_HOST=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-host --target tigris_host
export TIGRIS_HOST_LIBRARY=$PWD/build-host/host/libtigris_host.so # .dylib on macOS
tigris --version

On Windows with the Visual Studio generator, the configuration is chosen at build time and the DLL lands in a configuration subdirectory (PowerShell):

Terminal window
git clone --branch vX.Y.Z https://github.com/raws-labs/tigris-runtime.git
cd tigris-runtime
cmake -S . -B build-host -DTIGRIS_BUILD_HOST=ON
cmake --build build-host --config Release --target tigris_host
$env:TIGRIS_HOST_LIBRARY = "$PWD\build-host\host\Release\tigris_host.dll"
tigris --version
tigris, version X.Y.Z (runtime X.Y.Z; TIGRIS_HOST_LIBRARY=/path/to/libtigris_host.so)

Use the runtime release that matches your tigris-ml version. The library must implement the host ABI tigris-ml expects; a mismatch is refused. When the variable is set, run on a plan uses that library or fails with an error naming it; it never falls back to the bundled one. tigris --version, the first line of run output and run --json all show which library ran. See Host Library for the C API.