Memory Model

TiGrIS uses a three-region memory model designed for embedded devices with heterogeneous memory: small fast SRAM, large slow PSRAM, and read-only flash. The compiler fixes the operator schedule, stage and tile strategy, and activation-memory bounds. The runtime then assigns addresses from caller-provided fast and slow arenas with bounded bump allocation, reset, and compaction. The core executor does not call a general-purpose heap allocator during inference; it can still report out-of-memory when either arena is too small.

PSRAM is required for any model that compiles to more than one stage. Intermediate tensors spill from SRAM to PSRAM between stages because flash is read-only and cannot serve as spill storage. Models that fit in a single stage can run with SRAM only.

Memory regions#

SRAM (fast arena)#

The primary working memory for inference. All activation tensors during op execution live here.

• Bump allocator, pointer advances forward with alignment padding.
• Reset per stage or per tile iteration.
• Size set by the -m flag (e.g., -m 256K).
• An optional fast_reserved prefix at the start of the arena holds decompressed weight blocks and survives arena resets.

PSRAM (slow buffer)#

External RAM for inter-stage tensor storage. Required for any multi-stage plan. Present on targets like ESP32-S3 (2-16 MB PSRAM depending on variant).

• Bump allocator, persistent across stages.
• Compacted after each stage: dead tensors are reclaimed and live tensors are shifted down.
• Stage inputs are loaded from here into the fast arena. Stage outputs are spilled from the fast arena to here.
• Model inputs are pre-allocated here by the caller before inference.
• Model outputs remain here after inference completes.

Flash (read-only, XIP)#

Model weights are stored in flash and can be accessed via execute-in-place (XIP) memory mapping. With --xip on an uncompressed plan, the runtime reads weights directly from their flash addresses instead of copying the full weight blob to RAM. With weight compression enabled, stage weight blocks are still stored in flash but are decompressed into the fast arena as each stage runs.

tigris compile model.onnx -m 256K -f 4M --xip -o model.tgrs

The -f flag validates that the compiled plan (weights + metadata) fits within the flash budget.

Executor working storage#

Executor bookkeeping is not a tensor-memory region and is not included in the compiled SRAM budget. New integrations provide one caller-owned tigris_executor_workspace_t to tigris_run_with_workspace(). Its bounded size is derived from the runtime’s compile-time plan limits; query sizeof(tigris_executor_workspace_t) or tigris_executor_workspace_size() for the current build. Keep it in static, global, or heap-backed storage unless its full size is intentionally included in the task-stack budget. The compatibility tigris_run() entry point instead uses one process-global, non-re-entrant workspace.

Arena allocation#

The fast arena is a contiguous block of SRAM. Allocation works as follows:

Arena layout:

Low addr                                 High addr
┌───────────┬──────────┬──────────┬──────────────┐
│ reserved  │ Tensor A │ Tensor B │ free space   │
│ (weights) │ (aligned)│ (aligned)│              │
└───────────┴──────────┴──────────┴──────────────┘
             ^                     ^
             arena_base            bump_ptr

• The bump pointer starts after fast_reserved and advances with each allocation.
• Each allocation is padded to TIGRIS_TENSOR_ALIGN bytes.
• On arena reset (between stages or tiles), the bump pointer rewinds to arena_base, but fast_reserved is preserved.

OOM handling#

For a normal stage, if an activation allocation exceeds the remaining fast arena space:

1. Compact: Scan for tensors whose last consumer has run. Reclaim their space by shifting subsequent live tensors down. Retry the allocation.
2. Overflow: If compaction is insufficient, try the slow pool (PSRAM). A successful fallback incurs latency from slow memory access.

This is a bounded fallback, not a guarantee of success. Slow-arena exhaustion, tiled or chain workspace exhaustion, compressed-weight workspace exhaustion, and backend scratch failures are reported to the caller. The core runtime does not fall back to an unbounded heap allocation.

Tensor lifetime management#

Intra-stage (within a stage)#

The executor derives each tensor’s last consumer from the compiled operator order. After that consumer executes, the tensor pointer is invalidated and its arena range becomes eligible for reclamation during compaction.

Inter-stage (between stages)#

• Stage outputs: After a stage completes, tensors consumed by later stages are spilled (copied) to the slow buffer.
• Stage inputs: Before a stage executes, tensors produced by earlier stages are loaded from the slow buffer into the fast arena.
• The fast arena is reset between stages, so all intra-stage temporaries are reclaimed automatically.

Special tensors#

• Model inputs: Allocated in the slow buffer by the caller before invoking the executor. The first stage loads them.
• Model outputs: Left in the slow buffer after the last stage. The caller reads them directly.
• Constants/weights: Reside in flash. Not allocated in either arena. Accessed by pointer.

Alignment#

Tensor addresses are aligned to TIGRIS_TENSOR_ALIGN bytes. The value is auto-detected at compile time based on the target architecture.

Override with -DTIGRIS_TENSOR_ALIGN=N at build time. Backend-specific scratch buffers (e.g., ESP-NN SIMD) may require stricter alignment and are handled separately by the backend adapter.

The .tgrs buffer has a related base-address contract. Optimized CMSIS-NN kernels can read uncompressed XIP weights directly from offsets relative to the plan base, so the plan buffer itself must be aligned to TIGRIS_TENSOR_ALIGN—16 bytes on DSP-enabled Cortex-M. Embedded arrays, linker sections, custom flash mappings, and loaded buffers must all preserve that base alignment.

Measured fast-arena peak#

tigris_mem_t.fast_peak is the high-water mark of fast_used since the last tigris_mem_init(). It includes ordinary fast-arena allocations and direct compressed-weight bumps made by the executor, so inspect it after an executor call to observe the measured core-arena demand. Account for compressed-weight overhead when comparing it with the compiler’s activation bound. Backend workspace reserved outside the usable arena must be counted separately.

Memory budget sizing#

The -m flag sets the SRAM activation budget used during compilation. Choosing the right budget involves a tradeoff:

A larger budget means fewer temporal partitions, less data movement between SRAM and PSRAM, and fewer tiled iterations. A smaller budget frees SRAM for other uses like DMA buffers, RTOS stacks, or backend scratch buffers.

Backend scratch buffers#

Accelerated kernel backends need setup before inference. Call exactly one backend preparation function after tigris_mem_init() and before the first executor call, and treat a nonzero result as fatal:

• tigris_esp_nn_prepare(&plan, &mem) scans the plan and obtains aligned ESP-NN workspace through the platform allocator during initialization.
• tigris_cmsis_nn_prepare(&plan, &mem) reserves one 16-byte-aligned CMSIS-NN scratch region from the top of the fast buffer and reduces mem.fast_size.

The reference float32 and int8 backends do not require a preparation call. After successful preparation, neither accelerated dispatch path allocates model-dependent workspace during inference.

Fast-buffer layout after CMSIS-NN preparation:

Low addr                                   High addr
┌───────────┬──────────────────┬──────────────────┐
│ reserved  │ activation space │ backend scratch  │
│ (weights) │ (bump allocator) │ (SIMD scratch)   │
└───────────┴──────────────────┴──────────────────┘

tigris_cmsis_nn_scratch_required() returns the exact aligned CMSIS-NN reservation, including vendor scratch and scalar-to-per-channel quantization expansion. tigris_cmsis_nn_fast_arena_required() combines it with the core activation and compressed-weight requirement, including alignment between the core arena and adapter reservation, so a tight static fast buffer can be sized before tigris_mem_init(). The buffer base must already satisfy TIGRIS_TENSOR_ALIGN. Both helpers return UINT32_MAX for invalid or unrepresentable input.

The sweet spot depends on the model and backend. Example from a large int8 model on ESP32-S3:

The balanced configuration is roughly 4x faster because ESP-NN’s SIMD kernels use the scratch space effectively, and the additional tiling overhead from a smaller arena is more than offset by faster kernel execution.

Sizing guideline#

1. Run tigris analyze model.onnx -m <budget> with different budgets to see the stage count and tiling plan.
2. Reserve enough SRAM for the backend’s scratch requirements.
3. Start with the smallest budget that keeps the stage count reasonable (under ~20 stages for most models), then increase if latency is too high.