Understand the runtime paths you will compare

You will compare three CPU inference paths:

For many edge devices, it would not be possible to run a full PyTorch stack. You can on a Raspberry Pi 5, which allows you to compare and understand when you might consider moving from a full PyTorch stack, to using ExecuTorch.

The ExecuTorch workflow is split into three phases: program preparation, runtime preparation, and program execution. This module follows that same shape: prepare and lower the model on your development machine, copy the emitted .pte program to the Pi, then execute it from a small application. See the ExecuTorch architecture and components documentation for the full architecture.

ExecuTorch uses an ahead-of-time (AOT) workflow. This means that you prepare models ahead of use, on a separate, typically more compute-heavy, device. This stage includes lowering, where an exported PyTorch graph is converted into a lower-level representation that ExecuTorch can execute. By default, ExecuTorch can use what is called the portable kernel library. Portable kernels prioritize correctness and broad coverage, but are not necessarily the fastest CPU path.

An alternative is a backend delegate. These take supported subgraphs and hand them to a backend-specific execution path. XNNPACK is an example of a backend delegate for a CPU: a partitioner in ExecuTorch finds supported regions, lowers those regions, and leaves unsupported operators on the default ExecuTorch path. XNNPACK is optimized for Arm CPUs through KleidiAI.

KleidiAI is an open-source software library from Arm that provides highly optimized AI kernels for Arm CPUs. It accelerates common machine learning operations by taking advantage of Arm architecture features such as Neon and SME2, helping AI frameworks achieve better performance without requiring changes to model code.

For more detail on what the XNNPACK delegate supports and how it is used with XnnpackPartitioner, see the ExecuTorch XNNPACK backend documentation .

In this module, you will compare the portable path with XNNPACK delegation, as well as with PyTorch eager execution.

Once the model has been prepared in the AOT flow, you load it, along with your application, onto a target device. The model is used and executed by the application at runtime. This module uses Python to prepare inputs and drive the benchmarks, but the ExecuTorch inference engine is written in C++. The Python _load_for_executorch() binding loads the .pte program into the native C++ ExecuTorch runtime. Python remains responsible for tokenization, input shaping, output processing, and application behavior around the runtime call.

In summary:

• ExecuTorch is a deployment path, not simply “PyTorch but faster.”
• Portable ExecuTorch prioritizes broad operator coverage and deployability, not performance.
• XNNPACK can improve CPU inference when expensive supported regions delegate well.
• Unsupported operators remain on the default ExecuTorch path.
• Results depend on model structure, backend support, input shape, hardware, and runtime environment.