About this course

Welcome to Optimize and Deploy AI Inference with ExecuTorch on Arm. In these hands-on sessions, you will explore how PyTorch models can be exported and lowered to the ExecuTorch runtime for efficient, lightweight inference at the edge.

You will work across multiple deployment targets, including:

• CPU-based inference using XNNPACK and Arm KleidiAI

• deployment to Arm Ethos-U Neural Processing Units (NPUs)

If terms such as XNNPACK, KleidiAI, delegation, TOSA, or Vela are new to you, don’t worry: each is introduced and explained the first time it is used in the relevant module. You do not need to know them before you begin.

Edge AI refers to deploying machine learning models on small, low-power devices with stricter constraints on memory, compute, and energy. Because almost every target has a CPU, inference can be performed on a wide range of devices.

Edge AI is not only about performance. It also enables:

• processing sensitive data locally rather than sending it to the cloud

• operating without network connectivity

• deploying AI on lower-cost, lower-power devices

PyTorch is a popular open-source machine learning framework for building and training deep learning models. Originally developed by Meta’s AI Research lab (FAIR) in 2016, PyTorch has become a cornerstone of AI research and production thanks to its flexible, Pythonic design. It offers dynamic computation graphs, known as eager execution, an intuitive API, and a vast ecosystem, including TorchVision, Hugging Face Transformers, and others that make it easy for engineers to prototype and train neural networks. In essence, PyTorch is widely used to develop models on powerful hardware, such as GPUs and CPUs, accelerating the path from research prototyping to production deployments. However, the full PyTorch stack may be too large or complex for some edge and embedded deployments, especially where memory, startup time, dependency footprint, and packaging constraints matter.

ExecuTorch is part of the PyTorch workflow and is described as “PyTorch’s unified solution for deploying AI models on-device, from smartphones to microcontrollers, built for privacy, performance, and portability.” It is an inference runtime and toolchain that takes a PyTorch model and prepares it for efficient on-device execution. ExecuTorch is part of the PyTorch deployment workflow. It uses PyTorch export and ExecuTorch lowering to transform a model into a .pte program, without requiring conversion through ONNX or LiteRT. The runtime can include only the kernels, backends, and platform features required by the application, making its binary much smaller than a full PyTorch installation. Its final size depends on the target and build configuration. This difference enables deployment in environments where:

• memory is limited

• startup time matters

• dependencies must be minimal

In many edge scenarios, this determines whether deployment is possible at all.

A key idea you will meet throughout the course is delegation. When ExecuTorch prepares a model, it can hand supported parts of that model to a backend: a component optimized for particular hardware. Anything a backend does not support remains on the default ExecuTorch path. It can run only if the target runtime includes a compatible kernel. On Arm CPUs, the course uses the XNNPACK backend, a highly optimized library of neural-network operations, which in turn can call KleidiAI, Arm’s low-level routines that accelerate AI on Arm CPUs. Later, Module 4 delegates to an Arm Ethos-U NPU instead of a CPU. Each of these terms is explained again in context when you first use it.

Course modules

Module 0: Course overview and ExecuTorch deployment concepts

Learn what the course covers, how ExecuTorch fits into edge AI, and how to approach the resource.

Module 1: Deploy and compare transformer inference on Arm CPUs

Work with two transformer language models: GPT-2, which generates text, and DistilBERT, which predicts a masked (hidden) word in a sentence. Lower them to the ExecuTorch Portable Kernel Library and XNNPACK, inspect their graphs, benchmark them on a Raspberry Pi 5, and run a text-generation application (GPT-2) and an autocomplete application that suggests the next word as you type (DistilBERT).

Module 2: Optimize CNN inference with INT8 quantization

Prepare MobileNetV2 for image classification, delegate it to XNNPACK, compare PyTorch and FP32 ExecuTorch performance, then use post-training quantization to create and validate an INT8 image-classification application. An optional section shows how to select and integrate a pre-optimized YOLO object detection model from the Arm AI Portal. A second optional section uses ETRecord, ETDump, ExecuTorch Inspector, and Model Explorer to inspect runtime timing.

Module 3: Deploy INT8 semantic segmentation in an Android app

Work with LR-ASPP MobileNetV3, a semantic segmentation model that labels every pixel in an image with the class of object it belongs to (for example, separating a person or a dog from the background). Prepare FP32 and INT8 programs with XNNPACK, validate their segmentation output, and package the INT8 model in an Android object-cutout application, which uses those per-pixel labels to isolate the main subject and remove the background. Use the ExecuTorch Java API to load the model, create tensors, run inference, and manage native runtime resources.

Module 4: Deploy MobileNetV2 with Arm Ethos-U acceleration

Return to MobileNetV2 for image classification, but this time target an Arm Ethos-U NPU instead of a CPU. Quantize the model, lower supported regions through the TOSA intermediate representation and compile for the NPU with Vela. Run the program on a Corstone-320 Fixed Virtual Platform (FVP, a software simulation of the target hardware), and inspect the compiled graph and FVP output.

The modules are designed to be completed in order. Later modules reuse concepts from earlier ones, including .pte artifacts, model contracts, backend delegation, fallback work, and target validation.

What you need to complete the course

Use macOS (Apple silicon) or Ubuntu for the development-machine steps in Modules 1-3. Ubuntu under WSL is also supported. If you use WSL for Module 3, run the Android build and device commands from Windows. Module 4 requires an Ubuntu LTS host.

You should have basic Python experience and a working understanding of neural-network inference. Prior Java, Kotlin, or embedded C++ experience is not required. If you don’t have every target device, you can still run the ahead-of-time model-preparation steps and review the target-side commands and application code.