Profile ONNX model performance with SME2 using KleidiAI and ONNX Runtime
Introduction
ONNX Runtime architecture with SME2 acceleration
Integration of KleidiAI to ORT MLAS
Build ONNX Runtime with KleidiAI and SME2 for Android
Profile ONNX model performance
Next Steps
Profile ONNX model performance with SME2 using KleidiAI and ONNX Runtime
Who is this for?
This is an advanced topic for software developers, performance engineers, and AI practitioners.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Build ONNX Runtime with KleidiAI and SME2 support for Android
- Profile ONNX model performance using benchmark tools
- Analyze how KleidiAI kernels accelerate ONNX operators with SME2
- Compare performance improvements between standard and SME2-optimized execution
Prerequisites
Before starting, you will need the following:
- An Android device with Arm SME2 support
- Basic understanding of machine learning model inference
- Familiarity with Android NDK and cross-compilation
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll build ONNX Runtime for Android with KleidiAI SME2 microkernels and profile a model on a device. First, you’ll cross-compile with the Android NDK, deploy the binaries and model, and use
onnxruntime_perf_test to capture baseline and SME2 results. Then, you’ll compare KleidiAI dispatch, operator execution, and end-to-end inference time with ResNet-50 v2.Frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
MLAS checks CPU capabilities at runtime and dispatches to KleidiAI when SME2 is present. When enabled, GEMM and convolution operators use ArmKleidiAI kernels instead of the default MLAS paths.
Use ONNX Runtime v1.23.2.
Use Android NDK r26b or later. NDK r27 or later is recommended for the latest SME2 toolchain support.
Copy the archive to
/data/local/tmp and extract it there.Verify the Android device supports SME2, because MLAS enables KleidiAI only when SME2 is detected. If SME2 is unavailable, MLAS falls back to its default kernels (for example, Neon), and you can still profile that baseline.