Measure LLM inference performance with KleidiAI and SME2 on Android
Introduction
Understand how SME2 and KleidiAI accelerate LLM inference in llama.cpp
Trace how KleidiAI and SME2 accelerate llama.cpp from model load to token decode
Build llama.cpp with KleidiAI and SME2 enabled
Measure SME2 acceleration in llama.cpp on Android
Next Steps
Measure LLM inference performance with KleidiAI and SME2 on Android
Who is this for?
This is an advanced topic for software developers, performance engineers, and AI practitioners
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Build llama.cpp with KleidiAI and SME2 support
- Profile LLM inference performance on Android
- Understand how KleidiAI and SME2 accelerate LLM operators
Prerequisites
Before starting, you will need the following:
- Knowledge of KleidiAI and SME2
- A Linux host machine (x86_64 or aarch64) for building llama.cpp with the Arm GNU Toolchain
- Git, CMake, and Android Debug Bridge (ADB) installed on your host machine
- An Android device with Arm SME2 support for running and profiling the executable