Measure LLM inference performance with KleidiAI and SME2 on Android
Introduction
Explore SME2 and KleidiAI acceleration in llama.cpp
Trace KleidiAI and SME2 acceleration in llama.cpp
Build llama.cpp with KleidiAI and SME2 enabled
Measure SME2 acceleration in llama.cpp on Android
Next Steps
Measure LLM inference performance with KleidiAI and SME2 on Android
Review frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
Use the
aarch64 GCC cross-compile toolchain with the aarch64-none-linux-gnu- prefix from the Linux-hosted Arm GNU Toolchain. You need GCC version 14.2 or later to enable SME2.The build uses the Linux-hosted Arm GNU Toolchain. If you work on macOS or Windows, run the commands in a Linux virtual machine, container, or development machine.
Place the built
llama-cli executable and the Llama-3.2-3B-Instruct-Q4_0.gguf model on the device.Follow the verification step that confirms SME2 microkernels are active. The
ggml-cpu backend selects KleidiAI SME2 microkernels when the hardware supports them.Run the model on the Android device twice — once with SME2 enabled and once with it disabled. Compare the performance reported by
llama.cpp from both runs.