Measure LLM inference performance with KleidiAI and SME2 on Android
Introduction
Understand how SME2 and KleidiAI accelerate LLM inference in llama.cpp
Trace how KleidiAI and SME2 accelerate llama.cpp from model load to token decode
Build llama.cpp with KleidiAI and SME2 enabled
Measure SME2 acceleration in llama.cpp on Android
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Advanced
Reading time:  40 min
Last updated:  19 Aug 2026
Skill level:
  Advanced
Reading time:
  40 min
Last updated:
  19 Aug 2026
Author:Zenon Zhilong Xiu, Arm
Arm IP:Arm C1
Tags: ML Android Linux SME2 CPP llama.cpp
Author:
  • Zenon Zhilong Xiu, Arm
Arm IP:
Arm C1
Tags:
ML Android Linux SME2 CPP llama.cpp

Who is this for?

This is an advanced topic for software developers, performance engineers, and AI practitioners

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Build llama.cpp with KleidiAI and SME2 support
  • Profile large language model (LLM) inference performance on Android
  • Understand how KleidiAI and SME2 accelerate LLM operators

Prerequisites

Before starting, you will need the following:

  • Knowledge of KleidiAI and SME2
  • A Linux host machine (x86_64 or aarch64) for building llama.cpp with the Arm GNU Toolchain
  • Git, CMake, and Android Debug Bridge (adb) installed on your host machine
  • An Android device with Arm SME2 support for running and profiling the executable

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll build llama.cpp with KleidiAI and SME2, then measure LLM inference on Android. First, you’ll cross-compile llama-cli for aarch64, deploy it with adb, and run the Llama-3.2-3B-Instruct-Q4_0 model with SME2 enabled and disabled. Then, you’ll trace acceleration through ggml-cpu into KleidiAI, and verify that SME2 microkernels are active before comparing the reported performance.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
Which compiler and toolchain should I use to build with SME2?
Use the aarch64 GCC cross-compile toolchain with the aarch64-none-linux-gnu- prefix from the Linux-hosted Arm GNU Toolchain. You need GCC version 14.2 or later to enable SME2.
Why do I need a Linux host to build llama.cpp?
The build uses the Linux-hosted Arm GNU Toolchain. If you work on macOS or Windows, run the commands in a Linux virtual machine, container, or development machine.
What files must be on the Android device before measuring performance?
Place the built llama-cli executable and the Llama-3.2-3B-Instruct-Q4_0.gguf model on the device.
How do I verify that the SME2 path is used during inference?
Follow the verification step that confirms SME2 microkernels are active. The ggml-cpu backend selects KleidiAI SME2 microkernels when the hardware supports them.
How should I compare runs with SME2 enabled and disabled?
Run the model on the Android device twice — once with SME2 enabled and once with it disabled. Compare the performance reported by llama.cpp from both runs.
Next