Run LLM inference on Android with KleidiAI, MediaPipe, and XNNPACK
Introduction
Install dependencies
Run the Gemma 2B model using MediaPipe with XNNPACK
Benchmark the Gemma 2B Model with KleidiAI
Next Steps
Run LLM inference on Android with KleidiAI, MediaPipe, and XNNPACK
Who is this for?
This is an advanced topic for Android developers who want to efficiently run LLMs on-device.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Install the prerequisites for cross-compiling new inference engines for Android.
- Run LLM inference on an Android device with the Gemma 2B model using the Google AI Edge's MediaPipe framework.
- Benchmark LLM inference speed with and without the KleidiAI-enhanced Arm i8mm processor feature.
Prerequisites
Before starting, you will need the following:
- An x86_64 Linux machine running Ubuntu with approximately 500 MB of free space, or a docker daemon that can build and run a provided x86_64 Dockerfile.
- An Android phone with support for i8mm (tested on Google Pixel 8 Pro).
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll cross-compile MediaPipe with XNNPACK for Android
arm64 and run Gemma 2B on an i8mm-capable device. First, you’ll build and verify the binary with Bazel, then run inference. After running inference, you’ll create a second build without i8mm and benchmark both variants to assess the effect of KleidiAI integration on supported Android hardware.Frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
Use the Docker option if you have a working Docker daemon and prefer an isolated, reproducible environment. Choose the native Ubuntu option if you want to install dependencies directly on your
x86_64 Linux host.List the Bazel output directory. You should see
llm_inference_engine_cpu_main under bazel-bin/mediapipe/tasks/cc/genai/inference/c/.Build the target
mediapipe/tasks/cc/genai/inference/c:llm_inference_engine_cpu_main with --config=android_arm64 and --define=xnn_enable_arm_i8mm=true. These options select the Android arm64 build and enable the i8mm path integrated through XNNPACK.In the
llm_test Bazel command, use --define=xnn_enable_arm_i8mm=true with --define=xnn_enable_kleidiai=false. Retain -c opt, --config=android_arm64, --dynamic_mode=off, and the mediapipe/tasks/cc/genai/inference/utils/xnn_utils:llm_test target.Both builds enable i8mm. The first disables KleidiAI micro-kernels, while the second enables them by default, so you can compare the effect of KleidiAI on your device.