Measure LLM decode bandwidth limits on Arm with llama-roofline
Introduction
Understand why token generation is limited by memory bandwidth
Build llama.cpp and install llama-roofline
Measure the memory bandwidth ceiling
Diagnose whether decode is memory-bound
Choose thread count and model size from the measurements
Next Steps
Measure LLM decode bandwidth limits on Arm with llama-roofline
Check the Arm architecture features
Confirm that you’re on a 64-bit Arm system and list the CPU features:
uname -m
grep -m1 Features /proc/cpuinfo
On a Raspberry Pi 5, the output is similar to:
aarch64
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
The asimd flag is Neon. The asimddp flag is the DotProd extension. llama.cpp detects DotProd when you build it. It uses DotProd for its quantized kernels on the Cortex-A76. The Pi 5 doesn’t have i8mm or sve, which some newer Arm cores provide.
Install the build tools
Install the build tools and Python virtual-environment packages:
sudo apt update
sudo apt install -y git cmake build-essential python3-venv python3-pip
Build llama-bench
You need only the llama-bench tool from llama.cpp. Building one target with one compile job keeps memory use low on a 2 GB board:
cd $HOME
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git rev-parse --short HEAD
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-bench -j 1
The clone and build took about 11 minutes on the test Raspberry Pi 5.
On a board with more memory, you can increase -j to the number of cores to build faster. On a 2 GB Raspberry Pi 5, keep -j 1. Parallel compilation of large llama.cpp source files can exhaust memory and make the board unresponsive.
Record the commit hash printed by git rev-parse. You need it when you compare results. The outputs in the Learning Path come from llama.cpp commit 836d571.
To confirm that the build enabled DotProd, check the output of the cmake -B build command. Search for the line that starts with -- Adding CPU backend variant.
On a Raspberry Pi 5, the output is similar to:
-- Adding CPU backend variant ggml-cpu: -U__ARM_FEATURE_MATMUL_INT8;-U__ARM_FEATURE_SVE;-mcpu=cortex-a76+crypto+dotprod+noi8mm+nosve
Check that the build works:
./build/bin/llama-bench --help | head -5
The output is similar to:
usage: ./build/bin/llama-bench [options]
options:
-h, --help
--version show version and build info
Download three models
The tool fits a line through several models of different sizes, so you need at least three models. Use the official Qwen2.5 GGUF files, which fit in 2 GB of RAM and aren’t gated:
mkdir -p $HOME/models && cd $HOME/models
wget https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_0.gguf
wget https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf
wget https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/qwen2.5-1.5b-instruct-q4_0.gguf
ls -l *.gguf
Your user name and dates can differ, but the byte counts need to match.
The output is similar to:
-rw-r--r-- 1 manu manu 428730208 Oct 4 00:10 qwen2.5-0.5b-instruct-q4_0.gguf
-rw-r--r-- 1 manu manu 675710816 Oct 4 00:11 qwen2.5-0.5b-instruct-q8_0.gguf
-rw-r--r-- 1 manu manu 1066227232 Oct 4 00:13 qwen2.5-1.5b-instruct-q4_0.gguf
Use files from the same publisher and the same conversion pipeline for comparison. Two files with the same format label can contain different tensor types, depending on how they were produced. This changes their speed.
Install llama-roofline
Create a virtual environment and install the tool from PyPI:
python3 -m venv $HOME/roofline-venv
source $HOME/roofline-venv/bin/activate
pip install llama-roofline
llama-roofline --version
The output is similar to:
llama-roofline 0.2.0
What you’ve accomplished and what’s next
You’ve built llama-bench, downloaded three models of different sizes, and installed llama-roofline.
Next, you’ll measure the memory bandwidth ceiling of your system.