Measure LLM decode bandwidth limits on Arm with llama-roofline
Introduction
Understand why token generation is limited by memory bandwidth
Build llama.cpp and install llama-roofline
Measure the memory bandwidth ceiling
Diagnose whether decode is memory-bound
Choose thread count and model size from the measurements
Next Steps
Measure LLM decode bandwidth limits on Arm with llama-roofline
Who is this for?
This topic is for developers and performance engineers who want to choose llama.cpp settings from measured token throughput on Arm CPUs, using a Raspberry Pi 5 or another Arm Linux system.
What will you learn?
After completing this Learning Path, you'll be able to perform the following tasks:
- Measure the sustained memory-read bandwidth of an Arm Linux system.
- Determine whether llama.cpp token generation is memory-bound on your Arm CPU, and by how much, using llama-roofline.
- Choose a decode thread count and a model size from measured data instead of defaults.
- Identify why prompt processing and token generation need different thread counts.
Prerequisites
Before starting, you'll need the following prerequisites:
- A 64-bit Arm Linux system, such as a Raspberry Pi 5 with 2 GB of RAM and an active cooler
- Python 3 with the venv module
- About 3.3 GB of free disk space for llama.cpp and three GGUF models
- Familiarity with building software with CMake on Linux
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll use llama-roofline with llama.cpp on Arm Linux to find out whether token generation is limited by memory bandwidth. First, you’ll check your CPU features, build llama-bench, and prepare the system and models. Then, you’ll measure sustained read bandwidth and inspect how many bytes each model reads per token. Finally, you’ll compare decode and prefill across thread counts. You’ll diagnose memory limits, and choose a model size and thread settings.