# Profile llama.cpp performance with Arm Streamline and KleidiAI LLM kernels

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/)
- [Overview](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/1_overview/)
- [Explore llama.cpp architecture and the inference workflow](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/2_llama.cpp_intro/)
- [Integrate Streamline Annotations into llama.cpp](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/3_llama.cpp_annotation/)
- [Analyze token generation performance with Streamline profiling](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/4_analyze_token_prefill_decode/)
- [Implement operator-level performance analysis with channel annotations](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/5_operator_deepdive/)
- [Examine multi-threaded performance patterns in llama.cpp](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/6_multithread_analyze/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/_next-steps/)

## About this Learning Path

| Skill level:  | Advanced              |
|---------------|-----------------------|
| Reading time: | 1 hr                  |
| Last updated: | 17 Sep 2026           |

| Authors:                                                                                      |
|-----------------------------------------------------------------------------------------------|
| Zenon Zhilong Xiu, Arm [LinkedIn](https://linkedin.com/in/zenon-zhilong-xiu-491bb398)       |
| Odin Shen, Arm [GitHub](https://github.com/odincodeshen) [LinkedIn](https://linkedin.com/in/odin-shen-lmshen) |
| Arm IP: [Cortex-A](https://support.arm.com/?tab=compute-ip&Product%20Type=Application%20Processors) [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors) |
| Tags: [ML](https://learn.arm.com/tag/ml) [Linux](https://learn.arm.com/tag/linux) [Android](https://learn.arm.com/tag/android) [Arm Streamline](https://learn.arm.com/tag/arm-streamline) [CPP](https://learn.arm.com/tag/cpp) [llama.cpp](https://learn.arm.com/tag/llama.cpp) [Profiling](https://learn.arm.com/tag/profiling) |

### Who is this for?

This is an advanced topic for software developers, performance engineers, and AI practitioners who want to optimize llama.cpp performance on Arm-based CPUs.

### What will you learn?

Upon completion of this Learning Path, you will be able to:

- Profile llama.cpp architecture and identify the role of the Prefill and Decode stages.
- Integrate marker annotations into llama.cpp for fine-grained performance insights.
- Capture and interpret profiling data with Streamline.
- Analyze specific operators during token generation using channel annotations.
- Evaluate multi-core and multi-thread execution of llama.cpp on Arm CPUs.

### Prerequisites

Before starting, you will need the following:

- Basic understanding of llama.cpp
- Understanding of transformer models
- Knowledge of Arm Streamline usage
- An Arm Neoverse or Cortex-A hardware platform running Linux or Android

### Summary

You’ll profile `llama.cpp` on Arm CPUs with Arm Streamline, including runs that use KleidiAI LLM kernels. First, you’ll add marker annotations for prefill and decode phases and build `llama-cli`. Then, you’ll prepare the model files and `gator` daemon, and capture traces. Finally, you’ll inspect channel annotations and operator timing to distinguish pipeline stages and assess multi-core and multi-thread behavior during token generation.

### Frequently asked questions

**How do I know that the marker annotations are working in Streamline?**

During capture, look for marker annotations on the Streamline timeline. Each marker annotation is added before `llama_decode()` and records the start of a token-generation event. Marker annotations include values such as `past` and `n_eval`. If marker annotations don’t appear, rebuild `llama.cpp` with the annotation changes and run the annotated `llama-cli`.

**What do I need on the Arm target before starting a profiling capture?**

Ensure that the `gator` daemon is configured and running on the Arm system. Place the built `llama-cli` and the required model files on the target so that Streamline can capture a representative run.

**How do I differentiate prefill and decode phases in the results?**

Use the inserted marker annotations to identify each stage in the Streamline timeline. Prefill is compute-intensive and decode is memory-bound, so compare their annotated ranges to understand where time is spent.

**How can I analyze operator-level performance with channel annotations?**

Enable channel annotations in the annotation integration so that Streamline displays separate lanes for grouped operations. Inspect the channel annotation lanes to see operator timing and overlaps during token generation.

**How do I evaluate multi-core or multi-thread execution?**

Capture a token generation run and use the marker and channel annotations to correlate work across cores and threads. Compare behavior during prefill and decode phases to see how execution is distributed.