# Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/ai-agent-on-cpu/)
- [Introduction to AI Agents and Agent Use Cases](https://learn.arm.com/learning-paths/servers-and-cloud-computing/ai-agent-on-cpu/ai-agent/)
- [Set Up Your Local Environment to Run an AI Application](https://learn.arm.com/learning-paths/servers-and-cloud-computing/ai-agent-on-cpu/set-up/)
- [AI Agent Application](https://learn.arm.com/learning-paths/servers-and-cloud-computing/ai-agent-on-cpu/ai-agent-backend/)
- [Explore and Test Your AI Agent](https://learn.arm.com/learning-paths/servers-and-cloud-computing/ai-agent-on-cpu/agent-output/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/ai-agent-on-cpu/_next-steps/)

## About this Learning Path

| Skill level:      | Introductory          |
|-------------------|-----------------------|
| Reading time:     | 45 min                |
| Last updated:     | 31 Jul 2026           |

| Author:           | Andrew Choi           |
|-------------------|-----------------------|
| Arm IP:           | [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors) |
| Tags:             | ML, AWS, Microsoft Azure, Google Cloud, Oracle, Linux, Python, AWS Graviton, AI |

### Who is this for?
This is an introductory topic for software developers and ML engineers looking to deploy an optimized AI agent application.

### What will you learn?
Upon completion of this Learning Path, you will be able to:
- Set up llama-cpp-python optimized for Arm servers.
- Run optimized Large Language Models (LLMs).
- Create custom functions for LLMs.
- Deploy optimized AI agents for applications.

### Prerequisites
Before starting, you will need the following:
- An [Arm-based instance](https://learn.arm.com/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider or an on-premise Arm server.
- Basic understanding of Python and prompt engineering.
- Understanding of LLM fundamentals.

### Summary
You’ll build and run a function-calling AI agent on Arm servers using `llama.cpp` and the `llama-cpp-agent` framework with KleidiAI optimizations. After preparing an Arm-based Ubuntu environment and obtaining a quantized Llama 3.1 8B model, you’ll create `agent.py` to connect the model to tool functions and structured outputs. You’ll learn how the agent selects functions based on input intent, then test the end-to-end workflow to observe tool invocation and responses. By the end, you’ll run an AI agent backed by an optimized LLM, verify that the model loads, and inspect outputs that demonstrate function selection and result handling.

### Frequently asked questions

#### Which operating system and instance setup do the steps target?
The steps are designed for Arm servers running Ubuntu 22.04 LTS. They were tested on an Amazon EC2 `m7g.xlarge` instance powered by Graviton3, and expect at least four cores, 16 GB of memory, and 32 GB of disk.

#### What should be ready before running the `agent.py` script?
Build `llama.cpp` and download the quantized Llama 3.1 8B model. Ensure the Python dependencies, including `llama-cpp-agent` and `llama-cpp-python`, are installed in the active environment.

#### Where should the model file be placed for the script to find it?
Place the downloaded model at the path referenced by `model_path` in `agent.py`. Keep the directory structure consistent with the code so the model loads without modification.

#### How do I know the agent used a function call rather than replying directly?
Check the script’s output for a selected tool/function name and its arguments, followed by the function’s return data. If the agent replies directly, you will see a model-generated response without a function invocation.

#### How are functions exposed to the LLM in this application?
Functions are declared in `agent.py` and registered through `llama-cpp-agent`. The LLM analyzes the input and selects among the predefined functions based on intent and context.