Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI
Introduction
Introduction to AI Agents and Agent Use Cases
Set Up Your Local Environment to Run an AI Application
AI Agent Application
Explore and Test Your AI Agent
Next Steps
Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI
Who is this for?
This is an introductory topic for software developers and ML engineers looking to deploy an optimized AI agent application.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Set up llama-cpp-python optimized for Arm servers.
- Run optimized Large Language Models (LLMs).
- Create custom functions for LLMs.
- Deploy optimized AI agents for applications.
Prerequisites
Before starting, you will need the following:
- An Arm-based instance from a cloud service provider or an on-premise Arm server.
- Basic understanding of Python and prompt engineering.
- Understanding of LLM fundamentals.
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll build and run a function-calling AI agent on Arm servers using
llama.cpp and the llama-cpp-agent framework with KleidiAI optimizations. After preparing an Arm-based Ubuntu environment and obtaining a quantized Llama 3.1 8B model, you’ll create agent.py to connect the model to tool functions and structured outputs. You’ll learn how the agent selects functions based on input intent, then test the end-to-end workflow to observe tool invocation and responses. By the end, you’ll run an AI agent backed by an optimized LLM, verify that the model loads, and inspect outputs that demonstrate function selection and result handling.Frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
The steps are designed for Arm servers running Ubuntu 22.04 LTS. They were tested on an Amazon EC2
m7g.xlarge instance powered by Graviton3, and expect at least four cores, 16 GB of memory, and 32 GB of disk.Build
llama.cpp and download the quantized Llama 3.1 8B model. Ensure the Python dependencies, including llama-cpp-agent and llama-cpp-python, are installed in the active environment.Place the downloaded model at the path referenced by
model_path in agent.py. Keep the directory structure consistent with the code so the model loads without modification.Check the script’s output for a selected tool/function name and its arguments, followed by the function’s return data. If the agent replies directly, you will see a model-generated response without a function invocation.
Functions are declared in
agent.py and registered through llama-cpp-agent. The LLM analyzes the input and selects among the predefined functions based on intent and context.