Deploy an AI Agent on Arm with llama.cpp and llama-cpp-agent using KleidiAI
Introduction
Introduction to AI Agents and Agent Use Cases
Set Up Your Local Environment to Run an AI Application
AI Agent Application
Explore and Test Your AI Agent
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Introductory
Reading time:  45 min
Last updated:  26 Jun 2026
Skill level:
  Introductory
Reading time:
  45 min
Last updated:
  26 Jun 2026
Author:Andrew Choi
Arm IP:Neoverse
Tags: ML AWS Microsoft Azure Google Cloud Oracle Linux Python AWS Graviton AI
Author:
  • Andrew Choi
Arm IP:
Neoverse
Tags:
ML AWS Microsoft Azure Google Cloud Oracle Linux Python AWS Graviton AI

Who is this for?

This is an introductory topic for software developers and ML engineers looking to deploy an optimized AI agent application.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Set up llama-cpp-python optimized for Arm servers.
  • Run optimized Large Language Models (LLMs).
  • Create custom functions for LLMs.
  • Deploy optimized AI agents for applications.

Prerequisites

Before starting, you will need the following:

  • An Arm-based instance from a cloud service provider or an on-premise Arm server.
  • Basic understanding of Python and prompt engineering.
  • Understanding of LLM fundamentals.

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll build and run a function-calling AI agent on Arm servers using llama.cpp and the llama-cpp-agent framework with KleidiAI optimizations. After preparing an Arm-based Ubuntu environment and obtaining a quantized Llama 3.1 8B model, you’ll create agent.py to connect the model to tool functions and structured outputs. You’ll learn how the agent selects functions based on input intent, then test the end-to-end workflow to observe tool invocation and responses. By the end, you’ll run an AI agent backed by an optimized LLM, verify that the model loads, and inspect outputs that demonstrate function selection and result handling.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
Which operating system and instance setup do the steps target?
The steps are designed for Arm servers running Ubuntu 22.04 LTS. They were tested on an Amazon EC2 m7g.xlarge instance powered by Graviton3, and expect at least four cores, 16 GB of memory, and 32 GB of disk.
What should be ready before running the `agent.py` script?
Build llama.cpp and download the quantized Llama 3.1 8B model. Ensure the Python dependencies, including llama-cpp-agent and llama-cpp-python, are installed in the active environment.
Where should the model file be placed for the script to find it?
Place the downloaded model at the path referenced by model_path in agent.py. Keep the directory structure consistent with the code so the model loads without modification.
How do I know the agent used a function call rather than replying directly?
Check the script’s output for a selected tool/function name and its arguments, followed by the function’s return data. If the agent replies directly, you will see a model-generated response without a function invocation.
How are functions exposed to the LLM in this application?
Functions are declared in agent.py and registered through llama-cpp-agent. The LLM analyzes the input and selects among the predefined functions based on intent and context.
Next