Deploy a llama.cpp chatbot with KleidiAI on Arm servers
Introduction
Demo
Run a large language model chatbot on Arm servers
Access the chatbot using the OpenAI-compatible API
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

Explore related resources

Find more information about the topics in this Learning Path:

Give feedback

How would you rate this Learning Path?

Review frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
What result should I expect when I start the llama.cpp server?
The server starts and listens on port 8080. After the server starts running, you can send OpenAI-compatible requests without restarting the process between calls.
Do I need any extra tools to view API responses?
Yes. Install jq with sudo apt install jq -y. You’ll use jq to process JSON returned by the API.
Can I access the chatbot from another machine?
Yes. The server exposes an OpenAI-compatible API over the network, so a remote client can call the host running the LLM if it can reach port 8080.
Which model should I download before launching the server?
Use a pre-quantized Llama 3.1 8B model from Hugging Face. Download the model to the Arm server before starting the server.
How do I send a request to the running llama.cpp server?
Send a curl request to http://localhost:8080/v1/chat/completions with a JSON prompt, then pipe the response to jq -C. Save the request in curl-test.sh and run it with bash ./curl-test.sh.

Share

Tell others about this Learning Path.

Back
Return to Servers and Cloud Computing