Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers
Introduction
Demo
Run a Large Language model (LLM) chatbot on Arm servers
Access the chatbot using the OpenAI-compatible API
Review
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

What you've learned

You should now know how to:

  • Download and build llama.cpp on your Arm server.
  • Download a pre-quantized Llama 3.1 model from Hugging Face.
  • Run the pre-quantized model on your Arm CPU and measure the performance.

Knowledge Check

Can you run LLMs on Arm CPUs?

Can llama.cpp be built and run on CPU only?

Can you profile the time taken by the model to generate the output until the end of text?


Back
Next