Run distributed inference with llama.cpp on Arm-based AWS Graviton4 instances
Introduction
Convert model to GGUF and quantize
Configure the worker nodes
Configure the master node
Next Steps
Run distributed inference with llama.cpp on Arm-based AWS Graviton4 instances
Who is this for?
This introductory topic is for developers with some experience using llama.cpp who want to learn how to run distributed inference on Arm-based servers.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Set up a main host and worker nodes with llama.cpp
- Run a large quantized model (for example, Llama 3.1 405B) with distributed CPU inference on Arm machines
Prerequisites
Before starting, you will need the following:
- Three AWS c8g.4xlarge instances with at least 500 GB of EBS storage
- Python 3 installed on each instance
- Access to Meta’s gated repository for the Llama 3.1 model family and a Hugging Face token to download models
- Familiarity with the Learning Path Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers
- Familiarity with AWS
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
llama.cpp on AWS Graviton4 instances. You’ll convert Meta’s Llama 3.1 70B safetensors shards to one GGUF file, quantize the weights to 4-bit, configure worker RPC backends and a master through a comma-separated worker_ips list, and verify connectivity with telnet. You’ll then launch distributed inference across the Arm servers.Frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
worker_ips environment variable with a comma-separated list of host:port entries, for example, 172.31.110.11:50052,172.31.110.12:50052. Run this on the master node before starting inference.telnet <worker_ip> 50052. A successful connection confirms that the backend server on the worker is reachable.