Run distributed llama.cpp inference on AWS Graviton4
Introduction
Convert model to GGUF and quantize
Configure the worker nodes
Configure the master node
Next Steps
Run distributed llama.cpp inference on AWS Graviton4
Review frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
Export the
worker_ips environment variable with a comma-separated list of host:port entries, for example, 172.31.110.11:50052,172.31.110.12:50052. Run this on the master node before starting inference.From the master, run
telnet <worker_ip> 50052. A successful connection confirms that the backend server on the worker is reachable.Convert Meta’s safetensors files for Llama 3.1 70B to a single GGUF file, then quantize to 4-bit GGUF. Use the quantized GGUF for inference.
The process should create a new 4-bit GGUF weights file that is smaller than the 16-bit GGUF. Use this 4-bit file for the distributed run.
The example uses three AWS Graviton4 instances: one master node and two worker nodes. The master coordinates inference while the workers run the RPC backend.