# Run distributed inference with llama.cpp on Arm-based AWS Graviton4 instances

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/)
- [Convert model to GGUF and quantize](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-1/)
- [Configure the worker nodes](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-2/)
- [Configure the master node](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-3/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/_next-steps/)

## About this Learning Path

|  |  |
| --- | --- |
| Skill level: | Introductory |
| Reading time: | 30 min |
| Last updated: | 09 Sep 2026 |

|  |
| --- |
| Authors: |
| Aryan Bhusari , Arm<br>[LinkedIn](https://linkedin.com/in/aryanbhusari)<br>Joe Stech , Arm<br>[GitHub](https://github.com/JoeStech)<br>[LinkedIn](https://linkedin.com/in/joestech) |
| Arm IP: |
| [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors) |
| Tags: |
| [ML](https://learn.arm.com/tag/ml) [AWS Graviton](https://learn.arm.com/tag/aws-graviton) [Linux](https://learn.arm.com/tag/linux) [LLM](https://learn.arm.com/tag/llm) [Generative AI](https://learn.arm.com/tag/generative-ai) [AWS](https://learn.arm.com/tag/aws) |

|  |  |
| --- | --- |
| Authors: | Aryan Bhusari , Arm<br>[LinkedIn](https://linkedin.com/in/aryanbhusari)<br>Joe Stech , Arm<br>[GitHub](https://github.com/JoeStech)<br>[LinkedIn](https://linkedin.com/in/joestech) |
| Arm IP: | [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors) |
| Tags: | [ML](https://learn.arm.com/tag/ml) [AWS Graviton](https://learn.arm.com/tag/aws-graviton) [Linux](https://learn.arm.com/tag/linux) [LLM](https://learn.arm.com/tag/llm) [Generative AI](https://learn.arm.com/tag/generative-ai) [AWS](https://learn.arm.com/tag/aws) |

## Who is this for?
This introductory topic is for developers with some experience using llama.cpp who want to learn how to run distributed inference on Arm-based servers.

## What will you learn?
After completing this Learning Path, you'll be able to perform the following tasks:
- Set up a main host and worker nodes with llama.cpp
- Run a large quantized model (for example, Llama 3.1 405B) with distributed CPU inference on Arm machines

## Prerequisites
Before starting, you'll need the following prerequisites:
- Three AWS c8g.4xlarge instances with at least 500 GB of EBS storage
- Python 3 installed on each instance
- Access to Meta’s gated repository for the Llama 3.1 model family and a Hugging Face token to download models
- Familiarity with the Learning Path [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-cpu/)
- Familiarity with AWS

## Summary
You’ll run distributed CPU inference with `llama.cpp` on AWS Graviton4 instances. You’ll convert Meta’s Llama 3.1 70B safetensors shards to one GGUF file, quantize the weights to 4-bit, configure worker RPC backends and a master through a comma-separated `worker_ips` list, and verify connectivity with `telnet`. You’ll then launch distributed inference across the Arm servers.

## Frequently asked questions

### How do I pass the worker node addresses to the master?
Export the `worker_ips` environment variable with a comma-separated list of `host:port` entries, for example, `172.31.110.11:50052,172.31.110.12:50052`. Run this on the master node before starting inference.

### How do I verify the master can reach a worker before running a distributed job?
From the master, run `telnet <worker_ip> 50052`. A successful connection confirms that the backend server on the worker is reachable.

### Which model format should I use with llama.cpp in this path?
Convert Meta’s safetensors files for Llama 3.1 70B to a single GGUF file, then quantize to 4-bit GGUF. Use the quantized GGUF for inference.

### How do I confirm the quantization step worked?
The process should create a new 4-bit GGUF weights file that is smaller than the 16-bit GGUF. Use this 4-bit file for the distributed run.

### How many nodes are used in the example and what roles do they serve?
The example uses three AWS Graviton4 instances: one master node and two worker nodes. The master coordinates inference while the workers run the RPC backend.