# Run distributed llama.cpp inference on AWS Graviton4

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/)
- [Convert model to GGUF and quantize](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-1/)
- [Configure the worker nodes](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-2/)
- [Configure the master node](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/how-to-3/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/distributed-inference-with-llama-cpp/_next-steps/)

## About this Learning Path

- **Skill level:** Introductory  
- **Reading time:** 30 min  
- **Last updated:** 12 Nov 2026  

### Authors

- Aryan Bhusari, Arm  
  [LinkedIn](https://linkedin.com/in/aryanbhusari)  
- Joe Stech, Arm  
  [GitHub](https://github.com/JoeStech)  
  [LinkedIn](https://linkedin.com/in/joestech)  

**Arm IP:** [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors)

**Tags:**
- [AI/ML](https://learn.arm.com/tag/ai-ml/)
- [AWS Graviton](https://learn.arm.com/tag/aws-graviton/)
- [Linux](https://learn.arm.com/tag/linux/)
- [LLM](https://learn.arm.com/tag/llm/)
- [Generative AI](https://learn.arm.com/tag/generative-ai/)
- [AWS](https://learn.arm.com/tag/aws/)

### Who is this for?

This introductory topic is for developers with some experience using llama.cpp who want to learn how to run distributed inference on Arm-based servers.

### What will you learn?

After completing this Learning Path, you'll be able to perform the following tasks:

- Set up a main host and worker nodes with llama.cpp
- Run a large quantized model (for example, Llama 3.1 405B) with distributed CPU inference on Arm machines

### Prerequisites

Before starting, you'll need the following prerequisites:

- Three AWS c8g.4xlarge instances with at least 500 GB of EBS storage
- Python 3 installed on each instance
- Access to Meta’s gated repository for the Llama 3.1 model family and a Hugging Face token to download models
- Familiarity with the Learning Path [Deploy a Large Language Model (LLM) chatbot with llama.cpp using KleidiAI on Arm servers](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-cpu/)
- Familiarity with AWS

### Summary

You’ll run distributed CPU inference with `llama.cpp` on AWS Graviton4 instances. You’ll convert Meta’s Llama 3.1 70B safetensors shards to one GGUF file, quantize the weights to 4-bit, configure worker RPC backends and a master through a comma-separated `worker_ips` list, and verify connectivity with `telnet`. You’ll then launch distributed inference across the Arm servers.
