Explore related resources

Find more information about the topics in this Learning Path:

Give feedback

How would you rate this Learning Path?

Review frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
How do I pass the worker node addresses to the master?
Export the worker_ips environment variable with a comma-separated list of host:port entries, for example, 172.31.110.11:50052,172.31.110.12:50052. Run this on the master node before starting inference.
How do I verify the master can reach a worker before running a distributed job?
From the master, run telnet <worker_ip> 50052. A successful connection confirms that the backend server on the worker is reachable.
Which model format should I use with llama.cpp in this path?
Convert Meta’s safetensors files for Llama 3.1 70B to a single GGUF file, then quantize to 4-bit GGUF. Use the quantized GGUF for inference.
How do I confirm the quantization step worked?
The process should create a new 4-bit GGUF weights file that is smaller than the 16-bit GGUF. Use this 4-bit file for the distributed run.
How many nodes are used in the example and what roles do they serve?
The example uses three AWS Graviton4 instances: one master node and two worker nodes. The master coordinates inference while the workers run the RPC backend.

Share

Tell others about this Learning Path.

Back
Return to Servers and Cloud Computing