Deploy Arcee AFM-4.5B on Arm-based AWS Graviton4 with Llama.cpp
Introduction
Overview
Provision your Graviton4 environment
Configure your Graviton4 environment
Build Llama.cpp
Install Python dependencies
Download and optimize the AFM-4.5B model
Run inference with AFM-4.5B
Benchmark and evaluate the quantized models
Review what you built
Next Steps
Deploy Arcee AFM-4.5B on Arm-based AWS Graviton4 with Llama.cpp
Review frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
Use a
c8g.4xlarge (or larger) EC2 instance powered by Graviton4 and allocate at least 128 GB of available storage. Configure the volume size during instance launch in the EC2 service so the space is available before you build and run the model.Create the SSH key pair in the AWS Management Console before launching the instance. You will use it to connect to the Graviton4 host over SSH for setup, building, and running inference.
Clone the upstream repository at
https://github.com/ggerganov/llama.cpp as shown in the steps. Build from source because AFM-4.5B support was contributed upstream, so the standard repo contains the necessary modeling code.Yes. Creating a virtual environment, such as
env-llama-cpp, isolates the Python interpreter and packages required for the model workflow and avoids conflicts with system packages.After downloading AFM-4.5B and completing quantization, run the
llama.cpp steps that load the quantized model and perform evaluation. Expect the program to load the model without errors and print a perplexity value you can use to compare quality across runs.