Deploy Arcee AFM-4.5B on Arm-based AWS Graviton4 with Llama.cpp
Introduction
Overview
Provision your Graviton4 environment
Configure your Graviton4 environment
Build Llama.cpp
Install Python dependencies
Download and optimize the AFM-4.5B model
Run inference with AFM-4.5B
Benchmark and evaluate the quantized models
Review what you built
Next Steps
Deploy Arcee AFM-4.5B on Arm-based AWS Graviton4 with Llama.cpp
Who is this for?
This Learning Path is for developers and ML engineers who want to deploy Arcee's AFM-4.5B small language model on AWS Graviton4 instances using Llama.cpp.
What will you learn?
After completing this Learning Path, you'll be able to perform the following tasks:
- Launch an Arm-based EC2 instance on AWS Graviton4
- Build and install Llama.cpp from source
- Download and quantize the AFM-4.5B model from Hugging Face
- Run inference on the quantized model using Llama.cpp
- Evaluate model quality by measuring perplexity
Prerequisites
Before starting, you'll need the following prerequisites:
- An
AWS account
with permission to launch Graviton4 (
c8g.4xlargeor larger) instances - Basic familiarity with Linux and SSH
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll provision an Arm-based Amazon EC2 instance powered by AWS Graviton4, prepare a Linux environment, and build the
llama.cpp inference engine from source. With a dedicated Python virtual environment, you’ll retrieve Arcee’s AFM-4.5B model from Hugging Face, perform quantization, and run inference using llama.cpp on Graviton4. You’ll learn to make practical choices, including instance sizing, storage, dependency setup, and repository selection, so the model compiles and executes cleanly on Arm. By the end, you’ll perform a perplexity-based quality check to validate that the quantized model runs, and interpret the reported perplexity as an objective signal of model quality.