Deploy Arcee AFM-4.5B on Arm-based AWS Graviton4 with Llama.cpp
Introduction
Overview
Provision your Graviton4 environment
Configure your Graviton4 environment
Build Llama.cpp
Install Python dependencies
Download and optimize the AFM-4.5B model
Run inference with AFM-4.5B
Benchmark and evaluate the quantized models
Review what you built
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Introductory
Reading time:  30 min
Last updated:  26 Jun 2026
Skill level:
  Introductory
Reading time:
  30 min
Last updated:
  26 Jun 2026
Author:Julien Simon
Arm IP:Neoverse
Tags: ML AWS Linux AWS Hugging Face Python Llama.cpp
Author:
  • Julien Simon
Arm IP:
Neoverse
Tags:
ML AWS Linux AWS Hugging Face Python Llama.cpp

Who is this for?

This Learning Path is for developers and ML engineers who want to deploy Arcee's AFM-4.5B small language model on AWS Graviton4 instances using Llama.cpp.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Launch an Arm-based EC2 instance on AWS Graviton4
  • Build and install Llama.cpp from source
  • Download and quantize the AFM-4.5B model from Hugging Face
  • Run inference on the quantized model using Llama.cpp
  • Evaluate model quality by measuring perplexity

Prerequisites

Before starting, you will need the following:

  • An AWS account with permission to launch Graviton4 (c8g.4xlarge or larger) instances
  • Basic familiarity with Linux and SSH

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll provision an Arm-based Amazon EC2 instance powered by AWS Graviton4, prepare a Linux environment, and build the llama.cpp inference engine from source. With a dedicated Python virtual environment, you’ll retrieve Arcee’s AFM-4.5B model from Hugging Face, perform quantization, and run inference using llama.cpp on Graviton4. You’ll learn to make practical choices, including instance sizing, storage, dependency setup, and repository selection, so the model compiles and executes cleanly on Arm. By the end, you’ll perform a perplexity-based quality check to validate that the quantized model runs, and interpret the reported perplexity as an objective signal of model quality.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
Which EC2 instance type and storage size should I choose when launching?
Use a c8g.4xlarge (or larger) EC2 instance powered by Graviton4 and allocate at least 128 GB of available storage. Configure the volume size during instance launch in the EC2 service so the space is available before you build and run the model.
When should I create the SSH key pair and what is it used for?
Create the SSH key pair in the AWS Management Console before launching the instance. You will use it to connect to the Graviton4 host over SSH for setup, building, and running inference.
Which `llama.cpp` repository should I clone, and why build from source here?
Clone the upstream repository at https://github.com/ggerganov/llama.cpp as shown in the steps. Build from source because AFM-4.5B support was contributed upstream, so the standard repo contains the necessary modeling code.
Do I need a Python virtual environment for this workflow?
Yes. Creating a virtual environment, such as env-llama-cpp, isolates the Python interpreter and packages required for the model workflow and avoids conflicts with system packages.
How do I know that quantization and perplexity evaluation worked?
After downloading AFM-4.5B and completing quantization, run the llama.cpp steps that load the quantized model and perform evaluation. Expect the program to load the model without errors and print a perplexity value you can use to compare quality across runs.
Next