Deploy Arcee AFM-4.5B on Arm-based Google Cloud Axion with Llama.cpp
Introduction
AFM-4.5B deployment on Google Cloud Axion with Llama.cpp
Provision a Google Cloud Axion Arm64 environment
Configure your Google Cloud Axion Arm64 environment
Build Llama.cpp on Google Cloud Axion Arm64
Install Python dependencies for Llama.cpp
Download and optimize the AFM-4.5B model for Llama.cpp
Run inference with AFM-4.5B using Llama.cpp
Benchmark and evaluate AFM-4.5B quantized models on Axion
Review your AFM-4.5B deployment on Axion
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Introductory
Reading time:  30 min
Last updated:  26 Jun 2026
Skill level:
  Introductory
Reading time:
  30 min
Last updated:
  26 Jun 2026
Author:Julien Simon
Arm IP:Neoverse
Tags: ML Google Cloud Linux Google Cloud Hugging Face Python Llama.cpp
Author:
  • Julien Simon
Arm IP:
Neoverse
Tags:
ML Google Cloud Linux Google Cloud Hugging Face Python Llama.cpp

Who is this for?

This Learning Path is for developers and ML engineers who want to deploy Arcee's AFM-4.5B small language model on Google Cloud Axion instances using Llama.cpp.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Launch an Arm-based Compute Engine instance on Google Cloud Axion
  • Build and install Llama.cpp from source
  • Download and quantize the AFM-4.5B model from Hugging Face
  • Run inference on the quantized model using Llama.cpp
  • Evaluate model quality by measuring perplexity

Prerequisites

Before starting, you will need the following:

  • A Google Cloud account with permission to launch Axion (c4a-standard-16 or larger) instances
  • Basic familiarity with Linux and SSH

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll deploy Arcee’s AFM-4.5B model on Arm-based Google Cloud Axion using llama.cpp. First, you’ll provision an Axion-based VM, prepare the system and Python environment, and build llama.cpp from source. Then, you’ll download and quantize the model, and run inference. You’ll make concrete choices about instance size and storage, manage packages in an isolated environment, and use the upstream llama.cpp repository that includes the required AFM support. By the end, you’ll perform a perplexity run to assess model quality. A successful outcome includes a clean build, a loadable quantized model, token generation during a test prompt, and a reported perplexity value.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
Which Axion machine type should I choose to follow the steps?
Use c4a-standard-16 or larger, as specified. Smaller types are not listed for this workflow.
What should I check in my Google Cloud project before launching the Axion VM?
Confirm that your project has quota for Axion instances and that sufficient storage is available. The Learning Path calls for at least 128 GB to accommodate the model and dependencies.
Which `llama.cpp` repository should I clone for AFM-4.5B?
Clone the standard upstream repository from ggerganov/llama.cpp. Arcee AI has contributed the required modeling code upstream, so you don’t need a custom fork.
How do I know the `llama.cpp` build on Axion completed successfully?
The build should finish without errors and produce binaries that run as shown in the steps. Invoking the tool to display its help or version is a quick validation before proceeding.
What result should I expect after quantization and a test inference?
Quantization produces a smaller model artifact that llama.cpp can load. A brief prompt should generate tokens without errors, and a perplexity run should complete with a numeric value.
Next