Introduction
AFM-4.5B deployment on Google Cloud Axion with Llama.cpp
Provision a Google Cloud Axion Arm64 environment
Configure your Google Cloud Axion Arm64 environment
Build Llama.cpp on Google Cloud Axion Arm64
Install Python dependencies for Llama.cpp
Download and optimize the AFM-4.5B model for Llama.cpp
Run inference with AFM-4.5B using Llama.cpp
Benchmark and evaluate AFM-4.5B quantized models on Axion
Review your AFM-4.5B deployment on Axion
Next Steps
In this step, you’ll configure your Google Cloud Axion Arm64 instance with the system packages and Python environment required to build and run the Arcee Foundation Model using Llama.cpp.
Run the following command to update your local APT package index:
sudo apt-get update
This ensures you have the most recent metadata about available packages, versions, and dependencies, helping to prevent conflicts when installing new software.
Install the required build tools and Python environment:
sudo apt-get install cmake gcc g++ git python3 python3-pip python3-virtualenv libcurl4-openssl-dev unzip -y
This command installs the following:
.zip
filesThe -y
flag automatically approves the installation of all packages without prompting.
After completing these steps, your instance includes:
You’re now ready to build Llama.cpp and download the Arcee Foundation Model.