Run and benchmark vLLM mixture of experts models on Arm CPUs
Introduction
Set up vLLM for MoE inference
Understand MoE models and precision choices
Run MoE inference with vLLM
Benchmark MoE serving performance
Review benchmark results
Next Steps
Run and benchmark vLLM mixture of experts models on Arm CPUs
Who is this for?
This is an introductory topic for developers interested in running and benchmarking mixture of experts (MoE) models with vLLM on Arm CPUs.
What will you learn?
After completing this Learning Path, you'll be able to perform the following tasks:
- Install and verify vLLM for CPU inference on an Arm server.
- Serve BF16 and INT8/W8A8 MoE models and validate a response.
- Benchmark MoE serving performance with vLLM bench serve.
- Compare BF16 and INT8/W8A8 MoE performance using relative speedups.
Prerequisites
Before starting, you'll need the following prerequisites:
- An Arm-based Linux server running Ubuntu 24.04 LTS, with enough CPU cores, memory, and disk space for large MoE models
- CPU support for BF16 and I8MM instructions
- Python 3.12 and basic familiarity with Hugging Face models and vLLM serving
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll set up vLLM on an Arm-based Linux server to serve and benchmark MoE models. First, you’ll check CPU support and install vLLM. You’ll then choose BF16 and INT8/W8A8 checkpoints, serve each model, and validate a chat response. Next, you’ll prepare model-specific ShareGPT subsets and benchmark concurrent requests. Finally, you’ll compare throughput and latency using matching workloads and concurrency values.