Accelerate Generative AI workloads using KleidiAI
Introduction
KleidiAI and matrix multiplication
KleidiAI in a real software stack
Quantizing and packing micro-kernels
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Introductory
Reading time:  1 hr
Last updated:  03 Aug 2026
Skill level:
  Introductory
Reading time:
  1 hr
Last updated:
  03 Aug 2026
Author:Zach Lasiuk
Arm IP:Cortex-A Neoverse
Tags: ML Linux CPP Generative AI Neon Runbook
Author:
  • Zach Lasiuk
Arm IP:
Cortex-A Neoverse
Tags:
ML Linux CPP Generative AI Neon Runbook

Who is this for?

This is an introductory topic for developers who want to learn how to use KleidiAI to accelerate the execution of Generative AI workloads on hardware.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Describe how basic math operations power Large Language Models.
  • Describe how the KleidiAI micro-kernels speed up Generative AI inference performance.
  • Run a basic C++ matrix multiplication example to showcase the speedup that KleidiAI micro-kernels can deliver.

Prerequisites

Before starting, you will need the following:

  • An Arm-based Linux machine that implements the Int8 Matrix Multiplication (i8mm) architecture feature. The example in this Learning Path is run on an AWS Graviton 3 instance. Instructions on setting up an Arm-based server are found here .
  • A basic understanding of linear algebra terminology, such as dot product and matrix multiplication.

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll explore KleidiAI micro-kernels on Arm CPUs and see how matrix multiplication supports generative AI workloads. You’ll locate the matmul kernels and packing and quantization routines in the KleidiAI GitLab repository, review the architecture, and identify when a framework can invoke i8mm kernels. Then, you’ll run a C++ example that exercises the i8mm kernel and its supporting data path, so you can trace the optimized execution directly.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
How do I know if my ML framework will use KleidiAI automatically?
Your framework uses KleidiAI automatically when it integrates KleidiAI and your hardware supports the required Arm instructions, such as i8mm. If either condition is missing, run the standalone example to observe the micro-kernel behavior directly.
Where in the KleidiAI repository are the relevant micro-kernels?
Open /kai/ukernels/matmul in the KleidiAI GitLab repository. The pack directory contains quantization and packing routines. Adjacent directories contain the matrix multiplication routines.
Which example should I review to understand the i8mm execution path?
Open the KleidiAI example that demonstrates the i8mm matrix multiplication micro-kernel with its packing and quantization routines. The steps identify the example before you run it.
What should I expect when the C++ example runs successfully?
The example exercises the i8mm matmul micro-kernel with its packing and quantization steps. It demonstrates the micro-kernel data path and performance, not framework integration.
What should I check if I don’t observe the expected acceleration?
Verify that your hardware implements the required Arm instructions, such as i8mm, and that your software selects the optimized kernels. If you use a framework, confirm that it integrates KleidiAI. Otherwise, run the standalone example to validate the micro-kernel path.
Next