SME Insights

Arm Scalable Matrix Extension (SME) is an optional AArch64 feature for matrix-oriented computation. It adds a two-dimensional register array called ZA, instructions that accumulate vector outer products into ZA, and a Streaming SVE mode with its own vector length. SME2 extends this model with multi-vector operations, multi-vector predication, and lookup-table instructions. Both features are optional, so an Arm architecture version alone does not tell you whether a device supports them.

The central idea is to keep an output tile in ZA while a kernel processes the input data needed to build it. This can reduce the instructions and data movement needed for matrix multiplication. Beyond machine learning inference, signal processing, scientific computing, and computer vision workloads can also benefit when their hot operations map to optimized SME kernels. Actual gains depend on the processor and on whether the application reaches such a kernel.

How the Matrix Operation Works

ZA is a square array whose width and height each equal the streaming vector length (SVL) in bytes. At an SVL of 256 bits, each side is 32 bytes and ZA contains 32 × 32 = 1,024 bytes. The architectural SVL values are 128, 256, 512, 1024, and 2048 bits; an implementation can support a subset of these lengths. Code written without assuming a fixed SVL can run across implementations with different supported lengths.

An outer product multiplies every element of one vector by every element of another. If a and b are column vectors, their outer product is:

a × bᵀ =
┌ a0·b0  a0·b1  a0·b2 ┐
│ a1·b0  a1·b1  a1·b2 │
└ a2·b0  a2·b1  a2·b2 ┘

For C = A × B, a matrix kernel accumulates one such outer product for each position along the shared dimension: C = Σₖ A[:,k] × B[k,:]. SME outer-product instructions perform those updates directly in ZA. Predicates select active rows and columns, which helps handle partial tiles at matrix edges. The kernel loads input vectors, accumulates across the shared dimension, then stores the completed output tile.

Instruction variants cover floating-point and integer data. Their available precisions depend on the implemented features; for example, FP64 outer products require the optional FEAT_SME_F64F64 feature. SME2 adds operations on groups of vectors and lookup-table support useful for some low-precision kernels.

Streaming Mode and Arm Vector and Matrix Extensions

Streaming SVE mode changes which vector instructions are available and uses the streaming vector length, which can differ from the non-streaming SVE vector length. The architecture does not require either length to be larger. Streaming mode (PSTATE.SM) and ZA access (PSTATE.ZA) are separate state: ZA can be enabled without entering streaming mode, although matrix operations generally use both. SMSTART and SMSTOP change this state; C and C++ code can use ACLE function attributes to have the compiler manage mode transitions at function boundaries.

Applications can use ordinary scalar or Neon code and call SME kernels for suitable work. SVE/SVE2 code can also coexist where the processor supports those features. Streaming mode is an execution mode; it does not automatically stream matrix data from memory. A kernel still loads its input tiles and stores its results.

How Developers Use It

Most developers use an optimized library through a framework instead of writing SME instructions. One example is LiteRT → XNNPACK → KleidiAI → SME2 microkernel. XNNPACK can select such a kernel only when it was included in the build, the hardware and OS expose SME2, and the operator’s data type, quantization format, and shape match an available implementation. Other operations use a fallback such as Neon. Hardware support by itself therefore does not establish that a model is executing SME2 instructions.

Developers writing their own kernels can use SME intrinsics and function attributes from Arm C Language Extensions (ACLE), or assembly when they need direct control of ZA layout and instruction scheduling. Portable applications should check platform feature availability, such as HWCAP2 on Linux, before dispatching to SME code and retain a non-SME implementation for other devices.

Resources

• Arm SME and SME2 Programmer’s Guide for the programming model and feature details.
• Matrix multiplication with SME2 Learning Path for a guided implementation.
• Part 1: Arm SME Introduction for a conceptual introduction.
• SME2 from scratch: Build matrix multiplication (video) for a visual walkthrough.

Yes. Arm has hands-on material for learning SME and SME2. The best starting point depends on whether you want to inspect individual kernels, build a complete matrix multiplication example, or adapt existing SIMD code.

Start with a working kernel

Accelerate matrix multiplication performance with SME2 is the most direct end-to-end tutorial. It provides a CMake project with a plain C matrix multiplication baseline, an SME2 assembly version, and a version written with C intrinsics. Follow the path in order: set up and check the toolchain, build and run the baseline, study streaming mode and ZA state, implement the SME2 versions, then compare their results with the baseline. The path also covers debugging and benchmarking.

This is an advanced tutorial. It assumes you know C, basic SVE and SME concepts, and some Arm assembly. You can run the examples on supported hardware or use its Docker-based emulator to check that the code builds and produces the right results. Use native hardware for performance measurements; emulation does not model timing accurately.

Explore smaller examples

Learn SVE and SME programming with SIMD Loops uses self-contained loop kernels to show how scalar, Neon, SVE, and SME2 approaches differ. The project includes reference C implementations and correctness checks. Pick a kernel that interests you, build its supported variants, and compare the code and output. This is a manageable way to explore predication, scalable vectors, and ZA before tackling a full matrix multiplication kernel. It is particularly useful if you already know C and Neon intrinsics.

Follow a different starting point when it fits your work

• Porting x86 SIMD code? Migrate SIMD code to the Arm architecture maps x86 vector and matrix extensions to Neon, SVE, and SME and walks through a vectorized code example. It helps with migration choices, but its coding exercise is not an SME matrix kernel.
• Interested in integer matrix kernels? The SME2 INT8 matrix-multiplication example in Arm’s Programmer’s Guide is a worked reference for 8-bit inputs and 32-bit results, including input preprocessing. Use it after learning the basic programming model; it is more detailed than a first tutorial.

Arm Scalable Matrix Extension (SME) adds ZA, a two-dimensional architectural register array, and instructions that accumulate vector outer products into it. It also defines Streaming SVE mode, which uses a streaming vector length and a different set of available vector instructions. Neon and SVE/SVE2 primarily expose one-dimensional vectors; SME gives a matrix kernel a place to keep a two-dimensional block of partial results. This is useful when an operation can be organized around repeated matrix or outer-product updates, including suitable kernels in machine learning, signal processing, and scientific computing.

These are distinct architecture features that can coexist in one application. SME uses scalable vectors, but supporting SME does not by itself mean a processor supports non-streaming SVE or SVE2. Check the features your code actually requires.

The three programming models

Neon processes a fixed number of values per instruction: for example, four 32-bit floats fit in one 128-bit vector. Developers use its vector registers for arithmetic, shuffles, and suitably vectorized matrix kernels. A larger vector operation is built from multiple instructions.

SVE makes the vector length a property of the implementation rather than a constant in the program. Code written in a vector-length-agnostic style advances by the number of active elements and uses predicates for partial vectors. SVE also provides gather/scatter operations for some irregular access patterns. SVE2 extends that vector model with additional integer, DSP, and other instructions. SVE and SVE2 can implement matrix multiplication, but their accumulators are normally held in one-dimensional vector registers rather than ZA.

SME adds ZA and instructions that operate on its tiles and horizontal or vertical slices. ZA is a square byte array: at an SVL of 256 bits, each side is 32 bytes and ZA contains 32 × 32 = 1,024 bytes. Tile names and the number of tile views depend on element width; they are ways to access the same underlying ZA storage, not separate arrays. Horizontal and vertical slice access helps move or rearrange matrix data, including transposition.

Why ZA changes matrix multiplication

For a matrix product C = A × B, a kernel can compute one output block by repeatedly taking a column from A and a row from B. Each pair contributes an outer product to that block:

output tile += column from A × row from B

With SME, an instruction such as FMOPA accumulates that outer product directly into a ZA tile. The kernel can keep partial results there while it walks along the shared dimension, then store the completed tile. Predicates select which rows and columns are active when the output block reaches a matrix edge. SME also has integer and lower-precision outer-product variants; FP64 outer products require the optional FEAT_SME_F64F64 feature.

Neon and SVE kernels can perform the same mathematics, including with dot-product or matrix-oriented instructions where supported. Their code organizes the accumulation around vector registers and the instructions available on the target CPU. SME’s advantage is the programming model for a suitable kernel: ZA provides matrix-shaped accumulator storage, and outer-product instructions update it directly. Whether that runs faster depends on the processor, data layout, matrix shape, and quality of the implementation. ZA’s area grows with SVL squared, but that fact alone does not establish a performance gain.

Streaming mode and coexistence

Streaming SVE mode is controlled by PSTATE.SM; ZA access is controlled separately by PSTATE.ZA. The streaming vector length can differ from the length used by non-streaming SVE code, with no required ordering between them. Instruction availability differs from non-streaming mode, so a kernel cannot assume every Neon or SVE instruction is usable there. Assembly can use SMSTART and SMSTOP to change mode; in C or C++, Arm C Language Extensions (ACLE) function attributes let the compiler manage mode transitions and ZA state at function boundaries.

An application can keep its general work in scalar, Neon, or SVE/SVE2 code and call an SME kernel for a suitable operation. SME2 extends SME with multi-vector operations, predication for groups of vectors, and lookup-table support; it does not replace the other features. Portable software still needs a path for devices without the required SME features.

Resources

• Neon, SVE, and SME matrix multiplication compared shows how the same operation is expressed with each programming model.
• Arm SME and SME2 Programmer’s Guide explains ZA, streaming mode, and the related programming rules.

Streaming SVE mode and ZA are separate parts of Arm SME. Streaming mode changes the vector length and which vector instructions can execute. ZA is a two-dimensional register array that holds matrix data, especially partial results from outer-product operations. ZA’s dimensions come from the streaming vector length (SVL), even while the processor is outside streaming mode. A typical matrix kernel enables both features: streaming Z registers supply the input vectors, and the operation accumulates a result in ZA.

Two controls, four states

PSTATE.SM controls streaming mode; PSTATE.ZA controls whether ZA is enabled. They can be changed separately with SMSTART SM / SMSTOP SM and SMSTART ZA / SMSTOP ZA. An unqualified SMSTART enables both, and an unqualified SMSTOP disables both.

The separate controls make it possible to leave streaming mode while retaining an unfinished ZA result: use SMSTOP SM, leaving PSTATE.ZA set, and later use SMSTART SM to continue. Turning ZA off loses that live state; enabling it again starts with ZA cleared. Changing streaming mode zeroes Z0–Z31 and P0–P15 in the new mode, and FFR where it is implemented, so code must not carry their values across the transition. A branch or scalar loop does not itself require leaving streaming mode. Switching modes is useful when code needs instructions or a function interface that runs in non-streaming mode, but repeated switches have a cost.

Streaming mode uses SVL for its Z and predicate registers. If the processor also supports non-streaming SVE, its vector length can differ from SVL; neither length is required to be larger. Streaming mode supports a defined subset of vector instructions and the SME matrix operations. It does not automatically fetch matrix data from memory: a kernel still loads inputs and stores results.

SVL defines ZA’s shape

ZA is a square array with SVL in bytes on each side. For example, at SVL = 256 bits, each side is 32 bytes, so ZA contains 32 × 32 = 1,024 bytes. At SVL = 512 bits, it contains 64 × 64 = 4,096 bytes. This describes storage capacity; a larger ZA alone does not establish higher performance.

The same underlying bytes can be viewed in several ways:

• ZA array vectors provide a row-oriented view useful for saving and restoring state.
• ZA tiles divide the array into typed matrix views. The number of tiles depends on element width: one for 8-bit elements, two for 16-bit, four for 32-bit, eight for 64-bit, and sixteen for 128-bit. For instance, the 32-bit views are named ZA0.S through ZA3.S. They overlap the same ZA storage rather than forming four separate register arrays.
• Tile slices expose horizontal or vertical vectors within a tile. A horizontal slice such as ZA0H.S and a vertical slice such as ZA0V.S let instructions move or rearrange data along different axes, including for transposition.

The non-streaming SVE vector length does not change ZA’s size. If code retains ZA while switching out of streaming mode, its matrix state still has the dimensions set by SVL.

How a matrix kernel uses both

A simplified outer-product kernel enables streaming mode and ZA, clears the accumulator tile, then repeats these steps: load input vectors into streaming Z registers, accumulate their outer product into ZA, and advance along the shared dimension. It stores the finished tile before disabling ZA. Predicates select active rows and columns for a partial output tile.

The instruction requirements matter. An outer product such as FMOPA needs streaming mode and ZA enabled. ZA tile-slice LD1 and ST1 operations also belong inside streaming mode. The ZA array-vector LDR and STR operations used for state saving are distinct and can operate with ZA enabled outside streaming mode. This distinction is why retaining ZA across a mode switch does not make every ZA instruction available in non-streaming code.

In C or C++, Arm C Language Extensions (ACLE) function attributes make the state contract visible in a declaration:

void accumulate_tile(const float *a, const float *b)
    __arm_streaming __arm_inout("za");

void streaming_vector_op(float *data, unsigned count)
    __arm_streaming;

The first function runs in streaming mode and shares its caller’s ZA state, so its implementation can accumulate into that state. The second uses streaming vectors but declares no ZA state. These are interface declarations, not complete kernels; the caller and implementation must follow the ZA ownership rules. The compiler manages mode transitions and ZA preservation at call boundaries according to the attributes. Hand-written assembly must follow the calling convention itself.

Resources

• Arm’s streaming mode and ZA walkthrough explains the two state bits and their C/C++ programming model.
• Arm’s ZA array and tile mapping guide shows the array-vector, tile, and tile-slice views.
• Arm C Language Extensions defines the streaming and ZA function attributes.

The original SME provides streaming vectors, ZA tile storage, and outer-product instructions that accumulate into ZA. SME2 keeps those capabilities and adds instructions for groups of two or four vectors, a way to predicate those groups, lookup-table expansion of packed values, and binary outer products. These are different instruction families: not every SME2 instruction takes a two- or four-register group, and not every result goes into ZA.

Outer-product support in SME and SME2

SME outer-product instructions multiply values from two Z registers and accumulate or subtract the result in a ZA tile. Tile-slice moves and loads/stores transfer data between ZA, Z registers, and memory. The main outer-product input and accumulator combinations are:

Narrow inputs can contribute multiple products to each wider accumulator element. For example, an INT8 outer-product instruction sums four INT8 products into each INT32 result, while the FP32 form contributes one product per result.

What SME2 adds

Multi-vector arithmetic and data movement. SME2 can operate on groups of two or four Z registers or ZA vectors, commonly marked VGx2 and VGx4. Representative arithmetic instructions include multiply-accumulate (FMLA/FMLS), widening multiply-accumulate (FMLAL/FMLSL, SMLAL/UMLAL), and dot product (SDOT/UDOT, BFDOT). It also adds multi-vector addition, subtraction, shifts, and selection. Some instructions accumulate into ZA; others produce Z-register results. Multi-vector loads/stores (LD1/ST1), moves (MOVA), permutes (ZIP/UZP), and conversions help feed those calculations and handle their results. The supported element types vary by instruction.

For example, this MOVA transfers two Z registers into two ZA array-vector groups:

MOVA ZA.D[w8, #0, VGx2], { Z0.D-Z1.D }

The VGx2 operand describes the ZA destination grouping. It does not mean that every SME2 instruction uses VGx2 or writes ZA. A multi-vector LD1D, for instance, loads Z registers from memory. A widening arithmetic example is:

FMLAL ZA.S[W8, 0:1, VGx2], { Z0.H-Z1.H }, Z2.H

Here, FP16 source values from the Z registers are multiplied and accumulated as FP32 values in ZA. The 0:1 selects a double-vector group, and VGx2 makes this operation update two such groups.

Predicate-as-counter. SME2 introduces the PN interpretation of the existing predicate registers for multi-vector operations. A counter predicate can describe a run of active elements across two or four vectors, which is useful at the end of a loop when the remaining data does not fill the whole group. For example, WHILELT PN8.S, X0, X1, VLx2 generates a predicate for up to two vectors of 32-bit elements; PEXT can extract an ordinary predicate mask from it. The mechanism also supports 8-, 16-, and 64-bit elements. PN0 and P0 refer to the same underlying predicate register; the instruction determines how its bits are interpreted. The predicate-as-counter WHILELT form above restricts its destination to PN8–PN15. The encoding can also represent a run of inactive elements followed by active ones, so it is broader than just an active-prefix count.

Lookup-table expansion. SME2 adds the fixed 512-bit ZT0 register, arranged as sixteen 32-bit table entries. LUTI2 uses packed 2-bit indices to select among four entries; LUTI4 uses packed 4-bit indices to select among sixteen. The selected values expand into 8-, 16-, or 32-bit elements in Z registers. A kernel can load packed weights, expand them with LUTI2 or LUTI4, then pass the resulting Z values to SDOT or SMOPA for accumulation in ZA.

The lookup is a preparation step, not a 2-bit or 4-bit arithmetic operation. ZT0 can be loaded, stored, and cleared with LDR, STR, and ZERO. LUTI forms differ by output width and register count; in particular, the four-register, 8-bit LUTI4 form requires the optional FEAT_SME_LUTv2 extension.

Binary outer products. BMOPA and BMOPS use bitwise XNOR and population count to accumulate or subtract binary outer-product results in INT32 ZA elements. They serve workloads such as binary neural networks. These instructions use packed bits in their source vectors; they are not multi-vector arithmetic on 1-bit Z-register lanes.

Which data types are actually supported?

It helps to separate arithmetic inputs, ZA accumulators, and packed lookup indices:

LUTI can produce 8-bit patterns that represent FP8 values, but producing those patterns is different from having FP8 arithmetic instructions. When selecting a kernel, check the required optional feature as well as SME2 itself. Arm lists the SME FP8 arithmetic features as optional from Armv9.2, rather than making them a baseline Armv9.5 feature. The Arm C Language Extensions feature macros describe the compiler-visible distinctions.

Resources

• Arm’s SME2 instruction introduction illustrates multi-vector operands and predicate-as-counter with instruction examples.
• Arm’s LUTI walkthrough follows packed indices through ZT0, expanded Z registers, and a matrix instruction.

An SME outer-product instruction reads two streaming Z registers and updates a two-dimensional ZA tile. For FP32, FMOPA multiplies every active element of the first vector by every active element of the second, then adds each product to the corresponding tile element. Keeping that tile in ZA lets a matrix kernel accumulate many contributions before storing the result.

From a vector pair to a matrix product

For two-element vectors, the outer product looks like this:

[ a₀ ]                [ a₀b₀  a₀b₁ ]
[ a₁ ] × [ b₀ b₁ ] =  [ a₁b₀  a₁b₁ ]

An inner product pairs elements and reduces them to one number. An outer product forms a matrix of pairwise products. To multiply matrices, take a column of the left matrix and the matching row of the right matrix, form their outer product, and add it to the output. Repeat for every matching column and row. An SME kernel accumulates these contributions in the same ZA tile before storing it. Each input value contributes to several output elements.

The FMOPA instruction: inputs and result

Here is the FP32 instruction form:

FMOPA ZA0.S, P0/M, P1/M, Z0.S, Z1.S

Z0.S supplies the first input vector and Z1.S the second. ZA0.S is the FP32 destination tile, whose previous contents are accumulated into. P0/M selects active elements of Z0 (output rows); P1/M selects active elements of Z1 (output columns). For every selected row i and column j, the instruction performs ZA0[i, j] += Z0[i] × Z1[j]. Elements outside either predicate remain unchanged. FMOPS has the corresponding subtract behavior.

The instruction requires streaming mode and ZA enabled. A complete kernel must initialize or load the tile before the inner loop, place the appropriate input fragments in Z registers, and store the finished tile afterward. For a matrix stored row by row, obtaining column fragments efficiently often requires packing or another layout strategy; the mathematical column is not necessarily a contiguous memory load. At matrix edges, the kernel also predicates its input loads and tile stores so it stays within bounds. The FMOPA line above is a real instruction, but it is not a complete kernel.

Widening variants

The simple FP32 case is one outer product per instruction. With narrower inputs, an instruction combines several packed outer products into each wider ZA element:

Wider accumulation lets a kernel combine low-precision inputs while retaining more precision in its partial sums, though it does not eliminate rounding or integer overflow concerns.

For a concrete INT8 case, suppose the streaming vector length (SVL) is 128 bits. Each source Z register holds 16 bytes, arranged for this instruction as four groups of four INT8 values. One SMOPA or UMOPA updates a 4×4 INT32 tile: each output element receives a sum of four INT8 products. In general, the instruction treats each source as an (SVL/32)×4 group of INT8 values and updates an (SVL/32)×(SVL/32) INT32 tile. This is a short matrix product of depth four packed into one instruction, rather than the single rank-one update shown by FP32 FMOPA.

Why kernels use it

For FP32, let L = SVL / 32 be the number of FP32 elements in a streaming Z register. With all lanes active, one FMOPA updates an L×L tile, or L² accumulator elements, using two L-element source vectors. At a 512-bit SVL, L is 16, so that instruction performs 256 multiply-accumulate updates from 32 source elements. The number of updates grows quadratically with L while the source element count grows linearly. Each source value participates in multiple updates, and the partial sums stay in ZA across the inner loop.

This describes operand reuse and arithmetic work, not a guaranteed number of memory loads or a per-cycle throughput. Actual performance depends on input layout and packing, tile initialization and stores, the available bandwidth, the processor’s implementation, and how much work the kernel has. Vector-length-agnostic code can use the same instructions across supported SVLs by sizing its loops and data layout at runtime. A larger SVL changes the tile shape; it does not by itself guarantee a proportional speedup.

Independent row and column predicates let the same instruction update partial tiles when an output dimension is not a multiple of the tile size. They avoid changing inactive ZA elements; the surrounding load and store code still has to handle the boundaries correctly.

Resources

• Arm’s outer-product Learning Path shows how matrix multiplication becomes a sequence of tile updates and why input layout matters.
• Arm’s SME instruction walkthrough explains FMOPA, predication, and the widening variants with diagrams.

Arm publishes several official references for SME and SME2 because they answer different developer questions. Start with the Programmer’s Guide to understand the model. Use the A64 instruction pages for exact instruction behavior, Arm C Language Extensions (ACLE) for C/C++ code, and AAPCS64 for rules at function-call boundaries.

Learn the programming model: SME Programmer’s Guide

The SME Programmer’s Guide (document 109246) is the practical starting point. Its SME overview explains streaming SVE mode, the ZA array and tile views, and how SME2 extends the instruction set. The guide also has sections on toolchains and worked matrix-multiplication code, so you can see how a kernel enters streaming mode, computes in ZA, and stores a result. Use its support and feature-detection discussion when deciding which path your application can run.

Check an instruction precisely: A64 instruction reference

The Arm A-profile A64 instruction reference (DDI 0602) has a page for each SME or SME2 instruction. Look here when the exact assembler form or behavior matters. For example, find FMOPA to check its operands and predication, SMSTART to check state changes, or LUTI2 to check its lookup operands and required feature. The instruction pages give encodings, operand constraints, and a precise description of the operation. They are more useful for one instruction than a broad overview.

Write C or C++: Arm C Language Extensions

The ACLE SME section specifies <arm_sme.h>, SME and SME2 intrinsics, feature-test macros, and function attributes. Consult it to find the intrinsic for an instruction and to declare the state a function uses. For example, __arm_streaming says a function uses streaming mode, while __arm_inout("za") says it shares ZA state with its caller. A compiler feature macro describes what the current compilation target supports; it does not by itself prove that the processor and operating system running the program expose the feature.

Handle function calls and ZA state: AAPCS64

The AAPCS64 SME sections specify the procedure-call rules. Read them when an SME function calls other code, a library exposes an SME-aware function, or hand-written assembly must preserve state. They distinguish streaming and non-streaming interfaces, describe private and shared ZA state, and define when ZA needs to be saved. ACLE gives you the source-language attributes; AAPCS64 explains the calling contract behind them.

For a new kernel, a useful order is: learn streaming mode and ZA in the Programmer’s Guide, look up each instruction you use in the A64 reference, then check ACLE and AAPCS64 for how your function enters, shares, and leaves that state.

SME2 is a superset of SME, so SME2-capable hardware supports both instruction sets. Documented consumer devices include Apple A18/A19 and M4/M5 devices plus Android phones with SME2-enabled Arm C1 CPU clusters. The examples below have SME2, but do not assume every Armv9 device does. Check the exact device and whether its OS exposes the feature.

Note: Apple iPhone and iPad models with A18/A19 chips support SME2 natively, but Arm’s SME2 Learning Path demonstrates SME2 only on macOS (M4) and on Android phones with SME2 support — not on iPhone or iPad.

Devices with Native SME2 Support

Apple Devices

Apple’s A18 and M4 generations introduced SME2 in these product lines. The M5 generation adds further devices, including the M5 iPad Pro and MacBook Pro models with M5 Pro and M5 Max . These are device examples; use the feature check below before running SME2 code.

Android Phones

Arm C1 and C2 CPU Clusters with SME2

Arm’s C1 family is used in SME2-capable CPU clusters. SME2 execution is provided at the cluster level, so the C1 name alone is not a substitute for checking the finished device. The family includes:

These cores appear in SoCs like MediaTek Dimensity 9500 and Samsung Exynos 2600.

Arm announced the C2 CPU cluster in September 2026. It combines C2-Ultra and C2-Pro CPUs with two cluster-level SME2 units. This describes Arm’s CPU IP, not a guarantee that a particular phone exposes SME2 to applications. Check the finished device at runtime before selecting an SME2 code path.

Chips and Cores That Do NOT Support SME

Having Armv9, SVE, or SVE2 support does not imply SME support — SME and SME2 are optional architectural extensions. Notable modern processors that lack SME:

Rule of thumb: Treat SME/SME2 as optional and require an explicit vendor statement or runtime feature check. Product-family guesses are not a safe dispatch mechanism.

How to Verify SME2 on Your Device

On Linux or Android, you can check for SME2 at runtime using the HWCAP2 feature bits:

#include <sys/auxv.h>
#include <stdio.h>

#ifndef HWCAP2_SME2
#define HWCAP2_SME2 (1UL << 37)
#endif

int main(void) {
    unsigned long hwcap2 = getauxval(AT_HWCAP2);
    if (hwcap2 & HWCAP2_SME2)
        printf("SME2 is supported\n");
    else
        printf("SME2 is NOT supported\n");
    return 0;
}

On macOS (Apple Silicon), use sysctl:

sysctl hw.optional.arm.FEAT_SME2
# Returns 1 if supported

Development Without Native Hardware

If you don’t have access to SME2-capable hardware, you can use Arm’s Fixed Virtual Platform (FVP) to emulate an SME2-capable processor. Arm provides a Docker-based workflow that runs SME2 code in bare-metal mode on the FVP. See the Learning Path linked below for setup instructions.

Software and Toolchain Support

Software for building or using SME2 code includes:

• Arm KleidiAI — micro-kernel library for matrix operations, integrates with PyTorch, ExecuTorch, XNNPACK, ONNX Runtime, and llama.cpp
• LLVM/Clang 18+ and GCC 14+ — compiler support for SME/SME2 intrinsics and code generation
• Arm Compute Library (ACL) — backend support for ML frameworks

Resources

• Arm: devices with native SME2 support and setup options
• Arm: check SME2 availability on your device

SME devices are not all the same. They share a programming model with streaming mode and ZA tile storage, but can differ in SME generation, streaming vector length, optional instructions, matrix-unit sharing, and performance. The operating system and libraries also determine which capabilities an application can use. Check features at runtime and measure performance on the target device.

Key Differences Across SME Implementations

1. SME Generation (SME vs. SME2 vs. SME2.1)

The SME architecture has evolved through multiple generations, each adding significant functionality:

SME2 is a superset of SME, so any processor implementing SME2 also supports the base SME instructions.

2. Streaming Vector Length (SVL)

The most visible difference between implementations is the Streaming Vector Length (SVL), which is chosen by the hardware designer:

• The architecture permits SVL values of 128, 256, 512, 1024, or 2048 bits (any power of two in that range).
• SVL determines the width of vector operations in streaming mode and the dimensions of the ZA tile. On Linux, the effective SVL is a per-thread setting that the OS can limit to a supported value.
• A processor’s SVL does not have to match its non-streaming SVE vector length (NSVL). They are independent parameters.

3. ZA Tile Size (Scales Quadratically with SVL)

The ZA storage array is a 2D square of dimensions (SVL in bytes) × (SVL in bytes). Because the scaling is quadratic, small differences in SVL create large differences in available tile storage:

A wider SVL means more data processed per outer product instruction and a larger working set held in ZA without spilling to memory.

4. Optional Architectural Features

Hardware designers can selectively implement optional sub-features identified via the ID_AA64SMFR0_EL1 system register. These vary based on whether the target is HPC, mobile, or server:

Without FEAT_SME_FA64, streaming mode permits only a subset of AArch64 instructions. Code that executes an instruction unavailable in that mode can fault; keep non-streaming work outside the streaming region or check the needed feature.

5. Physical Matrix Units and Multithread Scaling

SME and SME2 define the architectural instructions and state, but they do not require a private matrix execution unit for every CPU core. A system can have one or more physical SME compute units shared by several cores. For example, an Arm C1-DSU cluster can be configured with one or two C1-SME2 units; Arm’s newer C2 cluster has two SME2 units. Other implementations may use a different topology or may not publish it.

This has important consequences for application threading:

• With one shared unit, a single SME-heavy thread may perform very well, while additional SME-heavy threads contend for the same resource. Scaling can flatten or even regress.
• With two units, two concurrent SME-heavy threads may scale better, but cache, memory-bandwidth, frequency, and thermal limits can still prevent linear scaling.
• On heterogeneous systems, scheduler placement and migration can change results. Pinning threads to a documented CPU set makes tests reproducible, but it does not make a shared SME unit private to those threads.
• Runtime feature detection such as HWCAP2_SME2 only proves that SME2 instructions are available. It does not report the number of physical matrix units or their sharing topology.

Do not choose the thread count from the number of CPU cores alone. Benchmark 1, 2, 4, and higher thread counts with fixed affinity, compare SME2-enabled and fallback runs at every count, and record the CPU set, power mode, frequency behavior, and temperature. Framework kernel tracing should also confirm that each tested operator actually selected an SME2 kernel.

6. Performance Characteristics

Even two processors with identical feature sets and the same SVL can differ in:

• Outer product instruction throughput (cycles per instruction)
• ZA access latency (read/write to tile rows and columns)
• Streaming mode entry/exit cost (penalty for SMSTART/SMSTOP transitions)
• Memory bandwidth and cache hierarchy (affects tile load/store performance)
• Clock speed and thermal limits (mobile vs. server envelopes)

7. System Enablement

Whether applications can actually exploit the hardware depends on:

• OS kernel support: saves and restores ZA and streaming state across context switches
• Compiler versions: provide SME intrinsics and code generation
• Hypervisor and firmware (EL2/EL3): must allow user-space SME execution
• Libraries: provide optimized kernels (for example, KleidiAI) and runtime dispatch

Runtime Feature Detection

Because features vary, software must query the hardware at runtime rather than assuming a fixed feature set.

Linux (HWCAP2 flags)

#include <sys/auxv.h>
#include <stdio.h>

// HWCAP2 flags for SME features; use system definitions when available.
#ifndef HWCAP2_SME
#define HWCAP2_SME         (1UL << 23)
#endif
#ifndef HWCAP2_SME2
#define HWCAP2_SME2        (1UL << 37)
#endif
#ifndef HWCAP2_SME_F64F64
#define HWCAP2_SME_F64F64  (1UL << 25)
#endif
#ifndef HWCAP2_SME_I16I64
#define HWCAP2_SME_I16I64  (1UL << 24)
#endif
#ifndef HWCAP2_SME_FA64
#define HWCAP2_SME_FA64    (1UL << 30)
#endif
#ifndef HWCAP2_SME2P1
#define HWCAP2_SME2P1     (1UL << 38)
#endif

int main(void) {
    unsigned long hwcap2 = getauxval(AT_HWCAP2);

    printf("SME:       %s\n", (hwcap2 & HWCAP2_SME)        ? "yes" : "no");
    printf("SME2:      %s\n", (hwcap2 & HWCAP2_SME2)       ? "yes" : "no");
    printf("SME2.1:    %s\n", (hwcap2 & HWCAP2_SME2P1)     ? "yes" : "no");
    printf("SME_F64:   %s\n", (hwcap2 & HWCAP2_SME_F64F64) ? "yes" : "no");
    printf("SME_I16I64:%s\n", (hwcap2 & HWCAP2_SME_I16I64) ? "yes" : "no");
    printf("SME_FA64:  %s\n", (hwcap2 & HWCAP2_SME_FA64)   ? "yes" : "no");

    return 0;
}

For manual inspection, /proc/cpuinfo also lists sme, sme2, and related flags. Use AT_HWCAP2 for program dispatch.

macOS

On Apple Silicon with SME support, use sysctlbyname:

#include <sys/sysctl.h>
#include <stdio.h>
#include <stdint.h>

int main(void) {
    int64_t sme = 0;
    size_t size = sizeof(sme);
    if (sysctlbyname("hw.optional.arm.FEAT_SME", &sme, &size, NULL, 0) == 0) {
        printf("SME: %s\n", sme ? "supported" : "not supported");
    } else {
        printf("SME: query unavailable\n");
    }
    size = sizeof(sme);
    if (sysctlbyname("hw.optional.arm.FEAT_SME2", &sme, &size, NULL, 0) == 0) {
        printf("SME2: %s\n", sme ? "supported" : "not supported");
    } else {
        printf("SME2: query unavailable\n");
    }
    return 0;
}

Writing Portable SME Code

The architectural intent is that Vector Length Agnostic (VLA) SME code remains functionally portable across all implementations. Best practices:

1. Never hard-code SVL. Use the ACLE svcntsb() intrinsic to get the streaming vector length in bytes, including when the caller is in non-streaming mode. svcntb() reports SVL only while running in streaming mode; otherwise it reports the non-streaming vector length.
2. Check optional features before using specialized instructions (FP64 outer products, FA64 mode).
3. Use libraries like KleidiAI that perform runtime detection and dispatch to the best available kernel for the current hardware.
4. Test on multiple SVL values using suitable hardware or an Arm FVP configured for different streaming vector lengths.

Resources

• Arm C Language Extensions: SME intrinsics and streaming-mode rules
• Arm C1-SME2 Technical Reference Manual, page 9: shared-unit topology
• Linux SME userspace interface: HWCAP2 and vector-length controls
• SqueezeSAM case study: shared SME2 resource and one-core vs. four-core scaling

SME (Scalable Matrix Extension) is an optional Arm architectural extension for matrix computation. It adds a streaming execution mode, scalable vector operations, and a two-dimensional register array called ZA. Outer-product instructions multiply values from two vectors and accumulate the results directly in ZA, which is useful for matrix multiplication and related kernels.

How it works

Suppose one vector holds two values from a column of matrix A and another holds two values from a row of matrix B. One outer-product step updates four accumulators in ZA: each value from A is multiplied by each value from B, and each product is added to its corresponding result. Repeating that step for successive columns and rows builds a matrix product while keeping partial results in ZA.

SME has three pieces that matter to a developer:

SME is most useful inside a compute kernel, such as matrix multiplication, convolution, or digital signal processing. The surrounding application can continue to use ordinary AArch64 and Neon code. Streaming mode has instruction restrictions and switching costs, so entering it for every small operation is not automatically beneficial.

How SME relates to SVE and SME2

SVE and SVE2 provide scalable vector instructions. SME adds streaming vector execution and ZA matrix state; it does not imply that the processor also exposes non-streaming SVE or SVE2. SME2 builds on SME with multi-vector operations, multi-vector predication, and the ZT0 lookup-table register. A lookup instruction can expand packed 2-bit or 4-bit indices into values used by a matrix kernel; this is not the same as native 2-bit or 4-bit arithmetic.

An Armv9 label alone does not guarantee SME. On AArch64 Linux or Android, check HWCAP2_SME or HWCAP2_SME2 before selecting a kernel. On macOS, check hw.optional.arm.FEAT_SME or hw.optional.arm.FEAT_SME2. Hardware support, operating-system exposure, and the library’s selected kernel are separate questions.

Resources

• Arm: introduction to SME’s programming model
• Arm: SME2 matrix multiplication Learning Path

An original SVE or SVE2 instruction is not streaming-only. SME and SME2 add instructions that require streaming mode; many existing SVE instructions work in either mode; some SVE and SVE2 forms require non-streaming mode. Classify the instruction form, including its addressing mode, rather than the mnemonic alone. For example, a contiguous LD1W and a gather LD1W have different rules.

This is a guide to the common cases, not a complete opcode list. Availability also depends on whether the processor implements the instruction’s separate feature requirements.

Streaming-only SME instructions

SME outer-product instructions such as FMOPA, BFMOPA, SMOPA, and UMOPA accumulate into ZA. ZA tile-slice LD1* and ST1* forms transfer tile data, and SME2 adds multi-vector operations. These need streaming mode because their vector operands and tile operations use streaming state. The similar-looking ordinary SVE LD1* forms have their own rules.

Some SME instructions are usable in either mode. RDSVL reads SVL even outside streaming mode, and SMSTART and SMSTOP control the mode. Whole-ZA LDR and STR can save or restore ZA when ZA is enabled; do not confuse them with the streaming-only tile-slice loads and stores.

SVE instructions that work in both modes

Common arithmetic such as ADD, FADD, and FMLA, predicate creation such as PTRUE and WHILELT, and contiguous LD1W/ST1W forms are streaming-compatible. Many logical operations, permutations, and reductions are too. Their vector width follows the current mode, so a loop that assumes a fixed element count is not portable between VL and SVL.

Other useful examples include:

For a particular intrinsic, check its streaming classification in ACLE . Some operations share a mnemonic but have encodings with different streaming behavior. Do not infer permission from a similar instruction name.

SVE instructions that require non-streaming mode

Without FEAT_SME_FA64, streaming mode excludes several instruction forms. Examples include vector-offset gather loads and scatter stores, first-fault loads (LDFF1*), operations on the First Fault Register such as RDFFR and SETFFR, and some cryptographic and in-vector matrix instructions. A gather LD1W can be excluded while a contiguous LD1W remains valid. Attempting to execute an unavailable form can raise an undefined-instruction exception, typically reported as SIGILL on Linux.

Other examples to check before moving a loop into streaming mode are non-faulting loads (LDNF1*), ordered floating-point reduction (FADDA), vector compaction (COMPACT), and SVE2 in-vector matrix multiply (FMMLA, BFMMLA, and integer *MMLA forms). These are distinct from SME outer products, which accumulate into ZA. Even a shared mnemonic can hide different addressing forms, so use the instruction’s specific streaming classification when porting hand-written assembly.

FEAT_SME_FA64 relaxes streaming-mode restrictions for otherwise unavailable A64 instructions, but it is optional. It does not create other missing architectural features. On Linux, test HWCAP2_SME_FA64 before relying on it. Code intended for a range of SME devices should keep such operations outside streaming mode unless they are essential to the kernel.

Express the mode contract in C or C++

ACLE function attributes let the compiler manage mode transitions at call boundaries and reject intrinsics used in the wrong mode:

// Body executes in streaming mode; a non-streaming caller can call it.
void matrix_kernel(float *a, float *b, float *c) __arm_streaming;

// Body can execute in either mode, so it may use only compatible operations.
void scalar_helper(int n) __arm_streaming_compatible;

// Ordinary functions have a non-streaming interface by default.
void gather_helper(const float *base, const unsigned *indices);

These are declarations that illustrate the mode contract, not complete implementations. A function’s mode attribute does not by itself enable ZA or guarantee that its target CPU has SME; compile for the appropriate feature and check runtime support before dispatch.

Resources

• Arm C Language Extensions: streaming-mode rules and function attributes
• Arm SME Programmer’s Guide: Streaming SVE mode

SME (Scalable Matrix Extension) was introduced to make matrix-heavy kernels more efficient on Arm CPUs. Neon and SVE/SVE2 can compute matrix products using vector instructions, but SME adds scalable ZA tile storage and outer-product instructions that update many results at once. SME2 adds multi-vector operations, lookup instructions useful with packed low-bit weights, and bitwise outer products for binary neural networks. These extensions give kernel authors more ways to reduce instruction and data-movement costs; the benefit still depends on the workload and implementation.

The Problem with Vectors for Matrix Workloads

Neon and SVE/SVE2 are fundamentally 1D vector processing extensions. They operate on rows of data elements one vector at a time. While you can build matrix multiplication from repeated vector operations (dot products, FMA loops), the hardware has no native concept of a 2D tile. The programmer must manually manage the blocking, accumulation, transposition, and register pressure that matrix workloads demand.

For workloads dominated by matrix-matrix multiplication—such as Machine Learning, Large Language Models, computer vision, and HPC—vector units face significant bottlenecks:

• High instruction count: Computing a matrix multiply with 1D vectors requires complex nested loops with many multiply-accumulate instructions.
• Excessive data movement: Partial results must be shuffled between vector registers or spilled to memory; input data must be transposed explicitly.
• Register pressure: Keeping accumulators, input vectors, and permutation state in a limited register file forces frequent spills.
• Instruction overhead: An outer-product instruction updates a two-dimensional group of accumulators, while a vector FMA updates a one-dimensional group.

SME uses a streaming vector mode with a subset of SVE2 instructions. It does not require the processor to expose non-streaming SVE or SVE2; check those features separately. Ordinary Neon code remains available outside streaming mode.

What SME Adds Beyond Neon and SVE2

ZA Storage (Matrix Tile Array)

SME introduces ZA storage: a dedicated two-dimensional architectural register array of SVL×SVL bytes (where SVL is the Streaming Vector Length). This is new register state, not a reuse of the Z registers. The ZA array can be accessed as:

• Tile slices — horizontal rows (ZA0H.S[n]) or vertical columns (ZA0V.S[n])
• Sub-tiles — square sub-arrays typed by element width (e.g., ZA0.S, ZA1.D)
• Full ZA vectors — for bulk save/restore operations

This allows accumulation of outer products directly into a 2D tile without consuming the ordinary vector register file.

Outer Product Instructions

Unlike SVE2’s dot-product instructions (which accumulate into a vector), SME’s outer product instructions (FMOPA, SMOPA, UMOPA, etc.) take a column vector and a row vector, compute the entire outer product matrix, and accumulate it directly into a ZA tile:

ZA += column_vector × row_vector^T

In a GEMM inner kernel, this can reduce instruction count and register pressure compared with building the same result from vector FMAs. Packing and memory traffic can still dominate the full operation.

Streaming SVE Mode

SME introduces a separate execution mode controlled via PSTATE.SM. In streaming mode:

• The core uses a Streaming Vector Length (SVL), which can differ from the normal SVE vector length (VL), if non-streaming SVE is implemented.
• SME instructions (outer products, tile loads/stores) become available.
• A subset of SVE2 instructions is available for supporting computation.
• Hardware can be optimized specifically for high-throughput streaming workloads.

Streaming mode is entered with SMSTART SM and exited with SMSTOP SM. A mode change invalidates Z and P register values across the boundary, so code must not carry live vector values through the transition.

On-the-Fly Transposition

ZA tile slices can be accessed horizontally or vertically during load, store, and extract operations. This can avoid some explicit shuffles when a kernel needs a different orientation, although it does not remove all packing or layout work from a matrix multiplication.

What SME2 Adds Beyond SME

SME’s initial scope was outer-product accumulation into ZA tiles. SME2 widens the applicability so that surrounding vector work does not bottleneck the matrix engine:

Multi-Vector Instructions

Operations consume or produce groups of 2 or 4 Z registers simultaneously (e.g., {Z0.D–Z1.D} or {Z0.S–Z3.S}), increasing useful work per instruction. This includes:

• Multi-vector multiply-accumulate — Z vectors as inputs, accumulating into ZA array vectors (including widening forms)
• Multi-vector loads and stores — pre-load multiple vectors in parallel
• Multi-vector moves, permutations, and conversions — efficiently transpose and shuffle data before feeding the matrix engine

ZT0 Lookup Table Register

SME2 adds a dedicated 512-bit architectural register (ZT0) for table lookups. The LUTI (Lookup Table) instructions can decompress packed/quantized weights on-the-fly:

• 2-bit or 4-bit compressed data → 8-bit, 16-bit, or 32-bit containers
• The expanded values can represent supported 8-, 16-, or 32-bit formats, depending on the instruction and other implemented features. The 2-bit or 4-bit values are lookup indices, not arithmetic operands.

This can help a kernel use compressed or quantized neural-network weights without first expanding the entire weight matrix in memory.

Predicate-as-Counter

SVE/SVE2 use bitmask predication where every bit corresponds to an active/inactive lane. SME2 introduces an alternative “predicate-as-counter” mechanism for multi-vector operations:

• The predicate register encodes a count, an element size, and an inversion flag. It can describe an active prefix followed by inactive elements, or an inactive prefix followed by active elements, across a vector group.
• This compact encoding supports loop-tail control in multi-vector loops.

Binary Neural Network Support

SME2 adds bitwise exclusive-NOR population count outer product instructions for 1-bit binary neural networks, extending the architecture’s ML capabilities to extremely compressed model representations.

Important Caveats

• SME does not replace Neon or SVE2. Ordinary vector loops and irregular code may still be simpler or faster with Neon/SVE2. SME/SME2 complement existing extensions.
• Mode transition cost. Entering and exiting streaming mode zeroes Z and P registers. Software must manage transitions carefully to avoid unnecessary overhead.
• Instruction restrictions. Only a subset of SVE2 instructions is available in streaming SVE mode. Code must be restructured for SME-compatible paths.
• SVL and VL can differ. The streaming vector length can be wider or narrower than the normal SVE vector length on the same implementation. Code must remain vector-length agnostic.
• Large context-save state. ZA can be up to 64 KB at a 2048-bit SVL. OS support must preserve the architectural state across context switches.
• OS and compiler support required. SME requires OS awareness for ZA state management and compiler support for function attributes (__arm_streaming, __arm_inout("za"), etc.).

Resources

• Arm: Neon, SVE, and SME matrix multiplication compared
• Arm SME Programmer’s Guide: SME and SME2
• Arm: how SME2 lookup instructions expand low-bit weights

They work alongside one another. Neon (Advanced SIMD) provides fixed-width vectors for general-purpose code. SVE2 provides scalable vectors for workloads such as signal processing and vectorized loops. SME adds streaming vector execution and ZA tile storage for matrix-oriented kernels; SME2 extends SME with multi-vector operations and a lookup-table register. A program can use different paths for different operations, but must check which optional extensions a device exposes.

How the features relate

SME2 includes SME, and SVE2 includes SVE. SME2 does not imply non-streaming SVE2. Streaming SVE provides a subset of SVE2-style operations in streaming mode, but non-streaming SVE2 is a separate feature. This distinction matters on devices that expose SME2 without exposing non-streaming SVE2.

Where execution happens

Most application code runs in non-streaming mode. Neon works there, as does SVE2 when the device implements it. A matrix kernel can enter streaming mode to use SME or SME2 instructions and ZA, then return to non-streaming mode. Streaming SVE operations use the streaming vector length (SVL), which may differ from the non-streaming SVE vector length.

A separate state bit enables ZA. ZA state can be live across a mode transition, but Z and P vector values cannot simply be carried through it. ACLE’s __arm_streaming and __arm_locally_streaming attributes help the compiler manage transitions while preserving the function’s mode contract.

For example, an image pipeline might use Neon or SVE2 for pixel conversion, call an SME2 matrix kernel for a model layer, and then return to ordinary code. Whether that is faster depends on how much work the kernel does relative to packing, memory traffic, and mode-switch costs. It is also valid to use Neon alone on a device with SME2.

Dispatch each optional feature separately

On AArch64 Linux and Android, HWCAP2_SVE2, HWCAP2_SME, and HWCAP2_SME2 report the relevant OS-exposed features. On macOS, use the corresponding hw.optional.arm.FEAT_* queries. Keep a baseline path and select a specialized kernel only when both the device and the implementation support it. A library such as KleidiAI may perform this dispatch internally, but its choice also depends on the operation, data type, and shape.

Resources

• Arm SME Programmer’s Guide: SME and SME2
• Arm: Neon, SVE, and SME matrix multiplication compared
• Arm ACLE: SME streaming-mode function attributes

SME2 includes SME, so code using base SME instructions can run on a processor and operating system that expose SME2. It does not imply non-streaming SVE2. Neon remains part of the AArch64 software baseline, but Neon instructions are not generally usable inside an SME streaming region. These are different kinds of compatibility: instruction-set support, execution-mode rules, and operating-system exposure.

The feature relationship is SME2 → SME and, separately, SVE2 → SVE. It is not one chain from SME2 through SVE2 to Neon. A device can expose streaming SVE operations for SME2 while lacking non-streaming SVE2.

Why the execution mode matters

SME instructions such as outer products operate in streaming mode with ZA enabled. Many SVE-style arithmetic operations also work there, using the streaming vector length (SVL). Other SVE2 forms, including common gather and first-fault operations, require non-streaming mode unless the optional FEAT_SME_FA64 feature is available. Neon code normally runs outside streaming mode.

Switching modes invalidates live Z and P vector values across the boundary. The SME calling convention and ACLE function attributes let a compiler manage transitions while preserving the function’s contract; they do not make arbitrary Neon or SVE2 instructions legal inside a streaming function.

Detect the paths you actually provide

On AArch64 Linux or Android, use the OS-provided HWCAP flags. This complete example reports the relevant features; it does not execute specialized instructions:

#include <stdio.h>
#include <sys/auxv.h>
#include <asm/hwcap.h>

int main(void) {
    unsigned long hwcap = getauxval(AT_HWCAP);
    unsigned long hwcap2 = getauxval(AT_HWCAP2);

    printf("Neon: %s\n", (hwcap & HWCAP_ASIMD) ? "yes" : "no");
    printf("SVE2: %s\n", (hwcap2 & HWCAP2_SVE2) ? "yes" : "no");
    printf("SME:  %s\n", (hwcap2 & HWCAP2_SME) ? "yes" : "no");
    printf("SME2: %s\n", (hwcap2 & HWCAP2_SME2) ? "yes" : "no");
    return 0;
}

On macOS, use the corresponding hw.optional.arm.FEAT_* queries. Compile specialized functions or translation units for their target features while keeping the entry point and fallback code compatible with the oldest device you support. Feature presence only makes a path eligible; the library or application’s dispatch must still select it.

Resources

• Arm ACLE: AArch64 feature dependencies
• Linux arm64 SME userspace interface
• Arm SME Programmer’s Guide: streaming mode

Yes. For one AArch64 platform and ABI, a binary can contain a baseline implementation plus separately compiled SVE2 and SME2 implementations. At runtime it checks the features that the operating system exposes and calls an eligible implementation. The entry point and baseline code must be built for the oldest device the binary supports.

Detect, then dispatch

On AArch64 Linux and Android, getauxval(AT_HWCAP2) reports whether the OS exposes SVE2 and SME2. Check them independently: SME2 does not imply non-streaming SVE2. Neon is the usual AArch64 baseline. On macOS, use sysctlbyname with keys such as hw.optional.arm.FEAT_SME2; the Linux HWCAP interface does not apply there.

This Linux example shows the dispatch contract. The three gemm_* functions are implementations supplied in separate source files with identical signatures:

#include <sys/auxv.h>
#include <asm/hwcap.h>

typedef void (*gemm_fn)(const float *, const float *, float *,
                        int, int, int);

extern void gemm_neon(const float *, const float *, float *,
                      int, int, int);
extern void gemm_sve2(const float *, const float *, float *,
                      int, int, int);
extern void gemm_sme2(const float *, const float *, float *,
                      int, int, int);

static gemm_fn select_gemm(void) {
    unsigned long features = getauxval(AT_HWCAP2);
    if (features & HWCAP2_SME2) return gemm_sme2;
    if (features & HWCAP2_SVE2) return gemm_sve2;
    return gemm_neon;
}

Call select_gemm() once during initialization and retain the returned pointer. This is a dispatch example, not a complete matrix multiplication program: each kernel must define its own layout, dimensions, and numerical behavior consistently.

Build the baseline and specialized source files separately. For example, with a compiler that supports SME2:

clang -O2 -march=armv8-a -c main.c gemm_neon.c
clang -O2 -march=armv9-a+sve2 -c gemm_sve2.c
clang -O2 -march=armv9-a+sme2 -c gemm_sme2.c
clang -o my_app main.o gemm_neon.o gemm_sve2.o gemm_sme2.o

Keep SME2 instructions inside the guarded kernel. Building the entire binary for SME2 can let the compiler put unsupported instructions in unguarded code, causing an illegal-instruction fault on older devices. Link-time optimization also needs checking: inspect the final binary or disable cross-inlining between the baseline and specialized code if the toolchain does not preserve the boundary.

Other ways to dispatch

Function multiversioning lets a supported compiler generate and choose versions of one function using target_version or target_clones. A default version supplies the fallback, and feature selection is generally fixed for the process. This is useful when the versions share an interface; exact compiler, linker, and streaming-mode requirements should be checked for the target platform. The Arm FMV Learning Path shows the syntax and selection rules.

Library dispatch can hide this choice from application code. A framework may use KleidiAI or another optimized backend to select an SME2 microkernel when one exists for the operator, data type, and shape. Hardware detection alone does not prove that a particular call used SME2; inspect backend logs or profile the selected kernel when that matters.

An SME2 kernel enters streaming mode and manages ZA according to the SME calling convention. That mode change is local to the specialized path; callers and fallback kernels can remain ordinary non-streaming code. Test the SME2 path and every fallback on suitable hardware or emulation, and verify both correctness and the kernel actually selected.

Resources

• Linux arm64 ELF hardware capabilities
• Arm ACLE: streaming-mode function attributes

Dense, regular matrix work is the strongest candidate, especially GEMM and operations implemented as blocked matrix multiplication. SME’s outer-product instructions update many ZA accumulators from two vectors while keeping partial results in the tile. That can reduce instruction and register-movement overhead compared with a vector kernel. It does not guarantee a particular speedup or turn a memory-bound workload into a compute-bound one.

The application domain alone does not decide. A convolution can map well to SME when the backend uses an eligible iGEMM kernel, while a small fully connected layer may spend more time moving weights than computing. Scientific solvers and beamforming can benefit where they spend meaningful time in dense blocked operations; their surrounding control flow and sparse work may not.

Why the outer product matters

For a 512-bit streaming vector length, an FP32 vector contains 16 lanes. One FP32 outer-product update can contribute to a 16-by-16 group of accumulators in ZA. This says something about work per instruction, not cycles per instruction, sustained throughput, or end-to-end speedup. A tuned Neon or SVE kernel also reuses loaded values, so a comparison must use similarly optimized implementations.

The Arm SME2 outer-product Learning Path shows how the two input vectors update a tile. The SqueezeSAM case study illustrates a measured case: operator-level GEMM and convolution improved substantially on one Android device, while full-model and four-core gains were smaller because data movement and shared resources still mattered.

When another path is a better fit

Element-wise loops, short reductions, sparse or gather-heavy work, and tiny one-off matrices often lack enough reusable tile work to offset setup and streaming-mode costs. Neon or SVE may be simpler and faster for those cases. Some operations also need instructions or data types that a particular SME implementation does not provide.

To evaluate a workload, profile first. Find the operators consuming absolute time, then check their exact shapes, data types, quantization and packing contracts against an available SME kernel. Compare SME enabled and disabled on the target device with the same thread count and power conditions. Measure end-to-end latency or energy as well as isolated kernel throughput.

No. SME accelerates matrix-style computation, regardless of the application’s domain. Outer-product instructions update ZA tiles from pairs of vectors, so the best candidates are dense, repeated multiply-accumulate operations with useful data reuse. Machine learning is one example; numerical solvers, communications, and some media kernels use the same pattern.

A concrete route outside ML

Consider a scientific application that calls SGEMM or DGEMM repeatedly during a blocked solver. Most of the solver can remain unchanged while an optimized BLAS library selects an SME kernel for those calls. Arm Performance Libraries 26.01 documents SME implementations for SGEMM and DGEMM on compatible Apple M-series Macs. This is library- and platform-specific evidence, not a promise that every BLAS call on every Arm processor uses SME.

A communications example is computing a bank of beamforming outputs from many input channels. If coefficients and samples can be arranged as a dense matrix product, an SME kernel can accumulate many outputs in ZA. If the workload instead processes only a few samples at a time or gathers scattered data, packing and mode-switch costs may outweigh the matrix benefit.

What to check before adopting it

SME can access ZA tile slices in horizontal and vertical orientations, which may reduce some explicit rearrangement. It does not eliminate all layout conversion or transposition cost. Likewise, an FFT or a small color transform is not automatically a good SME workload merely because it can be written mathematically as a matrix product; the practical algorithm may be faster with Neon or SVE.

Profile the existing implementation, identify the dense matrix kernels that dominate elapsed time, and verify support for their precision and shapes. Compare an SME-enabled implementation against a tuned baseline on the target device. Include packing, memory movement, mode transitions, and full-application latency in that comparison.

Resources

• Arm SME Programmer’s Guide: matrix operations and supported types

To decide whether SME is worthwhile, measure where the application spends time, check whether its costly operators can use suitable SME kernels, and compare end-to-end performance on the target device. There is no useful universal threshold for the percentage of matrix multiplication.

Step 1: Profile Your Workload to Identify the Hotspot

Before evaluating SME, measure what fraction of total execution time is spent in operations that can reach an SME implementation. These may include matrix multiplication, supported convolutions, attention projections, and dense linear algebra. The operator name alone does not establish that an SME kernel exists for its shape and data type.

Use profiling tools appropriate for Arm:

• Arm Streamline or perf with PMU counters to identify CPU-time hotspots
• Top-down microarchitecture analysis (using Arm’s topdown-tool or Arm Performance Studio) to classify whether you are backend-bound on compute vs. memory
• Annotation channels in your code to time specific operator types (e.g., what llama.cpp does to separately measure GEMM vs. GEMV time)
• APX code_hotspots recipe for quick profiling without instrumenting source

Record the fraction p of total runtime spent in these matrix-style kernels. If matrix operations don’t dominate your profile, gains from SME will be diluted regardless of how fast it makes the kernel.

Practical operator-level workflow for ML models

Profile the lowered model that actually runs, not only the original PyTorch, ONNX, or LiteRT graph. Delegation, fusion, layout conversion, and quantization can change both the operator set and the execution cost.

1. Run a timing-only baseline with SME2 disabled and collect per-operator latency.
2. Run the same model, inputs, thread count, affinity, and power mode with SME2 enabled.
3. For each expensive operator, record its lowered type, M×N×K or convolution dimensions, data types, quantization format, and whether it was delegated to XNNPACK, MLAS, or another supported backend.
4. Use framework or backend tracing in a separate, trace-enabled build to confirm whether the expected SME2 microkernel was selected. Do not use trace-enabled runs for timing.
5. Rank operators by absolute time saved, then inspect data movement, packing, layout conversion, and non-delegated work that becomes visible after matrix kernels accelerate.

The SqueezeSAM ExecuTorch case study demonstrates this workflow: ETDump operator profiling separates convolution/GEMM time from data movement and other overhead, while the accompanying sme-executorch-profiling repository provides reproducible SME2-on, SME2-off, timing, and trace configurations.

Step 2: Check Data Reuse and Matrix Shapes

SME outer-product instructions multiply elements from two input vectors and accumulate many results in the ZA tile. This can reduce instruction and data-movement overhead when a kernel reuses input data effectively. A well-tuned Neon or SVE kernel also reuses data, so a fixed multiply-to-load ratio does not predict the speedup. Inspect the actual kernel and its memory traffic.

Pay particular attention to skinny output matrices. Shapes such as 6×N or M×6 may leave much of an SME output tile unused even when the other dimension and K are large. Likewise, batch-1 inference often reduces a nominal matrix multiplication to GEMV (M=1), where weight movement can dominate compute. These operations may still have specialized SME2 kernels, but profitability depends on the exact shape, data type, packing cost, streaming vector length, and implementation. There is no universal minimum such as 64×64.

If your workload is already memory-bound (spending most time waiting on cache misses or DRAM bandwidth), wider compute units alone won’t help. Fix the memory bottleneck first, or choose narrower data types (INT8, BF16) to improve effective bandwidth.

Step 3: Evaluate Streaming Mode Transition Costs

SME requires entering Streaming SVE mode via the SMSTART instruction (and exiting with SMSTOP). These transitions carry costs:

• Live Z and P register values cannot be carried unchanged across a mode transition; the compiler or assembly code must respect the streaming-mode rules
• The Streaming Vector Length (SVL) may differ from the normal SVE vector length
• ZA tile state may need to be saved/restored across function calls
• Only streaming-compatible instructions can execute while streaming mode is active; calls to incompatible code require a mode transition

This overhead means you need the matrix work to be large enough per invocation to amortize the transition cost. Patterns that hurt:

• Frequent small matmuls interleaved with non-streaming code
• Scattered matrix operations that prevent hoisting SMSTART/SMSTOP to outer loops
• Library calls that implicitly switch modes

Patterns that help:

• Batching multiple outer-product accumulations before switching back
• Structuring code so all streaming work is grouped in a single call or loop
• Using __arm_streaming function attributes to let the compiler manage transitions

Step 4: Apply Amdahl’s Law

Even a large kernel speedup affects only the time spent in that kernel. If an operator takes 30% of the original runtime and becomes twice as fast, the whole application becomes about 1.18× faster before accounting for transitions, packing, or layout conversion. A smaller hotspot can still justify SME when an existing library supplies the kernel; a larger one may not help if its shapes or formats miss the available implementation.

Step 5: Benchmark on Real Hardware

Do not rely solely on theoretical analysis. Compare equivalent tuned implementations on actual SME-capable hardware:

1. Baseline: Your existing implementation under representative conditions
2. SME candidate: SME-enabled library (e.g., KleidiAI, Arm Performance Libraries) or hand-written kernel
3. End-to-end workload: Measure the full application, not only isolated GEMM throughput

Measure latency, throughput, CPU utilization, and energy. Note that emulators like Arm FVPs provide instruction counts for functional validation but do not model memory bandwidth, cache behavior, or latency — they are not suitable for performance benchmarking.

Make the Decision from Measured Return on Investment

Do not use a universal percentage or matrix-size cutoff. Adopt SME when the measured benefit on the target device justifies the integration and maintenance cost:

1. Identify the operators responsible for meaningful absolute runtime with representative inputs.
2. Confirm that the complete operator signature—shape, layout, types, quantization, and packing contract—can reach an SME implementation.
3. Measure SME2-on versus SME2-off on the actual device using identical runtime conditions.
4. Calculate the end-to-end latency, throughput, or energy saving rather than relying only on microkernel speedup.
5. Compare that saving with the cost of integration, validation, fallback support, and ongoing maintenance.

The adoption bar should be lower when a framework already dispatches to KleidiAI automatically and higher when you must write and maintain a custom kernel. Product constraints also differ: a 10% latency reduction may be decisive for an interactive deadline or energy budget, while a larger gain may still be irrelevant outside the user-visible critical path.

Resources

• Arm’s SME2 benchmarking guide
• Streaming mode and ZA state

There is no single expected SME2 speedup for convolution, fully connected layers, or an entire model. In one controlled SqueezeSAM deployment, Arm and PyTorch measured 3.4–4.1× faster GEMM, 4.4–9.0× faster convolution, and 1.83–3.90× faster single-core end-to-end inference, depending on precision. Those figures are useful evidence of what SME2 can achieve, but they are not general forecasts: operator shapes, data types, kernel coverage, thread count, hardware topology, memory behavior, and the comparison baseline all matter.

Published SqueezeSAM Case Study

The measurements below come from a specific ExecuTorch and Arm SME2 SqueezeSAM case study :

• Device: vivo X300 Android smartphone with an SME2-enabled Arm CPU
• Model: SqueezeSAM, a convolution-heavy interactive image-segmentation model
• Runtime path: ExecuTorch → XNNPACK → KleidiAI
• Comparison: the same model and device with SME2 enabled versus disabled
• Power setting: normal/default mobile mode

Operator-level results on one CPU core

These are category-level timings from this model, not universal properties of every GEMM or convolution shape.

End-to-end results

The gap between operator and end-to-end speedup illustrates Amdahl’s law: after convolution and GEMM accelerate, data movement, layout conversion, non-delegated operators, and framework overhead account for more of the remaining latency. The weaker four-core ratios also show why single-thread results must not be extrapolated linearly when SME2 compute resources or other system resources are shared.

Why SME2 Achieves These Gains

SME outer-product instructions accumulate many products into a two-dimensional ZA tile. Keeping partial sums there can reduce repeated movement through general vector registers and memory. SME2 also adds multi-vector operations that can help feed suitable kernels. These mechanisms explain the opportunity, but instruction counts alone do not establish a speedup: execution throughput, packing, cache traffic, and the quality of the comparison kernel also matter.

Why the operator results differ

In this SqueezeSAM configuration, convolution accounted for much more baseline time and achieved a larger speedup than GEMM. That does not establish a general rule that convolution always benefits more. A convolution lowered to a supported iGEMM path can have good data reuse, while a fully connected layer with small M or batch size can become bandwidth- or packing-bound; different shapes can reverse the outcome.

Why FP16 shows the larger ratio here

FP16 started from a substantially slower non-SME2 baseline in this experiment. With SME2 enabled, FP16 and INT8 both reached approximately 300 ms on one core, producing a larger relative ratio for FP16. This result demonstrates that SME2 can make FP16 practical for this model; it does not mean FP16 will generally outperform INT8 or always receive a larger speedup.

Factors That Affect Actual Speedup

How to Access These Speedups

The case study used ExecuTorch → XNNPACK → KleidiAI for supported operator shapes and types. Other runtimes can expose SME2 through optimized backends, but installation alone does not guarantee that a particular layer selects an SME2 kernel. Check the framework and backend versions, supported operator configurations, and runtime dispatch. Use trace or instruction profiling to confirm the selected kernel, then run timing in a separate measurement build.

How to Use Published Numbers

Use published speedups as a sanity check and a source of hypotheses, not as a performance commitment. A defensible estimate should always state:

• the device and number of SME compute units;
• model, operator shapes, and precision;
• framework, backend, and KleidiAI/XNNPACK versions;
• thread count, affinity, and power mode;
• whether the comparison disables only SME2 or changes other kernels or build settings;
• both operator-level and end-to-end latency.

For a new model, run an SME2-on/SME2-off A/B test under identical conditions and profile individual operators. Report a range only after measuring representative shapes and inputs on the intended device fleet.

Resources

• Benchmarking a LiteRT model with KleidiAI SME2 kernels
• LiteRT supported operator configurations

As of September 2026, documented consumer processor families with SME2 include Apple’s A18/A19 and M4/M5 generations and Arm C1-based clusters, such as MediaTek Dimensity 9500 and Samsung Exynos 2600. Arm has also announced a C2 cluster with two SME2 units. SME2 includes SME, but neither Armv9 nor SVE2 alone guarantees SME. Check the exact device and OS before selecting an SME code path.

Processor Families and Implementations

The C1 and C2 SME2 units are cluster resources; do not infer a separate matrix engine per core. An announced CPU design also differs from a shipping device whose OS exposes the feature. Apple’s SME2 implementation does not imply support for ordinary, non-streaming SVE2 instructions.

Arm’s device list and setup guide gives current hardware examples. Arm’s C2 cluster announcement describes the newer IP.

Check the Feature on the Target Device

On Linux or Android, query the kernel’s AT_HWCAP2 bits. This tests what the OS exposes to your process:

#include <sys/auxv.h>
#include <asm/hwcap.h>
#include <stdio.h>

int main(void) {
    unsigned long features = getauxval(AT_HWCAP2);
    printf("SME: %s\n", (features & HWCAP2_SME) ? "yes" : "no");
    printf("SME2: %s\n", (features & HWCAP2_SME2) ? "yes" : "no");
    return 0;
}

On macOS, run sysctl hw.optional.arm.FEAT_SME2; a result of 1 reports OS-exposed SME2 support. A compiler macro such as __ARM_FEATURE_SME2 says what a translation unit was built to use; it does not detect the current device. Arm’s FVP setup can validate functionality without SME2 hardware, but it cannot predict device performance.

There is no authoritative registry of SME-enabled libraries. As of September 2026, the projects below provide useful starting points. Distinguish a library that contains SME instructions from a framework that may reach them through a backend; even then, support depends on the exact build, operator, type, and shape.

Libraries with Direct SME/SME2 Implementations

These libraries contain native SME or SME2 assembly/intrinsic code:

KleidiAI Details

KleidiAI is Arm’s primary vehicle for making SME performance accessible without requiring developers to write SME intrinsics directly. Key characteristics:

• Tuned micro-kernels for specific Arm microarchitectures
• C/C++ API designed for integration into ML/AI frameworks
• Supports F32, FP16, INT8, and dynamically quantized matrix multiplication
• Source: KleidiAI GitLab Repository

KleidiCV’s multiversion build can select an implementation at runtime. To assess a particular operation, inspect its backend source and trace the selected function; the presence of an SME backend elsewhere in the library does not prove this call uses it.

Frameworks with SME Support (via KleidiAI/XNNPACK)

These higher-level frameworks obtain all or part of their SME acceleration through KleidiAI, XNNPACK, MLAS, or another native backend:

How the Integration Works (LiteRT Example)

When KleidiAI kernels are included in an XNNPACK build, XNNPACK may choose one for a supported operator configuration after checking the processor and kernel requirements. Operator type, data type, shape, layout, and quantization details all matter.

whisper.cpp reaches KleidiAI through its shared GGML CPU backend rather than through XNNPACK. Its upstream build system exposes GGML_CPU_KLEIDIAI, which is disabled by default. Enabling the option makes the KleidiAI buffer and kernels available, but it does not mean every Whisper operation uses SME2: dispatch still depends on the processor features and whether an operation, tensor type, and shape match a supported KleidiAI kernel. See the upstream whisper.cpp GGML build option and KleidiAI backend registration .

Important “Support” Distinctions

Do not treat “runs on an SME device” as proof of SME support. A project may contain native SME code (oneDNN, OpenBLAS, Eigen), delegate to a library that may dispatch SME (NumPy/SciPy through OpenBLAS, JAX/XLA through a specially built ACL path), or merely run on the same hardware while using Neon, SVE, or a GPU (MLX’s normal Metal path). Confirm the exact operator, data type, build option, and runtime-selected kernel.

BLIS has an open Apple-M4-focused SME2 implementation in pull request #921, but the current upstream release does not yet provide production SME support. MLX has no explicit SME backend. JAX/XLA does not natively lower ordinary JAX operations to SME, although custom calls or an ACL-enabled XLA build can reach external SME kernels.

Finding SME Support in Source Trees

When evaluating whether a library has genuine SME acceleration (vs. build-system boilerplate), search for these indicators in the source code:

• C keywords: __arm_streaming, __arm_new_za, __arm_inout("za")
• Assembly: smstart, smstop, fmopa, fmops
• Headers: #include <arm_sme.h>
• Build options that include SME kernels, such as XNNPACK_ENABLE_ARM_SME2 or OpenBLAS’s ARMV9SME target
• Architecture flags: -march=armv9.2-a+sme, -march=armv9.3-a+sme2

Caution: Some projects add SME build infrastructure without providing optimized kernels. For example, libyuv initially added an SME implementation file that was largely empty. Always verify actual kernel implementations exist, not just build scaffolding.

The Arm Software Ecosystem Dashboard is a broad discovery tool, not an SME support certification list. For a specific project, its own source and release notes are stronger evidence.

Resources

• Eigen vectorization and SME enablement
• OpenBLAS target and kernel coverage
• ArmPL’s documented SME GEMM support

Match each costly lowered operator to a kernel in the exact backend version your application uses, then verify that the runtime selected that kernel. A device with SME2 and a library containing SME2 code are necessary for that path, but neither establishes that your model’s operator reaches it.

1. Record What Actually Runs

Inspect the delegated or lowered graph, not only the original model. Fusion, quantization, and layout conversion can change its operations. For each expensive node, record:

Record actual runtime shapes for dynamic models. A layer called “INT8 matmul” is underspecified: a kernel for symmetric per-channel INT8 weights need not accept asymmetric per-tensor weights or a different packed representation.

2. Trace the Backend to a Kernel

First identify the execution path: for example, ExecuTorch or LiteRT can delegate eligible nodes to XNNPACK, which can use KleidiAI; llama.cpp can reach KleidiAI through GGML. The presence of KleidiAI in a source tree does not establish that the delegate was enabled in your build.

For the same KleidiAI commit embedded by your backend, inspect its kernel inventory and source . Match the operation, complete type and quantization signature, CPU feature requirement, tile sizes, packing functions, and any restrictions on dimensions or strides. Then inspect the backend’s registration and selection code to confirm that it exposes that kernel for the model operator. A kernel may handle a non-tile-aligned shape correctly through tail processing yet lose to another implementation on that shape.

As one concrete example, a packed 4-bit weight matrix with dynamically quantized 8-bit activations needs a kernel and packer that agree on the block layout, zero-point convention, and scale representation. Another 4-bit file format is not automatically compatible. The ExecuTorch node-to-KleidiAI guide shows how to connect a delegated node to its candidate microkernel; use its support table only for the versions it documents.

3. Prove the Selected Path

Check that the OS exposes SME2 on the target processor. On Linux or Android, use getauxval(AT_HWCAP2) with HWCAP2_SME2 in a C/C++ program; on macOS, query hw.optional.arm.FEAT_SME2. A compile-time __ARM_FEATURE_SME2 macro says only that the translation unit may use the instructions.

Next, collect evidence at the backend level. Depending on the runtime, use a dispatch trace, a breakpoint on the candidate kai_* symbol, or a sampled call stack. Inspect the selected function’s disassembly for SME instructions when necessary. Operator timing alone can show a benefit but cannot identify which instructions ran. For ExecuTorch, ETDump can reveal operator and delegate timing; combine it with backend tracing to identify the microkernel. For LiteRT, Arm’s benchmark and node-inspection guide explains one way to inspect the selected path.

Run timing with tracing disabled, then compare SME2-enabled and SME2-disabled builds under the same device, input, thread, and power conditions. A successful inference run proves that some path works; it does not prove SME2 selection.

Keep a Per-Operator Result

Classify each node as selected, eligible but not selected, needing conversion, unsupported, or still unverified. This distinction lets you improve the model or integration without confusing “runs correctly” with “uses SME2.”

There are now two distinct routes. Current source builds of PyTorch can include direct KleidiAI-backed eager CPU kernels, while ExecuTorch uses the mature deployment path ExecuTorch → XNNPACK → KleidiAI → SME2. Neither route means every operator uses SME2: kernel availability, data type, shape, quantization, hardware, and runtime dispatch must all match.

PyTorch vs. ExecuTorch: Where SME2 Lives

There is no separate “SME2 PyTorch wheel” analogous to a CUDA wheel. For a predictable edge-inference deployment and easy SME2-on/off comparison, the recommended workflow remains:

1. Develop and train your model in PyTorch
2. Export to ExecuTorch .pte format
3. Run inference with an ExecuTorch runner built with XNNPACK + KleidiAI

Prerequisites

• Compiler: Use an AArch64 toolchain that can compile the SME2 kernels in the pinned XNNPACK version; Clang 18+ and Android NDK r27+ are practical choices for the documented path. Check the build log because older toolchains may disable those kernels.
• Build tools: Install the versions required by the pinned ExecuTorch release, plus Ninja, Python, and Git.
• Target hardware: An AArch64 device implementing SME2 with OS/kernel support exposing SME2 to userspace
• Source: Clone ExecuTorch with --recurse-submodules to include XNNPACK and KleidiAI

Important: Do not globally add -march=...+sme2 to your build flags. XNNPACK builds separate micro-kernels with the appropriate architecture flags and performs runtime dispatch, preserving binary compatibility with non-SME2 Arm64 systems.

Building ExecuTorch with SME2 (Native Linux or macOS)

git clone --recurse-submodules https://github.com/pytorch/executorch.git
cd executorch

python3 -m venv .venv
source .venv/bin/activate

./install_executorch.sh

cmake -S . -B cmake-out/sme2 -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DEXECUTORCH_BUILD_XNNPACK=ON \
  -DEXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON \
  -DXNNPACK_ENABLE_ARM_SME2=ON \
  -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=ON \
  -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON

cmake --build cmake-out/sme2 --parallel

Key CMake Options

Note: The current ExecuTorch XNNPACK dependency configuration enables SME2 by default. The explicit option makes the intended build configuration visible, but inspect the pinned version and build output to confirm the kernels were actually compiled.

Building ExecuTorch with SME2 (Android)

Using the build script

export ANDROID_HOME=/path/to/android-sdk
export ANDROID_NDK_HOME=/path/to/android-ndk
export ANDROID_NDK="$ANDROID_NDK_HOME"

ANDROID_ABIS=arm64-v8a \
XNNPACK_ENABLE_ARM_SME2=ON \
./scripts/build_android_library.sh

Using CMake directly (custom runner)

cmake -S . -B cmake-out/android-sme2 -G Ninja \
  -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake" \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-26 \
  -DEXECUTORCH_BUILD_XNNPACK=ON \
  -DEXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON \
  -DXNNPACK_ENABLE_ARM_SME2=ON \
  -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=ON \
  -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build cmake-out/android-sme2 --parallel

Using CMake Presets (Recommended for Profiling)

The sme-executorch-profiling project provides ready-made presets that build SME2-on and SME2-off variants for comparison. The build script merges custom presets into ExecuTorch’s defaults:

{
  "name": "android-arm64-v9a",
  "displayName": "Android arm64 with XNNPACK (SME2 ON, timing)",
  "generator": "Ninja",
  "binaryDir": "${sourceDir}/cmake-out/android-arm64-v9a",
  "cacheVariables": {
    "EXECUTORCH_BUILD_XNNPACK": "ON",
    "EXECUTORCH_BUILD_DEVTOOLS": "ON",
    "EXECUTORCH_BUILD_EXECUTOR_RUNNER": "ON",
    "EXECUTORCH_ENABLE_LOGGING": "ON",
    "EXECUTORCH_ENABLE_EVENT_TRACER": "ON",
    "EXECUTORCH_BUILD_KERNELS_QUANTIZED": "ON",
    "EXECUTORCH_XNNPACK_ENABLE_KLEIDI": "ON",
    "CMAKE_BUILD_TYPE": "RelWithDebInfo"
  }
}

{
  "name": "mac-arm64",
  "displayName": "Mac arm64 with XNNPACK (SME2 ON, timing)",
  "generator": "Ninja",
  "binaryDir": "${sourceDir}/cmake-out/mac-arm64",
  "cacheVariables": {
    "EXECUTORCH_BUILD_XNNPACK": "ON",
    "EXECUTORCH_BUILD_DEVTOOLS": "ON",
    "EXECUTORCH_BUILD_EXECUTOR_RUNNER": "ON",
    "EXECUTORCH_ENABLE_LOGGING": "ON",
    "EXECUTORCH_ENABLE_EVENT_TRACER": "ON",
    "EXECUTORCH_BUILD_KERNELS_QUANTIZED": "ON",
    "EXECUTORCH_XNNPACK_ENABLE_KLEIDI": "ON",
    "CMAKE_BUILD_TYPE": "RelWithDebInfo"
  }
}

Build with:

bash model_profiling/scripts/build_runners.sh

Building PyTorch from Source with KleidiAI (Experimental)

For an experimental native AArch64 PyTorch build that includes KleidiAI integration:

git clone --recursive https://github.com/pytorch/pytorch.git
cd pytorch

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install --group dev

export CC=clang-18
export CXX=clang++-18
export USE_XNNPACK=1
export USE_KLEIDIAI=1
export MAX_JOBS="$(nproc)"

python -m pip install --no-build-isolation -v -e .

Verify KleidiAI was included:

import torch
print("KleidiAI:", torch.backends.kleidiai.is_available())

However, USE_KLEIDIAI=1 and is_available() establish build availability, not that an arbitrary eager operation executes SME2 instructions. The ExecuTorch route provides a separately inspectable deployment path for supported delegated operators.

Verifying the Build

Confirm the build configuration from the CMake cache:

grep -E 'EXECUTORCH_XNNPACK_ENABLE_KLEIDI|XNNPACK_ENABLE_ARM_SME2|XNNPACK_ENABLE_KLEIDIAI' \
  cmake-out/sme2/CMakeCache.txt

Example cache entries (types and exact names can differ by pinned revision):

EXECUTORCH_XNNPACK_ENABLE_KLEIDI:BOOL=ON
XNNPACK_ENABLE_ARM_SME2:BOOL=ON
XNNPACK_ENABLE_KLEIDIAI:BOOL=ON

Check the compiler version:

grep -E 'CMAKE_C_COMPILER:|CMAKE_C_COMPILER_VERSION' \
  cmake-out/sme2/CMakeCache.txt

For an A/B comparison, configure two otherwise identical builds, one with -DXNNPACK_ENABLE_ARM_SME2=ON and one with -DXNNPACK_ENABLE_ARM_SME2=OFF. Keep the full command and all other options the same.

A cache value of ON records the requested configuration. Check the build output for kernel compilation, then trace a candidate kernel at runtime. An A/B timing comparison measures the effect but alone does not identify the selected instructions.

Without SME2 Hardware

If you don’t have access to SME2-capable hardware, you can use Docker with Arm’s Fixed Virtual Platform (FVP) to emulate SME2 for development and testing. This allows compiling and running SME2 code in an emulated environment.

Resources

• Arm’s ExecuTorch SME2 setup and runner build
• ExecuTorch release notes for version-specific build behavior

Yes. llama.cpp can use SME-family microkernels through KleidiAI in its GGML CPU backend. Build an AArch64 version with GGML_CPU_KLEIDIAI=ON; the runtime then selects a compatible kernel for supported matrix operations when the OS exposes the required feature. This is operator-specific, so the option does not make every model layer run on SME2. The commands below follow llama.cpp’s upstream KleidiAI build guide .

Native Arm64 Build

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -S . -B build -DGGML_CPU_KLEIDIAI=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --parallel
./build/bin/llama-cli -m /path/to/model.gguf -p "Hello"

On a Mac, another backend such as Metal may take priority. For a CPU-only comparison, build with -DGGML_METAL=OFF, or select CPU execution at runtime where the installed llama.cpp version supports it.

Android Arm64 Build

Set ANDROID_NDK to the NDK root, then configure a portable arm64-v8a build:

cmake -S . -B build-android \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/build/cmake/android.toolchain.cmake" \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-28 \
  -DGGML_CPU_KLEIDIAI=ON \
  -DGGML_NATIVE=OFF \
  -DGGML_OPENMP=OFF \
  -DGGML_LLAMAFILE=OFF \
  -DLLAMA_OPENSSL=OFF
cmake --build build-android --config Release --parallel

Use the binary or installed artifact produced by that build on an Android device that exposes SME2. Keep global -march=...+sme2 flags off the general build: KleidiAI compiles specialized kernels separately, while global flags can make ordinary code incompatible with older devices.

What Dispatch Selects

The GGML CPU backend can use KleidiAI for selected GGML_OP_MUL_MAT paths, including supported floating-point and quantized formats. The exact Q4_0, Q8_0, F16, or other coverage depends on the bundled KleidiAI revision and llama.cpp’s selector. Runtime selection also depends on tensor type, shape, CPU feature detection, and whether another backend owns the operation. A single portable binary can fall back to another compatible CPU kernel on devices without SME2.

The GGML_KLEIDIAI_SME variable is a diagnostic override, not a normal deployment setting:

A positive value cannot enable unsupported instructions. To compare paths, run llama-bench on the same model, device, thread setting, and thermal state with the variable unset and then set to 0. Record prompt processing and token generation separately.

Verify That SME2 Ran

A CPU_KLEIDIAI model buffer size message shows that model tensors use the KleidiAI CPU buffer; it does not prove an SME2 microkernel ran. For a specific matrix operation, trace or break on the selected KleidiAI kernel and inspect its symbol or disassembly for SME instructions. Then repeat the timing run without tracing. Arm’s llama.cpp SME2 walkthrough demonstrates on-device execution and optional kernel-name instrumentation.

Cross-compile ONNX Runtime for Android with onnxruntime_USE_KLEIDIAI=ON passed through --cmake_extra_defines. This makes KleidiAI kernels available to MLAS. On an SME2-capable device, a supported operation can select an SME2 kernel at runtime; the build option alone does not establish that a given ONNX node uses one.

How KleidiAI and SME2 Work in ONNX Runtime

KleidiAI is integrated into ONNX Runtime’s MLAS engine, which handles performance-critical linear algebra operations (GEMM, convolutions, fully-connected layers).

• Runtime dispatch: MLAS checks OS-exposed CPU capabilities and selects from eligible kernels. The full operator signature and the particular ONNX Runtime build also determine whether a KleidiAI SME2 kernel is available.
• Fallback safety: On devices without SME2, MLAS falls back to NEON or SVE kernels. This means a single binary works across all arm64-v8a devices without risking SIGILL crashes on older hardware.
• No global architecture flags needed: You do not need to add -march=armv9-a+sme2 to the entire ONNX Runtime build. The KleidiAI micro-kernels are compiled with the appropriate flags internally, and MLAS handles dispatch.

Prerequisites

Set your environment variables:

# Linux host
export ANDROID_SDK_ROOT="$HOME/Android/Sdk"
export ANDROID_NDK_HOME="$ANDROID_SDK_ROOT/ndk/<ndk-version>"

# macOS host
export ANDROID_SDK_ROOT="$HOME/Library/Android/sdk"
export ANDROID_NDK_HOME="$ANDROID_SDK_ROOT/ndk/<ndk-version>"

Step-by-Step Build

1. Clone ONNX Runtime

Clone the repository and pin a release with KleidiAI support. The example below uses v1.23.2; check that release’s build options before substituting another revision:

git clone --recursive https://github.com/microsoft/onnxruntime.git onnxruntime.git
cd onnxruntime.git/
git checkout v1.23.2
git submodule update --init --recursive

2. Build with KleidiAI enabled

Run the build script from the repository root:

./build.sh \
  --android \
  --android_sdk_path $ANDROID_SDK_ROOT \
  --android_ndk_path $ANDROID_NDK_HOME \
  --android_abi arm64-v8a \
  --android_api 27 \
  --config RelWithDebInfo \
  --build_shared_lib \
  --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON \
  --cmake_generator Ninja \
  --parallel

Key flags explained

Note: Use the correct SDK path for --android_sdk_path. Some examples incorrectly pass the NDK path for both arguments. The SDK and NDK are separate directories.

3. Build an Android AAR (optional)

If your app uses the Java/Kotlin ONNX Runtime API, add --build_java to produce an AAR:

./build.sh \
  --android \
  --android_sdk_path $ANDROID_SDK_ROOT \
  --android_ndk_path $ANDROID_NDK_HOME \
  --android_abi arm64-v8a \
  --android_api 27 \
  --config Release \
  --build_shared_lib \
  --build_java \
  --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON \
  --cmake_generator Ninja \
  --parallel

Check the build output for the AAR. For the layout used by this example, it is under:

build/Android/Release/java/build/android/outputs/aar/

Build Outputs

For this build configuration, look for these artifacts under:

build/Android/RelWithDebInfo/
├── libonnxruntime.so          # Shared library to package into your Android app
└── onnxruntime_perf_test      # Benchmark binary for on-device testing

Deploy and Test on Device

Push artifacts to the device

adb push build/Android/RelWithDebInfo/onnxruntime_perf_test /data/local/tmp/
adb push build/Android/RelWithDebInfo/libonnxruntime.so /data/local/tmp/
adb push your_model.onnx /data/local/tmp/
adb shell chmod +x /data/local/tmp/onnxruntime_perf_test

Run a benchmark

adb shell 'cd /data/local/tmp && \
  LD_LIBRARY_PATH=/data/local/tmp \
  ./onnxruntime_perf_test \
    -e cpu \
    -m times \
    -r 20 \
    -x 1 \
    your_model.onnx'

Compare with and without KleidiAI

For meaningful performance comparison, build twice—once with onnxruntime_USE_KLEIDIAI=ON and once with onnxruntime_USE_KLEIDIAI=OFF—and run both versions with the same model, thread count, CPU affinity, and thermal conditions on an SME2-capable device.

Important Notes

• SME2 hardware required for acceleration: Building with onnxruntime_USE_KLEIDIAI=ON does not make SME2 available on older hardware. Cortex-X925 itself does not implement SME; use an explicitly documented SME2 SoC and verify sme2 at runtime.
• No SME2 override required: Normal runtime selection needs no SME2 environment variable. Verify the selected kernel when that distinction matters.
• NDK version matters: Use an NDK/toolchain combination supported by the pinned ONNX Runtime and KleidiAI revisions; inspect build output for SME2 kernel compilation.
• Build reproducibility: Pinning a specific ONNX Runtime version (e.g., v1.23.2) ensures reproducible builds and known-good KleidiAI integration.

Resources

• Arm’s ONNX Runtime Android build walkthrough
• Arm’s on-device profiling walkthrough
• ONNX Runtime’s Android build documentation

LiteRT can use SME2 through XNNPACK and KleidiAI. The documented source-build route passes Bazel --define flags that include the relevant microkernels. Defaults vary by release, so make the requested build configuration explicit and verify the selected kernel at runtime.

Architecture Overview

SME2 support in LiteRT flows through this software stack:

LiteRT
  └── XNNPACK delegate (operator execution)
        └── KleidiAI (Arm-optimized SME2 micro-kernels)

• LiteRT delegates quantized and floating-point operators to XNNPACK
• XNNPACK integrates KleidiAI, which provides SME2-accelerated matrix multiplication micro-kernels
• The xnn_enable_arm_sme2=true and xnn_enable_kleidiai=true flags activate these code paths
• At runtime, XNNPACK considers CPU features and the supported operator configuration before selecting an SME2 kernel; otherwise it uses a compatible fallback

Toolchain Requirements

Important: XNNPACK disables SME and SME2 support when building with Clang versions earlier than 18. Ensure your toolchain meets this requirement.

Bazel Build Flags (Primary Method)

Critical –define Flags

# XNNPACK and quantization paths
--define tflite_with_xnnpack=true
--define=tflite_with_xnnpack_qs8=true
--define=tflite_with_xnnpack_qu8=true
--define=tflite_with_xnnpack_dynamic_fully_connected=true

# SME/SME2 and KleidiAI enablement
--define=xnn_enable_arm_sme=true
--define=xnn_enable_arm_sme2=true
--define=xnn_enable_kleidiai=true

Flag Explanation

The quantization flags (qs8, qu8, dynamic_fully_connected) are particularly important because KleidiAI’s SME2 kernels primarily accelerate quantized matrix multiplication operations used in neural network inference.

Full Build Commands

Android arm64 Cross-Compilation

bazel build -c opt --config=android_arm64 \
  --define tflite_with_xnnpack=true \
  --define=tflite_with_xnnpack_qs8=true \
  --define=tflite_with_xnnpack_qu8=true \
  --define=tflite_with_xnnpack_dynamic_fully_connected=true \
  --define=xnn_enable_arm_sme=true \
  --define=xnn_enable_arm_sme2=true \
  --define=xnn_enable_kleidiai=true \
  "//litert/tools:benchmark_model" \
  --repo_env=HERMETIC_PYTHON_VERSION=3.12

Native Arm Linux Build

bazel build -c opt \
  --define tflite_with_xnnpack=true \
  --define=tflite_with_xnnpack_qs8=true \
  --define=tflite_with_xnnpack_qu8=true \
  --define=tflite_with_xnnpack_dynamic_fully_connected=true \
  --define=xnn_enable_arm_sme=true \
  --define=xnn_enable_arm_sme2=true \
  --define=xnn_enable_kleidiai=true \
  //litert/tools:benchmark_model

CMake Build Alternative

For a direct XNNPACK CMake build, the corresponding feature options are:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DXNNPACK_ENABLE_ARM_SME=ON \
  -DXNNPACK_ENABLE_ARM_SME2=ON \
  -DXNNPACK_ENABLE_KLEIDIAI=ON

Build Environment Setup with Docker

Google provides a Docker-based build environment for Android cross-compilation:

# Clone LiteRT
cd $WORKSPACE
git clone https://github.com/google-ai-edge/LiteRT.git

# Download and build the Docker image
wget https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/tools/tflite-android.Dockerfile
docker build . -t tflite-builder -f tflite-android.Dockerfile

# Launch the container
docker run -it -v $PWD:/host_dir tflite-builder bash

Inside the container:

# Install Android SDK/NDK components
sdkmanager \
  "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
  "platform-tools" \
  "platforms;android-${ANDROID_API_LEVEL}"

# Configure LiteRT
cd /host_dir/LiteRT
./configure

When prompted Would you like to interactively configure ./WORKSPACE for Android builds? [y/N], type y. The configuration script will detect SDK/NDK paths, set the toolchain version, configure the ABI to arm64-v8a, and initialize Bazel workspace rules.

Compiler-Level Architecture Flags

Within the Bazel build, you do not need to manually set -march flags—XNNPACK’s build system applies the correct architecture options to SME2 micro-kernel source files internally, enabling safe runtime dispatch.

However, if you are compiling SME2 translation units manually or integrating outside the standard build system, you need:

-march=armv9.3-a+sme2

Warning: Do not apply -march=...+sme2 globally to your entire application unless it will run exclusively on SME2 hardware. Globally enabling +sme2 can make startup or non-dispatched code illegal on older Arm CPUs. The XNNPACK/Bazel build handles this correctly by applying architecture flags only to specific micro-kernel targets.

Building Without SME2 (Baseline Comparison)

To build an SME-free baseline for performance comparison, explicitly disable both SME and SME2. Omitting the enable flags is insufficient: current XNNPACK Bazel defaults enable SME2 on AArch64. Keep KleidiAI and the quantization options the same as in the enabled build to isolate the contribution of SME/SME2:

bazel build -c opt --config=android_arm64 \
  --define tflite_with_xnnpack=true \
  --define=tflite_with_xnnpack_qs8=true \
  --define=tflite_with_xnnpack_qu8=true \
  --define=tflite_with_xnnpack_dynamic_fully_connected=true \
  --define=xnn_enable_arm_sme=false \
  --define=xnn_enable_arm_sme2=false \
  --define=xnn_enable_kleidiai=true \
  "//litert/tools:benchmark_model" \
  --repo_env=HERMETIC_PYTHON_VERSION=3.12

Use a separate output directory or copy each benchmark binary before rebuilding so both configurations remain available for comparison. The false settings disable the SME/SME2 paths while retaining KleidiAI. Check the selected kernels in both builds; the baseline may use other non-SME Arm extensions rather than Neon alone.

Resources

• Build the LiteRT benchmark tool with KleidiAI and SME2 enabled
• XNNPACK Bazel feature defaults and explicit settings
• LiteRT benchmark target
• XNNPACK CMakeLists.txt (SME2 build options)

Yes. Arm provides a dedicated Learning Path — “Profile ExecuTorch models with SME2 on Arm” — that walks through the complete workflow for benchmarking ExecuTorch models with and without SME2 acceleration. The guide uses the sme-executorch-profiling repository, which contains all scripts, configurations, and example models needed to run end-to-end performance comparisons and collect operator-level ETDump traces on both Android and macOS targets.

Prerequisites

Step 1: Clone the Profiling Repository

mkdir -p ~/sme2_analysis_work
cd ~/sme2_analysis_work
git clone https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling.git executorch_sme2_kit
cd executorch_sme2_kit

This creates a self-contained workspace. The Python virtual environment, ExecuTorch build outputs, models, and profiling runs all live under this directory.

Step 2: Set Up the Environment

bash model_profiling/scripts/setup_repo.sh
source .venv/bin/activate

The setup script:

• Creates and activates a Python virtual environment under .venv/
• Clones the ExecuTorch repository (if not already present)
• Initializes required submodules
• Installs ExecuTorch in editable mode

The executorch/ directory must keep that exact name — CMake presets depend on it.

Step 3: Build ExecuTorch Runners with SME2 Support

export ANDROID_NDK=/path/to/android-ndk   # Required for Android builds
bash model_profiling/scripts/build_runners.sh

The build script produces four runner variants for direct SME2 on/off comparison:

The build system uses CMake presets (merged from model_profiling/assets/cmake_presets.json) — no manual CMake flags needed. Key build settings:

• EXECUTORCH_ENABLE_EVENT_TRACER: ON — enables ETDump trace generation
• EXECUTORCH_BUILD_DEVTOOLS: ON — enables performance analysis tools
• EXECUTORCH_XNNPACK_ENABLE_KLEIDI: ON — enables Arm KleidiAI kernels
• EXECUTORCH_BUILD_KERNELS_QUANTIZED: ON — enables INT8 quantized kernel support

SME2 acceleration is enabled by default in XNNPACK when building for Arm architectures, so XNNPACK_ENABLE_ARM_SME2 is not needed explicitly.

Important: Runners are model-agnostic — build them once, reuse for all models. Only rebuild if you change CMake configurations.

Step 4: Stabilize the Target Device (Android)

adb devices
adb shell settings put global low_power 0
adb shell settings put global stay_on_while_plugged_in 15
adb shell svc power stayon true

Keep power management, thermal throttling, charging state, and thread affinity settings consistent across all comparison runs to ensure reproducible results.

Step 5: Run the Quick Smoke Test

python model_profiling/scripts/run_quick_test.py

This verifies that the environment, runners, and device connectivity are all working before committing to a full profiling run.

Step 6: Export Your Model to .pte Format

python model_profiling/export/export_model.py \
  --model toy_cnn \
  --dtype fp16 \
  --outdir model_profiling/out_toy_cnn/artifacts/

Built-in models include toy_cnn (smoke test) and mobilenet_v3_small. To add your own model:

1. Create model_profiling/models/<model_name>/ directory
2. Implement the EagerModelBase interface in model.py:
   • get_eager_model() — returns the PyTorch model
   • get_example_inputs() — returns tuple of example tensors
3. Register in model_profiling/models/__init__.py

Export dtype options include FP16, FP32 (floating-point), and INT8 (static or dynamic quantization).

Version compatibility: The .pte file and runners must use the same ExecuTorch revision.

Step 7: Run the Profiling Pipeline

For Android:

python model_profiling/scripts/android_pipeline.py \
  --config model_profiling/configs/examples/android_mobilenet_fp16.json

For macOS:

python model_profiling/scripts/mac_pipeline.py \
  --config model_profiling/configs/examples/mac_toy_cnn_fp16.json

The pipeline performs a two-run workflow per configuration:

1. Timing-only run — no trace logging overhead; provides accurate end-to-end latency measurements
2. Trace-enabled run — generates ETDump for operator-level analysis (XNNPACK trace logging alters timing, so use only for kernel selection verification)

Both SME2-on and SME2-off runners are executed automatically for comparison.

Step 8: Analyze Results

Analysis runs automatically after the pipeline completes. To rerun manually:

python model_profiling/scripts/analyze_results.py \
  --run-dir model_profiling/out_toy_cnn/runs/android

The analysis generates:

• analysis_summary.json — operator-category breakdown
• CSV files — per-operator timing data
• Comparison reports — SME2-on vs SME2-off side-by-side

Operator Categories

After SME2 accelerates CONV/GEMM, data movement typically becomes a larger percentage of total time — this is expected and reveals the next optimization target.

Benchmarking Best Practices

• Multiple repetitions: Run many iterations, discard warmups, and report median or percentile latency rather than a single measurement
• Separate timing from tracing: Use timing-only runners for latency comparisons; use trace-enabled runners only to verify kernel selection
• Consistent device state: Lock CPU frequency, disable thermal throttling if possible, and maintain consistent charging state
• Android vs macOS: Android provides the most representative results (real thermal, bandwidth, and scheduling constraints); macOS works for learning the workflow before deploying to hardware
• Version alignment: Keep model export and runner builds at the same ExecuTorch revision

Resources

• Environment setup and build instructions
• Model export and profiling
• sme-executorch-profiling GitHub repository

Build LiteRT-LM’s CPU runner for Arm64, make sure its pinned LiteRT/XNNPACK dependency includes KleidiAI SME2 kernels, and test a compatible .litertlm model on a device where the OS exposes SME2. LiteRT-LM’s upstream build guide documents the runner and CPU benchmark. The SME2 selection must be checked separately: CPU execution, a build define, or a profiling label alone is not proof that an SME2 instruction ran.

Build and Run the CPU Runner

Start from a pinned LiteRT-LM tag and follow its source-build guide . The Android example uses Bazel’s Arm64 configuration:

git clone https://github.com/google-ai-edge/LiteRT-LM.git
cd LiteRT-LM
git lfs pull
bazel build -c opt --config=android_arm64 \
  --enable_platform_specific_config \
  //runtime/engine:litert_lm_main

For a native Arm64 Linux or macOS build, follow the platform section of the same guide and omit --config=android_arm64. Use a .litertlm model compatible with the pinned runner version.

Deploy and run on Android:

adb shell mkdir -p /data/local/tmp/litert-lm
adb push bazel-bin/runtime/engine/litert_lm_main /data/local/tmp/litert-lm/
adb push /path/to/model.litertlm /data/local/tmp/litert-lm/model.litertlm
adb shell /data/local/tmp/litert-lm/litert_lm_main \
  --backend=cpu \
  --model_path=/data/local/tmp/litert-lm/model.litertlm

For a model-level measurement, add --benchmark --enable_profiling --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --async=false. The benchmark reports prompt processing and generation behavior; profiling can add overhead, so compare latency with profiling disabled as well.

Make the SME2 Kernels Available

LiteRT can build XNNPACK and KleidiAI with the Bazel defines below, as shown in Arm’s LiteRT SME2 build walkthrough :

--define=tflite_with_xnnpack=true
--define=xnn_enable_arm_sme=true
--define=xnn_enable_arm_sme2=true
--define=xnn_enable_kleidiai=true

Those flags are documented for LiteRT’s benchmark target. Before applying them to LiteRT-LM, inspect the pinned LiteRT-LM Bazel dependency and build action to confirm that the same defines reach its CPU runtime and that the SME2 kernels are compiled into the binary. LiteRT-LM’s published build guide does not itself establish that passing these defines to litert_lm_main enables an end-to-end SME2 path for every model. If you control LiteRT as a separate dependency, build and link the matching configured version rather than assuming an unrelated LiteRT build changes LiteRT-LM’s vendored dependency.

Avoid global -march=...+sme2 on a binary intended for older Arm64 devices. Specialized kernels should be built with their own target flags and selected through runtime dispatch.

Verify the Selected Kernel

1. Confirm OS exposure on the target device, for example through HWCAP2_SME2 on Linux/Android or sysctl hw.optional.arm.FEAT_SME2 on macOS. Check the actual device and CPU affinity; do not assume a fixed core number is SME2-capable.
2. Run --backend=cpu and inspect the lowered operators and XNNPACK delegate. A node label such as PF32 can indicate a packed or KleidiAI path in a particular XNNPACK version, but it is not by itself proof of SME2.
3. Trace the selected microkernel symbol or break on it, then inspect its disassembly for SME-family instructions. A kai_* symbol establishes a KleidiAI function; its feature suffix or instructions distinguish SME2 from another KleidiAI variant.
4. Compare a build with SME2 kernels available to a build with that path disabled, using the same model, input, thread count, affinity, and thermal conditions. Keep the trace run separate from timing.

If no SME2 kernel appears, check the model’s operator, shape, quantization and packing format against the pinned backend’s kernel table. A working CPU fallback is expected and should not be reported as SME2 acceleration.

Yes. As of oneDNN 3.13, oneDNN contains a native AArch64 JIT SME implementation for a constrained set of BRGEMM-backed matmul cases. This is separate from oneDNN’s optional Arm Compute Library (ACL) integration, which can also reach operation-specific SME kernels. It is therefore accurate to say that oneDNN supports SME, but not that every oneDNN primitive or every AArch64 build uses it.

Native oneDNN SME Path

The oneDNN 3.13.2 source contains all of the essential pieces of a direct implementation:

• an sme CPU ISA value and runtime feature test;
• an AArch64 JIT file, jit_brgemm_sme_kernel.cpp, that emits SME tile loads, FMOPA, and streaming-mode code;
• SME selection in the BRGEMM dispatcher; and
• an explicit brgemm_matmul_t<sme> instantiation.

The current kernel is focused on FP32 matrix multiplication. It does not accept bias, post-operations, or non-default source, weight, or destination scales. Requests outside those constraints are rejected by the SME implementation and can be handled by another oneDNN implementation. This is specialized matmul enablement, not a general SME backend for convolution, normalization, activations, and every other primitive.

SME2-capable hardware can run this baseline SME kernel because SME2 requires SME. The presence of SME2 does not turn the kernel into an SME2 implementation; the emitted operations and oneDNN dispatch level are what determine that.

ACL Is a Second, Indirect Route

oneDNN can also be built with Arm Compute Library support:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DONEDNN_TARGET_ARCH=AARCH64 \
  -DONEDNN_AARCH64_USE_ACL=ON \
  -DACL_ROOT_DIR=/path/to/ComputeLibrary
cmake --build build --parallel

Eligible primitives may then be delegated to ACL. A recent ACL build contains its own SME/SME2 kernels and runtime controls, so this path can use SME even when the executing code is owned by ACL rather than oneDNN. Delegation remains dependent on the operation, layout, data type, ACL version, and build options.

How to Verify the Selected Implementation

First verify that the OS exposes SME:

grep -m1 '^Features' /proc/cpuinfo | tr ' ' '\n' | grep '^sme'

Then enable oneDNN’s verbose log:

ONEDNN_VERBOSE=all ./your_workload 2>onednn.log

Check the implementation name and primitive description in the log. For a definitive check, profile or disassemble the selected JIT code and look for streaming-mode transitions and SME operations such as FMOPA. If testing the ACL route, use the verbose output to confirm ACL delegation and then verify the implementation selected inside ACL.

Benchmark the native SME path, ACL path, and non-SME AArch64 path with identical matrix dimensions, thread count, affinity, warm-up, and timing method. Small matrices or layouts requiring conversion can erase the benefit of a faster compute kernel.

Resources

• oneDNN releases
• oneDNN source repository

Not as a production feature in the current upstream BLIS release. BLIS can host architecture-specific SME microkernels, and an open pull request adds experimental Apple-M4-focused SME2 configurations, but that work is not yet equivalent to stable, generally available upstream SME support.

Current Status

BLIS separates its portable BLAS framework from architecture-specific packing and compute microkernels. Its released Arm configurations cover conventional AArch64/Neon and SVE targets. They do not currently include a released general-purpose SME or SME2 configuration.

Pull request flame/blis#921 adds two experimental configurations:

• m4sme_p, tuned for Apple M4 performance cores;
• m4sme_e, tuned for Apple M4 efficiency cores; and
• an armsme kernel directory containing SME2 packing and Level-3 GEMM kernels.

The proposal contains single- and double-precision GEMM microkernels and uses ZA when packing. Its kernels are streaming-vector-length agnostic, but its BLIS block sizes and context are tuned for Apple’s 512-bit streaming vector length. The work is primarily enabled for recent Clang on Darwin and remains an open pull request, so production users should treat its API, dispatch, compiler support, and performance as experimental.

Why BLIS Needs New Microkernels

BLIS’s established microkernel model maps naturally to 1D Neon or SVE registers. SME changes the inner kernel substantially:

• accumulation lives in the 2D ZA array;
• FMOPA-class instructions update matrix tiles by outer product;
• functions must obey streaming-mode and ZA ABI rules; and
• packing and block sizes must account for streaming vector length and the target cache hierarchy.

The BLIS framework is flexible enough to register those kernels, but an SVE kernel does not become an SME kernel merely by compiling with +sme.

Practical Choices

For evaluation on Apple M4/M5, test PR #921 in an isolated build and compare it with the official BLIS release and Apple Accelerate. Run BLIS correctness tests before performance tests and retain a stable fallback.

For production SME linear algebra today, alternatives with released code include OpenBLAS’s ARMV9SME/VORTEXM4 targets, selected Arm Performance Libraries routines, and Arm Compute Library. KleidiAI is usually a better fit for quantized AI microkernels than for a general BLAS API.

Resources

• Experimental BLIS SME2 pull request #921
• BLIS plugin guide

MLX has no explicit SME or SME2 backend. Its normal high-performance path is the Apple GPU through Metal; its CPU backend uses Apple CBLAS/Accelerate, BNNS for some types, and MLX’s own SIMD code. A closed-source Apple library called by MLX might internally dispatch to SME on M4/M5, but that is not documented, guaranteed, selectable, or observable as an MLX feature.

Direct Versus Indirect Support

Direct MLX SME support would require SME-specific source, build targets, CPU feature detection, streaming-mode/ZA ABI handling, and a public or internal dispatcher that selects those kernels. The current MLX tree does not expose an SME build switch or API and does not document SME kernels.

Indirect use is possible in this narrower sense:

MLX CPU operation
  → Accelerate CBLAS or BNNS
  → private Apple implementation
  → possibly SME on supported hardware

That possibility should not be presented as “MLX supports SME.” MLX requests a system-library operation; Apple owns any instruction selection behind the ABI. The implementation may vary by macOS release, matrix shape, type, and processor.

What to Use in Practice

• For normal MLX model training or inference, use the default Metal GPU path and benchmark it on the target Mac.
• To test the CPU path, select the MLX CPU device and compare identical workloads, but do not infer SME use from speed alone.
• To prove indirect SME execution, sample or disassemble the called Accelerate/BNNS implementation and look for SME streaming and outer-product instructions. This proves the system library used SME, not that MLX has an SME backend.
• For guaranteed control of ZA, streaming mode, and kernel dispatch, implement a native C/C++ extension using ACLE/assembly or call a library with documented SME kernels.

Apple M4 and M5 hardware supporting SME does not make every framework operation use SME automatically. Hardware capability, OS exposure, the selected MLX device, and the native implementation all matter.

Resources

• MLX source repository

Not as a first-class upstream JAX/XLA target. Ordinary JAX CPU compilation does not automatically lower jax.numpy matrix operations to ZA-based SME kernels, and standard jaxlib wheels should not be assumed to contain SME acceleration. LLVM and MLIR understand SME, but XLA does not automatically use MLIR’s ArmSME lowering simply because its LLVM dependency can encode SME instructions.

The Relevant Layers

JAX → StableHLO/HLO → XLA CPU backend → LLVM-generated code or library call

Upstream XLA’s AArch64 CPU feature handling covers Neon and limited SVE/SVE2 concerns; it does not expose an SME target or a documented path that converts HLO matmul into streaming-mode ZA operations. SME requires explicit tiling, data movement, streaming-mode attributes, and ZA lifetime management, so LLVM’s ordinary loop vectorizer is not a substitute for an SME matrix lowering.

MLIR does provide an ArmSME dialect and conversions capable of representing tiles and lowering matrix operations to SME intrinsics. That infrastructure can be used in custom compiler pipelines, but it is not the standard JAX/XLA CPU pipeline.

Ways a JAX Workload Can Still Reach SME

XLA Built with Arm Compute Library

XLA contains optional ACL integration controlled at build time. In a custom build, eligible operations may follow this route:

JAX → XLA CPU → ACL call → ACL runtime dispatch → SME/SME2 kernel

This is indirect support. Only operations, layouts, shapes, and data types accepted by both the XLA-to-ACL path and an ACL SME kernel benefit. Standard wheels may not include it.

JAX FFI or an XLA Custom Call

For deterministic use, wrap an SME implementation from Arm Compute Library, KleidiAI, or custom ACLE/assembly and invoke it using JAX FFI. The wrapper must define shape/layout contracts, runtime feature checks, fallback behavior, and any custom differentiation rule.

Custom MLIR Pipeline

A compiler project can lower StableHLO/linalg/vector operations through the ArmSME dialect and LLVM, but that is compiler engineering rather than an end-user JAX flag.

Verification

Confirm sme/sme2 in OS capabilities, inspect the installed jaxlib build configuration, dump XLA artifacts, and profile the native libraries. Search disassembly for SMSTART/SMSTOP, ZA loads/stores, and FMOPA/SMOPA-class operations. A compiler target flag alone does not prove the executing operator used SME.

Resources

• JAX FFI
• MLIR ArmSME dialect

Conditionally and indirectly. NumPy and SciPy do not expose SME-specific Python APIs or general SME ufunc kernels. Their BLAS-backed operations can nevertheless execute SME when the installed package is linked to a native BLAS implementation containing SME kernels—most concretely, an SME-enabled OpenBLAS build.

Where SME Can Appear

NumPy/SciPy execution has two relevant layers:

• NumPy’s own elementwise/reduction dispatcher handles operations such as add, exp, and many reductions. Its public Arm SIMD targets do not make SME a general ufunc backend.
• Dense linear algebra such as np.matmul, np.dot, and many numpy.linalg/scipy.linalg routines calls BLAS/LAPACK. The linked library can select its own architecture kernel.

OpenBLAS documents an ARMV9SME target and an Apple VORTEXM4 target with SME kernels for selected single-precision Level-3 BLAS operations, including SGEMM, SSYMM, STRMM, SSYRK, and SSYR2K. Therefore, a suitable contiguous FP32 matrix multiplication may use SME if NumPy/SciPy is linked to a build containing that target and OpenBLAS selects it at runtime.

This is partial support. It does not imply that FP64, every LAPACK routine, sparse algorithms, FFTs, image processing, or NumPy ufuncs use SME. Portable PyPI/Conda wheels may omit the SME target.

Inspect and Test the Installation

python - <<'PY'
import numpy as np
import scipy

np.show_config()
np.show_runtime()
scipy.show_config()
PY

threadpoolctl can identify the loaded BLAS library:

python -m pip install threadpoolctl
python -m threadpoolctl -i numpy scipy

If the backend is OpenBLAS, run a likely kernel with verbose dispatch:

OPENBLAS_VERBOSE=2 python -c \
'import numpy as np; a=np.ones((2048,2048),dtype=np.float32); print((a@a)[0,0])'

Look for an SME-capable OpenBLAS core name such as ARMV9SME or VORTEXM4. NumPy’s own SIMD report may still show Neon because the SME instructions execute inside OpenBLAS. For definitive proof, use openblas_get_corename(), profiling, or disassembly.

Resources

• OpenBLAS supported targets and SME kernels
• SciPy BLAS/LAPACK build selection

Yes, with a deliberately limited scope. ArmPL 26.01 introduced SME-optimized real SGEMM and DGEMM implementations for compatible Apple M-series processors on macOS. Applications keep using the standard BLAS API and ArmPL selects the implementation internally; this does not establish SME coverage across all ArmPL routines or platforms.

What Is Documented

Arm’s 26.01 announcement states that single- and double-precision real matrix-matrix multiplication uses SME rather than the previous Neon implementation on compatible Apple silicon. Arm reported up to 7× improvement for tested DGEMM matrix sizes relative to ArmPL 25.07.

The precise claims matter:

• supported routines called out publicly: real SGEMM and DGEMM;
• documented platform: compatible M-series Apple silicon running macOS;
• activation: transparent behind standard BLAS calls;
• version: ArmPL 26.01 or later.

Do not generalize this to all LAPACK, FFT, sparse, random-number, libamath, Linux, or Windows code. Higher-level LAPACK routines may benefit when their execution spends time in an accelerated GEMM, but that is an indirect consequence and should be measured.

Using and Verifying It

An application continues to call normal CBLAS or Fortran entry points:

#include <armpl.h>

/* cblas_sgemm(...) or cblas_dgemm(...), unchanged */

On macOS, confirm the feature is exposed:

sysctl -n hw.optional.arm.FEAT_SME
sysctl -n hw.optional.arm.FEAT_SME2

Then link the same benchmark against ArmPL 25.07 and 26.01 or later, keeping dimensions, transpose flags, leading dimensions, thread count, affinity, warm-up, and timing identical. Use ArmPL’s logging/performance tools or system profiling where available to confirm which BLAS calls dominate. If instruction-level proof is required, profile/disassemble the executing library routine.

SME support is internal to ArmPL, so application code does not need <arm_sme.h>, streaming attributes, or a global -march=...+sme flag merely to call the optimized BLAS routine.

Resources

• What’s new in Arm Performance Libraries 26.01
• Install Arm Performance Libraries

AngelSlim has demonstrated SME2-aware deployment for selected low-bit models, but it does not provide a general-purpose SME backend across the toolkit. The best-supported example is the 2-bit Hy-MT1.5 model path, where the compressed artifact is executed by a specialized runtime using SME2 kernels. Compression and hardware execution are separate layers.

What “Support” Means Here

AngelSlim performs model compression—quantization, pruning, distillation, and related transformations. It does not need to emit SME instructions while creating a checkpoint. SME2 is used later by an inference runtime:

AngelSlim compression/export
  → low-bit model artifact
  → compatible inference runtime and model format
  → KleidiAI or other SME2 microkernel
  → SME2-capable CPU

Tencent’s Hy-MT1.5 documentation describes a 2-bit SEQ model optimized for SME2-capable devices and shows an SME2-versus-Neon comparison. Its published llama.cpp pull-request instructions, however, are for the different 1.25-bit STQ1_0 format. Do not use that branch as evidence that the 2-bit GGUF can load or use SME2 there.

That evidence is model- and format-specific. It does not show that all AngelSlim algorithms, exported quantization formats, or standard upstream inference engines have turnkey SME2 coverage. In particular:

• baseline SME is not sufficient for a path requiring SME2;
• the runtime must understand the exact low-bit format produced by AngelSlim;
• the documented llama.cpp pull-request route is for 1.25-bit STQ1_0, not the 2-bit SEQ format;
• a kernel for STQ1_0 does not imply support for SEQ; and
• unsupported operations or formats may fall back to Neon or fail to load.

Validation Checklist

1. Identify the exact AngelSlim model, quantization format, and export revision.
2. Use the runtime and model format named by that model’s deployment documentation. For the 2-bit SEQ artifact, require explicit runtime support before attempting the 1.25-bit STQ1_0 pull-request instructions.
3. Check how that runtime implements its SME2 and fallback kernels; do not assume it uses KleidiAI because another llama.cpp path does.
4. Confirm sme2 is exposed by the target OS.
5. Verify the loaded kernel in a dispatch trace or profiler. A model-buffer type alone does not identify the executed instruction set.
6. Compare SME2 and Neon paths using the same artifact, thread count, affinity, prompt, and thermal state.

The published 2-bit comparison establishes a specific deployment claim, while the 1.25-bit instructions teach a separate model path. Keep those formats distinct when planning an integration.

Resources

• AngelSlim Hy-MT1.5 documentation

Use an inference runtime that already integrates KleidiAI, then package that runtime in your Android app. KleidiAI provides microkernels; it does not load models or run a whole app. This walkthrough uses ONNX Runtime → MLAS → KleidiAI because it gives a Java/Kotlin app a standard model API. The key checkpoints are the custom native build, the AAR the app loads, a supported model operator, and evidence that runtime dispatch selected an SME2 kernel.

1. Choose the Target and Model

Use an Android arm64 device whose OS exposes SME2, such as a documented Dimensity 9500 phone. Check the actual device:

adb shell 'grep -m1 "^Features" /proc/cpuinfo'

Look for sme2. This is a useful shell check; production native code should query getauxval(AT_HWCAP2) and HWCAP2_SME2. Use an ONNX model with operations and tensor formats supported by the ONNX Runtime/MLAS/KleidiAI revision you will build. An SME2 CPU does not make every Conv or MatMul node eligible.

Install a matching Android SDK, NDK, Python, CMake, Ninja, and ADB. Pin the ONNX Runtime release so the build and model behavior can be reproduced.

2. Build ONNX Runtime with KleidiAI

From an ONNX Runtime checkout with the KleidiAI option, build the Android shared library and Java AAR:

git clone --recursive https://github.com/microsoft/onnxruntime.git
cd onnxruntime
git checkout v1.23.2
git submodule update --init --recursive

./build.sh \
  --android \
  --android_sdk_path "$ANDROID_SDK_ROOT" \
  --android_ndk_path "$ANDROID_NDK_HOME" \
  --android_abi arm64-v8a \
  --android_api 27 \
  --config Release \
  --build_shared_lib \
  --build_java \
  --cmake_generator Ninja \
  --parallel \
  --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON

Set ANDROID_SDK_ROOT and ANDROID_NDK_HOME to real directories before running this command. Check the CMake cache and build log to confirm KleidiAI was included. Find the generated AAR under the build output rather than assuming a fixed filename. Arm’s Android ONNX Runtime build guide shows the release-specific workflow.

Do not put -march=...+sme2 in the app’s global compiler flags. The runtime builds specialized kernels and selects a compatible one at execution time, preserving a fallback for older arm64 devices.

3. Package the AAR and Run Inference

Copy the custom AAR to app/libs/ and include it in the Android app:

dependencies {
    implementation(files("libs/onnxruntime-release.aar"))
}

Make sure another ONNX Runtime Maven dependency is not replacing your custom native library. Put model.onnx in app/src/main/assets/, then create and close the session and tensors off the UI thread. This small example assumes a single float input and a model-specific output consumer:

import ai.onnxruntime.OnnxTensor
import ai.onnxruntime.OrtEnvironment
import ai.onnxruntime.OrtSession
import java.nio.FloatBuffer

val env = OrtEnvironment.getEnvironment()
val model = assets.open("model.onnx").use { it.readBytes() }
env.createSession(model, OrtSession.SessionOptions()).use { session ->
    val name = session.inputNames.first()
    OnnxTensor.createTensor(env, FloatBuffer.wrap(input), shape).use { tensor ->
        session.run(mapOf(name to tensor)).use { result ->
            consumeModelOutput(result[0].value) // Implement for your model's output type.
        }
    }
}

Run this in a worker or coroutine dispatcher; a large model or JNI call on the main thread will block the UI. input, shape, and consumeModelOutput are application-specific, so this is an integration sketch rather than a copy-and-run Activity.

4. Confirm Acceleration and Measure It

First ensure the app loads your custom AAR and successfully runs the model. Then inspect the backend’s selected function with profiling, a breakpoint, or disassembly. A kai_* symbol shows a KleidiAI function; its SME2 suffix or SME instructions establish the architecture path. Correct output or a faster run alone does not identify the executed kernel.

For a controlled comparison, build the same ONNX Runtime revision with onnxruntime_USE_KLEIDIAI=OFF, package that AAR in a separate app build, and measure the same inputs, thread count, CPU affinity, and device temperature. Report operator and end-to-end latency separately. Arm’s ONNX Runtime profiling guide provides an on-device benchmark workflow.

If your app uses a different model runtime, choose its documented path instead: LiteRT with XNNPACK and KleidiAI for .tflite models, or llama.cpp’s Arm64 KleidiAI build for GGUF LLMs. Each still needs model-specific kernel and runtime verification.

Yes. The clearest SME2-specific path is Arm’s llama.cpp on Android walkthrough , which builds an Arm64 CPU runner, loads a GGUF model, and compares SME-family kernels with a fallback. For an Android app rather than a command-line experiment, Arm also has an ExecuTorch chat-app walkthrough . That app demonstrates KleidiAI integration, but its documented acceleration path may use other Arm features; verify SME2 separately on the target device.

Start with llama.cpp to Measure the CPU Path

The llama.cpp example suits a developer who wants to see the build, deployment, model loading, and kernel-selection path before packaging an app. It teaches how GGML’s CPU backend reaches KleidiAI for supported GGML_OP_MUL_MAT operations.

For a current portable Android build, follow llama.cpp’s upstream Android Arm64 commands :

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -S . -B build-android \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK/build/cmake/android.toolchain.cmake" \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-28 \
  -DGGML_CPU_KLEIDIAI=ON \
  -DGGML_NATIVE=OFF \
  -DGGML_OPENMP=OFF \
  -DGGML_LLAMAFILE=OFF \
  -DLLAMA_OPENSSL=OFF
cmake --build build-android --config Release --parallel

Set ANDROID_NDK to the NDK root. Push the resulting llama-cli and a GGUF model to a writable location such as /data/local/tmp/, then run the model with CPU execution selected. The Arm walkthrough gives the exact transfer and benchmark steps for its pinned revision. Do not copy an older Linux GNU toolchain recipe with global +sme2 flags into a modern Android build; the NDK target and per-kernel dispatch matter.

Compare prompt processing and token generation separately with llama-bench. GGML_KLEIDIAI_SME=0 is a diagnostic way to disable SME-family kernels; leave it unset for normal automatic selection. A KleidiAI buffer message confirms buffer allocation, not that an SME2 instruction executed. Trace the selected microkernel or inspect its disassembly for that proof.

Then Build an App

The AI Chat library guide is useful if you want a Kotlin API and streaming tokens without first maintaining a custom JNI layer. The ExecuTorch guide is better when you need to export and quantize your own PyTorch model and inspect delegated operators. In either route, start with the guide’s stated model format and build version, run on an OS-exposed SME2 device, and keep an SME2-disabled comparison build.

These examples do not guarantee that an arbitrary LLM or quantization format uses SME2. Kernel coverage depends on weight packing, activation type, matrix shape, and the runtime’s bundled KleidiAI revision. In particular, a phone with an Exynos 2500 is not an SME2 test device merely because it is recent; check the actual SoC and sme2 capability before using it for this experiment.

ExecuTorch can delegate eligible model operators to XNNPACK, which can select KleidiAI microkernels for supported data types, quantization formats, and shapes. On an OS-exposed SME2 device, an eligible kernel can use SME2. There is no dependable general speedup factor: measure the selected operators and the complete model against an otherwise identical SME2-disabled build.

Where the Kernel Is Selected

1. Export a PyTorch model as an ExecuTorch .pte program, with eligible operations delegated to XNNPACK.
2. Build the runner with EXECUTORCH_BUILD_XNNPACK=ON and EXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON. In current ExecuTorch CMake source , XNNPACK_ENABLE_ARM_SME2 is on by default and can be set explicitly.
3. At runtime, XNNPACK selects a kernel based on CPU features and the operation’s complete contract. If a KleidiAI SME2 kernel does not match, another compatible kernel handles the operation.

Do not add -march=...+sme2 to the whole runner. XNNPACK builds specialized microkernels separately, allowing the rest of the binary to run on older Arm64 devices.

Typical candidates include fully connected layers, supported convolutions, and batch matrix multiplication. Their exact FP32, FP16, INT8, or packed 4-bit coverage is version-specific. Packed 4-bit weights do not mean SME2 performs native 4-bit arithmetic; the kernel decodes or expands values before accumulation. Arm’s ExecuTorch node-to-kernel guide shows how to check the supported signatures in a documented revision.

What Has Been Measured

In one controlled SqueezeSAM ExecuTorch/XNNPACK/KleidiAI case study , the authors compared SME2 enabled and disabled on the same vivo X300:

These are measurements for that model, device, precision, backend, and thread setting. They are not expected ranges for a new model. The smaller four-core ratios show why single-core operator gains cannot be multiplied into an application forecast; data movement, unaccelerated operators, scheduling, and shared compute resources also matter.

Verify and Estimate Your Own Result

Build SME2-on and SME2-off runners from the same ExecuTorch revision. Run the same .pte model, inputs, thread count, affinity, and power mode on the target device. Use ETDump for operator timings and inspect XNNPACK/KleidiAI dispatch or a sampled kernel call stack for the selected microkernel. ETDump timing alone does not prove SME2 instructions executed.

Rank operators by absolute time saved, then measure end-to-end latency separately. If the accelerated layers were a small fraction of baseline runtime, the whole model will gain much less than those layers. If packing or layout conversion becomes dominant, the next optimization may be outside the SME2 compute kernel. Arm’s ExecuTorch profiling workflow provides paired runners and trace/timing configurations.

As of September 2026, Arm’s native SME2 device list includes phones in the vivo X300 and OPPO Find X9 families with MediaTek Dimensity 9500, and SME2-capable variants of Samsung Galaxy S26 and Google Pixel 11. Check the exact regional model and OS-exposed feature before using a phone for testing. KleidiAI acceleration also requires a matching kernel and framework integration for your workload.

For a first performance investigation, the vivo X300 is a practical choice because Arm’s published workflows use it for ONNX Runtime profiling and other KleidiAI examples. That does not make it a universal reference for other phones or operators.

Check the Device

adb shell getprop ro.soc.model
adb shell 'grep -m1 "^Features" /proc/cpuinfo'

Look for sme2 in the kernel feature list. In app native code, query the process capability instead of parsing a device name:

#include <sys/auxv.h>
#include <asm/hwcap.h>
#include <stdbool.h>

bool has_sme2(void) {
    return (getauxval(AT_HWCAP2) & HWCAP2_SME2) != 0;
}

This answers whether SME2 is available to the process. It does not prove the app’s selected inference kernel uses SME2. Check the framework’s selected kai_* or other backend kernel and compare with an SME2-disabled build.

Older phones built around Cortex-X4 or Cortex-X925, such as Dimensity 9300/9400 devices, are useful fallback test targets but do not gain SME merely from being Armv9. For functional development without hardware, Arm’s FVP setup can execute SME2 examples; use a real phone for latency or energy measurements.

Build a KleidiCV example with the Android NDK, run it on a phone whose OS exposes SME, and verify which operation used the SME implementation. Passing correctness tests only proves that the built program runs; it does not prove that a runtime dispatcher chose SME. Arm’s Android KleidiCV Gaussian blur Learning Path provides a current, reproducible route using the 26.06 release.

Build and Run a Known SME Operation

Install a recent Android NDK, CMake, and ADB, and set ANDROID_NDK_HOME to the NDK directory. Check out the same KleidiCV release used by the guide:

git clone https://gitlab.arm.com/kleidi/kleidicv.git
cd kleidicv
git checkout --detach refs/tags/26.06

cmake -S examples/extract_one_operation \
  -B build/extract-android \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE="$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake" \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-21 \
  -DANDROID_STL=c++_static
cmake --build build/extract-android --target example_usage --parallel

Before running the SME example, check that the device exposes sme (and sme2 if you intend to test an SME2 operation):

adb shell 'grep -m1 "^Features" /proc/cpuinfo'
adb push build/extract-android/example_usage /data/local/tmp/
adb shell chmod 755 /data/local/tmp/example_usage
adb shell /data/local/tmp/example_usage

The example calls a particular SME Gaussian blur implementation directly. Compare its output with the guide’s expected pixels. This verifies that that SME routine runs and produces the expected result on the device; it does not test the default OpenCV or KleidiCV dispatcher.

Compare Backends and Test Dispatch Separately

The same Learning Path provides a small performance explorer that calls Neon, SVE2, and SME Gaussian blur implementations directly, checks them against a Neon reference, and reports per-operation timing. Its build and run step includes the extra source and CMake target needed for the 26.06 checkout. Do not expect gaussian_blur_benchmark in the unmodified release; the guide has you add it.

To verify runtime dispatch for your actual application, call the same KleidiCV or OpenCV operation your app uses, then inspect the resolved function or add temporary logging to its resolver. Current KleidiCV versions distinguish ordinary APIs, which normally choose Neon or SVE2, from _sme APIs that prefer SME/SME2 when available. Do not assume a global “SME2 before Neon” ordering for every call. Arm’s backend selection guide shows how to inspect the selected variant.

Measure latency on physical hardware with the same image, border mode, kernel size, CPU affinity, and thermal conditions. An FVP or emulator can help with functional testing but cannot predict phone performance.

KleidiCV has SME implementations for selected image-processing operations, including regular color conversion, fixed-stripe filtering, resize, morphology, and reductions. It can be used directly or through OpenCV’s HAL. The operation and its parameters must match an implemented path, and current KleidiCV APIs distinguish ordinary dispatch from _sme variants that prefer SME. Do not interpret every OpenCV speedup as an SME speedup.

Where SME Is Useful

The exact list changes by release and data type. Examples in KleidiCV’s documented backends include:

This is an overview, not a claim that every listed OpenCV call reaches SME. Some cases use Neon or SVE2, and unsupported parameters fall through to OpenCV’s own implementation. In KleidiCV 26.03 , Arm added more optimized remap, warp perspective, transpose, filtering, and optical-flow coverage. Those additions do not establish SME backends for every new variant.

Compare the Same Operation

OpenCV’s HAL lets an application continue calling, for example, cv::GaussianBlur or cv::cvtColor. With a KleidiCV-enabled OpenCV build, an eligible call may enter a KleidiCV implementation. Ordinary KleidiCV APIs in recent releases generally choose Neon or SVE2 by default; the _sme variants prefer SME or SME2 and can fall back. Verify the resolved implementation for the exact function, type, and parameters you use.

Arm reports gains of up to 14× for selected OpenCV with KleidiCV versus OpenCV without it operations on Cortex-A710. That core has no SME, so those figures are evidence for KleidiCV integration, not SME versus Neon. For a direct SME comparison, Arm’s Android Gaussian blur explorer runs Neon, SVE2, and SME implementations on the same input and checks output before timing. In its 15×15 example on one SME-capable phone, SME’s median latency was 1.45× to 2.01× lower than Neon’s across the tested resolutions. That result is specific to Gaussian blur, kernel size, device, and CPU affinity.

For an application-level comparison, build the same OpenCV version with and without KleidiCV, run identical image pipelines, and profile each call. Then test an explicit SME variant or trace the resolver to attribute any gain to SME. Include image decoding, transfers, and rendering in an end-to-end camera measurement; a fast filter alone does not predict frame rate.

Resources

• Arm’s KleidiCV backend and operation update
• OpenCV and KleidiCV Android app walkthrough

Arm’s OpenCV plus KleidiCV Android app processes images from assets, while its OpenCV face-detection app shows live camera capture. Together they provide a useful starting point, but they are not a published end-to-end camera app proving SME execution. You must connect the frame pipeline, build OpenCV with KleidiCV, and verify the selected backend for each operation.

Connect Camera Frames to OpenCV

Use CameraX ImageAnalysis with STRATEGY_KEEP_ONLY_LATEST so the processing pipeline does not queue stale frames. A simple first step is to process the Y plane as grayscale. CameraX planes have their own row strides and may not form one tightly packed NV21 buffer, so do not wrap the Y-plane pointer as a height * 3 / 2 NV21 image or assume the U/V planes follow it in memory.

The analyzer should close each ImageProxy even when processing fails:

val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
    .build()

analysis.setAnalyzer(cameraExecutor) { frame ->
    try {
        val y = frame.planes[0]
        processGrayFrame(
            y.buffer, y.rowStride, y.pixelStride,
            frame.width, frame.height
        )
    } finally {
        frame.close()
    }
}

processGrayFrame is an application JNI function to implement. For pixelStride == 1, native code can wrap the Y buffer with cv::Mat(height, width, CV_8UC1, data, rowStride) while the frame remains open, then call an out-of-place operation such as cv::GaussianBlur. For any other pixel stride, copy pixels into a packed grayscale buffer first. Do not retain the plane pointer after frame.close(); copy the processed result if it must outlive the callback.

If you need color, convert the three planes using their actual row and pixel strides or use a CameraX output format your conversion code explicitly supports. This avoids a common corruption bug in camera examples that silently treat YUV_420_888 as NV21.

Build and Verify the Backend

Follow Arm’s OpenCV/KleidiCV Android guide for a pinned OpenCV version and matching native library. The Maven artifact or an OpenCV build flag alone does not prove the packaged binary includes the desired KleidiCV release. Arm reports KleidiCV enabled by default for AArch64 OpenCV builds from 4.13; older 4.11-based examples need their documented build configuration.

After the app works, log Core.getBuildInformation() to confirm that the loaded OpenCV build contains KleidiCV. Then inspect the selected implementation for the particular Gaussian blur, color conversion, or resize call. Current KleidiCV ordinary APIs generally select Neon or SVE2; _sme variants prefer SME/SME2 and can fall back. An OpenCV HAL call is therefore not automatically an SME call. Arm’s KleidiCV backend inspection guide demonstrates resolver tracing.

Measure the whole frame path: capture, any YUV conversion, filtering, copy/JNI overhead, and display. Compare the same frame size and parameters on the same device with and without the SME implementation selected. A microkernel gain may be much smaller in the final camera frame time.

Neon is KleidiCV’s broad AArch64 baseline. Its SME/SME2 implementations cover a subset of operations and require an OS-exposed SME feature. Their streaming vectors can process more pixels per iteration on some hardware, but entering streaming mode and handling small or irregular work can erase the gain. The chosen backend depends on the API variant and operation, not just a fixed ranking of CPU features.

Selection and Coverage

Arm’s KleidiCV 26.03 update clarifies the important distinction:

• Ordinary KleidiCV APIs use Neon or SVE2 by default.
• APIs with an _sme postfix prefer an available SME or SME2 implementation and fall back when it is unavailable.
• KLEIDICV_PREFER_SME_BACKEND and build options can affect selection in a particular release. Check that release’s reference guide before relying on an override.

This is more accurate than saying every call always chooses SME2 before SME, SVE2, and Neon. A device can support SME2 while a given routine has only a Neon implementation.

Treat that table as representative: kernel coverage changes by release and depends on data type, channels, strides, and parameters. Some operations previously described as “Neon-only,” such as remap and warp perspective, gained selected optimized cases in later KleidiCV versions; inspect the exact implementation rather than relying on an old category list.

Performance Comparison

Arm’s headline “up to 14×” result for KleidiCV 26.03 compares OpenCV with KleidiCV against OpenCV without it on Cortex-A710. That processor has no SME, so the figure says nothing about SME versus Neon.

For a direct comparison, Arm’s Android Gaussian blur explorer calls the Neon, SVE2, and SME implementations on the same input, checks their output, and measures each on the same CPU. In its 15×15 grayscale example, SME’s median time was about 1.45× lower than Neon’s at 640×640 and 2.01× lower at 3840×2160. This is one operation and device, not a forecast for all KleidiCV calls.

To evaluate your app, trace the resolved implementation for a specific call, then compare matched images, kernel sizes, border modes, thread count, affinity, and thermal state. Time the full pipeline separately. The wider vector path can help long regular stripes; small images, transition overhead, memory bandwidth, and conversion work may favor another backend.

OpenCV is the best-documented framework integration, but it is not the only way to use KleidiCV. KleidiCV also exposes a standalone C API that an application or another computer-vision framework can call directly. As of September 2026, I would not claim another major public framework integration without identifying its source and release.

The KleidiCV 26.03 announcement describes direct integration as well as OpenCV adoption. It says OpenCV 4.13 and later enable KleidiCV by default in supported AArch64 builds; for earlier OpenCV versions, check the exact build configuration. Arm’s standalone build guide shows the library and API route without OpenCV.

Direct use is a capability, not evidence that a named commercial application already ships it. For an application audit, inspect its native dependency tree or symbols and confirm the selected KleidiCV operation. Likewise, the fact that KleidiAI appears in many ML runtimes says nothing about KleidiCV adoption.

If you plan a direct integration, start with one operation, for example a color conversion or Gaussian blur, and compare it against your existing implementation using identical image sizes and formats. Check the pinned KleidiCV release for the operation’s supported types and backend; current ordinary APIs generally choose Neon or SVE2, while _sme variants prefer SME/SME2 where available. The KleidiCV source is the place to verify what your version actually implements.

SME and SME2 ACLE (Arm C Language Extensions) intrinsics are supported by GCC 14+, Clang 18+, and Arm Compiler for Embedded 6.21+. These compilers can compile C/C++ source code containing SME intrinsics via the <arm_sme.h> header. Arm recommends always using the most recent available version of whichever compiler you choose.

Compiler Version Requirements

Key details per compiler

GCC 14+

• GCC 14 is the first version to support SME and SME2 ACLE intrinsics.
• GCC uses a separate assembler (GNU binutils). Use GNU binutils 2.41+ for SME2 kernels; older versions may support base SME but lack SME2 instructions.
• Versions prior to GCC 14 cannot compile C/C++ code that uses SME intrinsics.

Clang 18+

• Clang 18 provides the named <arm_sme.h> API for SME/SME2 ACLE intrinsics and is a practical starting point for these examples.
• Clang integrates its own assembler, so no separate binutils requirement exists.
• Earlier support was experimental. Clang 17 generated arm_sme_draft_spec_subject_to_change.h; that draft interface should not be treated as interchangeable with the <arm_sme.h> API used here.

Arm Compiler for Embedded 6

• A cross-platform compiler for bare-metal application development.
• SME support was first introduced in version 6.17.
• Additional SME and SME2 functionality was added in subsequent releases.
• Use version 6.21 or higher as a starting point, and check the release documentation for the required SME and SME2 intrinsic forms.

These version recommendations concern the SME/SME2 intrinsic sets available in those releases, not every newer optional SME extension. Check the compiler release and feature macros for any optional instructions your kernel requires.

Both Arm Compiler 6 and Clang also enable:

• Assembly of source code containing SME and SME2 instructions
• Disassembly of ELF object files containing SME and SME2 instructions

Required Compiler Flags

To enable SME intrinsics, pass the appropriate architecture extension flag:

# For SME (Armv9.2-A and later)
-march=armv9.2-a+sme

# For SME2 (Armv9.3-A and later)
-march=armv9.3-a+sme2

These flags enable the feature macros that guard the intrinsic declarations:

You can test for these macros in your code to conditionally compile SME paths:

#ifdef __ARM_FEATURE_SME2
#include <arm_sme.h>
// SME2 intrinsics available
#elif defined(__ARM_FEATURE_SME)
#include <arm_sme.h>
// SME intrinsics available
#else
// Fallback to SVE, Neon, or scalar
#endif

Header File

The primary header for SME intrinsics is:

#include <arm_sme.h>

This header also pulls in <arm_sve.h>, since SME builds on the SVE type system (using svfloat32_t, svint8_t, and other scalable vector types).

GNU Binutils Requirement

When using GCC (which uses a separate assembler/linker from the binutils package), use GNU binutils 2.41 or later for SME2. Older assemblers may support base SME without supporting SME2, and newer optional extensions may require a later release. Match the assembler to the instructions generated by your compiler.

This does not apply to Clang, which has an integrated assembler.

Practical Recommendations

1. Use the latest compiler version — Arm actively improves SME code generation quality, so newer releases produce better code.
2. Match your architecture flag to your target hardware — use armv9.2-a+sme for SME-only targets, armv9.3-a+sme2 for SME2.
3. Check binutils version when using GCC — a mismatched binutils is a common source of assembly errors.
4. Use pragmas for function-level control — If only some functions need SME, you can use target pragmas or function attributes rather than compiling the entire translation unit with SME enabled.

Resources

• Arm Programmer’s Guide – Compiler options and pragmas
• Arm Programmer’s Guide – Compiler support
• Clang 17 experimental SME header generation
• Clang 18 SME header generation

Include <arm_sme.h>, compile the SME translation unit for a target such as -march=armv9.2-a+sme or -march=armv9.3-a+sme2, and declare the function’s streaming-mode and ZA contract with ACLE attributes. The compiler can then insert required mode transitions and handle ABI state. Keep an ordinary Arm64 path and select the SME function only when the OS exposes the feature.

A Small Intrinsic Example

This function accumulates one FP32 outer product into a caller-owned ZA tile. It deliberately does not zero ZA or write a result matrix: a caller can accumulate several products, then store the tile. Its declaration is the important part of the example: __arm_streaming says the function body uses streaming mode, and __arm_inout("za") says it updates the caller’s ZA state.

#include <arm_sme.h>

void accumulate_fp32(const float *a, const float *b)
    __arm_streaming __arm_inout("za");

void accumulate_fp32(const float *a, const float *b)
    __arm_streaming __arm_inout("za")
{
    svbool_t pg = svptrue_b32();
    svfloat32_t left = svld1_f32(pg, a);
    svfloat32_t right = svld1_f32(pg, b);
    svmopa_za32_m(0, pg, pg, left, right);
}

Build the source as an object file, since the snippet has no main:

clang -O2 -march=armv9.2-a+sme -c accumulate.c -o accumulate.o
llvm-objdump -d accumulate.o

Look for an fmopa instruction in the disassembly. The compiler may also create mode-transition code at call sites; an object file containing fmopa proves compilation, not that your application executes the function. Arm’s worked SME2 matrix-multiply example shows tile initialization, repeated outer products, tail predicates, and result stores in a complete algorithm.

Choose the Function Contract

__arm_streaming and shared-ZA attributes are part of a function’s type and calling convention. A private-ZA attribute applies to a declaration/definition. Match the declaration and definition, and do not hide these contracts from callers behind an incompatible function pointer. For a regular application entry point, an __arm_locally_streaming wrapper with __arm_new("za") can own a tile internally and return through an ordinary interface.

SME2 intrinsics require the __ARM_FEATURE_SME2 compilation feature. Guard target-specific source and headers when building multiple variants. Do not put +sme2 on a whole portable binary: compiling an SME2 object and performing an OS-visible runtime check before calling it preserves fallback behavior. SME2 capability does not imply ordinary non-streaming SVE2 support.

For toolchain versions, start with GCC 14 or Clang 18 for SME/SME2 ACLE and verify the exact intrinsic set your pinned compiler implements. GCC builds also need an assembler that recognizes the selected instructions. The ACLE specification’s SME attributes and intrinsics is the precise syntax reference; Arm’s worked example is the better next step for a runnable kernel.

In C or C++, declare the function’s streaming contract with ACLE attributes and let the compiler place mode transitions around calls. In handwritten assembly or JIT code, use SMSTART and SMSTOP only after defining the function’s ABI contract and handling live vector and ZA state. Switching PSTATE.SM changes the interpretation and lifetime of Z/P values, so it cannot safely be hidden inside an ordinary C function with unmodelled inline assembly.

C/C++ Interfaces

For an ordinary public API whose implementation uses streaming mode, a locally streaming function is usually the simplest choice:

#include <arm_sme.h>

__arm_locally_streaming __arm_new("za")
void process_tile(const float *a, const float *b)
{
    svbool_t pg = svptrue_b32();
    svfloat32_t left = svld1_f32(pg, a);
    svfloat32_t right = svld1_f32(pg, b);
    svzero_za();
    svmopa_za32_m(0, pg, pg, left, right);
    /* Store the required ZA slices before returning. */
}

This is a contract example, not a complete matrix multiply: it initializes one ZA tile and accumulates one outer product but omits result storage. __arm_locally_streaming changes the function body without changing the caller’s ordinary interface. Check that the compiler defines __ARM_FEATURE_LOCALLY_STREAMING; that ACLE feature is optional.

For an internal chain that shares ZA and avoids repeated transitions, declare the streaming and ZA state in the function type:

void accumulate_tile(const float *a, const float *b)
    __arm_streaming __arm_inout("za");

The compiler can adapt calls between streaming and non-streaming functions according to their declared interfaces. __arm_streaming_compatible is for functions that can execute in either mode without a mode switch. __arm_new("za") gives a function private ZA state; __arm_inout("za") shares and changes the caller’s state. The ACLE streaming and ZA rules explain the exact contracts.

Assembly and JIT Code

SMSTART SM enables streaming mode; SMSTOP SM disables it. SMSTART ZA and SMSTOP ZA control ZA independently. Unqualified SMSTART and SMSTOP affect both SM and ZA. A kernel that uses ZA needs both the streaming mode required by its instructions and enabled ZA storage.

Before writing the assembly, decide whether the function has a streaming interface, creates private ZA, or shares the caller’s ZA. Preserve ABI-required registers and follow the AAPCS64 ZA lazy-save rules when applicable. Do not copy a short SMSTART/SMSTOP macro as a complete call wrapper; correct register preservation and ZA ownership depend on the surrounding ABI contract. Arm’s SME calling-convention guide is the appropriate starting point for assembly.

Treat Z and P values that are live across a streaming-mode change as invalid unless they have been saved according to the ABI. Streaming vector length can differ from the ordinary SVE vector length, so compute loop bounds for the active mode. On Linux, system calls and signals have documented SME state rules ; a JIT or runtime that crosses those boundaries must account for them.

Group enough work per streaming region to justify transitions, but do not manually hold streaming mode across arbitrary framework or library calls. Keep the mode and ZA contracts visible to the compiler or JIT and verify the generated instructions in disassembly.

Give each SME routine an explicit ZA ownership contract and keep that contract visible at every declaration and call site. For a self-contained kernel called from ordinary Neon/SVE code, use a normal external interface with __arm_locally_streaming and private __arm_new("za") state. If several streaming functions cooperate on one tile, use shared-ZA attributes such as __arm_inout("za"). Let an ACLE-aware compiler manage mode and ZA transitions; handwritten assembly must implement the matching AAPCS64 rules.

Separate Mode from Tile Ownership

PSTATE.SM controls streaming mode; PSTATE.ZA controls whether ZA storage is enabled. They can be controlled independently. A change to streaming mode affects live Z/P vector values, while ZA is a separate state resource that may remain enabled across a mode change. The ordinary SVE vector length and streaming vector length may differ.

__arm_new("za") does not promise that the tile’s numeric contents start at zero. Zero it explicitly with svzero_za() before accumulation if the algorithm needs a zero accumulator.

Two Useful Patterns

An isolated implementation can hide streaming and ZA details behind an ordinary API:

#include <arm_sme.h>

__arm_locally_streaming __arm_new("za")
void process_matrix_tile(const float *a, const float *b, float *result)
{
    svzero_za();
    /* Load vectors, accumulate into ZA, then store the required slices. */
}

This is an interface sketch: the algorithm and stores still need implementation. The caller can preprocess with Neon, call process_matrix_tile, and postprocess with non-streaming SVE using memory buffers between stages. Check __ARM_FEATURE_LOCALLY_STREAMING for compiler support.

For a pipeline that intentionally passes the same ZA tile between functions, put shared state in the declarations:

void initialize_tile(void) __arm_streaming __arm_out("za");
void accumulate_tile(const float *a, const float *b)
    __arm_streaming __arm_inout("za");
void store_tile(float *dst) __arm_streaming __arm_in("za");

A streaming wrapper with private ZA can call those three functions in sequence. Their declarations are part of the calling convention; do not call one through a function pointer that erases the attributes. The ACLE ZA examples explain the contract and its compiler handling.

Calls, Saves, and Performance

AAPCS64 defines a lazy-save protocol for caller-owned ZA when a callee needs private state. It avoids eagerly copying a potentially large tile on every call. The compiler and runtime handle it for correctly annotated C/C++ functions; do not implement a partial TPIDR2_EL0 protocol in an ordinary inline-assembly block. Arm’s ZA calling-convention guide explains when a save is committed.

Keep ordinary Neon/SVE helpers outside a hot streaming loop when possible; repeated mode transitions and any necessary state preservation have a cost. Pass stable data across the boundary through memory or ordinary scalar arguments. A Neon or SVE vector argument can be legal under the ABI in some cases, but do not assume its live register representation or vector length survives a mode change.

Finally, distinguish build support from runtime support. Compile SME source separately, check OS-exposed SME capability before dispatch, and retain a non-SME fallback. SME2 support does not imply ordinary non-streaming SVE2.

Expose an ordinary AArch64 function interface from the shared library, keep SME-specific functions internal, and dispatch to them only after checking OS-exposed SME capability. The key ABI rules are the function’s streaming-mode contract, whether it shares ZA with its caller, and the handling of vector state at a mode boundary. These contracts must agree across declarations, definitions, and function pointers.

Keep the Public Interface Conventional

A public header can use fixed-layout memory and scalar arguments:

/* matrix_api.h */
#include <stddef.h>
void matrix_multiply(size_t m, size_t n, size_t k,
                     const float *a, const float *b, float *out);

The public function has the normal non-streaming ABI. It can call a private SME implementation built in a separate target-specific translation unit:

/* SME translation unit: built with -march=armv9.2-a+sme */
#include <arm_sme.h>

__arm_locally_streaming __arm_new("za")
void matrix_multiply_sme(size_t m, size_t n, size_t k,
                         const float *a, const float *b, float *out)
{
    /* Initialize ZA explicitly, compute tiles, and store results. */
}

This is an ABI sketch, not a complete multiplication implementation. __arm_locally_streaming keeps the external function type ordinary while running its body in streaming mode; __arm_new("za") gives it private ZA state. An application may also give the SME symbol hidden visibility to keep the public ABI small. A non-SME fallback must implement the same numerical contract.

The dispatcher must use a runtime feature query rather than a compiler target macro. On Linux/Android, the OS-exposed HWCAP2 bits provide an explicit check:

/* Linux/Android dispatcher source, built for the portable arm64 baseline */
#include <sys/auxv.h>
#include <asm/hwcap.h>

static int os_exposes_sme(void) {
    return (getauxval(AT_HWCAP2) & HWCAP2_SME) != 0;
}

ACLE also defines __arm_has_sme() in <arm_sme.h> to report whether the current thread has access to SME; it is a valid runtime query where the toolchain and runtime provide it. It does not establish SME2 support. Choose HWCAP2_SME2 if the private kernel specifically requires SME2. Keep the feature-specific object isolated so the compiler does not put SME instructions in the public entry point or library initializer. An ordinary arm64 process can load a shared library containing specialized code; it must not execute that code on unsupported hardware.

Know Which Attributes Change the Call Contract

If a declaration omits __arm_streaming or a shared-ZA attribute that the definition requires, the caller may use the wrong convention. A public C++ header or callback typedef must carry any contract it exposes. The ACLE attribute rules and AAPCS64 are the authoritative details.

Keep scalable SVE types out of an ordinary public interface unless you have explicitly designed and tested that PCS. The ordinary SVE vector length can differ from the streaming vector length, and live Z/P values do not carry unchanged across a mode transition. Pointers to fixed-layout arrays make both the calling convention and fallback simpler.

ZA, ZT0, and the Dynamic Linker

When a caller owns ZA and a callee needs private ZA, AAPCS64 defines a lazy-save protocol. Correctly annotated compiler-generated code and its runtime support handle that protocol; a small handwritten SMSTART/SMSTOP wrapper is not a substitute. If you expose a streaming or shared-ZA function as a dynamic symbol, verify that the platform loader, toolchain, and runtime support the variant calling convention. An ordinary public wrapper keeps that concern internal.

SME2’s ZT0 has its own sharing attributes. A caller that needs its contents preserved across a call must use the corresponding ZT0 contract or save it explicitly; do not assume ZA’s lazy-save handling covers ZT0.

For validation, build the fallback and SME objects separately, inspect the public header and exported symbols, run on both an SME-capable and a non-SME device, and use a breakpoint or disassembly to confirm that the feature gate selects the intended object. Test callbacks and exception or nonlocal-control-flow paths if your library uses them, since they can cross the declared call boundary.

SME2 matrix multiplication works by decomposing C = A × B into vector outer products accumulated in ZA. If the streaming vector length is SVL_bits, ZA contains SVL_bytes × SVL_bytes bytes, where SVL_bytes = SVL_bits / 8. Each of its four FP32 tile views contains L × L elements, where L = SVL_bits / 32. This example preprocesses the left matrix into a column-contiguous layout, then accumulates K outer products per output tile in ZA0.S using svmopa_za32_m. It stores the result from ZA tile slices to memory and runs the kernel in streaming mode with ZA enabled.

Architecture Background

ZA contains SVL_bytes² bytes. For FP32 operations, it has four tile views, ZA0.S–ZA3.S, each containing L × L elements. At a 512-bit streaming vector length, ZA contains 4096 bytes and each tile is 16 × 16 FP32 elements. The FMOPA instruction (exposed via svmopa_za32_m) computes:

ZA0.S[i][j] += A_col[i] * B_row[j]    for all valid (i,j) within predicates

With all lanes active, one instruction performs an L × L rank-1 update. After K such accumulations, the tile holds an L × L block of the result matrix. Edge predicates restrict the update to valid output elements. In the code below, the variable named SVL is the FP32 lane count L, obtained with svcntsw(), rather than a length in bits or bytes.

Algorithm Overview

For matrices A (M×K), B (K×N), C (M×N):

1. Preprocess A — rearrange from row-major into column-contiguous blocks so that each column of a tile can be loaded with a single svld1
2. Tile loop — iterate over ceil(M/L) × ceil(N/L) output tiles, predicating partial edge tiles
3. K loop — for each tile, accumulate K outer products into ZA tile 0
4. Store — extract horizontal tile slices to the result matrix

Worked Implementation

The following kernel follows Arm’s SME2 intrinsics walkthrough . It teaches packing, tile accumulation, and storage. Pin the example source and compile it with your target toolchain before treating the excerpts as a drop-in library.

Header and Guards

// sme2_matmul.h
#ifndef SME2_MATMUL_H
#define SME2_MATMUL_H

#include <stdint.h>

// Public API — callable from non-streaming code
void sme2_matmul_f32(uint64_t M, uint64_t K, uint64_t N,
                     const float *restrict A,
                     const float *restrict B,
                     float *restrict A_packed,
                     float *restrict C);

#endif // SME2_MATMUL_H

Matrix Preprocessing (Intrinsics Version)

The left matrix must be transformed so that columns of each L × L tile are contiguous in memory. This uses ZA itself as a transposition buffer — data is written as rows and read back as columns:

// sme2_matmul.c
#include "sme2_matmul.h"
#include <arm_sme.h>

#ifndef __ARM_FEATURE_SME2
#error "Compile this source with SME2 enabled (-march=armv9.3-a+sme2)"
#endif

static void preprocess_l_intr(
    uint64_t M, uint64_t K, uint64_t SVL,
    const float *restrict a,
    float *restrict a_mod) __arm_streaming __arm_inout("za") {

    const uint64_t M_mod = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0));

    for (uint64_t row = 0; row < M; row += SVL) {
        svbool_t pMDim = svwhilelt_b32(row, M);

        for (uint64_t col = 0; col < K; col += 2 * SVL) {
            svcount_t pKDim = svwhilelt_c32(col, K, 2);

            // Load rows of A into ZA tiles (write-as-rows)
            for (uint64_t trow = 0; trow < SVL; trow += 4) {
                svcount_t p0 = svpsel_lane_c32(pKDim, pMDim, trow + 0);
                svcount_t p1 = svpsel_lane_c32(pKDim, pMDim, trow + 1);
                svcount_t p2 = svpsel_lane_c32(pKDim, pMDim, trow + 2);
                svcount_t p3 = svpsel_lane_c32(pKDim, pMDim, trow + 3);

                const uint64_t corner = (row + trow) * K + col;
                svfloat32x2_t zp0 = svld1_x2(p0, &a[corner + 0 * K]);
                svfloat32x2_t zp1 = svld1_x2(p1, &a[corner + 1 * K]);
                svfloat32x2_t zp2 = svld1_x2(p2, &a[corner + 2 * K]);
                svfloat32x2_t zp3 = svld1_x2(p3, &a[corner + 3 * K]);

                svfloat32x4_t zq0 = svcreate4(svget2(zp0, 0), svget2(zp1, 0),
                                              svget2(zp2, 0), svget2(zp3, 0));
                svfloat32x4_t zq1 = svcreate4(svget2(zp0, 1), svget2(zp1, 1),
                                              svget2(zp2, 1), svget2(zp3, 1));
                svwrite_hor_za32_f32_vg4(0, trow, zq0);
                svwrite_hor_za32_f32_vg4(1, trow, zq1);
            }

            // Read as columns and store (transposition complete)
            const uint64_t dest_0 = row * K + col * SVL;
            const uint64_t dest_1 = dest_0 + SVL * SVL;
            for (uint64_t tcol = 0; tcol < SVL; tcol += 4) {
                svcount_t p0 = svwhilelt_c32(dest_0 + tcol * SVL, K * M_mod, 4);
                svcount_t p1 = svwhilelt_c32(dest_1 + tcol * SVL, K * M_mod, 4);
                svfloat32x4_t zq0 = svread_ver_za32_f32_vg4(0, tcol);
                svfloat32x4_t zq1 = svread_ver_za32_f32_vg4(1, tcol);
                svst1(p0, &a_mod[dest_0 + tcol * SVL], zq0);
                svst1(p1, &a_mod[dest_1 + tcol * SVL], zq1);
            }
        }
    }
}

The Outer Product Kernel

The core computation iterates over L × L output tiles, accumulating K outer products per tile:

static void matmul_intr_impl(
    uint64_t M, uint64_t K, uint64_t N, uint64_t SVL,
    const float *restrict matLeft_mod,
    const float *restrict matRight,
    float *restrict matResult) __arm_streaming __arm_inout("za") {

    // Build the result matrix tile by tile
    for (uint64_t row = 0; row < M; row += SVL) {
        svbool_t pMDim = svwhilelt_b32(row, M);

        for (uint64_t col = 0; col < N; col += SVL) {
            svbool_t pNDim = svwhilelt_b32(col, N);

            // Zero the accumulator tile
            svzero_za();

            const uint64_t matLeft_pos = row * K;
            const uint64_t matRight_UL_corner = col;

            // Accumulate outer products over K
            for (uint64_t k = 0; k < K; k++) {
                // Load column from preprocessed left matrix
                svfloat32_t zL =
                    svld1(pMDim, &matLeft_mod[matLeft_pos + k * SVL]);

                // Load row from right matrix (naturally contiguous)
                svfloat32_t zR =
                    svld1(pNDim, &matRight[matRight_UL_corner + k * N]);

                // Outer product and accumulate into ZA tile 0
                svmopa_za32_m(0, pMDim, pNDim, zL, zR);
            }

            // Store ZA tile slices to the result matrix
            const uint64_t result_corner = row * N + col;
            for (uint64_t trow = 0; trow < SVL && row + trow < M; trow += 4) {
                svbool_t p0 = svpsel_lane_b32(pNDim, pMDim, row + trow + 0);
                svbool_t p1 = svpsel_lane_b32(pNDim, pMDim, row + trow + 1);
                svbool_t p2 = svpsel_lane_b32(pNDim, pMDim, row + trow + 2);
                svbool_t p3 = svpsel_lane_b32(pNDim, pMDim, row + trow + 3);

                svst1_hor_za32(0, trow + 0, p0,
                    &matResult[result_corner + (trow + 0) * N]);
                svst1_hor_za32(0, trow + 1, p1,
                    &matResult[result_corner + (trow + 1) * N]);
                svst1_hor_za32(0, trow + 2, p2,
                    &matResult[result_corner + (trow + 2) * N]);
                svst1_hor_za32(0, trow + 3, p3,
                    &matResult[result_corner + (trow + 3) * N]);
            }
        }
    }
}

Top-Level Entry Point

The public function uses __arm_locally_streaming (so callers need not be in streaming mode) and __arm_new("za") (to create fresh ZA state without destroying any caller’s ZA):

__arm_new("za") __arm_locally_streaming
void sme2_matmul_f32(
    uint64_t M, uint64_t K, uint64_t N,
    const float *restrict A,
    const float *restrict B,
    float *restrict A_packed,
    float *restrict C) {

    uint64_t SVL = svcntsw();  // Number of 32-bit elements per streaming vector
    preprocess_l_intr(M, K, SVL, A, A_packed);
    matmul_intr_impl(M, K, N, SVL, A_packed, B, C);
}

Integrating into Your Application

Workspace Allocation

The packed buffer size depends on the runtime SVL. Query it once and cache:

#include <stdlib.h>

// Call this from streaming-compatible context to get the workspace size
__arm_locally_streaming
size_t sme2_packed_a_size(uint64_t M, uint64_t K) {
    uint64_t SVL = svcntsw();
    uint64_t M_padded = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0));
    return M_padded * K * sizeof(float);
}

Runtime Feature Detection

Never assume SME2 is available. Check hardware capability before dispatching:

#include <stdbool.h>
#include <sys/auxv.h>
#include <asm/hwcap.h>

static bool cpu_has_sme2(void) {
#if defined(__aarch64__) && defined(HWCAP2_SME2)
    return (getauxval(AT_HWCAP2) & HWCAP2_SME2) != 0;
#else
    return false;
#endif
}

void application_matmul(uint64_t M, uint64_t K, uint64_t N,
                        const float *A, const float *B, float *C) {
    if (cpu_has_sme2()) {
        size_t bytes = sme2_packed_a_size(M, K);
        float *workspace = malloc(bytes);
        if (workspace == NULL) {
            matmul_reference_f32(M, K, N, A, B, C);
            return;
        }
        sme2_matmul_f32(M, K, N, A, B, workspace, C);
        free(workspace);
    } else {
        matmul_reference_f32(M, K, N, A, B, C);  // Fallback (Neon, SVE, scalar)
    }
}

Build System Integration

Isolate the SME2 kernel in its own translation unit and apply architecture flags only to that file:

# CMakeLists.txt
add_library(matmul_sme2 OBJECT sme2_matmul.c)
target_compile_options(matmul_sme2 PRIVATE -O3 -march=armv9.3-a+sme2)

add_executable(myapp main.c)
target_link_libraries(myapp PRIVATE matmul_sme2)
# main.c compiled without SME2 flags — stays portable

Direct compilation:

# SME2 kernel only
clang -O3 -march=armv9.3-a+sme2 -c sme2_matmul.c -o sme2_matmul.o

# Application (no SME2 flags needed)
clang -O3 -march=armv8.2-a main.c sme2_matmul.o -o myapp

Key ACLE Function Attributes

Key Intrinsics Reference

Compiler Requirements

Optimization Considerations

• Minimize streaming mode transitions — enter streaming mode once and perform all tile work before returning. Each mode transition has a cost and invalidates live Z/P values across the boundary.
• Reuse packed operands — if the same A matrix is used for multiple B matrices (e.g., batched GEMM), preprocess A once and reuse across calls.
• Unroll the K loop — issue multiple svmopa_za32_m calls per iteration to improve instruction-level parallelism and hide load latency.
• Use multi-vector SME2 data movement — grouped loads and ZA slice transfers can reduce the instructions needed for packing and storing tiles, as shown in the preprocessing code. The svmopa_za32_m outer-product intrinsic takes two individual vectors, not x2/x4 vector groups; multi-vector arithmetic forms use their own accumulator layouts.
• Fuse post-operations — apply bias addition, ReLU, or quantization while extracting tile slices, before the data leaves registers.
• Consider data types — SME2 supports FP16→FP32 (svmopa_za32_f16_m), BF16→FP32, INT8→INT32 and other widening outer products for ML workloads.
• Pre-arranged layouts — in production frameworks, arrange the left operand in column-major order upstream, eliminating the preprocessing step entirely.
• Benchmark realistic shapes — SME2 startup cost and packing overhead mean Neon or SVE may remain faster for small matrices (depending on the shape and selected kernel).

Resources

• Outer product approach and transposition — why left-matrix preprocessing is needed and how it works
• ACLE SME language extensions and intrinsics — official ACLE specification for SME

Use KleidiAI when Arm CPU inference is important to your users, a profiled hot operator matches a KleidiAI kernel’s complete type/shape/quantization contract, and an existing backend does not already give you that path. Prototype one operator and measure end-to-end benefit before committing to a new framework integration.

Check for an Existing Path First

If the framework already reaches KleidiAI, first test that route with the model you actually deploy. An installed package or successful inference run does not prove KleidiAI or SME2 executed. Trace the selected microkernel and compare latency with a controlled fallback.

When a Direct Integration Pays

KleidiAI supplies low-level microkernels and packing functions, not model loading, graph scheduling, or a universal “best kernel” policy. A direct integration has to:

1. Map a lowered operator to a kernel with matching input/weight/output types, quantization granularity, group size, layout, shape, and post-operations.
2. Query OS-exposed CPU features and select a compatible kernel. There is no universal SME2 → I8MM → DotProd ranking independent of operation and shape.
3. Allocate and manage packing/workspace buffers, including any weight packing that can be reused across inference calls.
4. Provide a correct fallback for unsupported hardware and operator signatures.
5. Integrate with your framework’s threading and scheduling, then test numerical correctness and performance.

Arm’s kernel selection guide explains how to match a kernel to a workload. The KleidiAI source inventory is the authority for the exact release you bundle.

Make the Decision from a Prototype

Profile representative models on the target Arm devices. If matrix multiplication or compatible convolutions account for meaningful absolute runtime, compare one candidate KleidiAI kernel with your existing tuned implementation at real shapes and thread counts. Include weight packing, activation quantization, layout conversions, and the full inference call in the measurement.

KleidiAI is less likely to justify a new integration when the CPU path is minor beside GPU/NPU work, the hot operators have no matching kernels, or one-shot packing costs dominate. Batch-1 matrix-vector operations can still have specialized kernels, so do not reject them from shape alone. If the prototype saves meaningful end-to-end time or energy and the fallback remains maintainable, expand coverage one operator at a time.

The ONNX Runtime integration walkthrough is a concrete example of the registration, packing, and dispatch work involved. Hardware SME2 support by itself is not the deciding factor: KleidiAI also targets other Arm features, and the best path depends on the workload.

Start with the library or framework already doing your matrix work. Run the same representative workload with its SME or SME2 kernel path enabled and disabled, confirm which kernel actually ran, and compare end-to-end time. This tests the opportunity without committing to a custom implementation. Compiling the application with +sme2 alone does not turn arbitrary loops into ZA outer-product kernels.

Find the work worth testing

Profile a representative run and identify the expensive operation and its shapes. For a native Linux program, perf record -g -- ./app and perf report can locate a hot GEMM, convolution, or packing routine. Framework operator profilers can show whether MatMul or Conv dominates an inference. Record the data type, matrix dimensions, quantization format, and thread count; these determine whether an available SME2 kernel can run.

Check that the OS exposes SME or SME2 on the machine you will measure. On Linux, inspect /proc/cpuinfo or use getauxval(AT_HWCAP2); a CPU marketing name alone is insufficient. On Android, test on the actual device. An emulator can check correctness, but its timings do not predict hardware speed.

Use an existing implementation

If no backend covers your operator and format, extract one hot kernel and build a small SME prototype. A function-specific target and ACLE attributes keep SME code isolated from the portable application. Include packing, dispatch, and streaming-mode transitions in the timing when they occur in production.

Make the result meaningful

Use identical inputs, precision, thread count, and core placement. Warm both variants, repeat measurements, and report the median plus variation. Compare the kernel time and the full request or inference time. Validate outputs against the same reference; different accumulation orders may require a stated floating-point tolerance.

A fast SME2 kernel is useful only when the framework dispatches to it and its savings exceed packing, mode-transition, and other application costs. If the overall time barely changes, the profile should show whether the operator was a small fraction of runtime, the workload missed kernel coverage, or another stage became dominant.

Resources

• Arm’s ONNX Runtime profiling walkthrough shows an operator-level and hardware-counter comparison.
• Arm’s llama.cpp SME2 performance walkthrough provides a concrete model benchmark.
• Arm Performance Libraries 26.01 update identifies the macOS SME GEMM support.

Yes. QEMU’s AArch64 emulation can run SME code for functional testing, including different streaming vector lengths. Arm Fixed Virtual Platforms (FVPs) are another route when you need a complete simulated system or bare-metal environment. Neither predicts the speed of a physical SME processor.

Start with QEMU user mode

Build an AArch64 Linux test binary with a compiler that supports the SME intrinsics you use, then run it through a recent qemu-aarch64. If the binary is dynamically linked, provide an AArch64 sysroot with -L or use a compatible static build. For example:

clang --target=aarch64-linux-gnu -O2 -march=armv9-a+sme2 test.c -o test
qemu-aarch64 -cpu max,sme=on,sme-default-vector-length=32 -L /path/to/aarch64-sysroot ./test

The sysroot path and link command depend on your toolchain. QEMU’s sme-default-vector-length value is bytes, so 32 selects a 256-bit default if that length is enabled. Try 16, 32, and 64 bytes with dimensions that are not neat tile multiples. QEMU’s max CPU may enable SME subfeatures such as SME2; check the reported capabilities of the QEMU version you use.

A tiny test can check that a locally streaming function runs and sees a streaming vector length:

#include <arm_sme.h>
#include <arm_sve.h>
#include <stdio.h>

#if defined(__ARM_FEATURE_LOCALLY_STREAMING)
__arm_locally_streaming
#endif
static unsigned words_per_streaming_vector(void) {
    return svcntsw();
}

int main(void) {
    printf("%u 32-bit lanes\n", words_per_streaming_vector());
}

Here svcntsw() counts 32-bit lanes in the streaming vector. The compiler handles entry and exit from streaming mode for the annotated function. This is a feature and ABI smoke test, not a speed test. If your compiler lacks __arm_locally_streaming, use an explicit streaming function interface supported by that toolchain.

Use a full-system platform when the OS matters

QEMU system mode and Arm FVPs can test boot, feature exposure, signals, and context switching with an appropriate guest image. Their setup depends on the exact machine model and image, so follow that platform’s instructions rather than copying a generic bare-metal command. Check that both the simulated CPU and guest kernel expose SME to user space.

A native SME-capable Apple M-series Mac is useful for macOS application testing. Its OS APIs and toolchain differ from Linux; do not use Linux HWCAP instructions as an Apple feature check.

Use emulation to find invalid instructions, ABI mistakes, tail bugs, and vector-length assumptions. Use real hardware for latency, throughput, power, cache behavior, and any Neon/SVE/SME comparison.

Resources

• QEMU Arm CPU feature documentation defines SME CPU properties and the user-mode vector-length setting.
• Arm’s SME2 environment check walks through compiler and runtime checks.

Prototype one hot operation at a time. Write a scalar reference, implement the same operation with SME intrinsics, test awkward dimensions and vector lengths, then measure on real hardware. Keep packing and streaming-mode costs visible: a fast inner instruction does not guarantee a fast application.

Choose a candidate and establish correctness

Profile the application first. Record the operation, input layout, data type, quantization parameters, and typical dimensions. Matrix multiplication and convolution lowered to matrix multiplication are natural candidates; tiny or memory-bound operations may not repay the setup cost.

Use a simple scalar implementation as the reference. For FP32 matrix multiplication, for example:

for (size_t m = 0; m < M; ++m)
    for (size_t n = 0; n < N; ++n) {
        float sum = 0.0f;
        for (size_t k = 0; k < K; ++k)
            sum += A[m*K + k] * B[k*N + n];
        C[m*N + n] = sum;
    }

Test tails such as M=17 and N=19, K boundaries, zero dimensions where permitted, and nonzero initial C if the kernel accumulates. Compare floating-point results with an explained tolerance because accumulation order can differ.

Build the smallest SME experiment

Arm’s SIMD Loops project has comparable scalar, Neon, SVE, and SME examples. It is a useful starting point for understanding build targets and checking output, but adapt its loop to your real dimensions and data layout.

An SME outer-product kernel repeatedly loads one vector from each operand, accumulates their pairwise products in ZA, and stores the output tile after the K loop. ACLE expresses the central step as:

// Inside a streaming function with ZA enabled; predicates and vectors
// must match the chosen tile and packed input layout.
svmopa_za32_m(0, row_predicate, column_predicate,
              lhs_vector, rhs_vector);

This is an illustrative inner step, not a complete matrix-multiply function. A complete kernel must choose tile dimensions from svcntsw(), pack or gather the left-hand columns, handle tails with predicates, initialize ZA when appropriate, and store each output slice. Use __arm_streaming and a suitable ZA interface such as __arm_new("za") for a self-contained leaf kernel; explicitly zero the accumulator before use. Build that source for SME with -march=armv9-a+sme or for SME2 with -march=armv9-a+sme2, according to the instructions used. Keep the rest of the application on a portable target and dispatch only to supported hardware.

Test, then measure

QEMU user mode or an Arm FVP can catch instruction, ABI, predication, and vector-length bugs. Try more than one streaming vector length. Their run time is not a hardware performance prediction.

On an SME-capable device, benchmark both kernels with identical inputs and thread placement. Warm up, repeat, report median and variation, and separately record packing, kernel, and full-application time. Check the binary or sampled instructions to confirm that the SME path ran. If the prototype loses, inspect matrix size, layout conversion, cache traffic, mode transitions, and ZA saves before changing the arithmetic.

Resources

• Arm’s SME2 matrix multiplication walkthrough provides a complete implementation to study.
• Arm SIMD Loops provides comparable loop implementations.
• ACLE SME intrinsics and attributes defines the function interface.

Use a profiler that attributes time to functions or model operators first. A counter cannot tell you how much time could be accelerated before an SME kernel exists. Once you have an SME-enabled implementation, compare the same workload and confirm which kernel ran.

Native Linux applications

perf provides a practical first pass:

perf record -g -- ./your_workload
perf report
perf list | grep -Ei 'sme|mopa|matrix'

Look for GEMM, convolution, and packing routines in the hot call paths. Available PMU event names vary by processor and kernel. If SME-related events exist, they can confirm instruction activity, but they do not by themselves measure saved time. Arm Streamline provides timeline and call-path views plus supported hardware counters, which help relate the matrix work to thread scheduling and memory stalls. Arm Performix and Linaro MAP can provide source-level hotspot views on supported systems.

For a BLAS application, call logging can identify how often SGEMM or DGEMM runs and with which dimensions. Check that an interposition tool actually intercepts the BLAS provider your executable uses; a preload library is not a universal substitute for relinking.

Framework models

An operator name or XNNPACK P prefix can identify a packed path, but does not alone prove SME2 execution. Use a kernel symbol, sampled instruction, or supported hardware event for that claim.

Compare before and after

Profile identical inputs, model, precision, thread count, and core placement with SME2 enabled and disabled in the backend. Repeat after warm-up and compare end-to-end time as well as the matrix operator. Packing and streaming-mode transitions count if they occur during real use. A faster GEMM may expose a new bottleneck in memory movement or other operators.

Resources

• Arm’s ONNX Runtime profiling example shows operator profiling and Streamline together.
• Arm’s ExecuTorch profiling guide shows how to compare operator timing.

Implement the same operation with the same input and output contract, verify correctness, and time each variant on one processor that supports all the extensions being compared. Report both kernel and end-to-end results. The fastest choice depends on operation, shape, layout, vector length, and implementation quality; SME does not automatically win because it has ZA.

Make the variants comparable

Neon is the AArch64 Advanced SIMD path; SVE supplies scalable vector operations and predication; SME adds streaming mode and ZA outer-product accumulation. For a matrix multiplication, each version should calculate the same output with the same precision and include equivalent packing or conversion costs. A compiler flag alone does not create a competitive SME kernel. Use explicit intrinsics, an established library, or verified disassembly to know what ran.

For example, compile source files that actually contain each implementation with appropriate targets:

clang -O3 -march=armv8-a+simd -c neon.c -o neon.o
clang -O3 -march=armv8-a+sve -c sve.c -o sve.o
clang -O3 -march=armv9-a+sme -c sme.c -o sme.o

Use SME2 when the kernel uses SME2-specific instructions. Function-specific targets can keep all variants in one dispatching application. Run only supported variants, and inspect disassembly or sampled symbols to confirm the intended path. Compare each output with a scalar reference using a stated tolerance for floating-point reorderings.

Control the measurement

Use representative small, medium, and large inputs. Keep core placement, thread count, input data, memory layout, and frequency/thermal state as consistent as possible. Warm up each variant and repeat enough times to report a median and variation. On Linux, perf stat -r 10 -- ./bench can add cycles and cache events; discover model-specific counters with perf list. Measure elapsed time directly as the primary result.

Report a ratio such as median_neon_time / median_sme_time, and state whether it covers the inner kernel or the whole operation. A scalar, unoptimized “vanilla” baseline is useful for validation but is a weak performance comparator. For an application decision, also compare an optimized existing backend.

Explain the result

A small tile may favor Neon because packing and streaming entry have little work to amortize. SVE may be effective for predicated loops that do not map well to ZA. SME can win on sufficiently large matrix work when operands arrive in a useful layout and the output stays in ZA through the reduction. The actual crossover is device and implementation specific. If SME loses, profile packing, memory stalls, mode transitions, and fallback dispatch before drawing an architectural conclusion.

Emulators can validate each variant’s behavior across vector lengths; use real hardware for timing.

Resources

• Arm SIMD Loops provides comparable kernels.
• Arm’s Neon, SVE, and SME matrix-multiplication comparison explains different mappings of the same work.

Check the framework’s dispatch path for your exact model, not just the KleidiAI kernel catalog. A kernel can exist in KleidiAI but be unavailable through your LiteRT, ExecuTorch, ONNX Runtime, or llama.cpp build. Eligibility depends on the operator, input and output types, weight format, quantization parameters, layout, shape, backend build, and device features.

Build an operator inventory

Export the model’s expensive nodes. For each, record the operator and effective matrix dimensions, activation and weight types, layout, quantization symmetry and zero points, per-tensor/channel/block scaling, and any fused activation. Dynamic shapes need representative runtime dimensions. A model viewer or a framework profiler is often quicker than reading graph source.

For ONNX, shape inference can fill in known static dimensions:

import onnx
from onnx import shape_inference

model = shape_inference.infer_shapes(onnx.load("model.onnx"))
for node in model.graph.node:
    print(node.name, node.op_type, list(node.input), list(node.output))

This lists nodes, but it does not prove that an execution provider delegates them or reveal every runtime shape.

Match the actual integration

Read the supported-operator table for the framework version you build, then follow its backend to the KleidiAI kernel it calls. For example, XNNPACK is the relevant path for many LiteRT and ExecuTorch CPU operators; MLAS is the path for ONNX Runtime; GGML’s type and operation dispatch matters for llama.cpp. “INT4” is not enough: block size, signedness, scale layout, and activation type must match. A kernel’s packing function and tail handling also matter.

Do not assume that a matrix dimension must be a multiple of a tile size. Many kernels handle tails, while some small shapes select a different implementation for performance. Check the particular dispatch condition.

Verify on the device

Run a representative input on SME2 hardware with the intended build. Use framework operator profiling to find which nodes were delegated and sampled native symbols or instruction disassembly to see which microkernel ran. In XNNPACK traces, a P node label indicates a packed path; it is not by itself proof of SME2. If a node misses, inspect the backend’s eligibility check, its exact tensor contract, and the device feature result.

Keep a small coverage sheet:

This separates static eligibility from measured dispatch, the distinction that matters when deciding whether to change a model or a build.

Resources

• LiteRT’s KleidiAI operator configurations gives one concrete support table.
• KleidiAI source defines kernel and packing contracts.
• Arm’s llama.cpp integration walkthrough explains that backend’s selection path.

Choose an output microtile that fits the ZA accumulators, keep it there through as much of the K reduction as practical, and store it once. Pack operands so each reduction step can load the A column values and B row values contiguously. The best tile shape depends on your data type, shape, cache, and hardware; “use every ZA tile” is a starting hypothesis, not a rule.

Derive the tile size

ZA is a square storage array whose side is the streaming vector length in bytes: SVL_bytes = SVL_bits / 8. It contains SVL_bytes² bytes. Its typed tile views share this storage. For 32-bit accumulation there are four ZA.S tiles, each containing L × L elements, where L = SVL_bits / 32. At a 512-bit SVL, ZA contains 4096 bytes and each tile is 16×16 32-bit elements. You can map the four tiles to a 2×2 output block of 32×32 elements or a 1×4 block of 16×64. The former reuses both operand panels; the latter can suit a narrow M dimension and wide N dimension. Query svcntsw() in streaming mode to get the number of 32-bit lanes. Do not bake 16 into the kernel.

A single output element is the sum of products across K. For a 2×2 tile arrangement, each K step loads two A vectors and two B vectors, then issues four outer products to update ZA0–ZA3. This is the core mapping:

for each output microtile:
    initialize its ZA accumulators
    for each k in the reduction:
        load A vectors for the output rows
        load B vectors for the output columns
        accumulate A0×B0, A0×B1, A1×B0, A1×B1 in ZA
    store the completed output microtile

A real implementation needs predicates for partial edge tiles and an explicit choice of whether C is overwritten or accumulated. If K is split into cache-sized blocks, carry partial sums across those blocks in ZA or load/add them back; zeroing ZA at each K block and overwriting C loses earlier products.

Pack and block around that kernel

For row-major A, the column values needed at one K step are strided. Pack panels so those values become consecutive, while B row values are already consecutive or can be packed into a panel that matches the selected N tile. Pack invariant weights once when possible. If packing dominates a small operation, a different algorithm or tile may win.

Choose outer M, N, and K cache blocks so packed panels are reused without overflowing the useful cache. The cache sizes and optimal block order are implementation-specific; benchmark realistic dimensions rather than assuming a fixed 2×2 arrangement. Initialize ZA before accumulation, keep streaming-mode transitions outside the hot tile loop where practical, and store only valid rows and columns with predicates.

The following shows the four accumulator updates. It is an inner-loop sketch, not a complete runnable matmul: the vectors and predicates must come from correctly packed panels, and the surrounding function must have a valid streaming/ZA interface.

svmopa_za32_m(0, pm0, pn0, a0, b0);
svmopa_za32_m(1, pm0, pn1, a0, b1);
svmopa_za32_m(2, pm1, pn0, a1, b0);
svmopa_za32_m(3, pm1, pn1, a1, b1);

A wrapper can use __arm_locally_streaming and __arm_new("za") where supported. The latter gives the function private ZA state; it does not initialize numeric accumulator contents, so zero the tiles explicitly. A portable caller should select this compiled kernel only after a runtime feature check.

Resources

• Arm’s SME2 matrix multiplication walkthrough gives a complete intrinsics example with packing and tails.
• Arm’s KleidiAI microkernel explanation shows production tiling choices.

Arrange each operand in the order the microkernel consumes it. For matrix multiplication, one outer-product step needs values from a column of A and a row of B. In ordinary row-major storage, B’s row is contiguous but A’s column is strided. Packing an A panel into column-contiguous vectors often removes expensive gathers from the reduction loop. Production kernels may pack both sides to improve reuse and match their exact instruction layout.

Follow the data through one tile

For a 32-bit accumulator tile, the conceptual inner loop is:

load consecutive A values for the tile's output rows at k
load consecutive B values for the tile's output columns at k
outer-product accumulate into ZA
repeat for k, then store valid output rows and columns

A 2×2 arrangement of ZA.S tiles can reuse two A vectors and two B vectors across four outer products. The point of packing is to make those loads simple and contiguous. A layout that looks tidy as a matrix may be poor for the actual instruction stream.

For example, an A panel can be stored as packed_A[k][row_lane] and a B panel as packed_B[k][column_lane]. This is an illustrative layout; use the selected kernel’s documented pack functions and strides for real integration. The packed format must define edge padding and the order of low-precision groups.

Account for cache and packing cost

Block M, N, and K so packed panels are reused in cache. If weights are constant, cache their packed representation rather than repacking at every inference. Keep the accumulator in ZA across the reduction; when you split K into blocks, preserve partial sums. Align panels as required by the load and pack routines, and allocate large temporary panels on the heap. An aligned allocation API may require the size to be a multiple of the alignment.

For BF16 and INT8 outer products, one instruction combines multiple narrow products into a wider accumulator. Their K grouping and lane order differ from FP32. Check the exact instruction and microkernel contract before packing; a generic “INT8 layout” is not enough. Quantized data also needs correct scales, zero points, and padding values, not just packed weights.

Use predicates or padded input panels for partial M and N tiles. Derive tile sizes from the streaming vector length, for example with svcntsw() for 32-bit lanes while streaming. Avoid calling ordinary non-streaming helpers inside a tight streaming loop when each call forces a mode transition.

Check the result

Benchmark a realistic matrix size with packing included. If speed falls short, profile strided loads, packed-panel cache misses, excess repacking, mode transitions, and output stores. A fast outer-product loop can still lose overall when its input format is expensive to create.

Resources

• Arm’s matrix preprocessing walkthrough shows why and how A is transposed for outer products.
• Arm’s KleidiAI packing explanation connects packed panels to a production microkernel.

Run both implementations on the same physical SME2 device with identical matrices and output semantics. Validate the results first, then compare repeated elapsed-time measurements. Report whether the measured interval includes packing, allocation, mode transitions, and stores. A scalar triple loop is a useful correctness reference, but an optimized Neon, SVE, or BLAS implementation is the meaningful performance baseline.

Set up a fair comparison

Use the same M, N, K, input values, output type, thread count, and core placement. Initialize C the same way for both variants; distinguish C = A×B from C += A×B. If one kernel accepts packed weights, time packing separately and include it in an end-to-end result when weights are repacked for each call. If weights stay constant across many calls, report the amortized setup case too.

Check output with both absolute and relative tolerances appropriate for the data type and reduction length. NaNs, infinities, and near-zero expected values need explicit handling; a fixed absolute tolerance alone can hide important errors or reject harmless accumulation differences. Exercise partial tiles, odd K values, and more than one streaming vector length if possible.

Collect timings

This C driver is complete apart from your sme2_matmul implementation, which must use the same overwrite contract and signature. It checks output, warms both paths, times them in alternating order, and reports a median. Choose error tolerances appropriate to your algorithm and K length.

#define _POSIX_C_SOURCE 200809L
#include <math.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

typedef void (*matmul_fn)(int, int, int, const float *,
                          const float *, float *);
extern void sme2_matmul(int, int, int, const float *,
                        const float *, float *);

static void scalar_matmul(int M, int K, int N,
                          const float *A, const float *B, float *C) {
    for (int m = 0; m < M; ++m)
        for (int n = 0; n < N; ++n) {
            float sum = 0.0f;
            for (int k = 0; k < K; ++k)
                sum += A[m*K + k] * B[k*N + n];
            C[m*N + n] = sum;
        }
}

static uint64_t now_ns(void) {
    struct timespec t;
    clock_gettime(CLOCK_MONOTONIC, &t);
    return (uint64_t)t.tv_sec * 1000000000ull + (uint64_t)t.tv_nsec;
}

static uint64_t time_one(matmul_fn fn, int M, int K, int N,
                         const float *A, const float *B, float *C) {
    uint64_t start = now_ns();
    fn(M, K, N, A, B, C);
    return now_ns() - start;
}

static int cmp_u64(const void *a, const void *b) {
    uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b;
    return (x > y) - (x < y);
}

int main(void) {
    enum { M = 257, K = 255, N = 259, RUNS = 21 };
    float *A = malloc((size_t)M*K*sizeof *A);
    float *B = malloc((size_t)K*N*sizeof *B);
    float *ref = malloc((size_t)M*N*sizeof *ref);
    float *got = malloc((size_t)M*N*sizeof *got);
    if (!A || !B || !ref || !got) return 1;
    for (size_t i = 0; i < (size_t)M*K; ++i)
        A[i] = (float)(i % 13) / 13.0f;
    for (size_t i = 0; i < (size_t)K*N; ++i)
        B[i] = (float)(i % 17) / 17.0f;

    scalar_matmul(M, K, N, A, B, ref);
    sme2_matmul(M, K, N, A, B, got);
    for (size_t i = 0; i < (size_t)M*N; ++i) {
        float tolerance = 1e-4f + 1e-4f * fabsf(ref[i]);
        if (!isfinite(got[i]) || fabsf(ref[i] - got[i]) > tolerance) {
            fprintf(stderr, "Mismatch at %zu: %g vs %g\n",
                    i, ref[i], got[i]);
            return 2;
        }
    }
    for (int i = 0; i < 10; ++i) {
        scalar_matmul(M, K, N, A, B, ref);
        sme2_matmul(M, K, N, A, B, got);
    }
    uint64_t base[RUNS], sme[RUNS];
    for (int i = 0; i < RUNS; ++i) {
        if (i & 1) {
            sme[i] = time_one(sme2_matmul, M, K, N, A, B, got);
            base[i] = time_one(scalar_matmul, M, K, N, A, B, ref);
        } else {
            base[i] = time_one(scalar_matmul, M, K, N, A, B, ref);
            sme[i] = time_one(sme2_matmul, M, K, N, A, B, got);
        }
    }
    qsort(base, RUNS, sizeof base[0], cmp_u64);
    qsort(sme, RUNS, sizeof sme[0], cmp_u64);
    printf("median: scalar=%llu ns SME2=%llu ns ratio=%.2fx\n",
           (unsigned long long)base[RUNS/2],
           (unsigned long long)sme[RUNS/2],
           (double)base[RUNS/2] / (double)sme[RUNS/2]);
    free(A); free(B); free(ref); free(got);
    return 0;
}

Compile the driver for a portable AArch64 target and link it with a separately compiled SME2 implementation. Replace the scalar comparator with an optimized Neon, SVE, or BLAS implementation before making a competitive speedup claim. The driver times only the function call, so state where packing and allocation happen in your implementation. For FP32 GEMM, 2×M×N×K divided by elapsed seconds estimates FLOP/s.

Sweep sizes below and above the expected cache and tile boundaries, including awkward dimensions. On Linux, taskset can pin a process and perf stat can collect cycles and cache events when available. Watch frequency and thermal behavior, especially on phones. An emulator is useful for correctness but cannot supply a physical-hardware speedup.

Interpret the result

If the SME2 inner loop wins but the full call loses, profile packing, memory traffic, allocation, mode switches, and edge handling. If a “vanilla” result is surprisingly slow, inspect its generated instructions and compare an established optimized library. Include a full-application or model measurement before claiming a user-visible gain.

Resources

• Arm’s SME2 matrix multiplication benchmark walkthrough offers a complete project to run and adapt.
• Arm’s matrix-multiplication comparison explains why the baseline implementation matters.

The recurring costs are entering and leaving streaming mode too often, saving live ZA state across incompatible calls, packing operands repeatedly, and using only a small part of each output tile. Diagnose them in a profile of the full operation before changing the kernel.

Too many mode transitions

A streaming function that calls ordinary non-streaming helpers in its inner loop can make the compiler insert repeated SMSTOP and SMSTART transitions. Mode changes also invalidate the Z and P register contents across the boundary, so the generated code may need extra saves and reloads. Move logging, allocation, and callbacks out of the hot loop. Give a cohesive SME kernel a streaming interface, then inspect disassembly around its calls. __arm_locally_streaming is useful for a non-streaming public caller when the compiler supports it; it does not make arbitrary calls inside the function free.

// Illustrative interfaces: the caller can remain non-streaming.
__arm_locally_streaming __arm_new("za")
void compute_tiles(const float *packed_a, const float *packed_b,
                   float *out, size_t tile_count) {
    // Initialize ZA before accumulating.
    // Process multiple tiles here; avoid logging inside the tile loop.
}

__arm_new("za") gives this function private ZA state, but does not zero its accumulator values. The body still needs explicit initialization. Where cooperating functions should share ZA, declare their interfaces accordingly rather than giving each helper a fresh ZA lifetime.

ZA save traffic

ZA storage grows with the square of streaming vector length: at 512-bit SVL it is 4 KiB, and at 2048 bits it is 64 KiB. The SME ABI’s lazy-save protocol can require memory traffic when a function with live ZA calls code that does not share that state, especially a callee creating private ZA. It does not mean every call necessarily copies the whole array. Keep the hot reduction within a clear ZA lifetime, use a sharing interface such as __arm_inout("za") for appropriate internal helpers, and inspect actual generated save/restore paths before blaming them.

Work that cannot amortize setup

Small matrices, partial tiles, and narrow reductions may spend more time on predicates, packing, ZA initialization, and dispatch than on arithmetic. Keep a measured Neon/SVE fallback for shapes where it wins. There is no universal M, N, or K threshold.

For row-major matrix multiplication, loading A columns directly can produce strided traffic. Pack panels in the microkernel’s consumption order, reuse packed invariant weights, and tune cache blocks on the target device. Preserve partial sums correctly when K is split into blocks. Measure the complete call as well as the inner loop; a high FMOPA rate can coexist with poor end-to-end latency.

Finally, do not assume ordinary SVE vector length equals SME streaming vector length. Derive streaming tile dimensions with a streaming query such as svcntsw(), and test non-multiple dimensions.

Resources

• ACLE SME function attributes defines the streaming and ZA contracts.
• Arm’s streaming-mode walkthrough explains the boundary behavior.

Compare profiles of the same workload with SME2 enabled and disabled, then rank the time that remains in the enabled run. Faster matrix code may expose packing, memory movement, elementwise operators, synchronization, or other work. There is no fixed “after SME2” time breakdown; measure your model and device.

Rebuild the hotspot map

Hold inputs, precision, threads, core placement, and warm-up procedure constant. Record repeated end-to-end times and per-operator or function timings. Use perf record -g for native Linux call paths, Arm Streamline for a timeline and available hardware counters, ETDump for ExecuTorch operators, or ONNX Runtime profiling for model nodes. Note which kernel actually executed; a backend build flag or packed-node label alone does not prove SME2 dispatch.

Create a before/after table with absolute time, not percentages alone:

A stage’s percentage can rise while its time stays flat because another stage got faster. Optimize the largest absolute remaining cost on the critical path.

Classify the new hotspot

If packing dominates, cache invariant packed weights or change the data layout at a producer/consumer boundary. If cache misses and memory stalls rise with size, test panel reuse and memory bandwidth; a high miss count alone is not proof of a bandwidth limit. If small shapes suffer, inspect mode transitions, ZA state handling, and dispatch thresholds. If multiple threads stop scaling, inspect scheduling, shared cache, bandwidth, and synchronization.

Sample hot symbols and inspect disassembly where needed to confirm the SME2 instruction path and look for unexpected transitions or spills. Counting SMSTART instructions in the entire binary is only a static clue; the hot execution path matters. Sweep representative shapes and thread counts to see whether the diagnosis holds beyond one test case.

Resources

• Arm’s ExecuTorch operator profiling guide shows an SME2 on/off workflow.
• Arm’s ONNX Runtime profiling example combines operator and hardware views.
• Arm Performix microarchitecture guide helps classify CPU bottlenecks.

Yes, when a supported framework already calls a KleidiAI microkernel for your operator and data format. KleidiAI supplies specialized packing and compute routines for Arm CPUs, including SME2 variants, so you can avoid writing those low-level kernels yourself. Integration and performance are still conditional: the framework must include the relevant path, its dispatcher must select it, and the cost of packing must be worthwhile.

If you use an existing framework

XNNPACK integrations can expose KleidiAI kernels to LiteRT and ExecuTorch; MLAS can expose them to ONNX Runtime; GGML can expose them to llama.cpp. Each framework enables a subset of the KleidiAI catalog. Check your version’s build configuration and operator coverage, then profile a representative run. A CPU feature check says a kernel could run, while sampled kernel symbols or instructions show what actually ran.

There is no universal selection order such as “SME2, then I8MM, then Dot Product.” The dispatcher considers operation, tensor type, shape, hardware features, and available implementations. A model may mix KleidiAI and other kernels.

If you build your own runtime

KleidiAI gives you small C APIs for the operation’s pack and matmul steps. You still own feature detection, kernel choice, allocation, threading, lifetime of packed weights, quantization metadata, and the fallback path. Follow the example for the exact kernel family you need, including its packing format. The library itself does not schedule a model or automatically choose the best kernel for an arbitrary operation.

check feature and tensor contract
→ choose matching pack and compute functions
→ pack weights and activations as required
→ run the microkernel
→ handle unsupported cases with another implementation

For constant weights, packing once and reusing the result can matter as much as the inner-kernel speed. For small operations or unsupported quantization formats, an existing backend may already be faster or simpler. Measure end-to-end latency and verify output before claiming a gain. KleidiAI targets selected AI operators on Arm CPUs; it does not accelerate every model node, training step, or GPU operation.

Resources

• KleidiAI repository and API overview explains the pack/compute interface and its integration responsibilities.
• Arm’s KleidiAI integration walkthrough shows how a framework selects a microkernel.

SME2 has a way to decode packed 2-bit indices, but it does not provide a general 2-bit multiply-accumulate instruction. LUTI2 uses each 2-bit index to select one of four values from a table, producing wider elements that a later instruction can multiply. Whether you can use this without writing a kernel depends on your framework, model format, and its current backend integration. Do not infer INT2 support from SME2 hardware alone.

What the instruction does

SME2’s ZT0 is a fixed 512-bit lookup-table register. A LUTI2 instruction reads packed 2-bit indices from a Z register and uses them to select table values. For a signed two-bit codebook such as −2, −1, 0, 1, the kernel can expand the weights to 8-bit values and feed those values to an INT8 dot-product or outer-product instruction. It must still apply the model’s scales, zero points, and packing rules correctly.

packed 2-bit indices → LUTI2 table lookup → wider values → matrix arithmetic

The table lookup is decompression, not direct INT2 arithmetic. Arm’s LUTI2 walkthrough shows ZT0 loading, packed indices, and the following matrix instruction in one worked example.

Find a no-code path for your model

Check the exact quantization format and backend, then verify runtime kernel selection. KleidiAI and its LiteRT/XNNPACK, ExecuTorch, ONNX Runtime, and llama.cpp integrations have format-specific kernels; an INT4 path does not imply a matching INT2 path. In particular, a GGUF format name containing “2” does not establish that its quantization scheme matches a LUTI2 kernel.

As of September 2026, AngelSlim’s Hy-MT1.5 documentation describes a 2-bit SEQ SME2 route. Its separate 1.25-bit STQ1_0 llama.cpp work is a different format and should not be presented as an interchangeable 2-bit backend. Follow the project’s own model and runtime instructions for the SEQ path, then profile the actual device to confirm the selected kernel.

If your framework has no kernel for your operator and quantization contract, you will need a custom kernel or a different supported format. The Arm LUTI Learning Path provides a functional example for building and checking such a kernel; emulation can check correctness, while speed requires physical SME2 hardware.

Stop inside the SME kernel while ZA is active, inspect SVCR first, then read the ZA register or a typed tile view. On Linux, an SME-aware GDB or LLDB obtains this state through the kernel’s SME ptrace register sets. When SVCR.ZA is clear, the debugger may display zeroes for ZA; those zeroes are not the previous accumulator values.

In GDB

Set a breakpoint after the kernel has initialized and updated a tile:

(gdb) break my_sme_kernel
(gdb) run
(gdb) info registers svcr svg
(gdb) print $za
(gdb) print $za0s

SVCR bit 0 reports streaming mode and bit 1 reports active ZA. SVG reports the streaming vector length in 64-bit granules, so SVG=8 means a 64-byte SVL. GDB also provides typed tile and slice pseudo-registers when the target advertises SME. For a 32-bit tile, inspect $za0s; use info all-registers to discover the slice names exposed by your GDB version. The full ZA array can be large: at a 64-byte SVL it holds 4096 bytes.

Stop at several points in the K loop to check whether accumulation changes the expected tile. If a tile unexpectedly becomes zero, inspect the transition and call immediately before that point. Avoid using GDB’s injected call command within live SME state; GDB documents limits in its SME-aware call handling.

In LLDB

(lldb) register read svcr svg
(lldb) register read za
(lldb) register read zt0

zt0 applies to SME2. LLDB presents ZA as a byte vector by default and supports formatting options, but does not infer the matrix element type. Use the program’s tile mapping to interpret it. svcr and svg are read-only in LLDB.

If ZA is missing, check the CPU and OS exposure, debugger version, local versus remote stub, and whether you stopped after ZA was enabled. GDB documents that a remote gdbserver may show the wrong ZA layout if the target changes SVL during the session. For automated numeric comparison, store a logical tile to an ordinary memory buffer within the kernel and compare it with a scalar result.

Resources

• GDB’s AArch64 SME register guide defines ZA tile views and debugger limitations.
• LLDB’s AArch64 Linux guide shows its SME register presentation.

SME adds architectural state for a debugger to expose: streaming-mode status, the streaming vector length, ZA, and, with SME2, ZT0. Ordinary breakpoints and memory watchpoints still work. On Linux, GDB and LLDB get SME register state through kernel ptrace interfaces; you do not need a special “SME breakpoint” facility.

What to inspect

On Linux, the kernel exposes streaming SVE state through NT_ARM_SSVE, ZA through NT_ARM_ZA, and ZT0 through NT_ARM_ZT when available. Whether a debugger can display them depends on the kernel, hardware, and debugger or remote stub. Do not infer support from a fixed minimum version without testing that combination.

GDB and LLDB workflow

(gdb) info registers svcr svg
(gdb) print $za0s
(gdb) print $zt0
(gdb) x/8i $pc

The tile view $za0s is useful for 32-bit accumulator elements. If SVCR.ZA is clear, GDB displays inactive ZA as zero; it does not reveal a saved earlier value. GDB supports ZA slice pseudo-registers and documents their naming and limits.

(lldb) register read svcr svg za zt0
(lldb) disassemble --pc

LLDB shows ZA as bytes unless you choose a format. It treats svcr and svg as read-only. When debugging a memory error, set a watchpoint on the input or output buffer, then inspect the faulting load/store and its predicate. A hardware watchpoint does not directly watch an element inside ZA.

For bare-metal work, an SME-aware debug probe and monitor can expose the same architectural state, but exact commands depend on the probe and target. For remote GDB, check whether the stub advertises the optional SME target feature. Avoid debugger-injected function calls while ZA is live unless the debugger documents correct ABI handling.

Resources

• Linux SME ptrace documentation defines the register sets.
• GDB AArch64 SME guide describes tile views and limitations.
• LLDB AArch64 Linux guide describes register display and state changes.

Store the logical tile slices you want to compare, rather than dumping all raw ZA storage. A horizontal-store ACLE intrinsic writes one row of a typed tile to memory. Loop over its rows in streaming mode, with ZA active, then compare that row-major buffer with the corresponding region of your reference output.

Store one 32-bit tile

This helper illustrates the interface and buffer contract. It assumes that the caller has active ZA and that the buffer has room for side * side 32-bit elements, where side is the number of 32-bit lanes in the streaming vector.

#include <arm_sme.h>
#include <arm_sve.h>
#include <stddef.h>
#include <stdint.h>

void dump_za0s(uint32_t *dst) __arm_streaming __arm_in("za") {
    size_t side = svcntsw();
    svbool_t all = svptrue_b32();
    for (size_t row = 0; row < side; ++row)
        svst1_hor_za32(0, row, all, dst + row * side);
}

The buffer holds one ZA0.S tile, not the entire ZA array. For a 512-bit SVL, side=16, so this example writes 16×16 32-bit values (1024 bytes). Four 32-bit tile views share the 4096-byte ZA storage. Adapt the tile number and element width to the accumulator you actually use.

Allocate the buffer before entering a streaming hot loop when possible. A private-ZA wrapper can perform the calculation, dump the tile, and return to a non-streaming caller. If it uses __arm_new("za"), initialize ZA explicitly before accumulation; that attribute does not zero numeric contents.

Compare with the reference

Map each stored tile row and column to the output coordinates chosen by your microkernel. Compare only active M and N lanes at edge tiles. For floating-point results, report the first mismatch, maximum absolute and relative error, and the inputs and K position that produced it. For integer accumulation, compare exact values unless the algorithm specifies saturation or rounding.

A raw ZA dump is useful for a debugger or context-state investigation, but typed tiles are interleaved within raw ZA storage. Its bytes cannot be treated as one row-major FP32 matrix without decoding the tile mapping. If you need an entire raw snapshot, use the architecture’s ZA array-vector store or the corresponding ACLE operation and size the buffer as streaming-vector-length-in-bytes squared.

Resources

• Arm’s ZA storage guide explains tile and slice mapping.
• ACLE SME intrinsics defines the store operations and function attributes.

Use two references. First write a plain scalar implementation for the mathematical result. Then write a scalar tiled version with the same M/N tiles and K-block order as the SME kernel. Compare their final outputs, and add checkpoints in the SME kernel only where you need to locate a mismatch. The tiled reference should mirror the logical work, not ZA’s physical register interleaving.

Start with a scalar result

For row-major FP32 matrices and overwrite semantics:

for (size_t m = 0; m < M; ++m)
    for (size_t n = 0; n < N; ++n) {
        float sum = 0.0f;
        for (size_t k = 0; k < K; ++k)
            sum += A[m*K + k] * B[k*N + n];
        C_ref[m*N + n] = sum;
    }

If your real operation adds to an existing C, includes bias, or uses quantization, put those semantics in the reference too.

Mirror the kernel’s tiling

The tiled reference below uses scalar arithmetic but makes the M/N and K boundaries visible. It illustrates one output element at a time; a real harness can keep a small local tile array to compare all elements after each K block.

for (size_t m0 = 0; m0 < M; m0 += MR)
    for (size_t n0 = 0; n0 < N; n0 += NR)
        for (size_t m = m0; m < M && m < m0 + MR; ++m)
            for (size_t n = n0; n < N && n < n0 + NR; ++n) {
                float sum = 0.0f;
                for (size_t k0 = 0; k0 < K; k0 += KC)
                    for (size_t k = k0; k < K && k < k0 + KC; ++k)
                        sum += A[m*K + k] * B[k*N + n];
                C_tiled[m*N + n] = sum;
            }

Compare C_tiled with C_ref before examining SME. Then test the packed A and B panels separately by decoding a few indices back to source coordinates. In the SME kernel, optionally store the accumulator tile after each K block to a debug buffer. Map each tile lane to its global output coordinate and compare it with the tiled reference at the same K checkpoint. Keep the normal fast path free of debug stores.

Build a useful test matrix

Include dimensions of 1, a full tile, one past a tile, and values just before and after K-block boundaries. Try more than one supported streaming vector length. Check zero padding, inactive predicates, and C overwrite versus accumulation. For FP32, use a tolerance based on the result magnitude and K length; report absolute and relative error, not a single unexplained threshold. For INT8/INT4 kernels, also check scales, zero points, unpack order, and overflow behavior.

A mismatch in both scalar references points to the reference or semantics. A mismatch only after packing points to layout. A mismatch at the first K checkpoint points to initialization, operand loading, or predicate mapping; one appearing later suggests K-block carry or accumulation order. This structure narrows the bug without emulating ZA’s raw byte layout.

Resources

• Arm’s SME2 matrix multiplication example shows the tiled implementation to validate.
• Arm’s ZA storage guide explains how logical tile slices map to ZA.

First distinguish leaving streaming mode from disabling ZA. SMSTOP SM clears streaming mode but can leave ZA active; SMSTOP ZA disables ZA, and later re-enabling ZA initializes its state rather than restoring your accumulator. A broad SMSTOP affects both. Check the actual instruction and SVCR bits at the point where the value changes.

Trace the failing boundary

Stop immediately before and after the suspicious call or mode transition. In GDB, info registers svcr shows bit 0 (SM) and bit 1 (ZA); in LLDB use register read svcr. Step through SMSTART and SMSTOP in disassembly. If ZA stays active but values change, inspect a callee that shares ZA, incorrect tile indices, or handwritten assembly. If ZA becomes inactive, find who disabled it and whether the calling contract allowed that.

Write down each function’s ACLE interface:

Do not insert manual SMSTART or SMSTOP into a function whose streaming state the compiler manages unless the assembly contract is fully accounted for. For a self-contained kernel with __arm_new("za"), explicitly initialize its accumulators before use. If hand-written assembly crosses a function boundary with live ZA, follow the SME ABI’s lazy-save protocol rather than assuming a normal AArch64 call preserves the matrix register.

Make the failure reproducible

In a debug build, store a small logical tile or checksum to ordinary memory before and after the boundary, and log only after leaving the hot streaming region. Avoid calling printf inside the active kernel just to inspect state: the debugger or compiler may need mode transitions for the call. Test both immediate and repeated calls, including signals or context switches when the issue appears only on a device. A supported OS should preserve architectural state across ordinary scheduling; do not treat every context switch as proof of an OS bug.

If a remote debugger changes or misreports SVL, repeat locally or check the kernel’s ptrace state. GDB documents limitations when gdbserver and GDB disagree after an SVL change.

Resources

• ACLE SME function attributes defines the interface contracts.
• Arm’s streaming-mode guide illustrates state transitions.
• GDB’s SME debugging notes describe remote and function-call limitations.

Separate three questions: did the intended SME2 path run, did it produce the right values, and did it improve the full workload? A build flag or an SME2 instruction somewhere in the binary answers none of these by itself.

Confirm the executed path

Check that the OS exposes SME2 to the process. On Linux, use getauxval(AT_HWCAP2) with HWCAP2_SME2; on macOS, use the documented feature sysctl. Record the device, OS, compiler, framework version, and streaming vector length. Inspect runtime logs and sample the hot symbols to see the selected kernel. Disassembly of that kernel can show SME2 instructions, but disassembly of an unused function is not execution proof.

Keep feature-targeted code isolated so a portable binary does not execute SME2 before the runtime check. If the application falls back, inspect the operator’s data type, quantization scheme, shape, backend configuration, and feature detection.

Localize a correctness difference

Compare a scalar reference and the device result for the same inputs. Start with a full tile, then non-multiple M/N/K dimensions and different vector lengths. If output diverges only at edges, inspect predicates and padded panels. If it diverges after a K block, inspect accumulator initialization and partial-sum carry. Decode a few packed elements back to source coordinates. For floating point, report absolute and relative error, NaNs, and the accumulation order; a bitwise mismatch can be legitimate while a large numeric error is not.

Audit every streaming and ZA function interface. __arm_new("za") supplies private ZA but does not zero its numeric contents. A mismatch between __arm_inout("za") declarations across translation units or a manual mode transition can corrupt state. Inspect SVCR and generated SMSTART/SMSTOP around the first failing call. Debuggers and emulators can help localize this even when they cannot predict speed.

Explain a performance difference

Run repeated native-device measurements with the same input, threads, affinity, warm-up, and thermal conditions as the baseline. Report median and variation. Time packing, the matrix kernel, other operators, and the full inference or request separately. If only the inner loop improves, packing or another stage may dominate. If the expected kernel ran but is slow, inspect tile utilization, memory stalls, mode transitions, and ZA save traffic. Compare with a genuinely optimized baseline, not solely a scalar reference.

Do not assume the simulator’s time, a different device’s speedup, or a fixed “SME2 gain” applies to this hardware. A short table of observed path, correctness, kernel time, and end-to-end time makes the difference actionable.

Resources

• Arm’s SME2 debugging walkthrough covers assembly and correctness checks.
• Linux SME userspace documentation covers feature exposure and state.
• Arm’s ONNX Runtime profiling example shows end-to-end and operator comparisons.

Use the operating system’s process-visible feature interface. On Linux and Android, test AT_HWCAP2 for HWCAP2_SME or HWCAP2_SME2; on macOS, query Apple’s hw.optional.arm.FEAT_SME and FEAT_SME2 selectors. A compiler macro says what the binary was built to use, not what the current device can run.

Linux and Android

#include <stdbool.h>
#include <stdio.h>
#include <sys/auxv.h>
#include <asm/hwcap.h>

int main(void) {
    unsigned long caps = getauxval(AT_HWCAP2);
    bool sme  = (caps & HWCAP2_SME) != 0;
    bool sme2 = (caps & HWCAP2_SME2) != 0;
    printf("SME=%d SME2=%d\n", sme, sme2);
    return 0;
}

Build this detection source for a portable AArch64 target. Current system headers define these bits; if you must support older headers, verify any local fallback values against the kernel UAPI before adding them. As a shell smoke test, /proc/cpuinfo commonly lists sme and sme2 in the Features line. Use HWCAP in application code because it represents what the kernel makes usable to the process, including its SME state-management interfaces.

If the code needs an optional SME subfeature, check its own HWCAP bit too. SME2 implies SME, but it does not imply every optional arithmetic format or ordinary non-streaming SVE2. Keep SME functions in a separate translation unit or otherwise isolate their target attributes; call them only after dispatch chooses a supported path.

macOS

sysctl hw.optional.arm.FEAT_SME
sysctl hw.optional.arm.FEAT_SME2

In C or C++, query the same selectors with sysctlbyname, checking the return value and returned size. A result of 1 indicates that macOS exposes the feature. Do not use Linux HWCAP interfaces on macOS.

For bare-metal or privileged software, architectural ID registers can describe the processor’s implemented features. They do not replace the OS-visible test in a user-space application. Do not probe by executing an SME instruction and waiting for SIGILL; Linux does not guarantee reliable feature discovery that way.

Resources

• Linux arm64 ELF HWCAP documentation defines process-visible feature tests.
• Linux SME userspace documentation explains the SME interfaces attached to those flags.

Collect evidence at three levels: the device exposes SME, the runtime chooses an SME implementation, and that implementation executes SME instructions during your representative workload. An SME instruction present in a binary proves only that it was compiled.

Check availability and dispatch

On Linux or Android, check getauxval(AT_HWCAP2) for SME/SME2; on macOS, check the corresponding feature sysctl selectors. Then inspect the framework’s operator profile or backend diagnostics. For a model, note the exact operator, data types, quantization format, shape, and selected kernel. A packed XNNPACK node label or llama.cpp CPU_KLEIDIAI model buffer can show part of the path but does not by itself prove SME2.

If the operator falls back, inspect its eligibility conditions and the build configuration. A device with SME2 can still run Neon for an unsupported shape or format. Modern libraries may use different kernel families for prefill and decode, or for full and partial tiles.

Prove the instruction ran

Sample the hot call path on the device, for example:

perf record -g -- ./your_application
perf report
perf annotate

Use simpleperf on Android when appropriate. A sample on a named SME kernel is stronger evidence than a build flag; disassemble that function and look for FMOPA, SMOPA, ZA accesses, or other SME instructions. A breakpoint at the kernel entry, followed by stepping into its arithmetic, is useful when sampling misses a short call. SMSTART alone shows a mode change, not that matrix arithmetic ran. Hardware event names and their availability depend on the core, so discover them with perf list rather than copying a generic SME event.

For a multiversioned library, inspect both the selected SME path and the fallback path. Record how many calls or what fraction of time uses each. A trace build can identify a kernel, but turn verbose logging off for timing because I/O distorts the result.

Resources

• Arm’s ONNX Runtime profiling walkthrough shows operator and hardware evidence.
• GDB’s AArch64 SME guide helps when stepping through a selected kernel.

Use a Mac whose CPU and macOS release expose SME or SME2, plus a compiler and SDK that accept the SME intrinsics you need. Apple M4 and M5 family Macs are documented SME2 development targets; do not infer support merely from “Apple silicon” or a model name. Check the feature selectors on the exact machine before building an SME-only binary.

Check the machine

sysctl hw.optional.arm.FEAT_SME
sysctl hw.optional.arm.FEAT_SME2

A value of 1 means the feature is exposed to applications. An unknown selector or zero means you should not dispatch to that path. M1–M3 Macs do not provide architectural SME. A MacBook Neo uses an A-series processor, so test its selectors rather than treating it as an M-series configuration.

You can make the same check in an application with sysctlbyname:

#include <stdbool.h>
#include <sys/sysctl.h>

static bool has_feature(const char *name) {
    int value = 0;
    size_t size = sizeof(value);
    return sysctlbyname(name, &value, &size, 0, 0) == 0
        && size == sizeof(value) && value == 1;
}

/* has_feature("hw.optional.arm.FEAT_SME2") */

A compiler target macro such as __ARM_FEATURE_SME2 describes the source being compiled; it is not a runtime feature check.

Build and test

Use a current Xcode command-line toolchain or another macOS AArch64 compiler with SME ACLE support. Compile a small source containing the required intrinsic and attribute first; toolchain support varies by version and intrinsic family. Keep this source in a feature-targeted object or function, while the dispatching code remains compatible with older Macs. If a build flag such as -march=armv9-a+sme2 is accepted by the chosen compiler, apply it to the SME2 source, not indiscriminately to the whole app.

Run a correctness test with odd matrix dimensions and inspect the selected function’s instructions or profile samples. For performance, measure the same work through an optimized baseline with the same thread count and thermal conditions. On macOS use sysctlbyname for feature detection; Linux getauxval(AT_HWCAP2) is not a macOS API.

Resources

• Arm’s macOS SME2 setup walkthrough identifies a working toolchain route.
• Arm’s SME2 environment check shows a small capability test.

Trace one node through framework delegation, backend operator creation, kernel configuration, and executed function. A static support table can tell you what is eligible, but shape, data type, build options, hardware features, and runtime heuristics determine what actually runs.

Follow one operator

1. Capture the model operator name, input/output shapes, layout, and quantization parameters in a framework profile.
2. Check whether the operator was delegated to XNNPACK or stayed in the framework’s own CPU path. ONNX Runtime’s MLAS and llama.cpp’s GGML paths are separate from XNNPACK; do not draw them as XNNPACK stages.
3. If delegated, inspect the backend’s operator type and kernel-selection trace for the exact build. One model node may become several backend operations, and a fused backend operation may cover several model nodes.
4. Sample native call stacks or set a debugger breakpoint at the selected xnn_* or kai_* symbol. Resolve stripped addresses against the matching unstripped binary or symbol file.
5. Disassemble the hot leaf function to distinguish SME2, SME, Neon, and scalar instructions. A P label in an XNNPACK operator name indicates a packed path, not necessarily an SME2 instruction.

For example, a Linear layer may use a matrix-multiply kernel for prompt prefill and a vector-matrix path for single-token decode. Record both phases rather than assigning the layer one permanent microkernel.

If the expected kernel is absent, check the supported type and quantization contract, the actual runtime shape, the build’s included microkernels, the device feature result, and the selected fallback. A model can mix SME2 and Neon paths.

Resources

• Arm’s LiteRT/XNNPACK tracing example explains packed node labels.
• Arm’s ExecuTorch profiling workflow connects model operators to timings.
• Arm’s llama.cpp KleidiAI trace shows the separate GGML route.

Build a separate diagnostic binary with XNNPACK logging enabled, run one representative inference, and inspect the logged microkernel name. Logging is compiled into XNNPACK; setting an environment variable cannot restore messages removed from a release build. Then confirm that the named function executes with a breakpoint or profiler, because registration logs and runtime execution are different evidence.

Enable logging in the build you actually ship through

In XNNPACK’s CMake build, the logging target defines XNN_LOG_LEVEL=5 for Debug and 0 for other configurations. A standalone debug build can start with:

cmake -S XNNPACK -B build-xnn-trace -DCMAKE_BUILD_TYPE=Debug
cmake --build build-xnn-trace -j

For ExecuTorch or LiteRT, XNNPACK is embedded in a larger build. Follow that project’s profiling preset or propagate its documented XNNPACK logging option through the dependency build; rebuilding only the top-level executable may leave the bundled XNNPACK library unchanged. Confirm the final binary contains the logging configuration. XNNPACK build options and log routing differ among CMake, Bazel, Android, and Apple builds, so do not assume one -D flag works everywhere.

Capture output from a short run, including Android logcat when that build uses platform logging. Look for the full microkernel symbol, for example a name containing sme2, neon, or scalar. Check sme2 before neon: an SME2 symbol may contain both strings. A kai_ prefix indicates a KleidiAI function family, but the exact architecture is in the full variant name.

Confirm the chosen path

A message that a kernel was registered or configured may precede a different operator path at execution time. Set a breakpoint on the logged symbol or sample the hot native call stack. Inspect its instructions for SME2 outer products or other SME2 operations. If the expected name never appears, check the device’s OS-exposed SME2 bit, the build’s compiled kernel set, the operator/data-format contract, and shape-specific fallback. Use a quiet optimized build for actual latency comparisons; debug logging and Debug optimization settings alter performance.

Resources

• XNNPACK CMake configuration shows its logging definition.
• Arm’s ExecuTorch SME2 profiling setup provides a trace-build workflow.

Yes for building, deploying, debugging, and general CPU profiling of an Android app that contains SME code. Android Studio does not have a dedicated view that proves SME instructions executed or displays ZA as a matrix. Use a physical SME-capable device and pair the IDE’s workflow with runtime feature checks and a native profiler or debugger.

Practical workflow

1. Install an Android NDK whose Clang supports the SME intrinsics or assembly you use. Build an arm64-v8a native library with SME code isolated behind runtime dispatch.
2. On the device, check sme or sme2 in /proc/cpuinfo as a smoke test. Production native code should read the Linux/Android AT_HWCAP2 flags and retain a compatible fallback.
3. Use Android Studio’s CPU Profiler to find expensive app and native functions. For instruction-level evidence, use simpleperf, a debugger breakpoint, or an SME-aware profiler, then inspect the selected native kernel’s instructions.
4. Compare equivalent runs with the accelerated path enabled and disabled, including full app latency. A function named “SME” in a stack trace or an arm64-v8a build does not establish that SME arithmetic ran.

A standard Android virtual device should not be used for hardware speed measurements. LLDB’s ability to show SME registers depends on the debugger, OS image, and target support; test it on the actual device.

Resources

• Arm’s Android SME2 environment guide covers NDK setup.
• Arm’s Android Studio profiling guide covers the IDE’s CPU view.

Keep the model in its existing format and first try an SME2-capable CPU runtime. ONNX Runtime can reach KleidiAI through MLAS; LiteRT can reach it through XNNPACK. Then profile the model, verify which operators actually use SME2 kernels, and change precision or graph structure only where the measured result justifies it. SME2 is a runtime implementation choice, not an ONNX or .tflite export format.

Establish a device baseline

Confirm the target OS exposes SME2 and run the unmodified model on the physical device. Record accuracy, cold-start and steady-state latency, memory use, thread count, and representative input shapes. Keep the same measurements for every later variant. A supported CPU does not imply every model operator has an SME2 kernel.

Use the matching runtime path

For an Android ONNX Runtime build, Arm’s KleidiAI build walkthrough documents the versioned build command and onnxruntime_USE_KLEIDIAI=ON option. Follow its Android SDK and NDK path parameters rather than treating the SDK path as the NDK path. For LiteRT, Arm’s benchmark-tool build step documents the matching Bazel options. These flags compile candidate kernels; they do not force each operator to use SME2.

In a LiteRT app, enable the XNNPACK delegate and check the return from ModifyGraphWithDelegate. For ONNX Runtime, use the CPU execution provider and an optimized session graph. If deployment uses a packaged runtime, inspect that binary’s build options instead of assuming it matches a source tutorial.

Optimize where profiling points

List the expensive MatMul, FullyConnected, Conv, and related operators with actual shapes, types, and quantization parameters. Check the framework’s support table for that full tuple, then inspect operator timing and sampled native symbols. If a hot operator misses the SME2 path, find the eligibility condition before rewriting the model.

Try FP32 or FP16 as a correctness baseline, then compare INT8 or INT4 only where the runtime has a matching kernel and the model meets its accuracy target. Quantization may introduce conversion and packing work or change operator coverage. Simplify redundant transposes and fuse operations only when the runtime supports the fused form and the model still produces acceptable outputs. Keep a portable CPU fallback for devices without SME2.

The result to report is the model’s accuracy and end-to-end latency on the device, plus the operators whose implementation changed. A faster isolated microkernel is useful evidence but not the final deployment result.

Resources

• Arm’s ONNX Runtime KleidiAI build and profile path gives a concrete Android route.
• Arm’s LiteRT operator configurations helps match model formats to available kernels.

Start with the model’s accuracy and memory target, then check actual backend kernel coverage for the chosen format. INT4 weights can cut weight storage for large models; INT8 can work well for vision and other quantized models; FP16 is a useful lower-conversion path when quantization hurts accuracy or coverage. None is inherently the fastest SME2 choice for every operator, shape, or device.

SME2’s LUTI4 can expand packed four-bit indices to wider values that an integer matrix instruction consumes. That is one possible kernel design, not a guarantee that every INT4 model format uses LUTI4 or even SME2. An INT4 GGUF Q4_0 path, an ONNX blockwise quantization scheme, and a LiteRT per-channel format have different contracts. Do not treat the bit width as the whole compatibility test.

For INT8, calibration data should represent deployment inputs. Per-channel weight scales can protect accuracy, but the runtime must support that scheme for the operator. Dynamic activation quantization avoids a fixed calibration range but adds per-inference work. FP16 reduces weight storage relative to FP32; its accuracy and accumulator behavior depend on the selected kernel. Test sensitive layers separately and consider a mixed-precision model if a few quantized layers cause the error.

Measure each candidate on the same device and inputs. Record model size, peak memory, quality metric, cold and warm latency, and which microkernels executed. Include packing and conversion in end-to-end time. Use a non-quantized baseline and do not extrapolate speedup from another model or a single matrix kernel.

Resources

• Arm’s KleidiAI quantization explanation explains kernel format choices.
• Arm’s LiteRT SME2 supported configurations provides one concrete backend coverage table.

First profile the current model to find expensive operators. Then check whether those operators have an SME or SME2 kernel for their exact types, shapes, and quantization scheme. Finally compare the same model with that backend path enabled and disabled on a physical device. Before the conversion, no profiler can directly measure “time SME would save.”

Get an operator breakdown

Use the profiler built into the runtime: ExecuTorch ETDump, ONNX Runtime JSON profiling, LiteRT’s benchmark/operator profile, or the equivalent for your framework. Record per-node latency, input shapes, data types, and thread count for representative inferences. Group related operations such as GEMM, convolution, packing, layout conversion, normalization, and data movement, but keep their absolute time visible. Percentages alone can mislead after one stage gets faster.

A matrix-heavy operator is a candidate, but its backend may lack a kernel for the model’s format. Check the exact framework version’s support table and whether delegation or fusion changes the operator shape. Small matrix-vector and edge-tile cases can favor a different implementation.

Compare an SME-enabled path

On Linux/Android, confirm the OS exposes SME2 through AT_HWCAP2; on macOS use feature sysctl. Build equivalent enabled and disabled runtime variants where the project supports that comparison. Run identical inputs, model, precision, threads, affinity, and warm-up. Profile operators and end-to-end latency in both runs. Sample native symbols or instructions to confirm the SME kernel actually executed; a packed-node label alone is insufficient.

If the matrix node improves but inference does not, inspect packing, tensor conversions, memory movement, and other operators. If the expected node does not improve, inspect eligibility, selected fallback, shape, and thermal or frequency effects. Report median latency and variation over repeated runs rather than one speedup number.

Resources

• Arm’s ExecuTorch profiling workflow shows per-operator comparison.
• Arm’s ONNX Runtime profiling example shows how to pair operator timing with hardware evidence.

Yes, for models and operators that ExecuTorch’s XNNPACK backend can lower and that its KleidiAI integration covers. The route is PyTorch export, optional quantization, XNNPACK partitioning, a .pte program, and an ExecuTorch runner built for the target. XNNPACK selects a suitable microkernel at runtime; exporting a .pte file alone does not prove SME2 acceleration.

Prepare and export

Start with an evaluated PyTorch model and representative example inputs. Check torch.export support for its operators and dynamic shapes, then compare exported output with eager PyTorch. For a minimal supported model, the structure is:

import torch
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.exir import to_edge_transform_and_lower

model = model.eval()                 # your prepared model
example_inputs = (sample_input,)    # representative shape and dtype
exported = torch.export.export(model, example_inputs)
edge = to_edge_transform_and_lower(
    exported, partitioner=[XnnpackPartitioner()]
)
program = edge.to_executorch()
with open("model.pte", "wb") as out:
    out.write(program.buffer)

This is an export sketch: define model and sample_input, and use APIs from a matching ExecuTorch revision. Real models may need operator replacements or shape constraints before export succeeds. If quantization is needed, follow ExecuTorch’s PT2E or current quantization recipe for the model family, calibrate with representative data, and recheck accuracy. Choose an INT4 or INT8 format only after checking the XNNPACK/KleidiAI kernel contract.

Build and deploy the matching runner

Build ExecuTorch with XNNPACK and its KleidiAI dependency for the target AArch64 platform. Pin the exporter and runner to the same ExecuTorch revision. Arm’s ExecuTorch profiling setup supplies versioned Android runner steps and SME2-on/off builds; use its current commands rather than assuming a CMake flag from a different release still works.

For that documented profiling kit, after cloning the kit repository and selecting a compatible ExecuTorch revision, the setup and build steps are:

bash model_profiling/scripts/setup_repo.sh
bash model_profiling/scripts/build_runners.sh

The script creates Android SME2-on and SME2-off runners under executorch/cmake-out/ when ANDROID_NDK is set, and analogous macOS runners for local development. It enables the XNNPACK/KleidiAI dependency in the on variant and provides a comparable off variant. Build the runners once per ExecuTorch revision, then use the same pair for each exported model. This gives a concrete starting point without embedding a long build command whose flags may drift between releases.

Push the .pte model and runner to the device, or package them in an app. Confirm that the OS exposes SME2. Run a correctness comparison against PyTorch or an unaccelerated backend with identical inputs. Enable ETDump/operator tracing in a diagnostic build to find delegated operators, then sample the native kernel path. An ETDump timing entry or XNNPACK P label does not alone identify an SME2 instruction.

Decide whether it helped

Compare SME2-on and SME2-off builds with the same model, quantization, inputs, thread count, and affinity. Measure end-to-end latency and per-operator time, including packing and data movement. Use a quiet timing build; verbose kernel logging changes latency. Unsupported operations may remain in portable ExecuTorch or another XNNPACK path, so report coverage rather than claiming the whole model uses SME2.

Resources

• ExecuTorch XNNPACK backend documentation describes partitioning and supported execution.
• Arm’s SME2 ExecuTorch profiling workflow gives a complete deploy-and-compare route.

Favor architectures whose expensive work becomes regular matrix operations with reusable weights, then confirm the target runtime has SME2 kernels for those operators and formats. Dense projections, sufficiently large matrix multiplies, and some convolutions are common opportunities. SME2 hardware alone does not make every Linear, attention, or Conv layer fast.

Layer and shape choices

For matrix multiplication, M is the number of output rows, N the output width, and K the reduction dimension. Larger regular M and N often use ZA tiles better, but tile dimensions depend on streaming vector length and the chosen microkernel. Avoid hard-coded rules such as “all dimensions must be multiples of 64 or 128.” Kernels can handle tails; changing model dimensions may cost accuracy or increase total work. Measure the actual shape distribution, including batch size and single-token decode. Decode can use specialized GEMV kernels even though M=1 does not fill a GEMM tile.

Prefer stable input layouts and constant weights that a runtime can pack once. Remove repeated transposes or conversions when profiling shows they dominate. For CNNs, a channel-last layout can fit some XNNPACK paths, but evaluate the whole graph before changing layout. Depthwise convolutions, softmax, normalization, and gathers may remain on different kernels; their time can limit end-to-end benefit.

Choose data formats with the backend

INT4 weight paths can reduce bandwidth for large models, but the block size and scales must match a kernel. INT8 may be easier to deploy for a calibrated vision model; FP16 may preserve accuracy with less conversion. SME2’s LUTI4 can expand four-bit indices for some kernels, but it does not mean every INT4 scheme is accelerated. Keep accuracy and memory measurements alongside latency.

Start with the existing architecture. Profile it, identify a costly unsupported or poorly tiled operator, and change one design choice at a time. Report accuracy, model size, operator coverage, and end-to-end time on the target device. This avoids optimizing tile alignment at the expense of model quality or total latency.

Resources

• Arm’s LiteRT SME2 operator coverage shows concrete supported formats.
• Arm’s special matrix-shape guidance explains why one tile size does not fit all workloads.