{ "schema_version": 1, "slug": "sme", "title": "SME Insights", "short_title": "SME", "description": "Answers about the Scalable Matrix Extension (SME), SME2, and the software ecosystem that supports them.", "template": "qa", "style": "sme", "fundamentals": { "title": "SME fundamentals", "paragraphs": [ "Arm Scalable Matrix Extension (SME) adds scalable matrix-processing capabilities to the Arm architecture. It introduces streaming vector execution, outer-product instructions, and ZA tile storage to accelerate matrix-heavy workloads such as machine learning, computer vision, signal processing, and scientific computing.", "This page brings together practical questions and detailed answers about SME, SME2, and the supporting software ecosystem. Use it to understand the fundamentals, find implementation guidance, and explore how SME is supported across tools, libraries, frameworks, and Arm-based devices." ] }, "entries": [ { "id": "s01-q00", "slug": "introductory-overview-scalable", "question": "Please provide an introductory overview of Arm Scalable Matrix Extension (SME).", "summary": "Arm Scalable Matrix Extension (SME) is an optional AArch64 feature for matrix-oriented computation. It adds a two-dimensional register array called ZA, instructions that accumulate vector outer products into ZA, and a Streaming SVE mode with its own vector length. SME2 extends this model with multi-vector operations, multi-vector predication, and lookup-table instructions. Both features are optional, so an Arm architecture version alone does not tell you whether a device supports them.", "answer": "Arm Scalable Matrix Extension (SME) is an optional AArch64 feature for matrix-oriented computation. It adds a two-dimensional register array called **ZA**, instructions that accumulate vector outer products into ZA, and a **Streaming SVE mode** with its own vector length. SME2 extends this model with multi-vector operations, multi-vector predication, and lookup-table instructions. Both features are optional, so an Arm architecture version alone does not tell you whether a device supports them.\n\nThe central idea is to keep an output tile in ZA while a kernel processes the input data needed to build it. This can reduce the instructions and data movement needed for matrix multiplication. Beyond machine learning inference, signal processing, scientific computing, and computer vision workloads can also benefit when their hot operations map to optimized SME kernels. Actual gains depend on the processor and on whether the application reaches such a kernel.\n\n## How the Matrix Operation Works\n\nZA is a square array whose width and height each equal the **streaming vector length (SVL)** in bytes. At an SVL of 256 bits, each side is 32 bytes and ZA contains 32 × 32 = 1,024 bytes. The architectural SVL values are 128, 256, 512, 1024, and 2048 bits; an implementation can support a subset of these lengths. Code written without assuming a fixed SVL can run across implementations with different supported lengths.\n\nAn outer product multiplies every element of one vector by every element of another. If a and b are column vectors, their outer product is:\n\n```text\na × bᵀ =\n┌ a0·b0 a0·b1 a0·b2 ┐\n│ a1·b0 a1·b1 a1·b2 │\n└ a2·b0 a2·b1 a2·b2 ┘\n```\n\nFor C = A × B, a matrix kernel accumulates one such outer product for each position along the shared dimension: C = Σₖ A[:,k] × B[k,:]. SME outer-product instructions perform those updates directly in ZA. Predicates select active rows and columns, which helps handle partial tiles at matrix edges. The kernel loads input vectors, accumulates across the shared dimension, then stores the completed output tile.\n\nInstruction variants cover floating-point and integer data. Their available precisions depend on the implemented features; for example, FP64 outer products require the optional FEAT_SME_F64F64 feature. SME2 adds operations on groups of vectors and lookup-table support useful for some low-precision kernels.\n\n## Streaming Mode and Arm Vector and Matrix Extensions\n\nStreaming SVE mode changes which vector instructions are available and uses the streaming vector length, which can differ from the non-streaming SVE vector length. The architecture does not require either length to be larger. Streaming mode (PSTATE.SM) and ZA access (PSTATE.ZA) are separate state: ZA can be enabled without entering streaming mode, although matrix operations generally use both. SMSTART and SMSTOP change this state; C and C++ code can use ACLE function attributes to have the compiler manage mode transitions at function boundaries.\n\n| Feature | Main programming model |\n|---------|------------------------|\n| Neon | Fixed-width, 128-bit vector operations |\n| SVE/SVE2 | Scalable one-dimensional vectors and predication |\n| SME | Streaming vectors and outer-product accumulation into two-dimensional ZA storage |\n| SME2 | SME plus multi-vector and lookup-table operations |\n\nApplications can use ordinary scalar or Neon code and call SME kernels for suitable work. SVE/SVE2 code can also coexist where the processor supports those features. Streaming mode is an execution mode; it does not automatically stream matrix data from memory. A kernel still loads its input tiles and stores its results.\n\n## How Developers Use It\n\nMost developers use an optimized library through a framework instead of writing SME instructions. One example is LiteRT → XNNPACK → KleidiAI → SME2 microkernel. XNNPACK can select such a kernel only when it was included in the build, the hardware and OS expose SME2, and the operator's data type, quantization format, and shape match an available implementation. Other operations use a fallback such as Neon. Hardware support by itself therefore does not establish that a model is executing SME2 instructions.\n\nDevelopers writing their own kernels can use SME intrinsics and function attributes from Arm C Language Extensions (ACLE), or assembly when they need direct control of ZA layout and instruction scheduling. Portable applications should check platform feature availability, such as HWCAP2 on Linux, before dispatching to SME code and retain a non-SME implementation for other devices.\n\n## Resources\n\n- [Arm SME and SME2 Programmer's Guide](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2?lang=en&utm_source=arm-mcp) for the programming model and feature details.\n- [Matrix multiplication with SME2 Learning Path](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/overview/?utm_source=arm-mcp) for a guided implementation.\n- [Part 1: Arm SME Introduction](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction?utm_source=arm-mcp#blog-title) for a conceptual introduction.\n- [SME2 from scratch: Build matrix multiplication (video)](https://www.youtube.com/watch?v=5peYlm5j0U8) for a visual walkthrough.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q01", "slug": "sme-tutorials-review", "question": "Are there any SME tutorials I can review?", "summary": "Yes. Arm has hands-on material for learning SME and SME2. The best starting point depends on whether you want to inspect individual kernels, build a complete matrix multiplication example, or adapt existing SIMD code.", "answer": "Yes. Arm has hands-on material for learning SME and SME2. The best starting point depends on whether you want to inspect individual kernels, build a complete matrix multiplication example, or adapt existing SIMD code.\n\n## Start with a working kernel\n\n[Accelerate matrix multiplication performance with SME2](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/) is the most direct end-to-end tutorial. It provides a CMake project with a plain C matrix multiplication baseline, an SME2 assembly version, and a version written with C intrinsics. Follow the path in order: set up and check the toolchain, build and run the baseline, study streaming mode and ZA state, implement the SME2 versions, then compare their results with the baseline. The path also covers debugging and benchmarking.\n\nThis is an advanced tutorial. It assumes you know C, basic SVE and SME concepts, and some Arm assembly. You can run the examples on supported hardware or use its Docker-based emulator to check that the code builds and produces the right results. Use native hardware for performance measurements; emulation does not model timing accurately.\n\n## Explore smaller examples\n\n[Learn SVE and SME programming with SIMD Loops](https://learn.arm.com/learning-paths/cross-platform/simd-loops/) uses self-contained loop kernels to show how scalar, Neon, SVE, and SME2 approaches differ. The project includes reference C implementations and correctness checks. Pick a kernel that interests you, build its supported variants, and compare the code and output. This is a manageable way to explore predication, scalable vectors, and ZA before tackling a full matrix multiplication kernel. It is particularly useful if you already know C and Neon intrinsics.\n\n## Follow a different starting point when it fits your work\n\n- **Porting x86 SIMD code?** [Migrate SIMD code to the Arm architecture](https://learn.arm.com/learning-paths/cross-platform/vectorization-comparison/) maps x86 vector and matrix extensions to Neon, SVE, and SME and walks through a vectorized code example. It helps with migration choices, but its coding exercise is not an SME matrix kernel.\n- **Interested in integer matrix kernels?** The [SME2 INT8 matrix-multiplication example in Arm's Programmer's Guide](https://developer.arm.com/documentation/109246/0101/matmul-int8--8-bit-integer-to-32-bit-integer-matrix-by-matrix-multiplication) is a worked reference for 8-bit inputs and 32-bit results, including input preprocessing. Use it after learning the basic programming model; it is more detailed than a first tutorial.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q02", "slug": "scalable-matrix-extension", "question": "What is the Scalable Matrix Extension (SME) and how does it differ from SVE and Neon?", "summary": "Arm Scalable Matrix Extension (SME) adds ZA, a two-dimensional architectural register array, and instructions that accumulate vector outer products into it. It also defines Streaming SVE mode, which uses a streaming vector length and a different set of available vector instructions. Neon and SVE/SVE2 primarily expose one-dimensional vectors; SME gives a matrix kernel a place to keep a two-dimensional block of partial results. This is useful when an operation can be organized around repeated matrix or outer-product updates, including suitable kernels in machine learning, signal processing, and scientific computing.", "answer": "Arm Scalable Matrix Extension (SME) adds **ZA**, a two-dimensional architectural register array, and instructions that accumulate vector outer products into it. It also defines Streaming SVE mode, which uses a streaming vector length and a different set of available vector instructions. Neon and SVE/SVE2 primarily expose one-dimensional vectors; SME gives a matrix kernel a place to keep a two-dimensional block of partial results. This is useful when an operation can be organized around repeated matrix or outer-product updates, including suitable kernels in machine learning, signal processing, and scientific computing.\n\nThese are distinct architecture features that can coexist in one application. SME uses scalable vectors, but supporting SME does not by itself mean a processor supports non-streaming SVE or SVE2. Check the features your code actually requires.\n\n## The three programming models\n\n| Feature | Neon (Advanced SIMD) | SVE and SVE2 | SME |\n|---------|----------------------|--------------|-----|\n| Main state | Thirty-two 128-bit vector registers | Thirty-two scalable vector registers and predicate registers | Streaming vectors plus the ZA array |\n| Vector length | Fixed at 128 bits | Implementation-selected, from 128 to 2048 bits | Streaming vector length (SVL), which can differ from the non-streaming SVE length |\n| Typical way to handle an incomplete vector | Mask or branch around the remaining elements, depending on the algorithm | Predicate the active lanes | Predicate the rows and columns of an outer-product update |\n| Matrix-kernel accumulator | Vector registers or memory | Vector registers or memory | Two-dimensional ZA tile storage |\n| Especially useful for | General fixed-width SIMD work | Scalable vector loops, including irregular memory access | Kernels that can reuse a matrix output tile across many updates |\n\n**Neon** processes a fixed number of values per instruction: for example, four 32-bit floats fit in one 128-bit vector. Developers use its vector registers for arithmetic, shuffles, and suitably vectorized matrix kernels. A larger vector operation is built from multiple instructions.\n\n**SVE** makes the vector length a property of the implementation rather than a constant in the program. Code written in a vector-length-agnostic style advances by the number of active elements and uses predicates for partial vectors. SVE also provides gather/scatter operations for some irregular access patterns. **SVE2** extends that vector model with additional integer, DSP, and other instructions. SVE and SVE2 can implement matrix multiplication, but their accumulators are normally held in one-dimensional vector registers rather than ZA.\n\n**SME** adds ZA and instructions that operate on its tiles and horizontal or vertical slices. ZA is a square byte array: at an SVL of 256 bits, each side is 32 bytes and ZA contains 32 × 32 = 1,024 bytes. Tile names and the number of tile views depend on element width; they are ways to access the same underlying ZA storage, not separate arrays. Horizontal and vertical slice access helps move or rearrange matrix data, including transposition.\n\n## Why ZA changes matrix multiplication\n\nFor a matrix product C = A × B, a kernel can compute one output block by repeatedly taking a column from A and a row from B. Each pair contributes an outer product to that block:\n\n```text\noutput tile += column from A × row from B\n```\n\nWith SME, an instruction such as FMOPA accumulates that outer product directly into a ZA tile. The kernel can keep partial results there while it walks along the shared dimension, then store the completed tile. Predicates select which rows and columns are active when the output block reaches a matrix edge. SME also has integer and lower-precision outer-product variants; FP64 outer products require the optional FEAT_SME_F64F64 feature.\n\nNeon and SVE kernels can perform the same mathematics, including with dot-product or matrix-oriented instructions where supported. Their code organizes the accumulation around vector registers and the instructions available on the target CPU. SME's advantage is the programming model for a suitable kernel: ZA provides matrix-shaped accumulator storage, and outer-product instructions update it directly. Whether that runs faster depends on the processor, data layout, matrix shape, and quality of the implementation. ZA's area grows with SVL squared, but that fact alone does not establish a performance gain.\n\n## Streaming mode and coexistence\n\nStreaming SVE mode is controlled by PSTATE.SM; ZA access is controlled separately by PSTATE.ZA. The streaming vector length can differ from the length used by non-streaming SVE code, with no required ordering between them. Instruction availability differs from non-streaming mode, so a kernel cannot assume every Neon or SVE instruction is usable there. Assembly can use SMSTART and SMSTOP to change mode; in C or C++, Arm C Language Extensions (ACLE) function attributes let the compiler manage mode transitions and ZA state at function boundaries.\n\nAn application can keep its general work in scalar, Neon, or SVE/SVE2 code and call an SME kernel for a suitable operation. SME2 extends SME with multi-vector operations, predication for groups of vectors, and lookup-table support; it does not replace the other features. Portable software still needs a path for devices without the required SME features.\n\n## Resources\n\n- [Neon, SVE, and SME matrix multiplication compared](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared) shows how the same operation is expressed with each programming model.\n- [Arm SME and SME2 Programmer's Guide](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2) explains ZA, streaming mode, and the related programming rules.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q03", "slug": "streaming-sve-mode", "question": "What is streaming SVE mode and how does it relate to the ZA tile storage in SME?", "summary": "Streaming SVE mode and ZA are separate parts of Arm SME. Streaming mode changes the vector length and which vector instructions can execute. ZA is a two-dimensional register array that holds matrix data, especially partial results from outer-product operations. ZA's dimensions come from the streaming vector length (SVL), even while the processor is outside streaming mode. A typical matrix kernel enables both features: streaming Z registers supply the input vectors, and the operation accumulates a result in ZA.", "answer": "Streaming SVE mode and ZA are separate parts of Arm SME. **Streaming mode** changes the vector length and which vector instructions can execute. **ZA** is a two-dimensional register array that holds matrix data, especially partial results from outer-product operations. ZA's dimensions come from the streaming vector length (SVL), even while the processor is outside streaming mode. A typical matrix kernel enables both features: streaming Z registers supply the input vectors, and the operation accumulates a result in ZA.\n\n## Two controls, four states\n\nPSTATE.SM controls streaming mode; PSTATE.ZA controls whether ZA is enabled. They can be changed separately with `SMSTART SM` / `SMSTOP SM` and `SMSTART ZA` / `SMSTOP ZA`. An unqualified SMSTART enables both, and an unqualified SMSTOP disables both.\n\n| SM | ZA | What the code can do |\n|------|------|----------------------|\n| 0 | 0 | Execute ordinary non-streaming code; ZA is unavailable. |\n| 1 | 0 | Use streaming vector operations at SVL without accessing ZA. |\n| 0 | 1 | Keep ZA contents available across a mode change. ZA array-vector load/store operations can save or restore its state, but most tile operations cannot run here. |\n| 1 | 1 | Use streaming vectors and ZA together for outer products and other matrix operations. |\n\nThe separate controls make it possible to leave streaming mode while retaining an unfinished ZA result: use `SMSTOP SM`, leaving PSTATE.ZA set, and later use `SMSTART SM` to continue. Turning ZA off loses that live state; enabling it again starts with ZA cleared. Changing streaming mode zeroes Z0–Z31 and P0–P15 in the new mode, and FFR where it is implemented, so code must not carry their values across the transition. A branch or scalar loop does not itself require leaving streaming mode. Switching modes is useful when code needs instructions or a function interface that runs in non-streaming mode, but repeated switches have a cost.\n\nStreaming mode uses SVL for its Z and predicate registers. If the processor also supports non-streaming SVE, its vector length can differ from SVL; neither length is required to be larger. Streaming mode supports a defined subset of vector instructions and the SME matrix operations. It does not automatically fetch matrix data from memory: a kernel still loads inputs and stores results.\n\n## SVL defines ZA's shape\n\nZA is a square array with **SVL in bytes** on each side. For example, at SVL = 256 bits, each side is 32 bytes, so ZA contains 32 × 32 = 1,024 bytes. At SVL = 512 bits, it contains 64 × 64 = 4,096 bytes. This describes storage capacity; a larger ZA alone does not establish higher performance.\n\nThe same underlying bytes can be viewed in several ways:\n\n- **ZA array vectors** provide a row-oriented view useful for saving and restoring state.\n- **ZA tiles** divide the array into typed matrix views. The number of tiles depends on element width: one for 8-bit elements, two for 16-bit, four for 32-bit, eight for 64-bit, and sixteen for 128-bit. For instance, the 32-bit views are named `ZA0.S` through `ZA3.S`. They overlap the same ZA storage rather than forming four separate register arrays.\n- **Tile slices** expose horizontal or vertical vectors within a tile. A horizontal slice such as `ZA0H.S` and a vertical slice such as `ZA0V.S` let instructions move or rearrange data along different axes, including for transposition.\n\nThe non-streaming SVE vector length does not change ZA's size. If code retains ZA while switching out of streaming mode, its matrix state still has the dimensions set by SVL.\n\n## How a matrix kernel uses both\n\nA simplified outer-product kernel enables streaming mode and ZA, clears the accumulator tile, then repeats these steps: load input vectors into streaming Z registers, accumulate their outer product into ZA, and advance along the shared dimension. It stores the finished tile before disabling ZA. Predicates select active rows and columns for a partial output tile.\n\nThe instruction requirements matter. An outer product such as FMOPA needs streaming mode and ZA enabled. ZA tile-slice LD1 and ST1 operations also belong inside streaming mode. The ZA array-vector LDR and STR operations used for state saving are distinct and can operate with ZA enabled outside streaming mode. This distinction is why retaining ZA across a mode switch does not make every ZA instruction available in non-streaming code.\n\nIn C or C++, Arm C Language Extensions (ACLE) function attributes make the state contract visible in a declaration:\n\n```c\nvoid accumulate_tile(const float *a, const float *b)\n __arm_streaming __arm_inout(\"za\");\n\nvoid streaming_vector_op(float *data, unsigned count)\n __arm_streaming;\n```\n\nThe first function runs in streaming mode and shares its caller's ZA state, so its implementation can accumulate into that state. The second uses streaming vectors but declares no ZA state. These are interface declarations, not complete kernels; the caller and implementation must follow the ZA ownership rules. The compiler manages mode transitions and ZA preservation at call boundaries according to the attributes. Hand-written assembly must follow the calling convention itself.\n\n## Resources\n\n- [Arm's streaming mode and ZA walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode/) explains the two state bits and their C/C++ programming model.\n- [Arm's ZA array and tile mapping guide](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-ZA-storage/ZA-array-vector-access-and-ZA-tile-mapping) shows the array-vector, tile, and tile-slice views.\n- [Arm C Language Extensions](https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics) defines the streaming and ZA function attributes.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q04", "slug": "instructions-sme2-add", "question": "What instructions does SME2 add beyond the original SME, and what data types do they support?", "summary": "The original SME provides streaming vectors, ZA tile storage, and outer-product instructions that accumulate into ZA. SME2 keeps those capabilities and adds instructions for groups of two or four vectors, a way to predicate those groups, lookup-table expansion of packed values, and binary outer products. These are different instruction families: not every SME2 instruction takes a two- or four-register group, and not every result goes into ZA.", "answer": "The original SME provides streaming vectors, ZA tile storage, and outer-product instructions that accumulate into ZA. SME2 keeps those capabilities and adds instructions for groups of two or four vectors, a way to predicate those groups, lookup-table expansion of packed values, and binary outer products. These are different instruction families: **not every SME2 instruction takes a two- or four-register group, and not every result goes into ZA.**\n\n## Outer-product support in SME and SME2\n\nSME outer-product instructions multiply values from two Z registers and accumulate or subtract the result in a ZA tile. Tile-slice moves and loads/stores transfer data between ZA, Z registers, and memory. The main outer-product input and accumulator combinations are:\n\n| Inputs | ZA accumulator | Availability |\n|---|---|---|\n| INT8 × INT8 | INT32 | SME |\n| INT16 × INT16 | INT32 | SME2 |\n| INT16 × INT16 | INT64 | Optional FEAT_SME_I16I64 |\n| FP16 × FP16, BF16 × BF16, or FP32 × FP32 | FP32 | SME |\n| FP64 × FP64 | FP64 | Optional FEAT_SME_F64F64 |\n\nNarrow inputs can contribute multiple products to each wider accumulator element. For example, an INT8 outer-product instruction sums four INT8 products into each INT32 result, while the FP32 form contributes one product per result.\n\n## What SME2 adds\n\n**Multi-vector arithmetic and data movement.** SME2 can operate on groups of two or four Z registers or ZA vectors, commonly marked `VGx2` and `VGx4`. Representative arithmetic instructions include multiply-accumulate (FMLA/FMLS), widening multiply-accumulate (FMLAL/FMLSL, SMLAL/UMLAL), and dot product (SDOT/UDOT, BFDOT). It also adds multi-vector addition, subtraction, shifts, and selection. Some instructions accumulate into ZA; others produce Z-register results. Multi-vector loads/stores (LD1/ST1), moves (MOVA), permutes (ZIP/UZP), and conversions help feed those calculations and handle their results. The supported element types vary by instruction.\n\nFor example, this MOVA transfers two Z registers into two ZA array-vector groups:\n\n```asm\nMOVA ZA.D[w8, #0, VGx2], { Z0.D-Z1.D }\n```\n\nThe `VGx2` operand describes the ZA destination grouping. It does not mean that every SME2 instruction uses `VGx2` or writes ZA. A multi-vector LD1D, for instance, loads Z registers from memory. A widening arithmetic example is:\n\n```asm\nFMLAL ZA.S[W8, 0:1, VGx2], { Z0.H-Z1.H }, Z2.H\n```\n\nHere, FP16 source values from the Z registers are multiplied and accumulated as FP32 values in ZA. The `0:1` selects a double-vector group, and `VGx2` makes this operation update two such groups.\n\n**Predicate-as-counter.** SME2 introduces the PN interpretation of the existing predicate registers for multi-vector operations. A counter predicate can describe a run of active elements across two or four vectors, which is useful at the end of a loop when the remaining data does not fill the whole group. For example, `WHILELT PN8.S, X0, X1, VLx2` generates a predicate for up to two vectors of 32-bit elements; PEXT can extract an ordinary predicate mask from it. The mechanism also supports 8-, 16-, and 64-bit elements. PN0 and P0 refer to the same underlying predicate register; the instruction determines how its bits are interpreted. The predicate-as-counter WHILELT form above restricts its destination to PN8–PN15. The encoding can also represent a run of inactive elements followed by active ones, so it is broader than just an active-prefix count.\n\n**Lookup-table expansion.** SME2 adds the fixed 512-bit ZT0 register, arranged as sixteen 32-bit table entries. LUTI2 uses packed 2-bit indices to select among four entries; LUTI4 uses packed 4-bit indices to select among sixteen. The selected values expand into 8-, 16-, or 32-bit elements in Z registers. A kernel can load packed weights, expand them with LUTI2 or LUTI4, then pass the resulting Z values to SDOT or SMOPA for accumulation in ZA.\n\nThe lookup is a preparation step, not a 2-bit or 4-bit arithmetic operation. ZT0 can be loaded, stored, and cleared with LDR, STR, and ZERO. LUTI forms differ by output width and register count; in particular, the four-register, 8-bit LUTI4 form requires the optional FEAT_SME_LUTv2 extension.\n\n**Binary outer products.** BMOPA and BMOPS use bitwise XNOR and population count to accumulate or subtract binary outer-product results in INT32 ZA elements. They serve workloads such as binary neural networks. These instructions use packed bits in their source vectors; they are not multi-vector arithmetic on 1-bit Z-register lanes.\n\n## Which data types are actually supported?\n\nIt helps to separate **arithmetic inputs**, **ZA accumulators**, and **packed lookup indices**:\n\n| Role | Formats and limits |\n|---|---|\n| SME and SME2 arithmetic inputs | INT8, INT16, and INT32; FP16, BF16, and FP32 in applicable instruction forms; binary values for BMOPA/BMOPS. Multi-vector elementwise operations can also use 64-bit integer lanes. |\n| ZA accumulators | INT32 and FP32; INT64 with FEAT_SME_I16I64; FP64 with FEAT_SME_F64F64 |\n| LUTI source codes | Packed 2-bit or 4-bit indices, expanded into 8-, 16-, or 32-bit Z elements before subsequent arithmetic |\n| FP8 arithmetic | Requires optional SME FP8 features, such as FEAT_SME_F8F32 or FEAT_SME_F8F16; it is not a baseline SME2 arithmetic format |\n\nLUTI can produce 8-bit patterns that represent FP8 values, but producing those patterns is different from having FP8 arithmetic instructions. When selecting a kernel, check the required optional feature as well as SME2 itself. Arm lists the SME FP8 arithmetic features as optional from Armv9.2, rather than making them a baseline Armv9.5 feature. The [Arm C Language Extensions feature macros](https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics) describe the compiler-visible distinctions.\n\n## Resources\n\n- [Arm's SME2 instruction introduction](https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/part4-arm-sme2-introduction) illustrates multi-vector operands and predicate-as-counter with instruction examples.\n- [Arm's LUTI walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/luti/02_luti_and_sme2/) follows packed indices through ZT0, expanded Z registers, and a matrix instruction.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q05", "slug": "outer-product-instruction", "question": "How does the outer product instruction work in SME, and why is it fundamental to matrix operations?", "summary": "An SME outer-product instruction reads two streaming Z registers and updates a two-dimensional ZA tile. For FP32, FMOPA multiplies every active element of the first vector by every active element of the second, then adds each product to the corresponding tile element. Keeping that tile in ZA lets a matrix kernel accumulate many contributions before storing the result.", "answer": "An SME outer-product instruction reads two streaming Z registers and updates a two-dimensional ZA tile. For FP32, FMOPA multiplies every active element of the first vector by every active element of the second, then adds each product to the corresponding tile element. Keeping that tile in ZA lets a matrix kernel accumulate many contributions before storing the result.\n\n## From a vector pair to a matrix product\n\nFor two-element vectors, the outer product looks like this:\n\n```text\n[ a₀ ] [ a₀b₀ a₀b₁ ]\n[ a₁ ] × [ b₀ b₁ ] = [ a₁b₀ a₁b₁ ]\n```\n\nAn inner product pairs elements and reduces them to one number. An outer product forms a matrix of pairwise products. To multiply matrices, take a column of the left matrix and the matching row of the right matrix, form their outer product, and add it to the output. Repeat for every matching column and row. An SME kernel accumulates these contributions in the same ZA tile before storing it. Each input value contributes to several output elements.\n\n## The FMOPA instruction: inputs and result\n\nHere is the FP32 instruction form:\n\n```asm\nFMOPA ZA0.S, P0/M, P1/M, Z0.S, Z1.S\n```\n\n`Z0.S` supplies the first input vector and `Z1.S` the second. `ZA0.S` is the FP32 destination tile, whose previous contents are accumulated into. `P0/M` selects active elements of `Z0` (output rows); `P1/M` selects active elements of `Z1` (output columns). For every selected row i and column j, the instruction performs **ZA0[i, j] += Z0[i] × Z1[j]**. Elements outside either predicate remain unchanged. FMOPS has the corresponding subtract behavior.\n\nThe instruction requires streaming mode and ZA enabled. A complete kernel must initialize or load the tile before the inner loop, place the appropriate input fragments in Z registers, and store the finished tile afterward. For a matrix stored row by row, obtaining column fragments efficiently often requires packing or another layout strategy; the mathematical column is not necessarily a contiguous memory load. At matrix edges, the kernel also predicates its input loads and tile stores so it stays within bounds. The FMOPA line above is a real instruction, but it is not a complete kernel.\n\n## Widening variants\n\nThe simple FP32 case is one outer product per instruction. With narrower inputs, an instruction combines several packed outer products into each wider ZA element:\n\n| Inputs | ZA accumulator | Instruction family | Contributions per result | Availability |\n|---|---|---|---|---|\n| FP32 × FP32 | FP32 | FMOPA / FMOPS | One FP32 product | SME |\n| FP64 × FP64 | FP64 | FMOPA / FMOPS | One FP64 product | Optional FEAT_SME_F64F64 |\n| FP16 × FP16 | FP32 | FMOPA / FMOPS | Two FP16 products | SME |\n| BF16 × BF16 | FP32 | BFMOPA / BFMOPS | Two BF16 products | SME |\n| Signed or unsigned INT8 × INT8 | INT32 | SMOPA / SMOPS or UMOPA / UMOPS | Four INT8 products | SME |\n| Signed or unsigned INT16 × INT16 | INT32 | SMOPA / SMOPS or UMOPA / UMOPS | Two INT16 products | SME2 |\n| Signed or unsigned INT16 × INT16 | INT64 | SMOPA / SMOPS or UMOPA / UMOPS | Four INT16 products | Optional FEAT_SME_I16I64 |\n\nWider accumulation lets a kernel combine low-precision inputs while retaining more precision in its partial sums, though it does not eliminate rounding or integer overflow concerns.\n\nFor a concrete INT8 case, suppose the streaming vector length (SVL) is 128 bits. Each source Z register holds 16 bytes, arranged for this instruction as four groups of four INT8 values. One SMOPA or UMOPA updates a 4×4 INT32 tile: each output element receives a sum of four INT8 products. In general, the instruction treats each source as an (SVL/32)×4 group of INT8 values and updates an (SVL/32)×(SVL/32) INT32 tile. This is a short matrix product of depth four packed into one instruction, rather than the single rank-one update shown by FP32 FMOPA.\n\n## Why kernels use it\n\nFor FP32, let **L = SVL / 32** be the number of FP32 elements in a streaming Z register. With all lanes active, one FMOPA updates an L×L tile, or L² accumulator elements, using two L-element source vectors. At a 512-bit SVL, L is 16, so that instruction performs 256 multiply-accumulate updates from 32 source elements. The number of updates grows quadratically with L while the source element count grows linearly. Each source value participates in multiple updates, and the partial sums stay in ZA across the inner loop.\n\nThis describes operand reuse and arithmetic work, not a guaranteed number of memory loads or a per-cycle throughput. Actual performance depends on input layout and packing, tile initialization and stores, the available bandwidth, the processor's implementation, and how much work the kernel has. Vector-length-agnostic code can use the same instructions across supported SVLs by sizing its loops and data layout at runtime. A larger SVL changes the tile shape; it does not by itself guarantee a proportional speedup.\n\nIndependent row and column predicates let the same instruction update partial tiles when an output dimension is not a multiple of the tile size. They avoid changing inactive ZA elements; the surrounding load and store code still has to handle the boundaries correctly.\n\n## Resources\n\n- [Arm's outer-product Learning Path](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product/) shows how matrix multiplication becomes a sequence of tile updates and why input layout matters.\n- [Arm's SME instruction walkthrough](https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction-p2) explains FMOPA, predication, and the widening variants with diagrams.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q06", "slug": "find-official-architecture", "question": "Where can I find the official Arm architecture reference material for SME and SME2?", "summary": "Arm publishes several official references for SME and SME2 because they answer different developer questions. Start with the Programmer's Guide to understand the model. Use the A64 instruction pages for exact instruction behavior, Arm C Language Extensions (ACLE) for C/C++ code, and AAPCS64 for rules at function-call boundaries.", "answer": "Arm publishes several official references for SME and SME2 because they answer different developer questions. Start with the Programmer's Guide to understand the model. Use the A64 instruction pages for exact instruction behavior, Arm C Language Extensions (ACLE) for C/C++ code, and AAPCS64 for rules at function-call boundaries.\n\n## Learn the programming model: SME Programmer's Guide\n\nThe [SME Programmer's Guide](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2) (document 109246) is the practical starting point. Its SME overview explains streaming SVE mode, the ZA array and tile views, and how SME2 extends the instruction set. The guide also has sections on toolchains and worked matrix-multiplication code, so you can see how a kernel enters streaming mode, computes in ZA, and stores a result. Use its support and feature-detection discussion when deciding which path your application can run.\n\n## Check an instruction precisely: A64 instruction reference\n\nThe [Arm A-profile A64 instruction reference](https://developer.arm.com/documentation/ddi0602/latest/SME-Instructions) (DDI 0602) has a page for each SME or SME2 instruction. Look here when the exact assembler form or behavior matters. For example, find FMOPA to check its operands and predication, SMSTART to check state changes, or LUTI2 to check its lookup operands and required feature. The instruction pages give encodings, operand constraints, and a precise description of the operation. They are more useful for one instruction than a broad overview.\n\n## Write C or C++: Arm C Language Extensions\n\nThe [ACLE SME section](https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics) specifies ``, SME and SME2 intrinsics, feature-test macros, and function attributes. Consult it to find the intrinsic for an instruction and to declare the state a function uses. For example, `__arm_streaming` says a function uses streaming mode, while `__arm_inout(\"za\")` says it shares ZA state with its caller. A compiler feature macro describes what the current compilation target supports; it does not by itself prove that the processor and operating system running the program expose the feature.\n\n## Handle function calls and ZA state: AAPCS64\n\nThe [AAPCS64 SME sections](https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst#62-sme-state) specify the procedure-call rules. Read them when an SME function calls other code, a library exposes an SME-aware function, or hand-written assembly must preserve state. They distinguish streaming and non-streaming interfaces, describe private and shared ZA state, and define when ZA needs to be saved. ACLE gives you the source-language attributes; AAPCS64 explains the calling contract behind them.\n\nFor a new kernel, a useful order is: learn streaming mode and ZA in the Programmer's Guide, look up each instruction you use in the A64 reference, then check ACLE and AAPCS64 for how your function enters, shares, and leaves that state.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q07", "slug": "devices-support-sme", "question": "What devices support SME and SME2?", "summary": "SME2 is a superset of SME, so SME2-capable hardware supports both instruction sets. Documented consumer devices include Apple A18/A19 and M4/M5 devices plus Android phones with SME2-enabled Arm C1 CPU clusters. The examples below have SME2, but do not assume every Armv9 device does. Check the exact device and whether its OS exposes the feature.", "answer": "SME2 is a superset of SME, so SME2-capable hardware supports both instruction sets. Documented consumer devices include Apple A18/A19 and M4/M5 devices plus Android phones with SME2-enabled Arm C1 CPU clusters. The examples below have SME2, but do not assume every Armv9 device does. Check the exact device and whether its OS exposes the feature.\n\n> Note: Apple iPhone and iPad models with A18/A19 chips support SME2 natively, but Arm's SME2 Learning Path demonstrates SME2 only on macOS (M4) and on Android phones with SME2 support — not on iPhone or iPad.\n\n## Devices with Native SME2 Support\n\n### Apple Devices\n\nApple's A18 and M4 generations introduced SME2 in these product lines. The M5 generation adds further devices, including the [M5 iPad Pro](https://www.apple.com/newsroom/2025/10/apple-introduces-the-powerful-new-ipad-pro-with-the-m5-chip/) and [MacBook Pro models with M5 Pro and M5 Max](https://www.apple.com/newsroom/2026/03/apple-introduces-macbook-pro-with-all-new-m5-pro-and-m5-max/). These are device examples; use the feature check below before running SME2 code.\n\n| Device | Release Date | Chip |\n|--------|-------------|------|\n| iPhone 16 | 2024 | A18 |\n| iPhone 17 | 2025 | A19 |\n| MacBook Neo (13-inch) | 2026 | A18 Pro |\n| iPad Pro (7th generation) | 2024 | M4 |\n| iPad Pro (2025) | 2025 | M5 |\n| iMac (2024) | 2024 | M4 |\n| Mac Mini (2024) | 2024 | M4 |\n| MacBook Pro (14-inch, 16-inch, 2024) | 2024 | M4 |\n| MacBook Air (2025) | 2025 | M4 |\n| MacBook Pro (14-inch, 2025) | 2025 | M5 |\n| MacBook Air (13-inch, 15-inch, 2026) | 2026 | M5 |\n| MacBook Pro (14-inch, 16-inch, 2026) | 2026 | M5 |\n\n### Android Phones\n\n| Device | Release Date | Chip |\n|--------|-------------|------|\n| Vivo X300 | 2025 | MediaTek Dimensity 9500 (8-core Arm C1 CPU, Arm G1-Ultra GPU) |\n| OPPO Find X9 | 2025 | MediaTek Dimensity 9500 (8-core Arm C1 CPU, Arm G1-Ultra GPU) |\n| Google Pixel 11 | 2026 | Google Tensor G6 (7-core Arm C1 CPU cluster) |\n| Samsung Galaxy S26 / S26+ | 2026 | Exynos 2600 variant (10-core Arm C1 CPU cluster) |\n| Samsung Galaxy Z Flip8 | 2026 | Exynos 2600 variant (10-core Arm C1 CPU cluster) |\n| Xiaomi Poco X8 Pro Max | 2026 | MediaTek Dimensity 9500 (8-core Arm C1 CPU, Arm G1-Ultra GPU) |\n| Xiaomi 18 Fold | 2026 | XRing O3 (10-core Arm C1 CPU cluster) |\n\n## Arm C1 and C2 CPU Clusters with SME2\n\nArm's C1 family is used in SME2-capable CPU clusters. SME2 execution is provided at the cluster level, so the C1 name alone is not a substitute for checking the finished device. The family includes:\n\n| Core | Target Segment |\n|------|---------------|\n| Arm C1-Ultra | Flagship performance |\n| Arm C1-Premium | High-performance |\n| Arm C1-Pro | Efficiency / balanced |\n| Arm C1-Nano | Area- and power-efficient |\n\nThese cores appear in SoCs like MediaTek Dimensity 9500 and Samsung Exynos 2600.\n\nArm [announced the C2 CPU cluster](https://www.arm.com/products/silicon-ip-cpu/c2-cpu-cluster) in September 2026. It combines C2-Ultra and C2-Pro CPUs with two cluster-level SME2 units. This describes Arm's CPU IP, not a guarantee that a particular phone exposes SME2 to applications. Check the finished device at runtime before selecting an SME2 code path.\n\n## Chips and Cores That Do NOT Support SME\n\nHaving Armv9, SVE, or SVE2 support does **not** imply SME support — SME and SME2 are optional architectural extensions. Notable modern processors that lack SME:\n\n| Chip / Core | Architecture | Notes |\n|-------------|-------------|-------|\n| MediaTek Dimensity 9400 | Armv9.2-A (Cortex-X925 / A725 / A520) | No SME |\n| Samsung Exynos 2500 | Armv9.2-A | No SME |\n| Arm Neoverse V3 / N3 | Armv9.2-A | SVE2 only, no SME |\n| Arm Cortex-X925, Cortex-A725 | Armv9.2-A | SVE2 only, no SME |\n\n**Rule of thumb:** Treat SME/SME2 as optional and require an explicit vendor statement or runtime feature check. Product-family guesses are not a safe dispatch mechanism.\n\n## How to Verify SME2 on Your Device\n\nOn Linux or Android, you can check for SME2 at runtime using the HWCAP2 feature bits:\n\n```c\n#include \n#include \n\n#ifndef HWCAP2_SME2\n#define HWCAP2_SME2 (1UL << 37)\n#endif\n\nint main(void) {\n unsigned long hwcap2 = getauxval(AT_HWCAP2);\n if (hwcap2 & HWCAP2_SME2)\n printf(\"SME2 is supported\\n\");\n else\n printf(\"SME2 is NOT supported\\n\");\n return 0;\n}\n```\n\nOn macOS (Apple Silicon), use sysctl:\n\n```bash\nsysctl hw.optional.arm.FEAT_SME2\n# Returns 1 if supported\n```\n\n## Development Without Native Hardware\n\nIf you don't have access to SME2-capable hardware, you can use Arm's Fixed Virtual Platform (FVP) to emulate an SME2-capable processor. Arm provides a Docker-based workflow that runs SME2 code in bare-metal mode on the FVP. See the Learning Path linked below for setup instructions.\n\n## Software and Toolchain Support\n\nSoftware for building or using SME2 code includes:\n\n- **Arm KleidiAI** — micro-kernel library for matrix operations, integrates with PyTorch, ExecuTorch, XNNPACK, ONNX Runtime, and llama.cpp\n- **LLVM/Clang 18+** and **GCC 14+** — compiler support for SME/SME2 intrinsics and code generation\n- **Arm Compute Library (ACL)** — backend support for ML frameworks\n\n## Resources\n\n- [Arm: devices with native SME2 support and setup options](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/#devices)\n- [Arm: check SME2 availability on your device](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment/#check-sme2-availability)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q08", "slug": "differences-among-sme", "question": "Are there differences among SME devices or are they all the same?", "summary": "SME devices are not all the same. They share a programming model with streaming mode and ZA tile storage, but can differ in SME generation, streaming vector length, optional instructions, matrix-unit sharing, and performance. The operating system and libraries also determine which capabilities an application can use. Check features at runtime and measure performance on the target device.", "answer": "SME devices are **not all the same**. They share a programming model with streaming mode and ZA tile storage, but can differ in SME generation, streaming vector length, optional instructions, matrix-unit sharing, and performance. The operating system and libraries also determine which capabilities an application can use. Check features at runtime and measure performance on the target device.\n\n## Key Differences Across SME Implementations\n\n### 1. SME Generation (SME vs. SME2 vs. SME2.1)\n\nThe SME architecture has evolved through multiple generations, each adding significant functionality:\n\n| Generation | Feature ID | Key Additions |\n|-----------|-----------|---------------|\n| **SME** | FEAT_SME | ZA tile storage, streaming SVE mode, outer product instructions, matrix load/store/insert/extract |\n| **SME2** | FEAT_SME2 | Multi-vector instructions (load, store, move, permute, convert), 512-bit lookup table register (ZT0) for weight dequantization, multi-vector predication, predicate-as-counter controls |\n| **SME2.1** | FEAT_SME2p1 | Additional multi-vector forms, including strided destination groups for ZT0 lookup instructions |\n\nSME2 is a superset of SME, so any processor implementing SME2 also supports the base SME instructions.\n\n### 2. Streaming Vector Length (SVL)\n\nThe most visible difference between implementations is the **Streaming Vector Length (SVL)**, which is chosen by the hardware designer:\n\n- The architecture permits SVL values of **128, 256, 512, 1024, or 2048 bits** (any power of two in that range).\n- SVL determines the width of vector operations in streaming mode and the dimensions of the ZA tile. On Linux, the effective SVL is a per-thread setting that the OS can limit to a supported value.\n- A processor's SVL does **not** have to match its non-streaming SVE vector length (NSVL). They are independent parameters.\n\n### 3. ZA Tile Size (Scales Quadratically with SVL)\n\nThe ZA storage array is a 2D square of dimensions (SVL in bytes) × (SVL in bytes). Because the scaling is **quadratic**, small differences in SVL create large differences in available tile storage:\n\n| SVL | ZA Dimensions | ZA Size |\n|-----|--------------|---------|\n| 128 bits (16 bytes) | 16 × 16 | **256 bytes** |\n| 256 bits (32 bytes) | 32 × 32 | **1,024 bytes** (1 KB) |\n| 512 bits (64 bytes) | 64 × 64 | **4,096 bytes** (4 KB) |\n| 1024 bits (128 bytes) | 128 × 128 | **16,384 bytes** (16 KB) |\n| 2048 bits (256 bytes) | 256 × 256 | **65,536 bytes** (64 KB) |\n\nA wider SVL means more data processed per outer product instruction and a larger working set held in ZA without spilling to memory.\n\n### 4. Optional Architectural Features\n\nHardware designers can selectively implement optional sub-features identified via the ID_AA64SMFR0_EL1 system register. These vary based on whether the target is HPC, mobile, or server:\n\n| Feature | Description | Typical Use Case |\n|---------|-------------|------------------|\n| FEAT_SME_F64F64 | Double-precision (FP64) outer products | HPC, scientific computing |\n| FEAT_SME_F16F16 | Half-precision (FP16) matrix operations | ML inference |\n| FEAT_SME_I16I64 | 16-bit integer inputs → 64-bit integer accumulation | Signal processing, HPC |\n| FEAT_SME_B16B16 | BFloat16 matrix operations | ML training/inference |\n| FEAT_SME_FA64 | Full A64 instruction set available in streaming mode | Reduces mode-switching overhead |\n\nWithout FEAT_SME_FA64, streaming mode permits only a subset of AArch64 instructions. Code that executes an instruction unavailable in that mode can fault; keep non-streaming work outside the streaming region or check the needed feature.\n\n### 5. Physical Matrix Units and Multithread Scaling\n\nSME and SME2 define the architectural instructions and state, but they do **not** require a private matrix execution unit for every CPU core. A system can have one or more physical SME compute units shared by several cores. For example, an Arm C1-DSU cluster can be configured with **one or two C1-SME2 units**; Arm's newer [C2 cluster](https://www.arm.com/products/silicon-ip-cpu/c2-cpu-cluster) has two SME2 units. Other implementations may use a different topology or may not publish it.\n\nThis has important consequences for application threading:\n\n- With **one shared unit**, a single SME-heavy thread may perform very well, while additional SME-heavy threads contend for the same resource. Scaling can flatten or even regress.\n- With **two units**, two concurrent SME-heavy threads may scale better, but cache, memory-bandwidth, frequency, and thermal limits can still prevent linear scaling.\n- On heterogeneous systems, scheduler placement and migration can change results. Pinning threads to a documented CPU set makes tests reproducible, but it does not make a shared SME unit private to those threads.\n- Runtime feature detection such as `HWCAP2_SME2` only proves that SME2 instructions are available. It does not report the number of physical matrix units or their sharing topology.\n\nDo not choose the thread count from the number of CPU cores alone. Benchmark 1, 2, 4, and higher thread counts with fixed affinity, compare SME2-enabled and fallback runs at every count, and record the CPU set, power mode, frequency behavior, and temperature. Framework kernel tracing should also confirm that each tested operator actually selected an SME2 kernel.\n\n### 6. Performance Characteristics\n\nEven two processors with identical feature sets and the same SVL can differ in:\n\n- **Outer product instruction throughput** (cycles per instruction)\n- **ZA access latency** (read/write to tile rows and columns)\n- **Streaming mode entry/exit cost** (penalty for SMSTART/SMSTOP transitions)\n- **Memory bandwidth and cache hierarchy** (affects tile load/store performance)\n- **Clock speed and thermal limits** (mobile vs. server envelopes)\n\n### 7. System Enablement\n\nWhether applications can actually exploit the hardware depends on:\n\n- **OS kernel support:** saves and restores ZA and streaming state across context switches\n- **Compiler versions:** provide SME intrinsics and code generation\n- **Hypervisor and firmware (EL2/EL3):** must allow user-space SME execution\n- **Libraries:** provide optimized kernels (for example, KleidiAI) and runtime dispatch\n\n## Runtime Feature Detection\n\nBecause features vary, software must query the hardware at runtime rather than assuming a fixed feature set.\n\n### Linux (HWCAP2 flags)\n\n```c\n#include \n#include \n\n// HWCAP2 flags for SME features; use system definitions when available.\n#ifndef HWCAP2_SME\n#define HWCAP2_SME (1UL << 23)\n#endif\n#ifndef HWCAP2_SME2\n#define HWCAP2_SME2 (1UL << 37)\n#endif\n#ifndef HWCAP2_SME_F64F64\n#define HWCAP2_SME_F64F64 (1UL << 25)\n#endif\n#ifndef HWCAP2_SME_I16I64\n#define HWCAP2_SME_I16I64 (1UL << 24)\n#endif\n#ifndef HWCAP2_SME_FA64\n#define HWCAP2_SME_FA64 (1UL << 30)\n#endif\n#ifndef HWCAP2_SME2P1\n#define HWCAP2_SME2P1 (1UL << 38)\n#endif\n\nint main(void) {\n unsigned long hwcap2 = getauxval(AT_HWCAP2);\n\n printf(\"SME: %s\\n\", (hwcap2 & HWCAP2_SME) ? \"yes\" : \"no\");\n printf(\"SME2: %s\\n\", (hwcap2 & HWCAP2_SME2) ? \"yes\" : \"no\");\n printf(\"SME2.1: %s\\n\", (hwcap2 & HWCAP2_SME2P1) ? \"yes\" : \"no\");\n printf(\"SME_F64: %s\\n\", (hwcap2 & HWCAP2_SME_F64F64) ? \"yes\" : \"no\");\n printf(\"SME_I16I64:%s\\n\", (hwcap2 & HWCAP2_SME_I16I64) ? \"yes\" : \"no\");\n printf(\"SME_FA64: %s\\n\", (hwcap2 & HWCAP2_SME_FA64) ? \"yes\" : \"no\");\n\n return 0;\n}\n```\n\nFor manual inspection, `/proc/cpuinfo` also lists `sme`, `sme2`, and related flags. Use `AT_HWCAP2` for program dispatch.\n\n### macOS\n\nOn Apple Silicon with SME support, use `sysctlbyname`:\n\n```c\n#include \n#include \n#include \n\nint main(void) {\n int64_t sme = 0;\n size_t size = sizeof(sme);\n if (sysctlbyname(\"hw.optional.arm.FEAT_SME\", &sme, &size, NULL, 0) == 0) {\n printf(\"SME: %s\\n\", sme ? \"supported\" : \"not supported\");\n } else {\n printf(\"SME: query unavailable\\n\");\n }\n size = sizeof(sme);\n if (sysctlbyname(\"hw.optional.arm.FEAT_SME2\", &sme, &size, NULL, 0) == 0) {\n printf(\"SME2: %s\\n\", sme ? \"supported\" : \"not supported\");\n } else {\n printf(\"SME2: query unavailable\\n\");\n }\n return 0;\n}\n```\n\n## Writing Portable SME Code\n\nThe architectural intent is that **Vector Length Agnostic (VLA) SME code remains functionally portable** across all implementations. Best practices:\n\n1. **Never hard-code SVL.** Use the ACLE `svcntsb()` intrinsic to get the streaming vector length in bytes, including when the caller is in non-streaming mode. `svcntb()` reports SVL only while running in streaming mode; otherwise it reports the non-streaming vector length.\n2. **Check optional features** before using specialized instructions (FP64 outer products, FA64 mode).\n3. **Use libraries like KleidiAI** that perform runtime detection and dispatch to the best available kernel for the current hardware.\n4. **Test on multiple SVL values** using suitable hardware or an Arm FVP configured for different streaming vector lengths.\n\n## Resources\n\n- [Arm C Language Extensions: SME intrinsics and streaming-mode rules](https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics)\n- [Arm C1-SME2 Technical Reference Manual, page 9: shared-unit topology](https://documentation-service.arm.com/static/68c0d4c3cccf2a5517016403#page=9)\n- [Linux SME userspace interface: HWCAP2 and vector-length controls](https://docs.kernel.org/arch/arm64/sme.html)\n- [SqueezeSAM case study: shared SME2 resource and one-core vs. four-core scaling](https://pytorch.org/blog/accelerating-on-device-ml-inference-with-executorch-and-arm-sme2/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q09", "slug": "sme", "question": "What is SME?", "summary": "SME (Scalable Matrix Extension) is an optional Arm architectural extension for matrix computation. It adds a streaming execution mode, scalable vector operations, and a two-dimensional register array called ZA. Outer-product instructions multiply values from two vectors and accumulate the results directly in ZA, which is useful for matrix multiplication and related kernels.", "answer": "**SME (Scalable Matrix Extension)** is an optional Arm architectural extension for matrix computation. It adds a streaming execution mode, scalable vector operations, and a two-dimensional register array called ZA. Outer-product instructions multiply values from two vectors and accumulate the results directly in ZA, which is useful for matrix multiplication and related kernels.\n\n## How it works\n\nSuppose one vector holds two values from a column of matrix A and another holds two values from a row of matrix B. One outer-product step updates four accumulators in ZA: each value from A is multiplied by each value from B, and each product is added to its corresponding result. Repeating that step for successive columns and rows builds a matrix product while keeping partial results in ZA.\n\nSME has three pieces that matter to a developer:\n\n| Feature | What it does |\n|---|---|\n| **Streaming SVE mode** | Enables the vector operations used by SME. Its streaming vector length (SVL) may differ from the processor's non-streaming SVE vector length. |\n| **ZA storage** | Holds partial matrix results as a scalable two-dimensional array, reducing the need to move each partial result through ordinary vector registers or memory. |\n| **Outer-product instructions** | Update many ZA elements from a pair of vectors in one instruction. |\n\nSME is most useful inside a compute kernel, such as matrix multiplication, convolution, or digital signal processing. The surrounding application can continue to use ordinary AArch64 and Neon code. Streaming mode has instruction restrictions and switching costs, so entering it for every small operation is not automatically beneficial.\n\n## How SME relates to SVE and SME2\n\nSVE and SVE2 provide scalable vector instructions. SME adds streaming vector execution and ZA matrix state; it does **not** imply that the processor also exposes non-streaming SVE or SVE2. SME2 builds on SME with multi-vector operations, multi-vector predication, and the ZT0 lookup-table register. A lookup instruction can expand packed 2-bit or 4-bit indices into values used by a matrix kernel; this is not the same as native 2-bit or 4-bit arithmetic.\n\nAn Armv9 label alone does not guarantee SME. On AArch64 Linux or Android, check `HWCAP2_SME` or `HWCAP2_SME2` before selecting a kernel. On macOS, check `hw.optional.arm.FEAT_SME` or `hw.optional.arm.FEAT_SME2`. Hardware support, operating-system exposure, and the library's selected kernel are separate questions.\n\n## Resources\n\n- [Arm: introduction to SME's programming model](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction)\n- [Arm: SME2 matrix multiplication Learning Path](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q10", "slug": "sve-instructions-only", "question": "What SVE instructions are only available in streaming mode, what are streaming-compatible, and which ones can only be used in non-streaming mode?", "summary": "An original SVE or SVE2 instruction is not streaming-only. SME and SME2 add instructions that require streaming mode; many existing SVE instructions work in either mode; some SVE and SVE2 forms require non-streaming mode. Classify the instruction form, including its addressing mode, rather than the mnemonic alone. For example, a contiguous LD1W and a gather LD1W have different rules.", "answer": "An original SVE or SVE2 instruction is not *streaming-only*. SME and SME2 add instructions that require streaming mode; many existing SVE instructions work in either mode; some SVE and SVE2 forms require non-streaming mode. Classify the **instruction form**, including its addressing mode, rather than the mnemonic alone. For example, a contiguous LD1W and a gather LD1W have different rules.\n\n| Category | Representative instructions or forms | Developer consequence |\n|---|---|---|\n| **Streaming-only** | SME outer products such as FMOPA and SMOPA; ZA tile-slice loads, stores, and moves; SME2 multi-vector matrix operations | Use them inside a streaming function or region. |\n| **Streaming-compatible** | Ordinary SVE arithmetic and logic; predicate operations; contiguous vector loads and stores; many permutations and reductions | They use the streaming vector length (SVL) in streaming mode and the non-streaming vector length (VL) otherwise. |\n| **Non-streaming-only without FA64** | SVE gather/scatter forms, first-fault loads and First Fault Register operations, some SVE2 crypto and in-vector matrix operations | Keep them outside a streaming region or use an alternative sequence. |\n\nThis is a guide to the common cases, not a complete opcode list. Availability also depends on whether the processor implements the instruction's separate feature requirements.\n\n## Streaming-only SME instructions\n\nSME outer-product instructions such as FMOPA, BFMOPA, SMOPA, and UMOPA accumulate into ZA. ZA tile-slice `LD1*` and `ST1*` forms transfer tile data, and SME2 adds multi-vector operations. These need streaming mode because their vector operands and tile operations use streaming state. The similar-looking ordinary SVE `LD1*` forms have their own rules.\n\nSome SME instructions are usable in either mode. RDSVL reads SVL even outside streaming mode, and SMSTART and SMSTOP control the mode. Whole-ZA LDR and STR can save or restore ZA when ZA is enabled; do not confuse them with the streaming-only tile-slice loads and stores.\n\n## SVE instructions that work in both modes\n\nCommon arithmetic such as ADD, FADD, and FMLA, predicate creation such as PTRUE and WHILELT, and contiguous LD1W/ST1W forms are streaming-compatible. Many logical operations, permutations, and reductions are too. Their vector width follows the current mode, so a loop that assumes a fixed element count is not portable between VL and SVL.\n\nOther useful examples include:\n\n| Kind of work | Representative compatible forms |\n|---|---|\n| Integer and floating-point arithmetic | SUB, MUL, FMUL, FMLS and predicated variants |\n| Predicate and loop control | PFALSE, WHILELO, predicate AND/ORR, PTEST |\n| Contiguous memory access | Ordinary vector LD1B/LD1H/LD1W/LD1D and matching `ST1*` forms |\n| Rearrangement and reduction | ZIP1, UZP1, REV, SADDV, FADDV |\n| Vector-length operations | CNTB, CNTW, INCB, ADDVL (using the length for the current mode) |\n\nFor a particular intrinsic, check its streaming classification in [ACLE](https://arm-software.github.io/acle/main/acle.html#controlling-the-use-of-streaming-mode). Some operations share a mnemonic but have encodings with different streaming behavior. Do not infer permission from a similar instruction name.\n\n## SVE instructions that require non-streaming mode\n\nWithout FEAT_SME_FA64, streaming mode excludes several instruction forms. Examples include vector-offset gather loads and scatter stores, first-fault loads (`LDFF1*`), operations on the First Fault Register such as RDFFR and SETFFR, and some cryptographic and in-vector matrix instructions. A gather LD1W can be excluded while a contiguous LD1W remains valid. Attempting to execute an unavailable form can raise an undefined-instruction exception, typically reported as SIGILL on Linux.\n\nOther examples to check before moving a loop into streaming mode are non-faulting loads (`LDNF1*`), ordered floating-point reduction (FADDA), vector compaction (COMPACT), and SVE2 in-vector matrix multiply (FMMLA, BFMMLA, and integer `*MMLA` forms). These are distinct from SME outer products, which accumulate into ZA. Even a shared mnemonic can hide different addressing forms, so use the instruction's specific streaming classification when porting hand-written assembly.\n\nFEAT_SME_FA64 relaxes streaming-mode restrictions for otherwise unavailable A64 instructions, but it is optional. It does not create other missing architectural features. On Linux, test `HWCAP2_SME_FA64` before relying on it. Code intended for a range of SME devices should keep such operations outside streaming mode unless they are essential to the kernel.\n\n## Express the mode contract in C or C++\n\nACLE function attributes let the compiler manage mode transitions at call boundaries and reject intrinsics used in the wrong mode:\n\n```c\n// Body executes in streaming mode; a non-streaming caller can call it.\nvoid matrix_kernel(float *a, float *b, float *c) __arm_streaming;\n\n// Body can execute in either mode, so it may use only compatible operations.\nvoid scalar_helper(int n) __arm_streaming_compatible;\n\n// Ordinary functions have a non-streaming interface by default.\nvoid gather_helper(const float *base, const unsigned *indices);\n```\n\nThese are declarations that illustrate the mode contract, not complete implementations. A function's mode attribute does not by itself enable ZA or guarantee that its target CPU has SME; compile for the appropriate feature and check runtime support before dispatch.\n\n## Resources\n\n- [Arm C Language Extensions: streaming-mode rules and function attributes](https://arm-software.github.io/acle/main/acle.html#controlling-the-use-of-streaming-mode)\n- [Arm SME Programmer's Guide: Streaming SVE mode](https://developer.arm.com/documentation/109246/0101/SME-Overview/Streaming-SVE-mode)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q11", "slug": "were-sme-sme2", "question": "Why Were SME and SME2 Introduced, and What Do They Add Beyond Neon and SVE2?", "summary": "SME (Scalable Matrix Extension) was introduced to make matrix-heavy kernels more efficient on Arm CPUs. Neon and SVE/SVE2 can compute matrix products using vector instructions, but SME adds scalable ZA tile storage and outer-product instructions that update many results at once. SME2 adds multi-vector operations, lookup instructions useful with packed low-bit weights, and bitwise outer products for binary neural networks. These extensions give kernel authors more ways to reduce instruction and data-movement costs; the benefit still depends on the workload and implementation.", "answer": "SME (Scalable Matrix Extension) was introduced to make matrix-heavy kernels more efficient on Arm CPUs. Neon and SVE/SVE2 can compute matrix products using vector instructions, but SME adds scalable ZA tile storage and outer-product instructions that update many results at once. SME2 adds multi-vector operations, lookup instructions useful with packed low-bit weights, and bitwise outer products for binary neural networks. These extensions give kernel authors more ways to reduce instruction and data-movement costs; the benefit still depends on the workload and implementation.\n\n## The Problem with Vectors for Matrix Workloads\n\nNeon and SVE/SVE2 are fundamentally 1D vector processing extensions. They operate on rows of data elements one vector at a time. While you can build matrix multiplication from repeated vector operations (dot products, FMA loops), the hardware has no native concept of a 2D tile. The programmer must manually manage the blocking, accumulation, transposition, and register pressure that matrix workloads demand.\n\nFor workloads dominated by matrix-matrix multiplication—such as Machine Learning, Large Language Models, computer vision, and HPC—vector units face significant bottlenecks:\n\n- **High instruction count**: Computing a matrix multiply with 1D vectors requires complex nested loops with many multiply-accumulate instructions.\n- **Excessive data movement**: Partial results must be shuffled between vector registers or spilled to memory; input data must be transposed explicitly.\n- **Register pressure**: Keeping accumulators, input vectors, and permutation state in a limited register file forces frequent spills.\n- **Instruction overhead**: An outer-product instruction updates a two-dimensional group of accumulators, while a vector FMA updates a one-dimensional group.\n\nSME uses a streaming vector mode with a subset of SVE2 instructions. It does not require the processor to expose non-streaming SVE or SVE2; check those features separately. Ordinary Neon code remains available outside streaming mode.\n\n## What SME Adds Beyond Neon and SVE2\n\n| Capability | Neon / SVE2 | SME |\n|---|---|---|\n| Register geometry | 1D vectors (128-bit fixed / scalable up to 2048-bit) | 2D tile array (\"ZA storage\") whose dimensions scale with SVL |\n| Core compute primitive | Dot product, FMA on vector lanes | Outer product of two SVE-length vectors accumulated directly into a tile |\n| Transposition | Requires explicit permute/shuffle instructions | On-the-fly transposition during load/store/extract of tile slices |\n| Execution mode | Standard A64 | Streaming SVE mode—a high-throughput data-processing mode |\n| Matrix state | None (programmer manages register blocking) | Dedicated architectural ZA state the OS can save/restore |\n| Results per instruction | Updates a vector of accumulators | Outer product updates a two-dimensional group of accumulators |\n\n### ZA Storage (Matrix Tile Array)\n\nSME introduces ZA storage: a dedicated two-dimensional architectural register array of SVL×SVL bytes (where SVL is the Streaming Vector Length). This is new register state, not a reuse of the Z registers. The ZA array can be accessed as:\n\n- **Tile slices** — horizontal rows (`ZA0H.S[n]`) or vertical columns (`ZA0V.S[n]`)\n- **Sub-tiles** — square sub-arrays typed by element width (e.g., `ZA0.S`, `ZA1.D`)\n- **Full ZA vectors** — for bulk save/restore operations\n\nThis allows accumulation of outer products directly into a 2D tile without consuming the ordinary vector register file.\n\n### Outer Product Instructions\n\nUnlike SVE2's dot-product instructions (which accumulate into a vector), SME's outer product instructions (FMOPA, SMOPA, UMOPA, etc.) take a column vector and a row vector, compute the entire outer product matrix, and accumulate it directly into a ZA tile:\n\n```text\nZA += column_vector × row_vector^T\n```\n\nIn a GEMM inner kernel, this can reduce instruction count and register pressure compared with building the same result from vector FMAs. Packing and memory traffic can still dominate the full operation.\n\n### Streaming SVE Mode\n\nSME introduces a separate execution mode controlled via PSTATE.SM. In streaming mode:\n\n- The core uses a Streaming Vector Length (SVL), which can differ from the normal SVE vector length (VL), if non-streaming SVE is implemented.\n- SME instructions (outer products, tile loads/stores) become available.\n- A subset of SVE2 instructions is available for supporting computation.\n- Hardware can be optimized specifically for high-throughput streaming workloads.\n\nStreaming mode is entered with `SMSTART SM` and exited with `SMSTOP SM`. A mode change invalidates Z and P register values across the boundary, so code must not carry live vector values through the transition.\n\n### On-the-Fly Transposition\n\nZA tile slices can be accessed horizontally or vertically during load, store, and extract operations. This can avoid some explicit shuffles when a kernel needs a different orientation, although it does not remove all packing or layout work from a matrix multiplication.\n\n## What SME2 Adds Beyond SME\n\nSME's initial scope was outer-product accumulation into ZA tiles. SME2 widens the applicability so that surrounding vector work does not bottleneck the matrix engine:\n\n### Multi-Vector Instructions\n\nOperations consume or produce groups of 2 or 4 Z registers simultaneously (e.g., `{Z0.D–Z1.D}` or `{Z0.S–Z3.S}`), increasing useful work per instruction. This includes:\n\n- **Multi-vector multiply-accumulate** — Z vectors as inputs, accumulating into ZA array vectors (including widening forms)\n- **Multi-vector loads and stores** — pre-load multiple vectors in parallel\n- **Multi-vector moves, permutations, and conversions** — efficiently transpose and shuffle data before feeding the matrix engine\n\n### ZT0 Lookup Table Register\n\nSME2 adds a dedicated 512-bit architectural register (ZT0) for table lookups. The LUTI (Lookup Table) instructions can decompress packed/quantized weights on-the-fly:\n\n- 2-bit or 4-bit compressed data → 8-bit, 16-bit, or 32-bit containers\n- The expanded values can represent supported 8-, 16-, or 32-bit formats, depending on the instruction and other implemented features. The 2-bit or 4-bit values are lookup indices, not arithmetic operands.\n\nThis can help a kernel use compressed or quantized neural-network weights without first expanding the entire weight matrix in memory.\n\n### Predicate-as-Counter\n\nSVE/SVE2 use bitmask predication where every bit corresponds to an active/inactive lane. SME2 introduces an alternative \"predicate-as-counter\" mechanism for multi-vector operations:\n\n- The predicate register encodes a count, an element size, and an inversion flag. It can describe an active prefix followed by inactive elements, or an inactive prefix followed by active elements, across a vector group.\n- This compact encoding supports loop-tail control in multi-vector loops.\n\n### Binary Neural Network Support\n\nSME2 adds bitwise exclusive-NOR population count outer product instructions for 1-bit binary neural networks, extending the architecture's ML capabilities to extremely compressed model representations.\n\n## Important Caveats\n\n- **SME does not replace Neon or SVE2.** Ordinary vector loops and irregular code may still be simpler or faster with Neon/SVE2. SME/SME2 complement existing extensions.\n- **Mode transition cost.** Entering and exiting streaming mode zeroes Z and P registers. Software must manage transitions carefully to avoid unnecessary overhead.\n- **Instruction restrictions.** Only a subset of SVE2 instructions is available in streaming SVE mode. Code must be restructured for SME-compatible paths.\n- **SVL and VL can differ.** The streaming vector length can be wider or narrower than the normal SVE vector length on the same implementation. Code must remain vector-length agnostic.\n- **Large context-save state.** ZA can be up to 64 KB at a 2048-bit SVL. OS support must preserve the architectural state across context switches.\n- **OS and compiler support required.** SME requires OS awareness for ZA state management and compiler support for function attributes (`__arm_streaming`, `__arm_inout(\"za\")`, etc.).\n\n## Resources\n\n- [Arm: Neon, SVE, and SME matrix multiplication compared](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared)\n- [Arm SME Programmer's Guide: SME and SME2](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2)\n- [Arm: how SME2 lookup instructions expand low-bit weights](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/luti/02_luti_and_sme2/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q12", "slug": "relationship-neon-sve2", "question": "What is the relationship between Neon, SVE2, SME, and SME2? Does SME2 replace them or work alongside them?", "summary": "They work alongside one another. Neon (Advanced SIMD) provides fixed-width vectors for general-purpose code. SVE2 provides scalable vectors for workloads such as signal processing and vectorized loops. SME adds streaming vector execution and ZA tile storage for matrix-oriented kernels; SME2 extends SME with multi-vector operations and a lookup-table register. A program can use different paths for different operations, but must check which optional extensions a device exposes.", "answer": "They work alongside one another. Neon (Advanced SIMD) provides fixed-width vectors for general-purpose code. SVE2 provides scalable vectors for workloads such as signal processing and vectorized loops. SME adds streaming vector execution and ZA tile storage for matrix-oriented kernels; SME2 extends SME with multi-vector operations and a lookup-table register. A program can use different paths for different operations, but must check which optional extensions a device exposes.\n\n## How the features relate\n\n| Extension | Main contribution | Typical use |\n|---|---|---|\n| **Neon** | Fixed-width 128-bit vector operations | Broad AArch64 SIMD code, media, signal processing |\n| **SVE2** | Scalable vector operations and predication | Vector-length-agnostic loops, DSP, media |\n| **SME** | Streaming SVE mode, ZA matrix state, outer products | Matrix multiplication and related compute kernels |\n| **SME2** | Multi-vector operations, predication, and the ZT0 lookup-table register | More efficient matrix-kernel data movement and packed-weight handling |\n\nSME2 includes SME, and SVE2 includes SVE. **SME2 does not imply non-streaming SVE2.** Streaming SVE provides a subset of SVE2-style operations in streaming mode, but non-streaming SVE2 is a separate feature. This distinction matters on devices that expose SME2 without exposing non-streaming SVE2.\n\n## Where execution happens\n\nMost application code runs in non-streaming mode. Neon works there, as does SVE2 when the device implements it. A matrix kernel can enter streaming mode to use SME or SME2 instructions and ZA, then return to non-streaming mode. Streaming SVE operations use the streaming vector length (SVL), which may differ from the non-streaming SVE vector length.\n\nA separate state bit enables ZA. ZA state can be live across a mode transition, but Z and P vector values cannot simply be carried through it. ACLE's `__arm_streaming` and `__arm_locally_streaming` attributes help the compiler manage transitions while preserving the function's mode contract.\n\nFor example, an image pipeline might use Neon or SVE2 for pixel conversion, call an SME2 matrix kernel for a model layer, and then return to ordinary code. Whether that is faster depends on how much work the kernel does relative to packing, memory traffic, and mode-switch costs. It is also valid to use Neon alone on a device with SME2.\n\n## Dispatch each optional feature separately\n\nOn AArch64 Linux and Android, `HWCAP2_SVE2`, `HWCAP2_SME`, and `HWCAP2_SME2` report the relevant OS-exposed features. On macOS, use the corresponding `hw.optional.arm.FEAT_*` queries. Keep a baseline path and select a specialized kernel only when both the device and the implementation support it. A library such as KleidiAI may perform this dispatch internally, but its choice also depends on the operation, data type, and shape.\n\n## Resources\n\n- [Arm SME Programmer's Guide: SME and SME2](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2)\n- [Arm: Neon, SVE, and SME matrix multiplication compared](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared)\n- [Arm ACLE: SME streaming-mode function attributes](https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q13", "slug": "sme2-backward-compatible", "question": "Is SME2 backward-compatible with SME, SVE2, and Neon?", "summary": "SME2 includes SME, so code using base SME instructions can run on a processor and operating system that expose SME2. It does not imply non-streaming SVE2. Neon remains part of the AArch64 software baseline, but Neon instructions are not generally usable inside an SME streaming region. These are different kinds of compatibility: instruction-set support, execution-mode rules, and operating-system exposure.", "answer": "**SME2 includes SME**, so code using base SME instructions can run on a processor and operating system that expose SME2. It does **not** imply non-streaming SVE2. Neon remains part of the AArch64 software baseline, but Neon instructions are not generally usable inside an SME streaming region. These are different kinds of compatibility: instruction-set support, execution-mode rules, and operating-system exposure.\n\n| Existing code | On an SME2-capable AArch64 device | Important condition |\n|---|---|---|\n| Neon | Runs in ordinary non-streaming mode | Leave streaming mode before calling Neon code unless a relevant optional feature changes the rule. |\n| SVE2 | Runs **if SVE2 is also exposed** | SME2 alone does not guarantee non-streaming SVE2. |\n| SME | Runs when the OS exposes SME2 | SME2 extends SME and preserves its instructions and ZA state. |\n| SME2 | Runs when the OS exposes SME2 | Select this path only after a runtime feature check. |\n\nThe feature relationship is SME2 → SME and, separately, SVE2 → SVE. It is not one chain from SME2 through SVE2 to Neon. A device can expose streaming SVE operations for SME2 while lacking non-streaming SVE2.\n\n## Why the execution mode matters\n\nSME instructions such as outer products operate in streaming mode with ZA enabled. Many SVE-style arithmetic operations also work there, using the streaming vector length (SVL). Other SVE2 forms, including common gather and first-fault operations, require non-streaming mode unless the optional FEAT_SME_FA64 feature is available. Neon code normally runs outside streaming mode.\n\nSwitching modes invalidates live Z and P vector values across the boundary. The SME calling convention and ACLE function attributes let a compiler manage transitions while preserving the function's contract; they do not make arbitrary Neon or SVE2 instructions legal inside a streaming function.\n\n## Detect the paths you actually provide\n\nOn AArch64 Linux or Android, use the OS-provided HWCAP flags. This complete example reports the relevant features; it does not execute specialized instructions:\n\n```c\n#include \n#include \n#include \n\nint main(void) {\n unsigned long hwcap = getauxval(AT_HWCAP);\n unsigned long hwcap2 = getauxval(AT_HWCAP2);\n\n printf(\"Neon: %s\\n\", (hwcap & HWCAP_ASIMD) ? \"yes\" : \"no\");\n printf(\"SVE2: %s\\n\", (hwcap2 & HWCAP2_SVE2) ? \"yes\" : \"no\");\n printf(\"SME: %s\\n\", (hwcap2 & HWCAP2_SME) ? \"yes\" : \"no\");\n printf(\"SME2: %s\\n\", (hwcap2 & HWCAP2_SME2) ? \"yes\" : \"no\");\n return 0;\n}\n```\n\nOn macOS, use the corresponding `hw.optional.arm.FEAT_*` queries. Compile specialized functions or translation units for their target features while keeping the entry point and fallback code compatible with the oldest device you support. Feature presence only makes a path eligible; the library or application's dispatch must still select it.\n\n## Resources\n\n- [Arm ACLE: AArch64 feature dependencies](https://arm-software.github.io/acle/main/acle.html#dependencies)\n- [Linux arm64 SME userspace interface](https://docs.kernel.org/arch/arm64/sme.html)\n- [Arm SME Programmer's Guide: streaming mode](https://developer.arm.com/documentation/109246/0101/SME-Overview/Streaming-SVE-mode)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s01-q14", "slug": "one-application-run", "question": "Can one application run on both SME2-capable and older Neon/SVE2 devices, and how is runtime fallback handled?", "summary": "Yes. For one AArch64 platform and ABI, a binary can contain a baseline implementation plus separately compiled SVE2 and SME2 implementations. At runtime it checks the features that the operating system exposes and calls an eligible implementation. The entry point and baseline code must be built for the oldest device the binary supports.", "answer": "Yes. For one AArch64 platform and ABI, a binary can contain a baseline implementation plus separately compiled SVE2 and SME2 implementations. At runtime it checks the features that the operating system exposes and calls an eligible implementation. The entry point and baseline code must be built for the oldest device the binary supports.\n\n## Detect, then dispatch\n\nOn AArch64 Linux and Android, `getauxval(AT_HWCAP2)` reports whether the OS exposes SVE2 and SME2. Check them independently: SME2 does not imply non-streaming SVE2. Neon is the usual AArch64 baseline. On macOS, use `sysctlbyname` with keys such as `hw.optional.arm.FEAT_SME2`; the Linux HWCAP interface does not apply there.\n\nThis Linux example shows the dispatch contract. The three `gemm_*` functions are implementations supplied in separate source files with identical signatures:\n\n```c\n#include \n#include \n\ntypedef void (*gemm_fn)(const float *, const float *, float *,\n int, int, int);\n\nextern void gemm_neon(const float *, const float *, float *,\n int, int, int);\nextern void gemm_sve2(const float *, const float *, float *,\n int, int, int);\nextern void gemm_sme2(const float *, const float *, float *,\n int, int, int);\n\nstatic gemm_fn select_gemm(void) {\n unsigned long features = getauxval(AT_HWCAP2);\n if (features & HWCAP2_SME2) return gemm_sme2;\n if (features & HWCAP2_SVE2) return gemm_sve2;\n return gemm_neon;\n}\n```\n\nCall `select_gemm()` once during initialization and retain the returned pointer. This is a dispatch example, not a complete matrix multiplication program: each kernel must define its own layout, dimensions, and numerical behavior consistently.\n\nBuild the baseline and specialized source files separately. For example, with a compiler that supports SME2:\n\n```bash\nclang -O2 -march=armv8-a -c main.c gemm_neon.c\nclang -O2 -march=armv9-a+sve2 -c gemm_sve2.c\nclang -O2 -march=armv9-a+sme2 -c gemm_sme2.c\nclang -o my_app main.o gemm_neon.o gemm_sve2.o gemm_sme2.o\n```\n\nKeep SME2 instructions inside the guarded kernel. Building the entire binary for SME2 can let the compiler put unsupported instructions in unguarded code, causing an illegal-instruction fault on older devices. Link-time optimization also needs checking: inspect the final binary or disable cross-inlining between the baseline and specialized code if the toolchain does not preserve the boundary.\n\n## Other ways to dispatch\n\n**Function multiversioning** lets a supported compiler generate and choose versions of one function using `target_version` or `target_clones`. A `default` version supplies the fallback, and feature selection is generally fixed for the process. This is useful when the versions share an interface; exact compiler, linker, and streaming-mode requirements should be checked for the target platform. The [Arm FMV Learning Path](https://learn.arm.com/learning-paths/cross-platform/function-multiversioning/semantics/) shows the syntax and selection rules.\n\n**Library dispatch** can hide this choice from application code. A framework may use KleidiAI or another optimized backend to select an SME2 microkernel when one exists for the operator, data type, and shape. Hardware detection alone does not prove that a particular call used SME2; inspect backend logs or profile the selected kernel when that matters.\n\nAn SME2 kernel enters streaming mode and manages ZA according to the SME calling convention. That mode change is local to the specialized path; callers and fallback kernels can remain ordinary non-streaming code. Test the SME2 path and every fallback on suitable hardware or emulation, and verify both correctness and the kernel actually selected.\n\n## Resources\n\n- [Linux arm64 ELF hardware capabilities](https://docs.kernel.org/arch/arm64/elf_hwcaps.html)\n- [Arm ACLE: streaming-mode function attributes](https://arm-software.github.io/acle/main/acle.html#sme-keyword-attributes-related-to-streaming-mode)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q01", "slug": "types-workloads-see", "question": "What types of workloads see the biggest performance gains from SME compared to Neon or SVE?", "summary": "Dense, regular matrix work is the strongest candidate, especially GEMM and operations implemented as blocked matrix multiplication. SME's outer-product instructions update many ZA accumulators from two vectors while keeping partial results in the tile. That can reduce instruction and register-movement overhead compared with a vector kernel. It does not guarantee a particular speedup or turn a memory-bound workload into a compute-bound one.", "answer": "Dense, regular matrix work is the strongest candidate, especially GEMM and operations implemented as blocked matrix multiplication. SME's outer-product instructions update many ZA accumulators from two vectors while keeping partial results in the tile. That can reduce instruction and register-movement overhead compared with a vector kernel. It does **not** guarantee a particular speedup or turn a memory-bound workload into a compute-bound one.\n\n| Workload | Why SME may help | What can limit the gain |\n|---|---|---|\n| Large GEMM and BLAS-3 kernels | Reuse input panels across many outer-product updates while holding an output tile in ZA | Packing, memory bandwidth, competing tuned kernels, and tile occupancy |\n| Convolution lowered to GEMM or iGEMM | Regular multiply-accumulate work with reusable input and filter data | Layout conversion, small channels or batches, unsupported shapes |\n| Fully connected and attention matrix products | Dense arithmetic can use SME or SME2 microkernels | Batch-1 matrix-vector shapes and weight bandwidth |\n| Blocked scientific and signal-processing kernels | Repeated dense matrix updates can reuse ZA | FP64 or other optional feature support, irregular portions of the algorithm |\n\nThe application domain alone does not decide. A convolution can map well to SME when the backend uses an eligible iGEMM kernel, while a small fully connected layer may spend more time moving weights than computing. Scientific solvers and beamforming can benefit where they spend meaningful time in dense blocked operations; their surrounding control flow and sparse work may not.\n\n## Why the outer product matters\n\nFor a 512-bit streaming vector length, an FP32 vector contains 16 lanes. One FP32 outer-product update can contribute to a 16-by-16 group of accumulators in ZA. This says something about **work per instruction**, not cycles per instruction, sustained throughput, or end-to-end speedup. A tuned Neon or SVE kernel also reuses loaded values, so a comparison must use similarly optimized implementations.\n\nThe [Arm SME2 outer-product Learning Path](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product/) shows how the two input vectors update a tile. The [SqueezeSAM case study](https://pytorch.org/blog/accelerating-on-device-ml-inference-with-executorch-and-arm-sme2/) illustrates a measured case: operator-level GEMM and convolution improved substantially on one Android device, while full-model and four-core gains were smaller because data movement and shared resources still mattered.\n\n## When another path is a better fit\n\nElement-wise loops, short reductions, sparse or gather-heavy work, and tiny one-off matrices often lack enough reusable tile work to offset setup and streaming-mode costs. Neon or SVE may be simpler and faster for those cases. Some operations also need instructions or data types that a particular SME implementation does not provide.\n\nTo evaluate a workload, profile first. Find the operators consuming absolute time, then check their exact shapes, data types, quantization and packing contracts against an available SME kernel. Compare SME enabled and disabled on the target device with the same thread count and power conditions. Measure end-to-end latency or energy as well as isolated kernel throughput.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q02", "slug": "sme-only-useful", "question": "Is SME only useful for machine learning inference, or can it accelerate other domains like signal processing or scientific computing?", "summary": "No. SME accelerates matrix-style computation, regardless of the application's domain. Outer-product instructions update ZA tiles from pairs of vectors, so the best candidates are dense, repeated multiply-accumulate operations with useful data reuse. Machine learning is one example; numerical solvers, communications, and some media kernels use the same pattern.", "answer": "No. SME accelerates matrix-style computation, regardless of the application's domain. Outer-product instructions update ZA tiles from pairs of vectors, so the best candidates are dense, repeated multiply-accumulate operations with useful data reuse. Machine learning is one example; numerical solvers, communications, and some media kernels use the same pattern.\n\n| Domain | Plausible SME work | Important qualification |\n|---|---|---|\n| Scientific computing | GEMM and blocked matrix updates inside LU, QR, Cholesky, or implicit solvers | The surrounding solver may be sparse, memory-bound, or dominated by other routines. FP64 outer products require the optional FEAT_SME_F64F64 feature. |\n| Communications and radar | Beamforming weights, channel-matrix operations, correlation and covariance | Gain depends on matrix dimensions, batch size, and how often data is reused. |\n| Signal and image processing | Filter banks, larger convolutions, block transforms | Small filters and simple element-wise stages may favor Neon or SVE. |\n| Machine learning | Fully connected layers, attention products, convolutions lowered to GEMM | Operator, shape, type, packing, and backend coverage determine whether SME is actually selected. |\n\n## A concrete route outside ML\n\nConsider a scientific application that calls SGEMM or DGEMM repeatedly during a blocked solver. Most of the solver can remain unchanged while an optimized BLAS library selects an SME kernel for those calls. [Arm Performance Libraries 26.01](https://developer.arm.com/community/arm-community-blogs/b/servers-and-cloud-computing-blog/posts/what-s-new-in-arm-performance-libraries-26-01) documents SME implementations for SGEMM and DGEMM on compatible Apple M-series Macs. This is library- and platform-specific evidence, not a promise that every BLAS call on every Arm processor uses SME.\n\nA communications example is computing a bank of beamforming outputs from many input channels. If coefficients and samples can be arranged as a dense matrix product, an SME kernel can accumulate many outputs in ZA. If the workload instead processes only a few samples at a time or gathers scattered data, packing and mode-switch costs may outweigh the matrix benefit.\n\n## What to check before adopting it\n\nSME can access ZA tile slices in horizontal and vertical orientations, which may reduce some explicit rearrangement. It does not eliminate all layout conversion or transposition cost. Likewise, an FFT or a small color transform is not automatically a good SME workload merely because it can be written mathematically as a matrix product; the practical algorithm may be faster with Neon or SVE.\n\nProfile the existing implementation, identify the dense matrix kernels that dominate elapsed time, and verify support for their precision and shapes. Compare an SME-enabled implementation against a tuned baseline on the target device. Include packing, memory movement, mode transitions, and full-application latency in that comparison.\n\n## Resources\n\n- [Arm SME Programmer's Guide: matrix operations and supported types](https://developer.arm.com/documentation/109246/0101/SME-Overview/SME-and-SME2)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q03", "slug": "detect-if-workload", "question": "How do I determine if my workload has enough matrix multiplication to justify adopting SME?", "summary": "To decide whether SME is worthwhile, measure where the application spends time, check whether its costly operators can use suitable SME kernels, and compare end-to-end performance on the target device. There is no useful universal threshold for the percentage of matrix multiplication.", "answer": "To decide whether SME is worthwhile, measure where the application spends time, check whether its costly operators can use suitable SME kernels, and compare end-to-end performance on the target device. There is no useful universal threshold for the percentage of matrix multiplication.\n\n## Step 1: Profile Your Workload to Identify the Hotspot\n\nBefore evaluating SME, measure what fraction of total execution time is spent in operations that can reach an SME implementation. These may include matrix multiplication, supported convolutions, attention projections, and dense linear algebra. The operator name alone does not establish that an SME kernel exists for its shape and data type.\n\nUse profiling tools appropriate for Arm:\n\n- **Arm Streamline** or **perf** with PMU counters to identify CPU-time hotspots\n- **Top-down microarchitecture analysis** (using Arm's topdown-tool or Arm Performance Studio) to classify whether you are backend-bound on compute vs. memory\n- **Annotation channels** in your code to time specific operator types (e.g., what llama.cpp does to separately measure GEMM vs. GEMV time)\n- **APX code_hotspots recipe** for quick profiling without instrumenting source\n\nRecord the fraction p of total runtime spent in these matrix-style kernels. If matrix operations don't dominate your profile, gains from SME will be diluted regardless of how fast it makes the kernel.\n\n### Practical operator-level workflow for ML models\n\nProfile the **lowered model that actually runs**, not only the original PyTorch, ONNX, or LiteRT graph. Delegation, fusion, layout conversion, and quantization can change both the operator set and the execution cost.\n\n1. Run a timing-only baseline with SME2 disabled and collect per-operator latency.\n2. Run the same model, inputs, thread count, affinity, and power mode with SME2 enabled.\n3. For each expensive operator, record its lowered type, M×N×K or convolution dimensions, data types, quantization format, and whether it was delegated to XNNPACK, MLAS, or another supported backend.\n4. Use framework or backend tracing in a separate, trace-enabled build to confirm whether the expected SME2 microkernel was selected. Do not use trace-enabled runs for timing.\n5. Rank operators by **absolute time saved**, then inspect data movement, packing, layout conversion, and non-delegated work that becomes visible after matrix kernels accelerate.\n\nThe [SqueezeSAM ExecuTorch case study](https://pytorch.org/blog/accelerating-on-device-ml-inference-with-executorch-and-arm-sme2/) demonstrates this workflow: ETDump operator profiling separates convolution/GEMM time from data movement and other overhead, while the accompanying [sme-executorch-profiling repository](https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling) provides reproducible SME2-on, SME2-off, timing, and trace configurations.\n\n## Step 2: Check Data Reuse and Matrix Shapes\n\nSME outer-product instructions multiply elements from two input vectors and accumulate many results in the ZA tile. This can reduce instruction and data-movement overhead when a kernel reuses input data effectively. A well-tuned Neon or SVE kernel also reuses data, so a fixed multiply-to-load ratio does not predict the speedup. Inspect the actual kernel and its memory traffic.\n\nPay particular attention to **skinny output matrices**. Shapes such as 6×N or M×6 may leave much of an SME output tile unused even when the other dimension and K are large. Likewise, batch-1 inference often reduces a nominal matrix multiplication to GEMV (M=1), where weight movement can dominate compute. These operations may still have specialized SME2 kernels, but profitability depends on the exact shape, data type, packing cost, streaming vector length, and implementation. There is no universal minimum such as 64×64.\n\nIf your workload is already memory-bound (spending most time waiting on cache misses or DRAM bandwidth), wider compute units alone won't help. Fix the memory bottleneck first, or choose narrower data types (INT8, BF16) to improve effective bandwidth.\n\n## Step 3: Evaluate Streaming Mode Transition Costs\n\nSME requires entering Streaming SVE mode via the SMSTART instruction (and exiting with SMSTOP). These transitions carry costs:\n\n- Live Z and P register values cannot be carried unchanged across a mode transition; the compiler or assembly code must respect the streaming-mode rules\n- The Streaming Vector Length (SVL) may differ from the normal SVE vector length\n- ZA tile state may need to be saved/restored across function calls\n- Only streaming-compatible instructions can execute while streaming mode is active; calls to incompatible code require a mode transition\n\nThis overhead means you need the matrix work to be **large enough per invocation** to amortize the transition cost. Patterns that hurt:\n\n- Frequent small matmuls interleaved with non-streaming code\n- Scattered matrix operations that prevent hoisting SMSTART/SMSTOP to outer loops\n- Library calls that implicitly switch modes\n\nPatterns that help:\n\n- Batching multiple outer-product accumulations before switching back\n- Structuring code so all streaming work is grouped in a single call or loop\n- Using `__arm_streaming` function attributes to let the compiler manage transitions\n\n## Step 4: Apply Amdahl's Law\n\nEven a large kernel speedup affects only the time spent in that kernel. If an operator takes 30% of the original runtime and becomes twice as fast, the whole application becomes about 1.18× faster before accounting for transitions, packing, or layout conversion. A smaller hotspot can still justify SME when an existing library supplies the kernel; a larger one may not help if its shapes or formats miss the available implementation.\n\n## Step 5: Benchmark on Real Hardware\n\nDo not rely solely on theoretical analysis. Compare equivalent tuned implementations on actual SME-capable hardware:\n\n1. **Baseline**: Your existing implementation under representative conditions\n2. **SME candidate**: SME-enabled library (e.g., KleidiAI, Arm Performance Libraries) or hand-written kernel\n3. **End-to-end workload**: Measure the full application, not only isolated GEMM throughput\n\nMeasure latency, throughput, CPU utilization, and energy. Note that emulators like Arm FVPs provide instruction counts for functional validation but **do not model memory bandwidth, cache behavior, or latency** — they are not suitable for performance benchmarking.\n\n## Make the Decision from Measured Return on Investment\n\nDo not use a universal percentage or matrix-size cutoff. Adopt SME when the measured benefit on the target device justifies the integration and maintenance cost:\n\n1. Identify the operators responsible for meaningful absolute runtime with representative inputs.\n2. Confirm that the complete operator signature—shape, layout, types, quantization, and packing contract—can reach an SME implementation.\n3. Measure SME2-on versus SME2-off on the actual device using identical runtime conditions.\n4. Calculate the end-to-end latency, throughput, or energy saving rather than relying only on microkernel speedup.\n5. Compare that saving with the cost of integration, validation, fallback support, and ongoing maintenance.\n\nThe adoption bar should be lower when a framework already dispatches to KleidiAI automatically and higher when you must write and maintain a custom kernel. Product constraints also differ: a 10% latency reduction may be decisive for an interactive deadline or energy budget, while a larger gain may still be irrelevant outside the user-visible critical path.\n\n## Resources\n\n- [Arm's SME2 benchmarking guide](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-benchmarking/)\n- [Streaming mode and ZA state](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q04", "slug": "expected-speedup-sme2", "question": "What is the expected speedup from SME2 for common neural network operations like convolution and fully-connected layers?", "summary": "There is no single expected SME2 speedup for convolution, fully connected layers, or an entire model. In one controlled SqueezeSAM deployment, Arm and PyTorch measured 3.4–4.1× faster GEMM, 4.4–9.0× faster convolution, and 1.83–3.90× faster single-core end-to-end inference, depending on precision. Those figures are useful evidence of what SME2 can achieve, but they are not general forecasts: operator shapes, data types, kernel coverage, thread count, hardware topology, memory behavior, and the comparison baseline all matter.", "answer": "There is no single expected SME2 speedup for convolution, fully connected layers, or an entire model. In one controlled SqueezeSAM deployment, Arm and PyTorch measured **3.4–4.1× faster GEMM**, **4.4–9.0× faster convolution**, and **1.83–3.90× faster single-core end-to-end inference**, depending on precision. Those figures are useful evidence of what SME2 can achieve, but they are not general forecasts: operator shapes, data types, kernel coverage, thread count, hardware topology, memory behavior, and the comparison baseline all matter.\n\n## Published SqueezeSAM Case Study\n\nThe measurements below come from a specific [ExecuTorch and Arm SME2 SqueezeSAM case study](https://pytorch.org/blog/accelerating-on-device-ml-inference-with-executorch-and-arm-sme2/):\n\n- **Device:** vivo X300 Android smartphone with an SME2-enabled Arm CPU\n- **Model:** SqueezeSAM, a convolution-heavy interactive image-segmentation model\n- **Runtime path:** ExecuTorch → XNNPACK → KleidiAI\n- **Comparison:** the same model and device with SME2 enabled versus disabled\n- **Power setting:** normal/default mobile mode\n\n### Operator-level results on one CPU core\n\n| Operation | Data Type | Measured Speedup (SME2 vs Non-SME2) |\n|-----------|-----------|--------------------------------------|\n| Fully-connected (GEMM) | INT8 | ~3.4× |\n| Fully-connected (GEMM) | FP16 | ~4.1× |\n| Convolution (iGEMM) | INT8 | ~4.4× |\n| Convolution (iGEMM) | FP16 | ~9.0× |\n\nThese are category-level timings from this model, not universal properties of every GEMM or convolution shape.\n\n### End-to-end results\n\n| Precision | CPU cores | SME2 off | SME2 on | Speedup |\n|-----------|----------:|---------:|--------:|--------:|\n| INT8 | 1 | 556 ms | 304 ms | 1.83× |\n| FP16 | 1 | 1,163 ms | 298 ms | 3.90× |\n| INT8 | 4 | 195 ms | 180 ms | 1.08× |\n| FP16 | 4 | 374 ms | 193 ms | 1.94× |\n\nThe gap between operator and end-to-end speedup illustrates Amdahl's law: after convolution and GEMM accelerate, data movement, layout conversion, non-delegated operators, and framework overhead account for more of the remaining latency. The weaker four-core ratios also show why single-thread results must not be extrapolated linearly when SME2 compute resources or other system resources are shared.\n\n## Why SME2 Achieves These Gains\n\nSME outer-product instructions accumulate many products into a two-dimensional ZA tile. Keeping partial sums there can reduce repeated movement through general vector registers and memory. SME2 also adds multi-vector operations that can help feed suitable kernels. These mechanisms explain the opportunity, but instruction counts alone do not establish a speedup: execution throughput, packing, cache traffic, and the quality of the comparison kernel also matter.\n\n### Why the operator results differ\n\nIn this SqueezeSAM configuration, convolution accounted for much more baseline time and achieved a larger speedup than GEMM. That does not establish a general rule that convolution always benefits more. A convolution lowered to a supported iGEMM path can have good data reuse, while a fully connected layer with small M or batch size can become bandwidth- or packing-bound; different shapes can reverse the outcome.\n\n### Why FP16 shows the larger ratio here\n\nFP16 started from a substantially slower non-SME2 baseline in this experiment. With SME2 enabled, FP16 and INT8 both reached approximately 300 ms on one core, producing a larger relative ratio for FP16. This result demonstrates that SME2 can make FP16 practical for this model; it does not mean FP16 will generally outperform INT8 or always receive a larger speedup.\n\n## Factors That Affect Actual Speedup\n\n| Factor | Impact |\n|--------|--------|\n| Matrix dimensions | Shapes that use the chosen kernel's tiles effectively tend to do better; small dimensions may still have specialized kernels |\n| Batch size | Changes whether a layer behaves like matrix multiplication or a bandwidth-sensitive matrix-vector operation |\n| Data type | Kernel availability and the quality of the baseline differ by precision; packed 4-bit formats also need unpacking or lookup work |\n| Memory bandwidth | Bandwidth-limited workloads (small batch, large models) see reduced gains |\n| Software optimization | Depends on an optimized kernel and successful dispatch to it |\n| Tile utilization | Matrices that don't evenly fill ZA tiles leave compute unused |\n| Streaming mode overhead | Frequent SMSTART/SMSTOP transitions reduce effective throughput |\n\n## How to Access These Speedups\n\nThe case study used **ExecuTorch → XNNPACK → KleidiAI** for supported operator shapes and types. Other runtimes can expose SME2 through optimized backends, but installation alone does not guarantee that a particular layer selects an SME2 kernel. Check the framework and backend versions, supported operator configurations, and runtime dispatch. Use trace or instruction profiling to confirm the selected kernel, then run timing in a separate measurement build.\n\n## How to Use Published Numbers\n\nUse published speedups as a sanity check and a source of hypotheses, not as a performance commitment. A defensible estimate should always state:\n\n- the device and number of SME compute units;\n- model, operator shapes, and precision;\n- framework, backend, and KleidiAI/XNNPACK versions;\n- thread count, affinity, and power mode;\n- whether the comparison disables only SME2 or changes other kernels or build settings;\n- both operator-level and end-to-end latency.\n\nFor a new model, run an SME2-on/SME2-off A/B test under identical conditions and profile individual operators. Report a range only after measuring representative shapes and inputs on the intended device fleet.\n\n## Resources\n\n- [Benchmarking a LiteRT model with KleidiAI SME2 kernels](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/4-benchmark/)\n- [LiteRT supported operator configurations](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-model/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q05", "slug": "processors-currently-support", "question": "Which Arm Processors Currently Support SME and SME2?", "summary": "As of September 2026, documented consumer processor families with SME2 include Apple's A18/A19 and M4/M5 generations and Arm C1-based clusters, such as MediaTek Dimensity 9500 and Samsung Exynos 2600. Arm has also announced a C2 cluster with two SME2 units. SME2 includes SME, but neither Armv9 nor SVE2 alone guarantees SME. Check the exact device and OS before selecting an SME code path.", "answer": "As of September 2026, documented consumer processor families with SME2 include Apple's A18/A19 and M4/M5 generations and Arm C1-based clusters, such as MediaTek Dimensity 9500 and Samsung Exynos 2600. Arm has also announced a C2 cluster with two SME2 units. SME2 includes SME, but neither Armv9 nor SVE2 alone guarantees SME. Check the exact device and OS before selecting an SME code path.\n\n## Processor Families and Implementations\n\n| Processor or CPU IP | SME status | Where it appears |\n|---|---|---|\n| Apple A18/A19 | SME2 in supported models | iPhone 16 and 17 generations |\n| Apple M4/M5 | SME2 in supported models | iPad Pro and Mac models |\n| Arm C1 cluster with C1-SME2 unit | SME2 when the SoC vendor includes the unit | Dimensity 9500 and Exynos 2600 devices |\n| Arm C2 cluster | SME2 in Arm's announced design | Processor IP; confirm support in a shipping device |\n| Earlier Cortex-X4/X925 and Neoverse V2/V3/N3 cores | No documented SME support | Armv9 and SVE2 are insufficient indicators |\n\nThe C1 and C2 SME2 units are **cluster resources**; do not infer a separate matrix engine per core. An announced CPU design also differs from a shipping device whose OS exposes the feature. Apple's SME2 implementation does not imply support for ordinary, non-streaming SVE2 instructions.\n\nArm's [device list and setup guide](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/#devices) gives current hardware examples. Arm's [C2 cluster announcement](https://newsroom.arm.com/blog/arm-css-for-mobile-2-and-c2-cpu-cluster) describes the newer IP.\n\n## Check the Feature on the Target Device\n\nOn Linux or Android, query the kernel's `AT_HWCAP2` bits. This tests what the OS exposes to your process:\n\n```c\n#include \n#include \n#include \n\nint main(void) {\n unsigned long features = getauxval(AT_HWCAP2);\n printf(\"SME: %s\\n\", (features & HWCAP2_SME) ? \"yes\" : \"no\");\n printf(\"SME2: %s\\n\", (features & HWCAP2_SME2) ? \"yes\" : \"no\");\n return 0;\n}\n```\n\nOn macOS, run `sysctl hw.optional.arm.FEAT_SME2`; a result of `1` reports OS-exposed SME2 support. A compiler macro such as `__ARM_FEATURE_SME2` says what a translation unit was built to use; it does not detect the current device. Arm's [FVP setup](https://developer.arm.com/documentation/109246/0101/Toolchains-and-model-support/How-to-run-an-SME-application) can validate functionality without SME2 hardware, but it cannot predict device performance.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q06", "slug": "list-libraries-include", "question": "Is there a list of libraries which include SME support?", "summary": "There is no authoritative registry of SME-enabled libraries. As of September 2026, the projects below provide useful starting points. Distinguish a library that contains SME instructions from a framework that may reach them through a backend; even then, support depends on the exact build, operator, type, and shape.", "answer": "There is no authoritative registry of SME-enabled libraries. As of September 2026, the projects below provide useful starting points. Distinguish a library that contains SME instructions from a framework that may reach them through a backend; even then, support depends on the exact build, operator, type, and shape.\n\n## Libraries with Direct SME/SME2 Implementations\n\nThese libraries contain native SME or SME2 assembly/intrinsic code:\n\n| Library | SME Level | Description |\n|---------|-----------|-------------|\n| **KleidiAI** | SME, SME2 | Open-source micro-kernels for AI workloads on Arm CPUs. Provides optimized GEMM, quantized matmul, and packing routines. Integrated into multiple major frameworks. |\n| **KleidiCV** | SME, SME2 | Computer vision kernels with runtime selection among available backends; coverage is operation-specific. |\n| **Arm Compute Library** | SME, SME2 | Native GEMM kernels plus bundled KleidiAI kernels; coverage depends on release, build, data type, and operator. |\n| **oneDNN** | SME in selected builds | AArch64 JIT path for selected matrix multiplication cases; check the installed version and selected primitive. |\n| **OpenBLAS** | SME | Includes an `ARMV9SME` build target with handwritten SME kernels for standard BLAS routines (e.g., SGEMM). |\n| **LIBXSMM** | SME | AArch64 SME code generator for small-matrix GEMM kernels. |\n| **Eigen** | SME in selected builds | Enablement depends on version and target flags. Current upstream selects its SME backend automatically when the target supports SME2 and no fixed `-msve-vector-bits` is specified; check the exact release and compiler configuration. |\n| **Apache TVM** | SME | Compiler integration through Arm SME TensorIntrin definitions and LLVM AArch64 SME intrinsics. |\n| **ArmPL 26.01+** | SME | Transparent SME SGEMM/DGEMM kernels documented for compatible Apple silicon on macOS; this is not library-wide SME coverage. |\n\n### KleidiAI Details\n\nKleidiAI is Arm's primary vehicle for making SME performance accessible without requiring developers to write SME intrinsics directly. Key characteristics:\n\n- Tuned micro-kernels for specific Arm microarchitectures\n- C/C++ API designed for integration into ML/AI frameworks\n- Supports F32, FP16, INT8, and dynamically quantized matrix multiplication\n- Source: [KleidiAI GitLab Repository](https://gitlab.arm.com/kleidi/kleidiai)\n\nKleidiCV's multiversion build can select an implementation at runtime. To assess a particular operation, inspect its backend source and trace the selected function; the presence of an SME backend elsewhere in the library does not prove this call uses it.\n\n## Frameworks with SME Support (via KleidiAI/XNNPACK)\n\nThese higher-level frameworks obtain all or part of their SME acceleration through KleidiAI, XNNPACK, MLAS, or another native backend:\n\n| Framework | Integration Path | Notes |\n|-----------|-----------------|-------|\n| **LiteRT** | LiteRT → XNNPACK → KleidiAI | Supported delegated operators can select SME2 kernels when built and configured for them |\n| **ONNX Runtime** | ORT MLAS → KleidiAI | Integrates KleidiAI for SGEMM, IGEMM, dynamic-quantized MatMul |\n| **ExecuTorch** | ExecuTorch → XNNPACK → KleidiAI | Meta's on-device inference framework |\n| **llama.cpp** | Direct KleidiAI integration | Runtime-selected SME2 acceleration for matrix multiplication |\n| **whisper.cpp** | whisper.cpp → GGML CPU backend → KleidiAI | Opt-in KleidiAI backend for compatible speech-model operations; enable it at build time with `-DGGML_CPU_KLEIDIAI=ON` |\n| **PyTorch** | Direct KleidiAI integration in current AArch64 source builds | `torch.backends.kleidiai.is_available()` reports whether the backend was built; operator coverage remains selective |\n| **MNN** | Direct integration | Alibaba Mobile Neural Network |\n| **MediaPipe** | Potentially via XNNPACK | Verify the graph, delegate, build, and selected kernel before claiming SME2 execution |\n\n### How the Integration Works (LiteRT Example)\n\nWhen KleidiAI kernels are included in an XNNPACK build, XNNPACK may choose one for a supported operator configuration after checking the processor and kernel requirements. Operator type, data type, shape, layout, and quantization details all matter.\n\nwhisper.cpp reaches KleidiAI through its shared GGML CPU backend rather than through XNNPACK. Its upstream build system exposes `GGML_CPU_KLEIDIAI`, which is disabled by default. Enabling the option makes the KleidiAI buffer and kernels available, but it does not mean every Whisper operation uses SME2: dispatch still depends on the processor features and whether an operation, tensor type, and shape match a supported KleidiAI kernel. See the upstream [whisper.cpp GGML build option](https://github.com/ggml-org/whisper.cpp/blob/master/ggml/CMakeLists.txt) and [KleidiAI backend registration](https://github.com/ggml-org/whisper.cpp/blob/master/ggml/src/ggml-cpu/ggml-cpu.cpp).\n\n## Important \"Support\" Distinctions\n\nDo not treat \"runs on an SME device\" as proof of SME support. A project may contain native SME code (oneDNN, OpenBLAS, Eigen), delegate to a library that may dispatch SME (NumPy/SciPy through OpenBLAS, JAX/XLA through a specially built ACL path), or merely run on the same hardware while using Neon, SVE, or a GPU (MLX's normal Metal path). Confirm the exact operator, data type, build option, and runtime-selected kernel.\n\nBLIS has an open Apple-M4-focused SME2 implementation in pull request #921, but the current upstream release does not yet provide production SME support. MLX has no explicit SME backend. JAX/XLA does not natively lower ordinary JAX operations to SME, although custom calls or an ACL-enabled XLA build can reach external SME kernels.\n\n## Finding SME Support in Source Trees\n\nWhen evaluating whether a library has genuine SME acceleration (vs. build-system boilerplate), search for these indicators in the source code:\n\n- C keywords: `__arm_streaming`, `__arm_new_za`, `__arm_inout(\"za\")`\n- Assembly: `smstart`, `smstop`, `fmopa`, `fmops`\n- Headers: `#include `\n- Build options that include SME kernels, such as `XNNPACK_ENABLE_ARM_SME2` or OpenBLAS's `ARMV9SME` target\n- Architecture flags: `-march=armv9.2-a+sme`, `-march=armv9.3-a+sme2`\n\n**Caution:** Some projects add SME build infrastructure without providing optimized kernels. For example, libyuv initially added an SME implementation file that was largely empty. Always verify actual kernel implementations exist, not just build scaffolding.\n\nThe [Arm Software Ecosystem Dashboard](https://www.arm.com/developer-hub/ecosystem-dashboard) is a broad discovery tool, not an SME support certification list. For a specific project, its own source and release notes are stronger evidence.\n\n## Resources\n\n- [Eigen vectorization and SME enablement](https://libeigen.gitlab.io/eigen/docs-master/TopicVectorization.html)\n- [OpenBLAS target and kernel coverage](https://github.com/OpenMathLib/OpenBLAS)\n- [ArmPL's documented SME GEMM support](https://developer.arm.com/community/arm-community-blogs/b/servers-and-cloud-computing-blog/posts/what-s-new-in-arm-performance-libraries-26-01)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s02-q08", "slug": "detect-whether-existing", "question": "How can I determine whether an existing SME or SME2 kernel supports my model's operators, data types, tensor shapes, and quantization formats?", "summary": "Match each costly lowered operator to a kernel in the exact backend version your application uses, then verify that the runtime selected that kernel. A device with SME2 and a library containing SME2 code are necessary for that path, but neither establishes that your model's operator reaches it.", "answer": "Match each costly **lowered operator** to a kernel in the **exact backend version** your application uses, then verify that the runtime selected that kernel. A device with SME2 and a library containing SME2 code are necessary for that path, but neither establishes that your model's operator reaches it.\n\n## 1. Record What Actually Runs\n\nInspect the delegated or lowered graph, not only the original model. Fusion, quantization, and layout conversion can change its operations. For each expensive node, record:\n\n| Property | Details that can change kernel eligibility |\n|---|---|\n| Operation | Matrix multiply, fully connected, convolution, batch matmul, or another lowered operator |\n| Shape | M, N, K, batch size, convolution kernel/stride, and dynamic ranges |\n| Tensor contract | Input, weight, accumulator, and output types; layout, transposition, and strides |\n| Quantization | Signedness, symmetric/asymmetric zero points, scale granularity, group size, scale type, and packed format |\n| Postprocessing | Bias, clamp, activation, and requantization that a kernel must implement or combine with other work |\n\nRecord actual runtime shapes for dynamic models. A layer called \"INT8 matmul\" is underspecified: a kernel for symmetric per-channel INT8 weights need not accept asymmetric per-tensor weights or a different packed representation.\n\n## 2. Trace the Backend to a Kernel\n\nFirst identify the execution path: for example, ExecuTorch or LiteRT can delegate eligible nodes to XNNPACK, which can use KleidiAI; llama.cpp can reach KleidiAI through GGML. The presence of KleidiAI in a source tree does not establish that the delegate was enabled in your build.\n\nFor the **same KleidiAI commit embedded by your backend**, inspect its [kernel inventory and source](https://github.com/ARM-software/kleidiai). Match the operation, complete type and quantization signature, CPU feature requirement, tile sizes, packing functions, and any restrictions on dimensions or strides. Then inspect the backend's registration and selection code to confirm that it exposes that kernel for the model operator. A kernel may handle a non-tile-aligned shape correctly through tail processing yet lose to another implementation on that shape.\n\nAs one concrete example, a packed 4-bit weight matrix with dynamically quantized 8-bit activations needs a kernel and packer that agree on the block layout, zero-point convention, and scale representation. Another 4-bit file format is not automatically compatible. The [ExecuTorch node-to-KleidiAI guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/measure-kleidiai-kernel-performance-on-executorch/03-executorch-node-kai-kernel/) shows how to connect a delegated node to its candidate microkernel; use its support table only for the versions it documents.\n\n## 3. Prove the Selected Path\n\nCheck that the OS exposes SME2 on the target processor. On Linux or Android, use `getauxval(AT_HWCAP2)` with `HWCAP2_SME2` in a C/C++ program; on macOS, query `hw.optional.arm.FEAT_SME2`. A compile-time `__ARM_FEATURE_SME2` macro says only that the translation unit may use the instructions.\n\nNext, collect evidence at the backend level. Depending on the runtime, use a dispatch trace, a breakpoint on the candidate `kai_*` symbol, or a sampled call stack. Inspect the selected function's disassembly for SME instructions when necessary. Operator timing alone can show a benefit but cannot identify which instructions ran. For ExecuTorch, ETDump can reveal operator and delegate timing; combine it with backend tracing to identify the microkernel. For LiteRT, Arm's [benchmark and node-inspection guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/4-benchmark/) explains one way to inspect the selected path.\n\nRun timing with tracing disabled, then compare SME2-enabled and SME2-disabled builds under the same device, input, thread, and power conditions. A successful inference run proves that some path works; it does not prove SME2 selection.\n\n## Keep a Per-Operator Result\n\n| Node | Lowered operation and full signature | Candidate kernel | Backend registration | Observed dispatch | Result |\n|---|---|---|---|---|---|\n| Encoder projection | FP16 GEMM, M/N/K and layout recorded | Version-matched FP16 kernel | Confirmed | Trace or breakpoint | Supported and measured |\n| Decoder projection | Packed 4-bit weights, dynamic activations | Check exact packing contract | Pending | Not observed | Unverified |\n| Depthwise convolution | Shape and quantization recorded | None found | None | Fallback | No SME2 path in this build |\n\nClassify each node as selected, eligible but not selected, needing conversion, unsupported, or still unverified. This distinction lets you improve the model or integration without confusing \"runs correctly\" with \"uses SME2.\"", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q01", "slug": "build-install-pytorch", "question": "How do I build or install PyTorch or ExecuTorch with SME2 support enabled for Arm targets?", "summary": "There are now two distinct routes. Current source builds of PyTorch can include direct KleidiAI-backed eager CPU kernels, while ExecuTorch uses the mature deployment path ExecuTorch → XNNPACK → KleidiAI → SME2. Neither route means every operator uses SME2: kernel availability, data type, shape, quantization, hardware, and runtime dispatch must all match.", "answer": "There are now two distinct routes. Current source builds of **PyTorch** can include direct KleidiAI-backed eager CPU kernels, while **ExecuTorch** uses the mature deployment path ExecuTorch → XNNPACK → KleidiAI → SME2. Neither route means every operator uses SME2: kernel availability, data type, shape, quantization, hardware, and runtime dispatch must all match.\n\n## PyTorch vs. ExecuTorch: Where SME2 Lives\n\n| Component | SME2 Support | Notes |\n|-----------|-------------|-------|\n| **PyTorch (eager CPU)** | Yes, for supported KleidiAI-backed operations | Source builds expose `torch.backends.kleidiai`; coverage is operation-specific |\n| **ExecuTorch (inference)** | Yes, via XNNPACK + KleidiAI | SME2 kernels selected at runtime for supported operators |\n\nThere is no separate \"SME2 PyTorch wheel\" analogous to a CUDA wheel. For a predictable edge-inference deployment and easy SME2-on/off comparison, the recommended workflow remains:\n\n1. Develop and train your model in PyTorch\n2. Export to ExecuTorch `.pte` format\n3. Run inference with an ExecuTorch runner built with XNNPACK + KleidiAI\n\n## Prerequisites\n\n- **Compiler:** Use an AArch64 toolchain that can compile the SME2 kernels in the pinned XNNPACK version; Clang 18+ and Android NDK r27+ are practical choices for the documented path. Check the build log because older toolchains may disable those kernels.\n- **Build tools:** Install the versions required by the pinned ExecuTorch release, plus Ninja, Python, and Git.\n- **Target hardware:** An AArch64 device implementing SME2 with OS/kernel support exposing SME2 to userspace\n- **Source:** Clone ExecuTorch with `--recurse-submodules` to include XNNPACK and KleidiAI\n\n> **Important:** Do not globally add `-march=...+sme2` to your build flags. XNNPACK builds separate micro-kernels with the appropriate architecture flags and performs runtime dispatch, preserving binary compatibility with non-SME2 Arm64 systems.\n\n## Building ExecuTorch with SME2 (Native Linux or macOS)\n\n```bash\ngit clone --recurse-submodules https://github.com/pytorch/executorch.git\ncd executorch\n\npython3 -m venv .venv\nsource .venv/bin/activate\n\n./install_executorch.sh\n\ncmake -S . -B cmake-out/sme2 -G Ninja \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DEXECUTORCH_BUILD_XNNPACK=ON \\\n -DEXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON \\\n -DXNNPACK_ENABLE_ARM_SME2=ON \\\n -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=ON \\\n -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON\n\ncmake --build cmake-out/sme2 --parallel\n```\n\n### Key CMake Options\n\n| Flag | Purpose |\n|------|---------|\n| `EXECUTORCH_BUILD_XNNPACK=ON` | Enables the XNNPACK delegate backend |\n| `EXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON` | Integrates Arm KleidiAI optimized kernels |\n| `XNNPACK_ENABLE_ARM_SME2=ON` | Enables SME2 kernel compilation when the toolchain and target support it |\n| `EXECUTORCH_BUILD_KERNELS_QUANTIZED=ON` | Enables quantized kernel support (INT8 models) |\n\n**Note:** The current [ExecuTorch XNNPACK dependency configuration](https://github.com/pytorch/executorch/blob/main/backends/xnnpack/cmake/Dependencies.cmake) enables SME2 by default. The explicit option makes the intended build configuration visible, but inspect the pinned version and build output to confirm the kernels were actually compiled.\n\n## Building ExecuTorch with SME2 (Android)\n\n### Using the build script\n\n```bash\nexport ANDROID_HOME=/path/to/android-sdk\nexport ANDROID_NDK_HOME=/path/to/android-ndk\nexport ANDROID_NDK=\"$ANDROID_NDK_HOME\"\n\nANDROID_ABIS=arm64-v8a \\\nXNNPACK_ENABLE_ARM_SME2=ON \\\n./scripts/build_android_library.sh\n```\n\n### Using CMake directly (custom runner)\n\n```bash\ncmake -S . -B cmake-out/android-sme2 -G Ninja \\\n -DCMAKE_TOOLCHAIN_FILE=\"$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake\" \\\n -DANDROID_ABI=arm64-v8a \\\n -DANDROID_PLATFORM=android-26 \\\n -DEXECUTORCH_BUILD_XNNPACK=ON \\\n -DEXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON \\\n -DXNNPACK_ENABLE_ARM_SME2=ON \\\n -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=ON \\\n -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \\\n -DCMAKE_BUILD_TYPE=Release\n\ncmake --build cmake-out/android-sme2 --parallel\n```\n\n## Using CMake Presets (Recommended for Profiling)\n\nThe [sme-executorch-profiling](https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling) project provides ready-made presets that build SME2-on and SME2-off variants for comparison. The build script merges custom presets into ExecuTorch's defaults:\n\n```json\n{\n \"name\": \"android-arm64-v9a\",\n \"displayName\": \"Android arm64 with XNNPACK (SME2 ON, timing)\",\n \"generator\": \"Ninja\",\n \"binaryDir\": \"${sourceDir}/cmake-out/android-arm64-v9a\",\n \"cacheVariables\": {\n \"EXECUTORCH_BUILD_XNNPACK\": \"ON\",\n \"EXECUTORCH_BUILD_DEVTOOLS\": \"ON\",\n \"EXECUTORCH_BUILD_EXECUTOR_RUNNER\": \"ON\",\n \"EXECUTORCH_ENABLE_LOGGING\": \"ON\",\n \"EXECUTORCH_ENABLE_EVENT_TRACER\": \"ON\",\n \"EXECUTORCH_BUILD_KERNELS_QUANTIZED\": \"ON\",\n \"EXECUTORCH_XNNPACK_ENABLE_KLEIDI\": \"ON\",\n \"CMAKE_BUILD_TYPE\": \"RelWithDebInfo\"\n }\n}\n```\n\n```json\n{\n \"name\": \"mac-arm64\",\n \"displayName\": \"Mac arm64 with XNNPACK (SME2 ON, timing)\",\n \"generator\": \"Ninja\",\n \"binaryDir\": \"${sourceDir}/cmake-out/mac-arm64\",\n \"cacheVariables\": {\n \"EXECUTORCH_BUILD_XNNPACK\": \"ON\",\n \"EXECUTORCH_BUILD_DEVTOOLS\": \"ON\",\n \"EXECUTORCH_BUILD_EXECUTOR_RUNNER\": \"ON\",\n \"EXECUTORCH_ENABLE_LOGGING\": \"ON\",\n \"EXECUTORCH_ENABLE_EVENT_TRACER\": \"ON\",\n \"EXECUTORCH_BUILD_KERNELS_QUANTIZED\": \"ON\",\n \"EXECUTORCH_XNNPACK_ENABLE_KLEIDI\": \"ON\",\n \"CMAKE_BUILD_TYPE\": \"RelWithDebInfo\"\n }\n}\n```\n\nBuild with:\n```bash\nbash model_profiling/scripts/build_runners.sh\n```\n\n## Building PyTorch from Source with KleidiAI (Experimental)\n\nFor an experimental native AArch64 PyTorch build that includes KleidiAI integration:\n\n```bash\ngit clone --recursive https://github.com/pytorch/pytorch.git\ncd pytorch\n\npython3 -m venv .venv\nsource .venv/bin/activate\n\npython -m pip install --upgrade pip\npython -m pip install --group dev\n\nexport CC=clang-18\nexport CXX=clang++-18\nexport USE_XNNPACK=1\nexport USE_KLEIDIAI=1\nexport MAX_JOBS=\"$(nproc)\"\n\npython -m pip install --no-build-isolation -v -e .\n```\n\nVerify KleidiAI was included:\n\n```python\nimport torch\nprint(\"KleidiAI:\", torch.backends.kleidiai.is_available())\n```\n\nHowever, `USE_KLEIDIAI=1` and `is_available()` establish build availability, not that an arbitrary eager operation executes SME2 instructions. The ExecuTorch route provides a separately inspectable deployment path for supported delegated operators.\n\n## Verifying the Build\n\nConfirm the build configuration from the CMake cache:\n\n```bash\ngrep -E 'EXECUTORCH_XNNPACK_ENABLE_KLEIDI|XNNPACK_ENABLE_ARM_SME2|XNNPACK_ENABLE_KLEIDIAI' \\\n cmake-out/sme2/CMakeCache.txt\n```\n\nExample cache entries (types and exact names can differ by pinned revision):\n\n```text\nEXECUTORCH_XNNPACK_ENABLE_KLEIDI:BOOL=ON\nXNNPACK_ENABLE_ARM_SME2:BOOL=ON\nXNNPACK_ENABLE_KLEIDIAI:BOOL=ON\n```\n\nCheck the compiler version:\n\n```bash\ngrep -E 'CMAKE_C_COMPILER:|CMAKE_C_COMPILER_VERSION' \\\n cmake-out/sme2/CMakeCache.txt\n```\n\nFor an A/B comparison, configure two otherwise identical builds, one with `-DXNNPACK_ENABLE_ARM_SME2=ON` and one with `-DXNNPACK_ENABLE_ARM_SME2=OFF`. Keep the full command and all other options the same.\n\nA cache value of `ON` records the requested configuration. Check the build output for kernel compilation, then trace a candidate kernel at runtime. An A/B timing comparison measures the effect but alone does not identify the selected instructions.\n\n## Without SME2 Hardware\n\nIf you don't have access to SME2-capable hardware, you can use Docker with Arm's Fixed Virtual Platform (FVP) to emulate SME2 for development and testing. This allows compiling and running SME2 code in an emulated environment.\n\n## Resources\n\n- [Arm's ExecuTorch SME2 setup and runner build](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline/)\n- [ExecuTorch release notes for version-specific build behavior](https://github.com/pytorch/executorch/releases)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q02", "slug": "llama-cpp-support", "question": "Does llama.cpp support SME2 acceleration, and how do I enable it when building from source?", "summary": "Yes. llama.cpp can use SME-family microkernels through KleidiAI in its GGML CPU backend. Build an AArch64 version with GGML_CPU_KLEIDIAI=ON; the runtime then selects a compatible kernel for supported matrix operations when the OS exposes the required feature. This is operator-specific, so the option does not make every model layer run on SME2. The commands below follow llama.cpp's upstream KleidiAI build guide.", "answer": "Yes. llama.cpp can use SME-family microkernels through KleidiAI in its GGML CPU backend. Build an AArch64 version with `GGML_CPU_KLEIDIAI=ON`; the runtime then selects a compatible kernel for supported matrix operations when the OS exposes the required feature. This is operator-specific, so the option does not make every model layer run on SME2. The commands below follow llama.cpp's [upstream KleidiAI build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#arm-kleidiai).\n\n## Native Arm64 Build\n\n```bash\ngit clone https://github.com/ggml-org/llama.cpp.git\ncd llama.cpp\ncmake -S . -B build -DGGML_CPU_KLEIDIAI=ON -DCMAKE_BUILD_TYPE=Release\ncmake --build build --config Release --parallel\n./build/bin/llama-cli -m /path/to/model.gguf -p \"Hello\"\n```\n\nOn a Mac, another backend such as Metal may take priority. For a CPU-only comparison, build with `-DGGML_METAL=OFF`, or select CPU execution at runtime where the installed llama.cpp version supports it.\n\n## Android Arm64 Build\n\nSet `ANDROID_NDK` to the NDK root, then configure a portable arm64-v8a build:\n\n```bash\ncmake -S . -B build-android \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DCMAKE_TOOLCHAIN_FILE=\"$ANDROID_NDK/build/cmake/android.toolchain.cmake\" \\\n -DANDROID_ABI=arm64-v8a \\\n -DANDROID_PLATFORM=android-28 \\\n -DGGML_CPU_KLEIDIAI=ON \\\n -DGGML_NATIVE=OFF \\\n -DGGML_OPENMP=OFF \\\n -DGGML_LLAMAFILE=OFF \\\n -DLLAMA_OPENSSL=OFF\ncmake --build build-android --config Release --parallel\n```\n\nUse the binary or installed artifact produced by that build on an Android device that exposes SME2. Keep global `-march=...+sme2` flags off the general build: KleidiAI compiles specialized kernels separately, while global flags can make ordinary code incompatible with older devices.\n\n## What Dispatch Selects\n\nThe GGML CPU backend can use KleidiAI for selected `GGML_OP_MUL_MAT` paths, including supported floating-point and quantized formats. The exact Q4_0, Q8_0, F16, or other coverage depends on the bundled KleidiAI revision and llama.cpp's selector. Runtime selection also depends on tensor type, shape, CPU feature detection, and whether another backend owns the operation. A single portable binary can fall back to another compatible CPU kernel on devices without SME2.\n\nThe `GGML_KLEIDIAI_SME` variable is a **diagnostic override**, not a normal deployment setting:\n\n| Value | Effect |\n|---|---|\n| Unset | Automatic detection and selection |\n| `0` | Disable SME-family kernels for comparison |\n| Positive integer `n` | Allow compatible SME-family kernels and cap selected quantized-kernel threads at `n` |\n\nA positive value cannot enable unsupported instructions. To compare paths, run `llama-bench` on the same model, device, thread setting, and thermal state with the variable unset and then set to `0`. Record prompt processing and token generation separately.\n\n## Verify That SME2 Ran\n\nA `CPU_KLEIDIAI model buffer size` message shows that model tensors use the KleidiAI CPU buffer; it does **not** prove an SME2 microkernel ran. For a specific matrix operation, trace or break on the selected KleidiAI kernel and inspect its symbol or disassembly for SME instructions. Then repeat the timing run without tracing. Arm's [llama.cpp SME2 walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_llama_cpp_sme2/run_llm/) demonstrates on-device execution and optional kernel-name instrumentation.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q03", "slug": "build-onnx-runtime", "question": "How do I build ONNX Runtime with KleidiAI and SME2 support for Android?", "summary": "Cross-compile ONNX Runtime for Android with onnxruntime_USE_KLEIDIAI=ON passed through --cmake_extra_defines. This makes KleidiAI kernels available to MLAS. On an SME2-capable device, a supported operation can select an SME2 kernel at runtime; the build option alone does not establish that a given ONNX node uses one.", "answer": "Cross-compile ONNX Runtime for Android with `onnxruntime_USE_KLEIDIAI=ON` passed through `--cmake_extra_defines`. This makes KleidiAI kernels available to MLAS. On an SME2-capable device, a supported operation can select an SME2 kernel at runtime; the build option alone does not establish that a given ONNX node uses one.\n\n## How KleidiAI and SME2 Work in ONNX Runtime\n\nKleidiAI is integrated into ONNX Runtime's MLAS engine, which handles performance-critical linear algebra operations (GEMM, convolutions, fully-connected layers).\n\n- **Runtime dispatch:** MLAS checks OS-exposed CPU capabilities and selects from eligible kernels. The full operator signature and the particular ONNX Runtime build also determine whether a KleidiAI SME2 kernel is available.\n- **Fallback safety:** On devices without SME2, MLAS falls back to NEON or SVE kernels. This means a single binary works across all arm64-v8a devices without risking SIGILL crashes on older hardware.\n- **No global architecture flags needed:** You do not need to add `-march=armv9-a+sme2` to the entire ONNX Runtime build. The KleidiAI micro-kernels are compiled with the appropriate flags internally, and MLAS handles dispatch.\n\n## Prerequisites\n\n| Requirement | Details |\n|-------------|---------|\n| **Android NDK** | Use the version required by the pinned ONNX Runtime release; r27+ is a practical SME2-capable choice |\n| **CMake** | 3.28+ |\n| **Ninja** | Required as the build generator |\n| **Python** | 3.10+ |\n| **Host OS** | Linux, macOS, or Windows (cross-compilation target is Android) |\n| **Target device** | A compatible arm64-v8a device can use the portable fallback; SME2 acceleration requires OS-exposed SME2 and a matching kernel |\n\nSet your environment variables:\n\n```bash\n# Linux host\nexport ANDROID_SDK_ROOT=\"$HOME/Android/Sdk\"\nexport ANDROID_NDK_HOME=\"$ANDROID_SDK_ROOT/ndk/\"\n\n# macOS host\nexport ANDROID_SDK_ROOT=\"$HOME/Library/Android/sdk\"\nexport ANDROID_NDK_HOME=\"$ANDROID_SDK_ROOT/ndk/\"\n```\n\n## Step-by-Step Build\n\n### 1. Clone ONNX Runtime\n\nClone the repository and pin a release with KleidiAI support. The example below uses v1.23.2; check that release's build options before substituting another revision:\n\n```bash\ngit clone --recursive https://github.com/microsoft/onnxruntime.git onnxruntime.git\ncd onnxruntime.git/\ngit checkout v1.23.2\ngit submodule update --init --recursive\n```\n\n### 2. Build with KleidiAI enabled\n\nRun the build script from the repository root:\n\n```bash\n./build.sh \\\n --android \\\n --android_sdk_path $ANDROID_SDK_ROOT \\\n --android_ndk_path $ANDROID_NDK_HOME \\\n --android_abi arm64-v8a \\\n --android_api 27 \\\n --config RelWithDebInfo \\\n --build_shared_lib \\\n --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON \\\n --cmake_generator Ninja \\\n --parallel\n```\n\n#### Key flags explained\n\n| Flag | Purpose |\n|------|---------|\n| `--android_abi arm64-v8a` | Targets 64-bit Arm Android devices |\n| `--android_api 27` | Minimum Android API level (Android 8.1+) |\n| `--config RelWithDebInfo` | Optimized build with debug symbols for profiling |\n| `--build_shared_lib` | Produces `libonnxruntime.so` for app integration |\n| `--cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON` | Integrates KleidiAI micro-kernels into MLAS |\n| `--cmake_generator Ninja` | Uses Ninja for fast parallel builds |\n\n> **Note:** Use the correct SDK path for `--android_sdk_path`. Some examples incorrectly pass the NDK path for both arguments. The SDK and NDK are separate directories.\n\n### 3. Build an Android AAR (optional)\n\nIf your app uses the Java/Kotlin ONNX Runtime API, add `--build_java` to produce an AAR:\n\n```bash\n./build.sh \\\n --android \\\n --android_sdk_path $ANDROID_SDK_ROOT \\\n --android_ndk_path $ANDROID_NDK_HOME \\\n --android_abi arm64-v8a \\\n --android_api 27 \\\n --config Release \\\n --build_shared_lib \\\n --build_java \\\n --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON \\\n --cmake_generator Ninja \\\n --parallel\n```\n\nCheck the build output for the AAR. For the layout used by this example, it is under:\n\n```\nbuild/Android/Release/java/build/android/outputs/aar/\n```\n\n## Build Outputs\n\nFor this build configuration, look for these artifacts under:\n\n```\nbuild/Android/RelWithDebInfo/\n├── libonnxruntime.so # Shared library to package into your Android app\n└── onnxruntime_perf_test # Benchmark binary for on-device testing\n```\n\n## Deploy and Test on Device\n\n### Push artifacts to the device\n\n```bash\nadb push build/Android/RelWithDebInfo/onnxruntime_perf_test /data/local/tmp/\nadb push build/Android/RelWithDebInfo/libonnxruntime.so /data/local/tmp/\nadb push your_model.onnx /data/local/tmp/\nadb shell chmod +x /data/local/tmp/onnxruntime_perf_test\n```\n\n### Run a benchmark\n\n```bash\nadb shell 'cd /data/local/tmp && \\\n LD_LIBRARY_PATH=/data/local/tmp \\\n ./onnxruntime_perf_test \\\n -e cpu \\\n -m times \\\n -r 20 \\\n -x 1 \\\n your_model.onnx'\n```\n\n### Compare with and without KleidiAI\n\nFor meaningful performance comparison, build twice—once with `onnxruntime_USE_KLEIDIAI=ON` and once with `onnxruntime_USE_KLEIDIAI=OFF`—and run both versions with the same model, thread count, CPU affinity, and thermal conditions on an SME2-capable device.\n\n## Important Notes\n\n- **SME2 hardware required for acceleration:** Building with `onnxruntime_USE_KLEIDIAI=ON` does not make SME2 available on older hardware. Cortex-X925 itself does not implement SME; use an explicitly documented SME2 SoC and verify `sme2` at runtime.\n- **No SME2 override required:** Normal runtime selection needs no SME2 environment variable. Verify the selected kernel when that distinction matters.\n- **NDK version matters:** Use an NDK/toolchain combination supported by the pinned ONNX Runtime and KleidiAI revisions; inspect build output for SME2 kernel compilation.\n- **Build reproducibility:** Pinning a specific ONNX Runtime version (e.g., v1.23.2) ensures reproducible builds and known-good KleidiAI integration.\n\n## Resources\n\n- [Arm's ONNX Runtime Android build walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/build_ort/)\n- [Arm's on-device profiling walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/)\n- [ONNX Runtime's Android build documentation](https://onnxruntime.ai/docs/build/android.html)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q04", "slug": "compiler-flags-build", "question": "What compiler flags and build options are needed to enable SME2 in LiteRT on Arm?", "summary": "LiteRT can use SME2 through XNNPACK and KleidiAI. The documented source-build route passes Bazel --define flags that include the relevant microkernels. Defaults vary by release, so make the requested build configuration explicit and verify the selected kernel at runtime.", "answer": "LiteRT can use SME2 through XNNPACK and KleidiAI. The documented source-build route passes Bazel `--define` flags that include the relevant microkernels. Defaults vary by release, so make the requested build configuration explicit and verify the selected kernel at runtime.\n\n## Architecture Overview\n\nSME2 support in LiteRT flows through this software stack:\n\n```\nLiteRT\n └── XNNPACK delegate (operator execution)\n └── KleidiAI (Arm-optimized SME2 micro-kernels)\n```\n\n- **LiteRT** delegates quantized and floating-point operators to XNNPACK\n- **XNNPACK** integrates KleidiAI, which provides SME2-accelerated matrix multiplication micro-kernels\n- The `xnn_enable_arm_sme2=true` and `xnn_enable_kleidiai=true` flags activate these code paths\n- At runtime, XNNPACK considers CPU features and the supported operator configuration before selecting an SME2 kernel; otherwise it uses a compatible fallback\n\n## Toolchain Requirements\n\n| Requirement | Minimum | Recommended |\n|-------------|---------|-------------|\n| Clang/LLVM | 18.0 | Latest stable |\n| GCC | 14.1 | Latest stable |\n| Binutils | 2.41 | Latest stable |\n| Bazel | As specified by the pinned LiteRT checkout | Matching version |\n| Android NDK | r27 | Latest compatible r27+ release |\n| Python | 3.10 | 3.12 |\n\n**Important:** XNNPACK disables SME and SME2 support when building with Clang versions earlier than 18. Ensure your toolchain meets this requirement.\n\n## Bazel Build Flags (Primary Method)\n\n### Critical --define Flags\n\n```bash\n# XNNPACK and quantization paths\n--define tflite_with_xnnpack=true\n--define=tflite_with_xnnpack_qs8=true\n--define=tflite_with_xnnpack_qu8=true\n--define=tflite_with_xnnpack_dynamic_fully_connected=true\n\n# SME/SME2 and KleidiAI enablement\n--define=xnn_enable_arm_sme=true\n--define=xnn_enable_arm_sme2=true\n--define=xnn_enable_kleidiai=true\n```\n\n### Flag Explanation\n\n| Flag | Purpose |\n|------|---------|\n| `tflite_with_xnnpack=true` | Enables XNNPACK delegate for operator acceleration |\n| `tflite_with_xnnpack_qs8=true` | Enables signed 8-bit quantization paths |\n| `tflite_with_xnnpack_qu8=true` | Enables unsigned 8-bit quantization paths |\n| `tflite_with_xnnpack_dynamic_fully_connected=true` | Enables dynamic quantization for fully-connected layers |\n| `xnn_enable_arm_sme=true` | Enables SME (v1) micro-kernels in XNNPACK |\n| `xnn_enable_arm_sme2=true` | Enables SME2 micro-kernels in XNNPACK |\n| `xnn_enable_kleidiai=true` | Enables KleidiAI integration for Arm-optimized kernels |\n\nThe quantization flags (`qs8`, `qu8`, `dynamic_fully_connected`) are particularly important because KleidiAI's SME2 kernels primarily accelerate quantized matrix multiplication operations used in neural network inference.\n\n## Full Build Commands\n\n### Android arm64 Cross-Compilation\n\n```bash\nbazel build -c opt --config=android_arm64 \\\n --define tflite_with_xnnpack=true \\\n --define=tflite_with_xnnpack_qs8=true \\\n --define=tflite_with_xnnpack_qu8=true \\\n --define=tflite_with_xnnpack_dynamic_fully_connected=true \\\n --define=xnn_enable_arm_sme=true \\\n --define=xnn_enable_arm_sme2=true \\\n --define=xnn_enable_kleidiai=true \\\n \"//litert/tools:benchmark_model\" \\\n --repo_env=HERMETIC_PYTHON_VERSION=3.12\n```\n\n### Native Arm Linux Build\n\n```bash\nbazel build -c opt \\\n --define tflite_with_xnnpack=true \\\n --define=tflite_with_xnnpack_qs8=true \\\n --define=tflite_with_xnnpack_qu8=true \\\n --define=tflite_with_xnnpack_dynamic_fully_connected=true \\\n --define=xnn_enable_arm_sme=true \\\n --define=xnn_enable_arm_sme2=true \\\n --define=xnn_enable_kleidiai=true \\\n //litert/tools:benchmark_model\n```\n\n## CMake Build Alternative\n\nFor a direct XNNPACK CMake build, the corresponding feature options are:\n\n```bash\ncmake -S . -B build \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DXNNPACK_ENABLE_ARM_SME=ON \\\n -DXNNPACK_ENABLE_ARM_SME2=ON \\\n -DXNNPACK_ENABLE_KLEIDIAI=ON\n```\n\n## Build Environment Setup with Docker\n\nGoogle provides a Docker-based build environment for Android cross-compilation:\n\n```bash\n# Clone LiteRT\ncd $WORKSPACE\ngit clone https://github.com/google-ai-edge/LiteRT.git\n\n# Download and build the Docker image\nwget https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/tools/tflite-android.Dockerfile\ndocker build . -t tflite-builder -f tflite-android.Dockerfile\n\n# Launch the container\ndocker run -it -v $PWD:/host_dir tflite-builder bash\n```\n\nInside the container:\n\n```bash\n# Install Android SDK/NDK components\nsdkmanager \\\n \"build-tools;${ANDROID_BUILD_TOOLS_VERSION}\" \\\n \"platform-tools\" \\\n \"platforms;android-${ANDROID_API_LEVEL}\"\n\n# Configure LiteRT\ncd /host_dir/LiteRT\n./configure\n```\n\nWhen prompted `Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]`, type `y`. The configuration script will detect SDK/NDK paths, set the toolchain version, configure the ABI to arm64-v8a, and initialize Bazel workspace rules.\n\n## Compiler-Level Architecture Flags\n\nWithin the Bazel build, you **do not** need to manually set `-march` flags—XNNPACK's build system applies the correct architecture options to SME2 micro-kernel source files internally, enabling safe runtime dispatch.\n\nHowever, if you are compiling SME2 translation units manually or integrating outside the standard build system, you need:\n\n```bash\n-march=armv9.3-a+sme2\n```\n\n**Warning:** Do not apply `-march=...+sme2` globally to your entire application unless it will run exclusively on SME2 hardware. Globally enabling `+sme2` can make startup or non-dispatched code illegal on older Arm CPUs. The XNNPACK/Bazel build handles this correctly by applying architecture flags only to specific micro-kernel targets.\n\n## Building Without SME2 (Baseline Comparison)\n\nTo build an SME-free baseline for performance comparison, explicitly disable both SME and SME2. Omitting the enable flags is insufficient: current XNNPACK Bazel defaults enable SME2 on AArch64. Keep KleidiAI and the quantization options the same as in the enabled build to isolate the contribution of SME/SME2:\n\n```bash\nbazel build -c opt --config=android_arm64 \\\n --define tflite_with_xnnpack=true \\\n --define=tflite_with_xnnpack_qs8=true \\\n --define=tflite_with_xnnpack_qu8=true \\\n --define=tflite_with_xnnpack_dynamic_fully_connected=true \\\n --define=xnn_enable_arm_sme=false \\\n --define=xnn_enable_arm_sme2=false \\\n --define=xnn_enable_kleidiai=true \\\n \"//litert/tools:benchmark_model\" \\\n --repo_env=HERMETIC_PYTHON_VERSION=3.12\n```\n\nUse a separate output directory or copy each benchmark binary before rebuilding so both configurations remain available for comparison. The false settings disable the SME/SME2 paths while retaining KleidiAI. Check the selected kernels in both builds; the baseline may use other non-SME Arm extensions rather than Neon alone.\n\n## Resources\n\n- [Build the LiteRT benchmark tool with KleidiAI and SME2 enabled](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/3-build-tool/?utm_source=arm-mcp#build-the-litert-benchmark-tool-with-kleidiai-and-sme2-enabled)\n- [XNNPACK Bazel feature defaults and explicit settings](https://github.com/google/XNNPACK/blob/master/BUILD.bazel)\n- [LiteRT benchmark target](https://github.com/google-ai-edge/LiteRT/blob/main/litert/tools/BUILD)\n- [XNNPACK CMakeLists.txt (SME2 build options)](https://github.com/google/XNNPACK/blob/master/CMakeLists.txt)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q05", "slug": "step-by-guide", "question": "Is there a step-by-step guide for benchmarking ExecuTorch on an SME2-capable device?", "summary": "Yes. Arm provides a dedicated Learning Path — \"Profile ExecuTorch models with SME2 on Arm\" — that walks through the complete workflow for benchmarking ExecuTorch models with and without SME2 acceleration. The guide uses the sme-executorch-profiling repository, which contains all scripts, configurations, and example models needed to run end-to-end performance comparisons and collect operator-level ETDump traces on both Android and macOS targets.", "answer": "Yes. Arm provides a dedicated Learning Path — **\"Profile ExecuTorch models with SME2 on Arm\"** — that walks through the complete workflow for benchmarking ExecuTorch models with and without SME2 acceleration. The guide uses the `sme-executorch-profiling` repository, which contains all scripts, configurations, and example models needed to run end-to-end performance comparisons and collect operator-level ETDump traces on both Android and macOS targets.\n\n## Prerequisites\n\n| Requirement | Details |\n|---|---|\n| Host machine | Apple Silicon macOS or Arm Linux with Python 3.9+ and CMake 3.29+ |\n| Android NDK | r27 or later (set `ANDROID_NDK` for Android builds) |\n| Target device | A documented SME2 device, such as Apple M4/M5 or Android hardware using Dimensity 9500, Exynos 2600, or Tensor G6 |\n| ADB | Required for Android target communication |\n| Familiarity | Basic PyTorch and ExecuTorch knowledge |\n\n## Step 1: Clone the Profiling Repository\n\n```bash\nmkdir -p ~/sme2_analysis_work\ncd ~/sme2_analysis_work\ngit clone https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling.git executorch_sme2_kit\ncd executorch_sme2_kit\n```\n\nThis creates a self-contained workspace. The Python virtual environment, ExecuTorch build outputs, models, and profiling runs all live under this directory.\n\n## Step 2: Set Up the Environment\n\n```bash\nbash model_profiling/scripts/setup_repo.sh\nsource .venv/bin/activate\n```\n\nThe setup script:\n- Creates and activates a Python virtual environment under `.venv/`\n- Clones the ExecuTorch repository (if not already present)\n- Initializes required submodules\n- Installs ExecuTorch in editable mode\n\nThe `executorch/` directory must keep that exact name — CMake presets depend on it.\n\n## Step 3: Build ExecuTorch Runners with SME2 Support\n\n```bash\nexport ANDROID_NDK=/path/to/android-ndk # Required for Android builds\nbash model_profiling/scripts/build_runners.sh\n```\n\nThe build script produces **four runner variants** for direct SME2 on/off comparison:\n\n| Runner | Path | Purpose |\n|--------|------|---------|\n| Android SME2-on | `executorch/cmake-out/android-arm64-v9a/executor_runner` | SME2-accelerated mobile benchmarks |\n| Android SME2-off | `executorch/cmake-out/android-arm64-v9a-sme2-off/executor_runner` | Baseline mobile benchmarks |\n| macOS SME2-on | `executorch/cmake-out/mac-arm64/executor_runner` | SME2-accelerated desktop benchmarks |\n| macOS SME2-off | `executorch/cmake-out/mac-arm64-sme2-off/executor_runner` | Baseline desktop benchmarks |\n\nThe build system uses CMake presets (merged from `model_profiling/assets/cmake_presets.json`) — no manual CMake flags needed. Key build settings:\n\n- `EXECUTORCH_ENABLE_EVENT_TRACER: ON` — enables ETDump trace generation\n- `EXECUTORCH_BUILD_DEVTOOLS: ON` — enables performance analysis tools\n- `EXECUTORCH_XNNPACK_ENABLE_KLEIDI: ON` — enables Arm KleidiAI kernels\n- `EXECUTORCH_BUILD_KERNELS_QUANTIZED: ON` — enables INT8 quantized kernel support\n\nSME2 acceleration is enabled by default in XNNPACK when building for Arm architectures, so `XNNPACK_ENABLE_ARM_SME2` is not needed explicitly.\n\n**Important:** Runners are model-agnostic — build them once, reuse for all models. Only rebuild if you change CMake configurations.\n\n## Step 4: Stabilize the Target Device (Android)\n\n```bash\nadb devices\nadb shell settings put global low_power 0\nadb shell settings put global stay_on_while_plugged_in 15\nadb shell svc power stayon true\n```\n\nKeep power management, thermal throttling, charging state, and thread affinity settings consistent across all comparison runs to ensure reproducible results.\n\n## Step 5: Run the Quick Smoke Test\n\n```bash\npython model_profiling/scripts/run_quick_test.py\n```\n\nThis verifies that the environment, runners, and device connectivity are all working before committing to a full profiling run.\n\n## Step 6: Export Your Model to .pte Format\n\n```bash\npython model_profiling/export/export_model.py \\\n --model toy_cnn \\\n --dtype fp16 \\\n --outdir model_profiling/out_toy_cnn/artifacts/\n```\n\nBuilt-in models include `toy_cnn` (smoke test) and `mobilenet_v3_small`. To add your own model:\n\n1. Create `model_profiling/models//` directory\n2. Implement the `EagerModelBase` interface in `model.py`:\n - `get_eager_model()` — returns the PyTorch model\n - `get_example_inputs()` — returns tuple of example tensors\n3. Register in `model_profiling/models/__init__.py`\n\nExport dtype options include FP16, FP32 (floating-point), and INT8 (static or dynamic quantization).\n\n**Version compatibility:** The `.pte` file and runners must use the same ExecuTorch revision.\n\n## Step 7: Run the Profiling Pipeline\n\nFor Android:\n```bash\npython model_profiling/scripts/android_pipeline.py \\\n --config model_profiling/configs/examples/android_mobilenet_fp16.json\n```\n\nFor macOS:\n```bash\npython model_profiling/scripts/mac_pipeline.py \\\n --config model_profiling/configs/examples/mac_toy_cnn_fp16.json\n```\n\nThe pipeline performs a **two-run workflow** per configuration:\n\n1. **Timing-only run** — no trace logging overhead; provides accurate end-to-end latency measurements\n2. **Trace-enabled run** — generates ETDump for operator-level analysis (XNNPACK trace logging alters timing, so use only for kernel selection verification)\n\nBoth SME2-on and SME2-off runners are executed automatically for comparison.\n\n## Step 8: Analyze Results\n\nAnalysis runs automatically after the pipeline completes. To rerun manually:\n\n```bash\npython model_profiling/scripts/analyze_results.py \\\n --run-dir model_profiling/out_toy_cnn/runs/android\n```\n\nThe analysis generates:\n\n- **`analysis_summary.json`** — operator-category breakdown\n- **CSV files** — per-operator timing data\n- **Comparison reports** — SME2-on vs SME2-off side-by-side\n\n### Operator Categories\n\n| Category | Description | Expected SME2 impact |\n|----------|-------------|---------------------|\n| Convolution | Conv2d operations | Significant speedup |\n| GEMM | Linear/MatMul operations | Significant speedup |\n| Data Movement | Transpose, permute, copy | Often becomes dominant % after SME2 |\n| Elementwise | Add, mul, relu, etc. | Usually small |\n| Other | Everything else | Check for unexpected operations |\n\nAfter SME2 accelerates CONV/GEMM, data movement typically becomes a larger percentage of total time — this is expected and reveals the next optimization target.\n\n## Benchmarking Best Practices\n\n- **Multiple repetitions:** Run many iterations, discard warmups, and report median or percentile latency rather than a single measurement\n- **Separate timing from tracing:** Use timing-only runners for latency comparisons; use trace-enabled runners only to verify kernel selection\n- **Consistent device state:** Lock CPU frequency, disable thermal throttling if possible, and maintain consistent charging state\n- **Android vs macOS:** Android provides the most representative results (real thermal, bandwidth, and scheduling constraints); macOS works for learning the workflow before deploying to hardware\n- **Version alignment:** Keep model export and runner builds at the same ExecuTorch revision\n\n## Resources\n\n- [Environment setup and build instructions](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline/?utm_source=arm-mcp#build-executorch-runners-with-sme2-support)\n- [Model export and profiling](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/03-model-onboarding-and-profiling/?utm_source=arm-mcp#onboard-a-model-and-analyze-its-performance)\n- [sme-executorch-profiling GitHub repository](https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q06", "slug": "build-run-litert", "question": "How do I build and run LiteRT-LM with SME2 acceleration on a supported Arm device, and how can I verify that the SME2 path is being used?", "summary": "Build LiteRT-LM's CPU runner for Arm64, make sure its pinned LiteRT/XNNPACK dependency includes KleidiAI SME2 kernels, and test a compatible .litertlm model on a device where the OS exposes SME2. LiteRT-LM's upstream build guide documents the runner and CPU benchmark. The SME2 selection must be checked separately: CPU execution, a build define, or a profiling label alone is not proof that an SME2 instruction ran.", "answer": "Build LiteRT-LM's CPU runner for Arm64, make sure its **pinned LiteRT/XNNPACK dependency** includes KleidiAI SME2 kernels, and test a compatible `.litertlm` model on a device where the OS exposes SME2. LiteRT-LM's upstream build guide documents the runner and CPU benchmark. The SME2 selection must be checked separately: CPU execution, a build define, or a profiling label alone is not proof that an SME2 instruction ran.\n\n## Build and Run the CPU Runner\n\nStart from a pinned LiteRT-LM tag and follow its [source-build guide](https://github.com/google-ai-edge/LiteRT-LM/blob/main/docs/getting-started/build-and-run.md). The Android example uses Bazel's Arm64 configuration:\n\n```bash\ngit clone https://github.com/google-ai-edge/LiteRT-LM.git\ncd LiteRT-LM\ngit lfs pull\nbazel build -c opt --config=android_arm64 \\\n --enable_platform_specific_config \\\n //runtime/engine:litert_lm_main\n```\n\nFor a native Arm64 Linux or macOS build, follow the platform section of the same guide and omit `--config=android_arm64`. Use a `.litertlm` model compatible with the pinned runner version.\n\nDeploy and run on Android:\n\n```bash\nadb shell mkdir -p /data/local/tmp/litert-lm\nadb push bazel-bin/runtime/engine/litert_lm_main /data/local/tmp/litert-lm/\nadb push /path/to/model.litertlm /data/local/tmp/litert-lm/model.litertlm\nadb shell /data/local/tmp/litert-lm/litert_lm_main \\\n --backend=cpu \\\n --model_path=/data/local/tmp/litert-lm/model.litertlm\n```\n\nFor a model-level measurement, add `--benchmark --enable_profiling --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --async=false`. The benchmark reports prompt processing and generation behavior; profiling can add overhead, so compare latency with profiling disabled as well.\n\n## Make the SME2 Kernels Available\n\nLiteRT can build XNNPACK and KleidiAI with the Bazel defines below, as shown in Arm's [LiteRT SME2 build walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/3-build-tool/):\n\n```text\n--define=tflite_with_xnnpack=true\n--define=xnn_enable_arm_sme=true\n--define=xnn_enable_arm_sme2=true\n--define=xnn_enable_kleidiai=true\n```\n\nThose flags are documented for **LiteRT's benchmark target**. Before applying them to LiteRT-LM, inspect the pinned LiteRT-LM Bazel dependency and build action to confirm that the same defines reach its CPU runtime and that the SME2 kernels are compiled into the binary. LiteRT-LM's published build guide does not itself establish that passing these defines to `litert_lm_main` enables an end-to-end SME2 path for every model. If you control LiteRT as a separate dependency, build and link the matching configured version rather than assuming an unrelated LiteRT build changes LiteRT-LM's vendored dependency.\n\nAvoid global `-march=...+sme2` on a binary intended for older Arm64 devices. Specialized kernels should be built with their own target flags and selected through runtime dispatch.\n\n## Verify the Selected Kernel\n\n1. Confirm OS exposure on the target device, for example through `HWCAP2_SME2` on Linux/Android or `sysctl hw.optional.arm.FEAT_SME2` on macOS. Check the actual device and CPU affinity; do not assume a fixed core number is SME2-capable.\n2. Run `--backend=cpu` and inspect the lowered operators and XNNPACK delegate. A node label such as `PF32` can indicate a packed or KleidiAI path in a particular XNNPACK version, but it is **not by itself proof of SME2**.\n3. Trace the selected microkernel symbol or break on it, then inspect its disassembly for SME-family instructions. A `kai_*` symbol establishes a KleidiAI function; its feature suffix or instructions distinguish SME2 from another KleidiAI variant.\n4. Compare a build with SME2 kernels available to a build with that path disabled, using the same model, input, thread count, affinity, and thermal conditions. Keep the trace run separate from timing.\n\nIf no SME2 kernel appears, check the model's operator, shape, quantization and packing format against the pinned backend's kernel table. A working CPU fallback is expected and should not be reported as SME2 acceleration.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q07", "slug": "onednn-support-sme", "question": "Does oneDNN support SME?", "summary": "Yes. As of oneDNN 3.13, oneDNN contains a native AArch64 JIT SME implementation for a constrained set of BRGEMM-backed matmul cases. This is separate from oneDNN's optional Arm Compute Library (ACL) integration, which can also reach operation-specific SME kernels. It is therefore accurate to say that oneDNN supports SME, but not that every oneDNN primitive or every AArch64 build uses it.", "answer": "Yes. As of oneDNN 3.13, oneDNN contains a native AArch64 JIT SME implementation for a constrained set of BRGEMM-backed `matmul` cases. This is separate from oneDNN's optional Arm Compute Library (ACL) integration, which can also reach operation-specific SME kernels. It is therefore accurate to say that oneDNN supports SME, but not that every oneDNN primitive or every AArch64 build uses it.\n\n## Native oneDNN SME Path\n\nThe oneDNN 3.13.2 source contains all of the essential pieces of a direct implementation:\n\n- an `sme` CPU ISA value and runtime feature test;\n- an AArch64 JIT file, `jit_brgemm_sme_kernel.cpp`, that emits SME tile loads, FMOPA, and streaming-mode code;\n- SME selection in the BRGEMM dispatcher; and\n- an explicit `brgemm_matmul_t` instantiation.\n\nThe current kernel is focused on FP32 matrix multiplication. It does not accept bias, post-operations, or non-default source, weight, or destination scales. Requests outside those constraints are rejected by the SME implementation and can be handled by another oneDNN implementation. This is specialized matmul enablement, not a general SME backend for convolution, normalization, activations, and every other primitive.\n\nSME2-capable hardware can run this baseline SME kernel because SME2 requires SME. The presence of SME2 does not turn the kernel into an SME2 implementation; the emitted operations and oneDNN dispatch level are what determine that.\n\n## ACL Is a Second, Indirect Route\n\noneDNN can also be built with Arm Compute Library support:\n\n```bash\ncmake -S . -B build \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DONEDNN_TARGET_ARCH=AARCH64 \\\n -DONEDNN_AARCH64_USE_ACL=ON \\\n -DACL_ROOT_DIR=/path/to/ComputeLibrary\ncmake --build build --parallel\n```\n\nEligible primitives may then be delegated to ACL. A recent ACL build contains its own SME/SME2 kernels and runtime controls, so this path can use SME even when the executing code is owned by ACL rather than oneDNN. Delegation remains dependent on the operation, layout, data type, ACL version, and build options.\n\n## How to Verify the Selected Implementation\n\nFirst verify that the OS exposes SME:\n\n```bash\ngrep -m1 '^Features' /proc/cpuinfo | tr ' ' '\\n' | grep '^sme'\n```\n\nThen enable oneDNN's verbose log:\n\n```bash\nONEDNN_VERBOSE=all ./your_workload 2>onednn.log\n```\n\nCheck the implementation name and primitive description in the log. For a definitive check, profile or disassemble the selected JIT code and look for streaming-mode transitions and SME operations such as FMOPA. If testing the ACL route, use the verbose output to confirm ACL delegation and then verify the implementation selected inside ACL.\n\nBenchmark the native SME path, ACL path, and non-SME AArch64 path with identical matrix dimensions, thread count, affinity, warm-up, and timing method. Small matrices or layouts requiring conversion can erase the benefit of a faster compute kernel.\n\n## Resources\n\n- [oneDNN releases](https://github.com/uxlfoundation/oneDNN/releases)\n- [oneDNN source repository](https://github.com/uxlfoundation/oneDNN)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q08", "slug": "blis-blas-like", "question": "Does BLIS (BLAS-like Library Instantiation Software) support SME?", "summary": "Not as a production feature in the current upstream BLIS release. BLIS can host architecture-specific SME microkernels, and an open pull request adds experimental Apple-M4-focused SME2 configurations, but that work is not yet equivalent to stable, generally available upstream SME support.", "answer": "Not as a production feature in the current upstream BLIS release. BLIS can host architecture-specific SME microkernels, and an open pull request adds experimental Apple-M4-focused SME2 configurations, but that work is not yet equivalent to stable, generally available upstream SME support.\n\n## Current Status\n\nBLIS separates its portable BLAS framework from architecture-specific packing and compute microkernels. Its released Arm configurations cover conventional AArch64/Neon and SVE targets. They do not currently include a released general-purpose SME or SME2 configuration.\n\nPull request [flame/blis#921](https://github.com/flame/blis/pull/921) adds two experimental configurations:\n\n- `m4sme_p`, tuned for Apple M4 performance cores;\n- `m4sme_e`, tuned for Apple M4 efficiency cores; and\n- an `armsme` kernel directory containing SME2 packing and Level-3 GEMM kernels.\n\nThe proposal contains single- and double-precision GEMM microkernels and uses ZA when packing. Its kernels are streaming-vector-length agnostic, but its BLIS block sizes and context are tuned for Apple's 512-bit streaming vector length. The work is primarily enabled for recent Clang on Darwin and remains an open pull request, so production users should treat its API, dispatch, compiler support, and performance as experimental.\n\n## Why BLIS Needs New Microkernels\n\nBLIS's established microkernel model maps naturally to 1D Neon or SVE registers. SME changes the inner kernel substantially:\n\n- accumulation lives in the 2D ZA array;\n- FMOPA-class instructions update matrix tiles by outer product;\n- functions must obey streaming-mode and ZA ABI rules; and\n- packing and block sizes must account for streaming vector length and the target cache hierarchy.\n\nThe BLIS framework is flexible enough to register those kernels, but an SVE kernel does not become an SME kernel merely by compiling with `+sme`.\n\n## Practical Choices\n\nFor evaluation on Apple M4/M5, test PR #921 in an isolated build and compare it with the official BLIS release and Apple Accelerate. Run BLIS correctness tests before performance tests and retain a stable fallback.\n\nFor production SME linear algebra today, alternatives with released code include OpenBLAS's `ARMV9SME`/`VORTEXM4` targets, selected Arm Performance Libraries routines, and Arm Compute Library. KleidiAI is usually a better fit for quantized AI microkernels than for a general BLAS API.\n\n## Resources\n\n- [Experimental BLIS SME2 pull request #921](https://github.com/flame/blis/pull/921)\n- [BLIS plugin guide](https://github.com/flame/blis/blob/master/docs/PluginHowTo.md)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q09", "slug": "mlx-apple-s", "question": "Does MLX (Apple's array/machine-learning framework for Apple silicon) support SME?", "summary": "MLX has no explicit SME or SME2 backend. Its normal high-performance path is the Apple GPU through Metal; its CPU backend uses Apple CBLAS/Accelerate, BNNS for some types, and MLX's own SIMD code. A closed-source Apple library called by MLX might internally dispatch to SME on M4/M5, but that is not documented, guaranteed, selectable, or observable as an MLX feature.", "answer": "MLX has no explicit SME or SME2 backend. Its normal high-performance path is the Apple GPU through Metal; its CPU backend uses Apple CBLAS/Accelerate, BNNS for some types, and MLX's own SIMD code. A closed-source Apple library called by MLX might internally dispatch to SME on M4/M5, but that is not documented, guaranteed, selectable, or observable as an MLX feature.\n\n## Direct Versus Indirect Support\n\nDirect MLX SME support would require SME-specific source, build targets, CPU feature detection, streaming-mode/ZA ABI handling, and a public or internal dispatcher that selects those kernels. The current MLX tree does not expose an SME build switch or API and does not document SME kernels.\n\nIndirect use is possible in this narrower sense:\n\n```text\nMLX CPU operation\n → Accelerate CBLAS or BNNS\n → private Apple implementation\n → possibly SME on supported hardware\n```\n\nThat possibility should not be presented as \"MLX supports SME.\" MLX requests a system-library operation; Apple owns any instruction selection behind the ABI. The implementation may vary by macOS release, matrix shape, type, and processor.\n\n## What to Use in Practice\n\n- For normal MLX model training or inference, use the default Metal GPU path and benchmark it on the target Mac.\n- To test the CPU path, select the MLX CPU device and compare identical workloads, but do not infer SME use from speed alone.\n- To prove indirect SME execution, sample or disassemble the called Accelerate/BNNS implementation and look for SME streaming and outer-product instructions. This proves the system library used SME, not that MLX has an SME backend.\n- For guaranteed control of ZA, streaming mode, and kernel dispatch, implement a native C/C++ extension using ACLE/assembly or call a library with documented SME kernels.\n\nApple M4 and M5 hardware supporting SME does not make every framework operation use SME automatically. Hardware capability, OS exposure, the selected MLX device, and the native implementation all matter.\n\n## Resources\n\n- [MLX source repository](https://github.com/ml-explore/mlx)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q10", "slug": "jax-xla-google", "question": "Does JAX / XLA (Google's ML framework and its XLA compiler) support SME?", "summary": "Not as a first-class upstream JAX/XLA target. Ordinary JAX CPU compilation does not automatically lower jax.numpy matrix operations to ZA-based SME kernels, and standard jaxlib wheels should not be assumed to contain SME acceleration. LLVM and MLIR understand SME, but XLA does not automatically use MLIR's ArmSME lowering simply because its LLVM dependency can encode SME instructions.", "answer": "Not as a first-class upstream JAX/XLA target. Ordinary JAX CPU compilation does not automatically lower `jax.numpy` matrix operations to ZA-based SME kernels, and standard jaxlib wheels should not be assumed to contain SME acceleration. LLVM and MLIR understand SME, but XLA does not automatically use MLIR's ArmSME lowering simply because its LLVM dependency can encode SME instructions.\n\n## The Relevant Layers\n\n```text\nJAX → StableHLO/HLO → XLA CPU backend → LLVM-generated code or library call\n```\n\nUpstream XLA's AArch64 CPU feature handling covers Neon and limited SVE/SVE2 concerns; it does not expose an SME target or a documented path that converts HLO matmul into streaming-mode ZA operations. SME requires explicit tiling, data movement, streaming-mode attributes, and ZA lifetime management, so LLVM's ordinary loop vectorizer is not a substitute for an SME matrix lowering.\n\nMLIR does provide an ArmSME dialect and conversions capable of representing tiles and lowering matrix operations to SME intrinsics. That infrastructure can be used in custom compiler pipelines, but it is not the standard JAX/XLA CPU pipeline.\n\n## Ways a JAX Workload Can Still Reach SME\n\n### XLA Built with Arm Compute Library\n\nXLA contains optional ACL integration controlled at build time. In a custom build, eligible operations may follow this route:\n\n```text\nJAX → XLA CPU → ACL call → ACL runtime dispatch → SME/SME2 kernel\n```\n\nThis is indirect support. Only operations, layouts, shapes, and data types accepted by both the XLA-to-ACL path and an ACL SME kernel benefit. Standard wheels may not include it.\n\n### JAX FFI or an XLA Custom Call\n\nFor deterministic use, wrap an SME implementation from Arm Compute Library, KleidiAI, or custom ACLE/assembly and invoke it using JAX FFI. The wrapper must define shape/layout contracts, runtime feature checks, fallback behavior, and any custom differentiation rule.\n\n### Custom MLIR Pipeline\n\nA compiler project can lower StableHLO/linalg/vector operations through the ArmSME dialect and LLVM, but that is compiler engineering rather than an end-user JAX flag.\n\n## Verification\n\nConfirm `sme`/`sme2` in OS capabilities, inspect the installed jaxlib build configuration, dump XLA artifacts, and profile the native libraries. Search disassembly for SMSTART/SMSTOP, ZA loads/stores, and FMOPA/SMOPA-class operations. A compiler target flag alone does not prove the executing operator used SME.\n\n## Resources\n\n- [JAX FFI](https://docs.jax.dev/en/latest/ffi.html)\n- [MLIR ArmSME dialect](https://mlir.llvm.org/docs/Dialects/ArmSME/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q11", "slug": "numpy-scipy-python", "question": "Does NumPy / SciPy (Python numerical/scientific computing libraries) support SME?", "summary": "Conditionally and indirectly. NumPy and SciPy do not expose SME-specific Python APIs or general SME ufunc kernels. Their BLAS-backed operations can nevertheless execute SME when the installed package is linked to a native BLAS implementation containing SME kernels—most concretely, an SME-enabled OpenBLAS build.", "answer": "Conditionally and indirectly. NumPy and SciPy do not expose SME-specific Python APIs or general SME ufunc kernels. Their BLAS-backed operations can nevertheless execute SME when the installed package is linked to a native BLAS implementation containing SME kernels—most concretely, an SME-enabled OpenBLAS build.\n\n## Where SME Can Appear\n\nNumPy/SciPy execution has two relevant layers:\n\n- NumPy's own elementwise/reduction dispatcher handles operations such as `add`, `exp`, and many reductions. Its public Arm SIMD targets do not make SME a general ufunc backend.\n- Dense linear algebra such as `np.matmul`, `np.dot`, and many `numpy.linalg`/`scipy.linalg` routines calls BLAS/LAPACK. The linked library can select its own architecture kernel.\n\nOpenBLAS documents an `ARMV9SME` target and an Apple `VORTEXM4` target with SME kernels for selected single-precision Level-3 BLAS operations, including SGEMM, SSYMM, STRMM, SSYRK, and SSYR2K. Therefore, a suitable contiguous FP32 matrix multiplication may use SME if NumPy/SciPy is linked to a build containing that target and OpenBLAS selects it at runtime.\n\nThis is partial support. It does not imply that FP64, every LAPACK routine, sparse algorithms, FFTs, image processing, or NumPy ufuncs use SME. Portable PyPI/Conda wheels may omit the SME target.\n\n## Inspect and Test the Installation\n\n```bash\npython - <<'PY'\nimport numpy as np\nimport scipy\n\nnp.show_config()\nnp.show_runtime()\nscipy.show_config()\nPY\n```\n\nthreadpoolctl can identify the loaded BLAS library:\n\n```bash\npython -m pip install threadpoolctl\npython -m threadpoolctl -i numpy scipy\n```\n\nIf the backend is OpenBLAS, run a likely kernel with verbose dispatch:\n\n```bash\nOPENBLAS_VERBOSE=2 python -c \\\n'import numpy as np; a=np.ones((2048,2048),dtype=np.float32); print((a@a)[0,0])'\n```\n\nLook for an SME-capable OpenBLAS core name such as `ARMV9SME` or `VORTEXM4`. NumPy's own SIMD report may still show Neon because the SME instructions execute inside OpenBLAS. For definitive proof, use `openblas_get_corename()`, profiling, or disassembly.\n\n## Resources\n\n- [OpenBLAS supported targets and SME kernels](https://github.com/OpenMathLib/OpenBLAS)\n- [SciPy BLAS/LAPACK build selection](https://docs.scipy.org/doc/scipy/building/blas_lapack.html)", "tags": [ "SME" ], "golden": true }, { "id": "s03-q12", "slug": "performance-libraries-armpl", "question": "Does Arm Performance Libraries (ArmPL, Arm's optimized math/BLAS/LAPACK/FFT library) support SME?", "summary": "Yes, with a deliberately limited scope. ArmPL 26.01 introduced SME-optimized real SGEMM and DGEMM implementations for compatible Apple M-series processors on macOS. Applications keep using the standard BLAS API and ArmPL selects the implementation internally; this does not establish SME coverage across all ArmPL routines or platforms.", "answer": "Yes, with a deliberately limited scope. ArmPL 26.01 introduced SME-optimized real SGEMM and DGEMM implementations for compatible Apple M-series processors on macOS. Applications keep using the standard BLAS API and ArmPL selects the implementation internally; this does not establish SME coverage across all ArmPL routines or platforms.\n\n## What Is Documented\n\nArm's 26.01 announcement states that single- and double-precision real matrix-matrix multiplication uses SME rather than the previous Neon implementation on compatible Apple silicon. Arm reported up to 7× improvement for tested DGEMM matrix sizes relative to ArmPL 25.07.\n\nThe precise claims matter:\n\n- supported routines called out publicly: real SGEMM and DGEMM;\n- documented platform: compatible M-series Apple silicon running macOS;\n- activation: transparent behind standard BLAS calls;\n- version: ArmPL 26.01 or later.\n\nDo not generalize this to all LAPACK, FFT, sparse, random-number, libamath, Linux, or Windows code. Higher-level LAPACK routines may benefit when their execution spends time in an accelerated GEMM, but that is an indirect consequence and should be measured.\n\n## Using and Verifying It\n\nAn application continues to call normal CBLAS or Fortran entry points:\n\n```c\n#include \n\n/* cblas_sgemm(...) or cblas_dgemm(...), unchanged */\n```\n\nOn macOS, confirm the feature is exposed:\n\n```bash\nsysctl -n hw.optional.arm.FEAT_SME\nsysctl -n hw.optional.arm.FEAT_SME2\n```\n\nThen link the same benchmark against ArmPL 25.07 and 26.01 or later, keeping dimensions, transpose flags, leading dimensions, thread count, affinity, warm-up, and timing identical. Use ArmPL's logging/performance tools or system profiling where available to confirm which BLAS calls dominate. If instruction-level proof is required, profile/disassemble the executing library routine.\n\nSME support is internal to ArmPL, so application code does not need ``, streaming attributes, or a global `-march=...+sme` flag merely to call the optimized BLAS routine.\n\n## Resources\n\n- [What's new in Arm Performance Libraries 26.01](https://developer.arm.com/community/arm-community-blogs/b/servers-and-cloud-computing-blog/posts/what-s-new-in-arm-performance-libraries-26-01)\n- [Install Arm Performance Libraries](https://learn.arm.com/install-guides/armpl/?utm_source=arm-mcp#linux)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s03-q13", "slug": "angelslim-tencent-s", "question": "Does AngelSlim (Tencent's LLM compression toolkit) support SME?", "summary": "AngelSlim has demonstrated SME2-aware deployment for selected low-bit models, but it does not provide a general-purpose SME backend across the toolkit. The best-supported example is the 2-bit Hy-MT1.5 model path, where the compressed artifact is executed by a specialized runtime using SME2 kernels. Compression and hardware execution are separate layers.", "answer": "AngelSlim has demonstrated SME2-aware deployment for selected low-bit models, but it does not provide a general-purpose SME backend across the toolkit. The best-supported example is the 2-bit Hy-MT1.5 model path, where the compressed artifact is executed by a specialized runtime using SME2 kernels. Compression and hardware execution are separate layers.\n\n## What \"Support\" Means Here\n\nAngelSlim performs model compression—quantization, pruning, distillation, and related transformations. It does not need to emit SME instructions while creating a checkpoint. SME2 is used later by an inference runtime:\n\n```text\nAngelSlim compression/export\n → low-bit model artifact\n → compatible inference runtime and model format\n → KleidiAI or other SME2 microkernel\n → SME2-capable CPU\n```\n\nTencent's Hy-MT1.5 documentation describes a **2-bit SEQ** model optimized for SME2-capable devices and shows an SME2-versus-Neon comparison. Its published llama.cpp pull-request instructions, however, are for the **different 1.25-bit STQ1_0 format**. Do not use that branch as evidence that the 2-bit GGUF can load or use SME2 there.\n\nThat evidence is model- and format-specific. It does not show that all AngelSlim algorithms, exported quantization formats, or standard upstream inference engines have turnkey SME2 coverage. In particular:\n\n- baseline SME is not sufficient for a path requiring SME2;\n- the runtime must understand the exact low-bit format produced by AngelSlim;\n- the documented llama.cpp pull-request route is for 1.25-bit STQ1_0, not the 2-bit SEQ format;\n- a kernel for STQ1_0 does not imply support for SEQ; and\n- unsupported operations or formats may fall back to Neon or fail to load.\n\n## Validation Checklist\n\n1. Identify the exact AngelSlim model, quantization format, and export revision.\n2. Use the runtime and model format named by that model's deployment documentation. For the 2-bit SEQ artifact, require explicit runtime support before attempting the 1.25-bit STQ1_0 pull-request instructions.\n3. Check how that runtime implements its SME2 and fallback kernels; do not assume it uses KleidiAI because another llama.cpp path does.\n4. Confirm `sme2` is exposed by the target OS.\n5. Verify the loaded kernel in a dispatch trace or profiler. A model-buffer type alone does not identify the executed instruction set.\n6. Compare SME2 and Neon paths using the same artifact, thread count, affinity, prompt, and thermal state.\n\nThe published 2-bit comparison establishes a specific deployment claim, while the 1.25-bit instructions teach a separate model path. Keep those formats distinct when planning an integration.\n\n## Resources\n\n- [AngelSlim Hy-MT1.5 documentation](https://github.com/Tencent/AngelSlim/blob/main/docs/source/models/Hy-MT1.5/hy-mt1.5.md)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s04-q01", "slug": "walk-through-building", "question": "Can you walk me through building an SME2-accelerated app on Android using KleidiAI?", "summary": "Use an inference runtime that already integrates KleidiAI, then package that runtime in your Android app. KleidiAI provides microkernels; it does not load models or run a whole app. This walkthrough uses ONNX Runtime → MLAS → KleidiAI because it gives a Java/Kotlin app a standard model API. The key checkpoints are the custom native build, the AAR the app loads, a supported model operator, and evidence that runtime dispatch selected an SME2 kernel.", "answer": "Use an inference runtime that already integrates KleidiAI, then package that runtime in your Android app. KleidiAI provides microkernels; it does not load models or run a whole app. This walkthrough uses **ONNX Runtime → MLAS → KleidiAI** because it gives a Java/Kotlin app a standard model API. The key checkpoints are the custom native build, the AAR the app loads, a supported model operator, and evidence that runtime dispatch selected an SME2 kernel.\n\n## 1. Choose the Target and Model\n\nUse an Android arm64 device whose OS exposes SME2, such as a documented Dimensity 9500 phone. Check the actual device:\n\n```bash\nadb shell 'grep -m1 \"^Features\" /proc/cpuinfo'\n```\n\nLook for `sme2`. This is a useful shell check; production native code should query `getauxval(AT_HWCAP2)` and `HWCAP2_SME2`. Use an ONNX model with operations and tensor formats supported by the ONNX Runtime/MLAS/KleidiAI revision you will build. An SME2 CPU does not make every `Conv` or `MatMul` node eligible.\n\nInstall a matching Android SDK, NDK, Python, CMake, Ninja, and ADB. Pin the ONNX Runtime release so the build and model behavior can be reproduced.\n\n## 2. Build ONNX Runtime with KleidiAI\n\nFrom an ONNX Runtime checkout with the KleidiAI option, build the Android shared library and Java AAR:\n\n```bash\ngit clone --recursive https://github.com/microsoft/onnxruntime.git\ncd onnxruntime\ngit checkout v1.23.2\ngit submodule update --init --recursive\n\n./build.sh \\\n --android \\\n --android_sdk_path \"$ANDROID_SDK_ROOT\" \\\n --android_ndk_path \"$ANDROID_NDK_HOME\" \\\n --android_abi arm64-v8a \\\n --android_api 27 \\\n --config Release \\\n --build_shared_lib \\\n --build_java \\\n --cmake_generator Ninja \\\n --parallel \\\n --cmake_extra_defines onnxruntime_USE_KLEIDIAI=ON\n```\n\nSet `ANDROID_SDK_ROOT` and `ANDROID_NDK_HOME` to real directories before running this command. Check the CMake cache and build log to confirm KleidiAI was included. Find the generated AAR under the build output rather than assuming a fixed filename. Arm's [Android ONNX Runtime build guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/build_ort/) shows the release-specific workflow.\n\nDo not put `-march=...+sme2` in the app's global compiler flags. The runtime builds specialized kernels and selects a compatible one at execution time, preserving a fallback for older arm64 devices.\n\n## 3. Package the AAR and Run Inference\n\nCopy the **custom** AAR to `app/libs/` and include it in the Android app:\n\n```kotlin\ndependencies {\n implementation(files(\"libs/onnxruntime-release.aar\"))\n}\n```\n\nMake sure another ONNX Runtime Maven dependency is not replacing your custom native library. Put `model.onnx` in `app/src/main/assets/`, then create and close the session and tensors off the UI thread. This small example assumes a single float input and a model-specific output consumer:\n\n```kotlin\nimport ai.onnxruntime.OnnxTensor\nimport ai.onnxruntime.OrtEnvironment\nimport ai.onnxruntime.OrtSession\nimport java.nio.FloatBuffer\n\nval env = OrtEnvironment.getEnvironment()\nval model = assets.open(\"model.onnx\").use { it.readBytes() }\nenv.createSession(model, OrtSession.SessionOptions()).use { session ->\n val name = session.inputNames.first()\n OnnxTensor.createTensor(env, FloatBuffer.wrap(input), shape).use { tensor ->\n session.run(mapOf(name to tensor)).use { result ->\n consumeModelOutput(result[0].value) // Implement for your model's output type.\n }\n }\n}\n```\n\nRun this in a worker or coroutine dispatcher; a large model or JNI call on the main thread will block the UI. `input`, `shape`, and `consumeModelOutput` are application-specific, so this is an integration sketch rather than a copy-and-run Activity.\n\n## 4. Confirm Acceleration and Measure It\n\nFirst ensure the app loads your custom AAR and successfully runs the model. Then inspect the backend's selected function with profiling, a breakpoint, or disassembly. A `kai_*` symbol shows a KleidiAI function; its SME2 suffix or SME instructions establish the architecture path. Correct output or a faster run alone does not identify the executed kernel.\n\nFor a controlled comparison, build the same ONNX Runtime revision with `onnxruntime_USE_KLEIDIAI=OFF`, package that AAR in a separate app build, and measure the same inputs, thread count, CPU affinity, and device temperature. Report operator and end-to-end latency separately. Arm's [ONNX Runtime profiling guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) provides an on-device benchmark workflow.\n\nIf your app uses a different model runtime, choose its documented path instead: [LiteRT with XNNPACK and KleidiAI](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/3-build-tool/) for `.tflite` models, or llama.cpp's [Arm64 KleidiAI build](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#arm-kleidiai) for GGUF LLMs. Each still needs model-specific kernel and runtime verification.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s04-q02", "slug": "end-examples-running", "question": "Are there end-to-end examples of running an LLM on a mobile device using KleidiAI with SME2 acceleration?", "summary": "Yes. The clearest SME2-specific path is Arm's llama.cpp on Android walkthrough, which builds an Arm64 CPU runner, loads a GGUF model, and compares SME-family kernels with a fallback. For an Android app rather than a command-line experiment, Arm also has an ExecuTorch chat-app walkthrough. That app demonstrates KleidiAI integration, but its documented acceleration path may use other Arm features; verify SME2 separately on the target device.", "answer": "Yes. The clearest SME2-specific path is Arm's [llama.cpp on Android walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_llama_cpp_sme2/), which builds an Arm64 CPU runner, loads a GGUF model, and compares SME-family kernels with a fallback. For an Android app rather than a command-line experiment, Arm also has an [ExecuTorch chat-app walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/build-llama3-chat-android-app-using-executorch-and-xnnpack/). That app demonstrates KleidiAI integration, but its documented acceleration path may use other Arm features; verify SME2 separately on the target device.\n\n## Start with llama.cpp to Measure the CPU Path\n\nThe llama.cpp example suits a developer who wants to see the build, deployment, model loading, and kernel-selection path before packaging an app. It teaches how GGML's CPU backend reaches KleidiAI for supported `GGML_OP_MUL_MAT` operations.\n\nFor a current portable Android build, follow llama.cpp's [upstream Android Arm64 commands](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#arm-kleidiai):\n\n```bash\ngit clone https://github.com/ggml-org/llama.cpp.git\ncd llama.cpp\ncmake -S . -B build-android \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DCMAKE_TOOLCHAIN_FILE=\"$ANDROID_NDK/build/cmake/android.toolchain.cmake\" \\\n -DANDROID_ABI=arm64-v8a \\\n -DANDROID_PLATFORM=android-28 \\\n -DGGML_CPU_KLEIDIAI=ON \\\n -DGGML_NATIVE=OFF \\\n -DGGML_OPENMP=OFF \\\n -DGGML_LLAMAFILE=OFF \\\n -DLLAMA_OPENSSL=OFF\ncmake --build build-android --config Release --parallel\n```\n\nSet `ANDROID_NDK` to the NDK root. Push the resulting `llama-cli` and a GGUF model to a writable location such as `/data/local/tmp/`, then run the model with CPU execution selected. The Arm walkthrough gives the exact transfer and benchmark steps for its pinned revision. Do not copy an older Linux GNU toolchain recipe with global `+sme2` flags into a modern Android build; the NDK target and per-kernel dispatch matter.\n\nCompare prompt processing and token generation separately with `llama-bench`. `GGML_KLEIDIAI_SME=0` is a diagnostic way to disable SME-family kernels; leave it unset for normal automatic selection. A KleidiAI buffer message confirms buffer allocation, not that an SME2 instruction executed. Trace the selected microkernel or inspect its disassembly for that proof.\n\n## Then Build an App\n\n| Example | Best use | SME2 qualification |\n|---|---|---|\n| Arm's ExecuTorch Android chat app | Complete Android UI, model export, XNNPACK delegate, and KleidiAI integration | The tutorial teaches an app path; inspect the bundled backend and selected kernel on an SME2 device |\n| Arm AI Chat library | Smallest app integration for a GGUF chat model | Check the packaged native library and runtime selection; an SDK dependency alone is not evidence of SME2 |\n\nThe [AI Chat library guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/android-ai-chat-lib/) is useful if you want a Kotlin API and streaming tokens without first maintaining a custom JNI layer. The ExecuTorch guide is better when you need to export and quantize your own PyTorch model and inspect delegated operators. In either route, start with the guide's stated model format and build version, run on an OS-exposed SME2 device, and keep an SME2-disabled comparison build.\n\nThese examples do not guarantee that an arbitrary LLM or quantization format uses SME2. Kernel coverage depends on weight packing, activation type, matrix shape, and the runtime's bundled KleidiAI revision. In particular, a phone with an Exynos 2500 is not an SME2 test device merely because it is recent; check the actual SoC and `sme2` capability before using it for this experiment.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s04-q03", "slug": "kleidiai-integrate-executorch", "question": "How does KleidiAI integrate into ExecuTorch, and what performance improvement can I expect on SME2 hardware?", "summary": "ExecuTorch can delegate eligible model operators to XNNPACK, which can select KleidiAI microkernels for supported data types, quantization formats, and shapes. On an OS-exposed SME2 device, an eligible kernel can use SME2. There is no dependable general speedup factor: measure the selected operators and the complete model against an otherwise identical SME2-disabled build.", "answer": "ExecuTorch can delegate eligible model operators to XNNPACK, which can select KleidiAI microkernels for supported data types, quantization formats, and shapes. On an OS-exposed SME2 device, an eligible kernel can use SME2. There is **no dependable general speedup factor**: measure the selected operators and the complete model against an otherwise identical SME2-disabled build.\n\n## Where the Kernel Is Selected\n\n1. Export a PyTorch model as an ExecuTorch `.pte` program, with eligible operations delegated to XNNPACK.\n2. Build the runner with `EXECUTORCH_BUILD_XNNPACK=ON` and `EXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON`. In current [ExecuTorch CMake source](https://github.com/pytorch/executorch/blob/main/backends/xnnpack/cmake/Dependencies.cmake), `XNNPACK_ENABLE_ARM_SME2` is on by default and can be set explicitly.\n3. At runtime, XNNPACK selects a kernel based on CPU features and the operation's complete contract. If a KleidiAI SME2 kernel does not match, another compatible kernel handles the operation.\n\nDo not add `-march=...+sme2` to the whole runner. XNNPACK builds specialized microkernels separately, allowing the rest of the binary to run on older Arm64 devices.\n\nTypical candidates include fully connected layers, supported convolutions, and batch matrix multiplication. Their exact FP32, FP16, INT8, or packed 4-bit coverage is version-specific. Packed 4-bit **weights** do not mean SME2 performs native 4-bit arithmetic; the kernel decodes or expands values before accumulation. Arm's [ExecuTorch node-to-kernel guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/measure-kleidiai-kernel-performance-on-executorch/03-executorch-node-kai-kernel/) shows how to check the supported signatures in a documented revision.\n\n## What Has Been Measured\n\nIn one controlled [SqueezeSAM ExecuTorch/XNNPACK/KleidiAI case study](https://pytorch.org/blog/accelerating-on-device-ml-inference-with-executorch-and-arm-sme2/), the authors compared SME2 enabled and disabled on the same vivo X300:\n\n| Measurement | INT8 | FP16 |\n|---|---:|---:|\n| GEMM operator speedup, one core | About 3.4× | About 4.1× |\n| Convolution operator speedup, one core | About 4.4× | About 9.0× |\n| End-to-end inference, one core | 556 ms → 304 ms (1.83×) | 1,163 ms → 298 ms (3.90×) |\n| End-to-end inference, four cores | 195 ms → 180 ms (1.08×) | 374 ms → 193 ms (1.94×) |\n\nThese are measurements for that model, device, precision, backend, and thread setting. They are not expected ranges for a new model. The smaller four-core ratios show why single-core operator gains cannot be multiplied into an application forecast; data movement, unaccelerated operators, scheduling, and shared compute resources also matter.\n\n## Verify and Estimate Your Own Result\n\nBuild SME2-on and SME2-off runners from the same ExecuTorch revision. Run the same `.pte` model, inputs, thread count, affinity, and power mode on the target device. Use ETDump for operator timings and inspect XNNPACK/KleidiAI dispatch or a sampled kernel call stack for the selected microkernel. ETDump timing alone does not prove SME2 instructions executed.\n\nRank operators by **absolute time saved**, then measure end-to-end latency separately. If the accelerated layers were a small fraction of baseline runtime, the whole model will gain much less than those layers. If packing or layout conversion becomes dominant, the next optimization may be outside the SME2 compute kernel. Arm's [ExecuTorch profiling workflow](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline/) provides paired runners and trace/timing configurations.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s04-q04", "slug": "android-devices-currently", "question": "What Android devices currently support SME and can be used to test KleidiAI-accelerated apps?", "summary": "As of September 2026, Arm's native SME2 device list includes phones in the vivo X300 and OPPO Find X9 families with MediaTek Dimensity 9500, and SME2-capable variants of Samsung Galaxy S26 and Google Pixel 11. Check the exact regional model and OS-exposed feature before using a phone for testing. KleidiAI acceleration also requires a matching kernel and framework integration for your workload.", "answer": "As of September 2026, Arm's [native SME2 device list](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/#devices) includes phones in the vivo X300 and OPPO Find X9 families with MediaTek Dimensity 9500, and SME2-capable variants of Samsung Galaxy S26 and Google Pixel 11. Check the exact regional model and OS-exposed feature before using a phone for testing. KleidiAI acceleration also requires a matching kernel and framework integration for your workload.\n\n| Device family | Relevant SoC or variant | Testing note |\n|---|---|---|\n| vivo X300 | Dimensity 9500, Arm C1 cluster | Used in several Arm on-device SME2 examples |\n| OPPO Find X9 | Dimensity 9500, Arm C1 cluster | Check the exact model and software version |\n| Samsung Galaxy S26 | Exynos 2600 variant | A Snapdragon regional variant is a different processor |\n| Google Pixel 11 | Tensor G6 variant listed by Arm | Check the specific device and OS release |\n\nFor a first performance investigation, the vivo X300 is a practical choice because Arm's published workflows use it for [ONNX Runtime profiling](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) and other KleidiAI examples. That does not make it a universal reference for other phones or operators.\n\n## Check the Device\n\n```bash\nadb shell getprop ro.soc.model\nadb shell 'grep -m1 \"^Features\" /proc/cpuinfo'\n```\n\nLook for `sme2` in the kernel feature list. In app native code, query the process capability instead of parsing a device name:\n\n```c\n#include \n#include \n#include \n\nbool has_sme2(void) {\n return (getauxval(AT_HWCAP2) & HWCAP2_SME2) != 0;\n}\n```\n\nThis answers whether SME2 is available to the process. It does **not** prove the app's selected inference kernel uses SME2. Check the framework's selected `kai_*` or other backend kernel and compare with an SME2-disabled build.\n\nOlder phones built around Cortex-X4 or Cortex-X925, such as Dimensity 9300/9400 devices, are useful fallback test targets but do not gain SME merely from being Armv9. For functional development without hardware, Arm's [FVP setup](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/) can execute SME2 examples; use a real phone for latency or energy measurements.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s05-q01", "slug": "build-test-kleidicv", "question": "How do I build and test KleidiCV on Android to verify SME backend support?", "summary": "Build a KleidiCV example with the Android NDK, run it on a phone whose OS exposes SME, and verify which operation used the SME implementation. Passing correctness tests only proves that the built program runs; it does not prove that a runtime dispatcher chose SME. Arm's Android KleidiCV Gaussian blur Learning Path provides a current, reproducible route using the 26.06 release.", "answer": "Build a KleidiCV example with the Android NDK, run it on a phone whose OS exposes SME, and verify **which operation** used the SME implementation. Passing correctness tests only proves that the built program runs; it does not prove that a runtime dispatcher chose SME. Arm's [Android KleidiCV Gaussian blur Learning Path](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/explore-kleidicv-gaussian-blur-on-android/) provides a current, reproducible route using the 26.06 release.\n\n## Build and Run a Known SME Operation\n\nInstall a recent Android NDK, CMake, and ADB, and set `ANDROID_NDK_HOME` to the NDK directory. Check out the same KleidiCV release used by the guide:\n\n```bash\ngit clone https://gitlab.arm.com/kleidi/kleidicv.git\ncd kleidicv\ngit checkout --detach refs/tags/26.06\n\ncmake -S examples/extract_one_operation \\\n -B build/extract-android \\\n -DCMAKE_BUILD_TYPE=Release \\\n -DCMAKE_TOOLCHAIN_FILE=\"$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake\" \\\n -DANDROID_ABI=arm64-v8a \\\n -DANDROID_PLATFORM=android-21 \\\n -DANDROID_STL=c++_static\ncmake --build build/extract-android --target example_usage --parallel\n```\n\nBefore running the SME example, check that the device exposes `sme` (and `sme2` if you intend to test an SME2 operation):\n\n```bash\nadb shell 'grep -m1 \"^Features\" /proc/cpuinfo'\nadb push build/extract-android/example_usage /data/local/tmp/\nadb shell chmod 755 /data/local/tmp/example_usage\nadb shell /data/local/tmp/example_usage\n```\n\nThe example calls a particular SME Gaussian blur implementation directly. Compare its output with the guide's expected pixels. This verifies that **that** SME routine runs and produces the expected result on the device; it does not test the default OpenCV or KleidiCV dispatcher.\n\n## Compare Backends and Test Dispatch Separately\n\nThe same Learning Path provides a small performance explorer that calls Neon, SVE2, and SME Gaussian blur implementations directly, checks them against a Neon reference, and reports per-operation timing. Its [build and run step](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/explore-kleidicv-gaussian-blur-on-android/3-build-and-customize-the-performance-explorer/) includes the extra source and CMake target needed for the 26.06 checkout. Do not expect `gaussian_blur_benchmark` in the unmodified release; the guide has you add it.\n\nTo verify **runtime dispatch** for your actual application, call the same KleidiCV or OpenCV operation your app uses, then inspect the resolved function or add temporary logging to its resolver. Current KleidiCV versions distinguish ordinary APIs, which normally choose Neon or SVE2, from `_sme` APIs that prefer SME/SME2 when available. Do not assume a global \"SME2 before Neon\" ordering for every call. Arm's [backend selection guide](https://learn.arm.com/learning-paths/laptops-and-desktops/kleidicv-on-mac/run-test-2/) shows how to inspect the selected variant.\n\nMeasure latency on physical hardware with the same image, border mode, kernel size, CPU affinity, and thermal conditions. An FVP or emulator can help with functional testing but cannot predict phone performance.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s05-q02", "slug": "computer-vision-operations", "question": "What computer vision operations does KleidiCV accelerate using SME, and how do they compare to standard OpenCV?", "summary": "KleidiCV has SME implementations for selected image-processing operations, including regular color conversion, fixed-stripe filtering, resize, morphology, and reductions. It can be used directly or through OpenCV's HAL. The operation and its parameters must match an implemented path, and current KleidiCV APIs distinguish ordinary dispatch from _sme variants that prefer SME. Do not interpret every OpenCV speedup as an SME speedup.", "answer": "KleidiCV has SME implementations for **selected** image-processing operations, including regular color conversion, fixed-stripe filtering, resize, morphology, and reductions. It can be used directly or through OpenCV's HAL. The operation and its parameters must match an implemented path, and current KleidiCV APIs distinguish ordinary dispatch from `_sme` variants that prefer SME. Do not interpret every OpenCV speedup as an SME speedup.\n\n## Where SME Is Useful\n\nThe exact list changes by release and data type. Examples in KleidiCV's documented backends include:\n\n| Area | Examples | Eligibility detail to check |\n|---|---|---|\n| Filtering | Fixed-kernel Gaussian blur, separable filtering, Sobel/Scharr, some median blur paths | Kernel size, image type, channels, border mode |\n| Color conversion | RGB/BGR/RGBA and selected YUV formats | Pixel layout and chroma representation |\n| Geometry and morphology | Linear resize, selected downsample, erode, dilate | Scale, strides, and image dimensions |\n| Reductions and arithmetic | Min/max, sum, exp, threshold, selected type conversions | Input/output type and whether location indices are requested |\n\nThis is an overview, not a claim that every listed OpenCV call reaches SME. Some cases use Neon or SVE2, and unsupported parameters fall through to OpenCV's own implementation. In [KleidiCV 26.03](https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/what-s-new-in-kleidicv-26-03-for-computer-vision-on-arm-cpus), Arm added more optimized remap, warp perspective, transpose, filtering, and optical-flow coverage. Those additions do not establish SME backends for every new variant.\n\n## Compare the Same Operation\n\nOpenCV's HAL lets an application continue calling, for example, `cv::GaussianBlur` or `cv::cvtColor`. With a KleidiCV-enabled OpenCV build, an eligible call may enter a KleidiCV implementation. Ordinary KleidiCV APIs in recent releases generally choose Neon or SVE2 by default; the `_sme` variants prefer SME or SME2 and can fall back. Verify the resolved implementation for the exact function, type, and parameters you use.\n\nArm reports gains of up to 14× for selected **OpenCV with KleidiCV versus OpenCV without it** operations on Cortex-A710. That core has no SME, so those figures are evidence for KleidiCV integration, **not** SME versus Neon. For a direct SME comparison, Arm's [Android Gaussian blur explorer](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/explore-kleidicv-gaussian-blur-on-android/3-build-and-customize-the-performance-explorer/) runs Neon, SVE2, and SME implementations on the same input and checks output before timing. In its 15×15 example on one SME-capable phone, SME's median latency was 1.45× to 2.01× lower than Neon's across the tested resolutions. That result is specific to Gaussian blur, kernel size, device, and CPU affinity.\n\nFor an application-level comparison, build the same OpenCV version with and without KleidiCV, run identical image pipelines, and profile each call. Then test an explicit SME variant or trace the resolver to attribute any gain to SME. Include image decoding, transfers, and rendering in an end-to-end camera measurement; a fast filter alone does not predict frame rate.\n\n## Resources\n\n- [Arm's KleidiCV backend and operation update](https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/what-s-new-in-kleidicv-26-03-for-computer-vision-on-arm-cpus)\n- [OpenCV and KleidiCV Android app walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/android_opencv_kleidicv/)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s05-q03", "slug": "example-integrating-kleidicv", "question": "Is there an example of integrating KleidiCV into an Android camera application for real-time image processing with SME?", "summary": "Arm's OpenCV plus KleidiCV Android app processes images from assets, while its OpenCV face-detection app shows live camera capture. Together they provide a useful starting point, but they are not a published end-to-end camera app proving SME execution. You must connect the frame pipeline, build OpenCV with KleidiCV, and verify the selected backend for each operation.", "answer": "Arm's [OpenCV plus KleidiCV Android app](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/android_opencv_kleidicv/) processes images from assets, while its [OpenCV face-detection app](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/android_opencv_facedetection/) shows live camera capture. Together they provide a useful starting point, but they are **not** a published end-to-end camera app proving SME execution. You must connect the frame pipeline, build OpenCV with KleidiCV, and verify the selected backend for each operation.\n\n## Connect Camera Frames to OpenCV\n\nUse CameraX `ImageAnalysis` with `STRATEGY_KEEP_ONLY_LATEST` so the processing pipeline does not queue stale frames. A simple first step is to process the Y plane as grayscale. CameraX planes have their own row strides and may not form one tightly packed NV21 buffer, so do **not** wrap the Y-plane pointer as a `height * 3 / 2` NV21 image or assume the U/V planes follow it in memory.\n\nThe analyzer should close each `ImageProxy` even when processing fails:\n\n```kotlin\nval analysis = ImageAnalysis.Builder()\n .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)\n .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)\n .build()\n\nanalysis.setAnalyzer(cameraExecutor) { frame ->\n try {\n val y = frame.planes[0]\n processGrayFrame(\n y.buffer, y.rowStride, y.pixelStride,\n frame.width, frame.height\n )\n } finally {\n frame.close()\n }\n}\n```\n\n`processGrayFrame` is an application JNI function to implement. For `pixelStride == 1`, native code can wrap the Y buffer with `cv::Mat(height, width, CV_8UC1, data, rowStride)` while the frame remains open, then call an out-of-place operation such as `cv::GaussianBlur`. For any other pixel stride, copy pixels into a packed grayscale buffer first. Do not retain the plane pointer after `frame.close()`; copy the processed result if it must outlive the callback.\n\nIf you need color, convert the three planes using their actual row and pixel strides or use a CameraX output format your conversion code explicitly supports. This avoids a common corruption bug in camera examples that silently treat YUV_420_888 as NV21.\n\n## Build and Verify the Backend\n\nFollow Arm's OpenCV/KleidiCV Android guide for a **pinned OpenCV version and matching native library**. The Maven artifact or an OpenCV build flag alone does not prove the packaged binary includes the desired KleidiCV release. Arm reports KleidiCV enabled by default for AArch64 OpenCV builds from **4.13**; older 4.11-based examples need their documented build configuration.\n\nAfter the app works, log `Core.getBuildInformation()` to confirm that the loaded OpenCV build contains KleidiCV. Then inspect the selected implementation for the particular Gaussian blur, color conversion, or resize call. Current KleidiCV ordinary APIs generally select Neon or SVE2; `_sme` variants prefer SME/SME2 and can fall back. An OpenCV HAL call is therefore **not automatically an SME call**. Arm's [KleidiCV backend inspection guide](https://learn.arm.com/learning-paths/laptops-and-desktops/kleidicv-on-mac/run-test-2/) demonstrates resolver tracing.\n\nMeasure the whole frame path: capture, any YUV conversion, filtering, copy/JNI overhead, and display. Compare the same frame size and parameters on the same device with and without the SME implementation selected. A microkernel gain may be much smaller in the final camera frame time.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s05-q04", "slug": "kleidicv-s-sme", "question": "How does KleidiCV's SME backend differ from its Neon backend in terms of supported operations and performance?", "summary": "Neon is KleidiCV's broad AArch64 baseline. Its SME/SME2 implementations cover a subset of operations and require an OS-exposed SME feature. Their streaming vectors can process more pixels per iteration on some hardware, but entering streaming mode and handling small or irregular work can erase the gain. The chosen backend depends on the API variant and operation, not just a fixed ranking of CPU features.", "answer": "Neon is KleidiCV's broad AArch64 baseline. Its SME/SME2 implementations cover a **subset** of operations and require an OS-exposed SME feature. Their streaming vectors can process more pixels per iteration on some hardware, but entering streaming mode and handling small or irregular work can erase the gain. The chosen backend depends on the **API variant and operation**, not just a fixed ranking of CPU features.\n\n## Selection and Coverage\n\nArm's [KleidiCV 26.03 update](https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/what-s-new-in-kleidicv-26-03-for-computer-vision-on-arm-cpus) clarifies the important distinction:\n\n- Ordinary KleidiCV APIs use Neon or SVE2 by default.\n- APIs with an `_sme` postfix prefer an available SME or SME2 implementation and fall back when it is unavailable.\n- `KLEIDICV_PREFER_SME_BACKEND` and build options can affect selection in a particular release. Check that release's reference guide before relying on an override.\n\nThis is more accurate than saying every call always chooses SME2 before SME, SVE2, and Neon. A device can support SME2 while a given routine has only a Neon implementation.\n\n| Kind of work | SME examples in documented releases | Neon role |\n|---|---|---|\n| Regular filtering | Fixed-stripe Gaussian blur, selected median, Sobel/Scharr, separable filters | Covers additional sizes, types, border modes, and fallbacks |\n| Pixel conversion | Selected RGB/YUV layouts and type conversions | Covers other formats and parameter combinations |\n| Geometry and morphology | Selected linear resize, downsample, erode/dilate | Handles more general or irregular cases |\n| Reductions | Selected min/max and sum paths | Handles cases such as location tracking or unsupported types |\n\nTreat that table as representative: kernel coverage changes by release and depends on data type, channels, strides, and parameters. Some operations previously described as \"Neon-only,\" such as remap and warp perspective, gained **selected** optimized cases in later KleidiCV versions; inspect the exact implementation rather than relying on an old category list.\n\n## Performance Comparison\n\nArm's headline \"up to 14×\" result for KleidiCV 26.03 compares **OpenCV with KleidiCV against OpenCV without it** on Cortex-A710. That processor has no SME, so the figure says nothing about SME versus Neon.\n\nFor a direct comparison, Arm's [Android Gaussian blur explorer](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/explore-kleidicv-gaussian-blur-on-android/3-build-and-customize-the-performance-explorer/) calls the Neon, SVE2, and SME implementations on the same input, checks their output, and measures each on the same CPU. In its 15×15 grayscale example, SME's median time was about 1.45× lower than Neon's at 640×640 and 2.01× lower at 3840×2160. This is one operation and device, not a forecast for all KleidiCV calls.\n\nTo evaluate your app, trace the resolved implementation for a specific call, then compare matched images, kernel sizes, border modes, thread count, affinity, and thermal state. Time the full pipeline separately. The wider vector path can help long regular stripes; small images, transition overhead, memory bandwidth, and conversion work may favor another backend.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s05-q05", "slug": "applications-kleidicv-beyond", "question": "Are there any applications that use KleidiCV beyond OpenCV or is this the only place it is used?", "summary": "OpenCV is the best-documented framework integration, but it is not the only way to use KleidiCV. KleidiCV also exposes a standalone C API that an application or another computer-vision framework can call directly. As of September 2026, I would not claim another major public framework integration without identifying its source and release.", "answer": "OpenCV is the best-documented **framework integration**, but it is not the only way to use KleidiCV. KleidiCV also exposes a standalone C API that an application or another computer-vision framework can call directly. As of September 2026, I would not claim another major public framework integration without identifying its source and release.\n\n| Route | What the developer calls | When it fits |\n|---|---|---|\n| OpenCV integration | Standard OpenCV operations such as `cv::GaussianBlur` | An existing OpenCV application |\n| Direct KleidiCV API | KleidiCV image-processing functions | A custom camera, pre/post-processing, or embedded pipeline that does not need all of OpenCV |\n\nThe [KleidiCV 26.03 announcement](https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/what-s-new-in-kleidicv-26-03-for-computer-vision-on-arm-cpus) describes direct integration as well as OpenCV adoption. It says OpenCV 4.13 and later enable KleidiCV by default in supported AArch64 builds; for earlier OpenCV versions, check the exact build configuration. Arm's [standalone build guide](https://learn.arm.com/learning-paths/laptops-and-desktops/kleidicv-on-mac/build-1/) shows the library and API route without OpenCV.\n\nDirect use is a **capability**, not evidence that a named commercial application already ships it. For an application audit, inspect its native dependency tree or symbols and confirm the selected KleidiCV operation. Likewise, the fact that KleidiAI appears in many ML runtimes says nothing about KleidiCV adoption.\n\nIf you plan a direct integration, start with one operation, for example a color conversion or Gaussian blur, and compare it against your existing implementation using identical image sizes and formats. Check the pinned KleidiCV release for the operation's supported types and backend; current ordinary APIs generally choose Neon or SVE2, while `_sme` variants prefer SME/SME2 where available. The [KleidiCV source](https://gitlab.arm.com/kleidi/kleidicv) is the place to verify what your version actually implements.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s06-q00", "slug": "cpp-compiler-versions", "question": "What C/C++ compiler versions support SME intrinsics?", "summary": "SME and SME2 ACLE (Arm C Language Extensions) intrinsics are supported by GCC 14+, Clang 18+, and Arm Compiler for Embedded 6.21+. These compilers can compile C/C++ source code containing SME intrinsics via the header. Arm recommends always using the most recent available version of whichever compiler you choose.", "answer": "SME and SME2 ACLE (Arm C Language Extensions) intrinsics are supported by **GCC 14+**, **Clang 18+**, and **Arm Compiler for Embedded 6.21+**. These compilers can compile C/C++ source code containing SME intrinsics via the `` header. Arm recommends always using the most recent available version of whichever compiler you choose.\n\n## Compiler Version Requirements\n\n| Compiler | Minimum Version | SME2 Support | Notes |\n|----------|----------------|--------------|-------|\n| **GCC** | 14 | Yes (from 14) | First version to support SME and SME2 ACLE intrinsics |\n| **Clang / LLVM** | 18 | Yes (from 18) | SME/SME2 intrinsics through ``; Clang 17 provided an experimental draft header |\n| **Arm Compiler for Embedded 6** | 6.17 (initial) | 6.21+ recommended | Initial SME support from 6.17; check the selected release for the required SME/SME2 intrinsics |\n\n### Key details per compiler\n\n**GCC 14+**\n- GCC 14 is the first version to support SME and SME2 ACLE intrinsics.\n- GCC uses a separate assembler (GNU binutils). Use **GNU binutils 2.41+** for SME2 kernels; older versions may support base SME but lack SME2 instructions.\n- Versions prior to GCC 14 cannot compile C/C++ code that uses SME intrinsics.\n\n**Clang 18+**\n- Clang 18 provides the named `` API for SME/SME2 ACLE intrinsics and is a practical starting point for these examples.\n- Clang integrates its own assembler, so no separate binutils requirement exists.\n- Earlier support was experimental. Clang 17 generated `arm_sme_draft_spec_subject_to_change.h`; that draft interface should not be treated as interchangeable with the `` API used here.\n\n**Arm Compiler for Embedded 6**\n- A cross-platform compiler for bare-metal application development.\n- SME support was first introduced in version 6.17.\n- Additional SME and SME2 functionality was added in subsequent releases.\n- Use version 6.21 or higher as a starting point, and check the release documentation for the required SME and SME2 intrinsic forms.\n\nThese version recommendations concern the SME/SME2 intrinsic sets available in those releases, not every newer optional SME extension. Check the compiler release and feature macros for any optional instructions your kernel requires.\n\nBoth Arm Compiler 6 and Clang also enable:\n- Assembly of source code containing SME and SME2 instructions\n- Disassembly of ELF object files containing SME and SME2 instructions\n\n## Required Compiler Flags\n\nTo enable SME intrinsics, pass the appropriate architecture extension flag:\n\n```sh\n# For SME (Armv9.2-A and later)\n-march=armv9.2-a+sme\n\n# For SME2 (Armv9.3-A and later)\n-march=armv9.3-a+sme2\n```\n\nThese flags enable the feature macros that guard the intrinsic declarations:\n\n| Flag | Feature Macro Defined |\n|------|----------------------|\n| `+sme` | `__ARM_FEATURE_SME` |\n| `+sme2` | `__ARM_FEATURE_SME2` |\n\nYou can test for these macros in your code to conditionally compile SME paths:\n\n```c\n#ifdef __ARM_FEATURE_SME2\n#include \n// SME2 intrinsics available\n#elif defined(__ARM_FEATURE_SME)\n#include \n// SME intrinsics available\n#else\n// Fallback to SVE, Neon, or scalar\n#endif\n```\n\n## Header File\n\nThe primary header for SME intrinsics is:\n\n```c\n#include \n```\n\nThis header also pulls in ``, since SME builds on the SVE type system (using `svfloat32_t`, `svint8_t`, and other scalable vector types).\n\n## GNU Binutils Requirement\n\nWhen using GCC (which uses a separate assembler/linker from the binutils package), use **GNU binutils 2.41 or later** for SME2. Older assemblers may support base SME without supporting SME2, and newer optional extensions may require a later release. Match the assembler to the instructions generated by your compiler.\n\nThis does not apply to Clang, which has an integrated assembler.\n\n## Practical Recommendations\n\n1. **Use the latest compiler version** — Arm actively improves SME code generation quality, so newer releases produce better code.\n2. **Match your architecture flag** to your target hardware — use `armv9.2-a+sme` for SME-only targets, `armv9.3-a+sme2` for SME2.\n3. **Check binutils version** when using GCC — a mismatched binutils is a common source of assembly errors.\n4. **Use pragmas for function-level control** — If only some functions need SME, you can use target pragmas or function attributes rather than compiling the entire translation unit with SME enabled.\n\n## Resources\n\n- [Arm Programmer's Guide – Compiler options and pragmas](https://developer.arm.com/documentation/109246/0101/Toolchains-and-model-support/Compiler-support/Compiler-options-and-pragmas?lang=en&utm_source=arm-mcp)\n- [Arm Programmer's Guide – Compiler support](https://developer.arm.com/documentation/109246/0101/Toolchains-and-model-support/Compiler-support?lang=en&utm_source=arm-mcp)\n- [Clang 17 experimental SME header generation](https://github.com/llvm/llvm-project/blob/llvmorg-17.0.1/clang/lib/Headers/CMakeLists.txt)\n- [Clang 18 SME header generation](https://github.com/llvm/llvm-project/blob/llvmorg-18.1.0/clang/lib/Headers/CMakeLists.txt)", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s06-q01", "slug": "sme-intrinsics-cpp", "question": "How do I use SME intrinsics in C/C++ code, and what header files and compiler flags are required?", "summary": "Include , compile the SME translation unit for a target such as -march=armv9.2-a+sme or -march=armv9.3-a+sme2, and declare the function's streaming-mode and ZA contract with ACLE attributes. The compiler can then insert required mode transitions and handle ABI state. Keep an ordinary Arm64 path and select the SME function only when the OS exposes the feature.", "answer": "Include ``, compile the SME translation unit for a target such as `-march=armv9.2-a+sme` or `-march=armv9.3-a+sme2`, and declare the function's streaming-mode and ZA contract with ACLE attributes. The compiler can then insert required mode transitions and handle ABI state. Keep an ordinary Arm64 path and select the SME function only when the OS exposes the feature.\n\n## A Small Intrinsic Example\n\nThis function accumulates **one FP32 outer product into a caller-owned ZA tile**. It deliberately does not zero ZA or write a result matrix: a caller can accumulate several products, then store the tile. Its declaration is the important part of the example: `__arm_streaming` says the function body uses streaming mode, and `__arm_inout(\"za\")` says it updates the caller's ZA state.\n\n```c\n#include \n\nvoid accumulate_fp32(const float *a, const float *b)\n __arm_streaming __arm_inout(\"za\");\n\nvoid accumulate_fp32(const float *a, const float *b)\n __arm_streaming __arm_inout(\"za\")\n{\n svbool_t pg = svptrue_b32();\n svfloat32_t left = svld1_f32(pg, a);\n svfloat32_t right = svld1_f32(pg, b);\n svmopa_za32_m(0, pg, pg, left, right);\n}\n```\n\nBuild the source as an object file, since the snippet has no `main`:\n\n```bash\nclang -O2 -march=armv9.2-a+sme -c accumulate.c -o accumulate.o\nllvm-objdump -d accumulate.o\n```\n\nLook for an `fmopa` instruction in the disassembly. The compiler may also create mode-transition code at call sites; an object file containing `fmopa` proves compilation, not that your application executes the function. Arm's [worked SME2 matrix-multiply example](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr/) shows tile initialization, repeated outer products, tail predicates, and result stores in a complete algorithm.\n\n## Choose the Function Contract\n\n| Attribute | Use it when |\n|---|---|\n| `__arm_streaming` | The function's interface requires streaming mode; callers can be adapted by the compiler |\n| `__arm_locally_streaming` | The function has an ordinary external interface but its body runs in streaming mode; check `__ARM_FEATURE_LOCALLY_STREAMING` in the toolchain |\n| `__arm_streaming_compatible` | The function is valid in either mode and does not require a mode switch |\n| `__arm_new(\"za\")` | The function creates private ZA state |\n| `__arm_inout(\"za\")` | The function shares and changes the caller's ZA state |\n\n`__arm_streaming` and shared-ZA attributes are part of a function's type and calling convention. A private-ZA attribute applies to a declaration/definition. Match the declaration and definition, and do not hide these contracts from callers behind an incompatible function pointer. For a regular application entry point, an `__arm_locally_streaming` wrapper with `__arm_new(\"za\")` can own a tile internally and return through an ordinary interface.\n\nSME2 intrinsics require the `__ARM_FEATURE_SME2` compilation feature. Guard target-specific source and headers when building multiple variants. Do not put `+sme2` on a whole portable binary: compiling an SME2 object and performing an OS-visible runtime check before calling it preserves fallback behavior. SME2 capability does not imply ordinary non-streaming SVE2 support.\n\nFor toolchain versions, start with GCC 14 or Clang 18 for SME/SME2 ACLE and verify the exact intrinsic set your pinned compiler implements. GCC builds also need an assembler that recognizes the selected instructions. The [ACLE specification's SME attributes and intrinsics](https://arm-software.github.io/acle/main/acle.html#sme-keyword-attributes) is the precise syntax reference; Arm's worked example is the better next step for a runnable kernel.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s06-q02", "slug": "correct-way-enter", "question": "What is the correct way to enter and exit streaming SVE mode when writing SME code in my own framework?", "summary": "In C or C++, declare the function's streaming contract with ACLE attributes and let the compiler place mode transitions around calls. In handwritten assembly or JIT code, use SMSTART and SMSTOP only after defining the function's ABI contract and handling live vector and ZA state. Switching PSTATE.SM changes the interpretation and lifetime of Z/P values, so it cannot safely be hidden inside an ordinary C function with unmodelled inline assembly.", "answer": "In C or C++, declare the function's streaming contract with **ACLE attributes** and let the compiler place mode transitions around calls. In handwritten assembly or JIT code, use SMSTART and SMSTOP only after defining the function's ABI contract and handling live vector and ZA state. Switching PSTATE.SM changes the interpretation and lifetime of Z/P values, so it cannot safely be hidden inside an ordinary C function with unmodelled inline assembly.\n\n## C/C++ Interfaces\n\nFor an ordinary public API whose implementation uses streaming mode, a locally streaming function is usually the simplest choice:\n\n```c\n#include \n\n__arm_locally_streaming __arm_new(\"za\")\nvoid process_tile(const float *a, const float *b)\n{\n svbool_t pg = svptrue_b32();\n svfloat32_t left = svld1_f32(pg, a);\n svfloat32_t right = svld1_f32(pg, b);\n svzero_za();\n svmopa_za32_m(0, pg, pg, left, right);\n /* Store the required ZA slices before returning. */\n}\n```\n\nThis is a **contract example**, not a complete matrix multiply: it initializes one ZA tile and accumulates one outer product but omits result storage. `__arm_locally_streaming` changes the function body without changing the caller's ordinary interface. Check that the compiler defines `__ARM_FEATURE_LOCALLY_STREAMING`; that ACLE feature is optional.\n\nFor an internal chain that shares ZA and avoids repeated transitions, declare the streaming and ZA state in the function type:\n\n```c\nvoid accumulate_tile(const float *a, const float *b)\n __arm_streaming __arm_inout(\"za\");\n```\n\nThe compiler can adapt calls between streaming and non-streaming functions according to their declared interfaces. `__arm_streaming_compatible` is for functions that can execute in either mode without a mode switch. `__arm_new(\"za\")` gives a function private ZA state; `__arm_inout(\"za\")` shares and changes the caller's state. The [ACLE streaming and ZA rules](https://arm-software.github.io/acle/main/acle.html#managing-streaming-mode-across-function-boundaries) explain the exact contracts.\n\n## Assembly and JIT Code\n\n`SMSTART SM` enables streaming mode; `SMSTOP SM` disables it. `SMSTART ZA` and `SMSTOP ZA` control ZA independently. Unqualified SMSTART and SMSTOP affect both SM and ZA. A kernel that uses ZA needs both the streaming mode required by its instructions and enabled ZA storage.\n\nBefore writing the assembly, decide whether the function has a streaming interface, creates private ZA, or shares the caller's ZA. Preserve ABI-required registers and follow the AAPCS64 ZA lazy-save rules when applicable. Do not copy a short SMSTART/SMSTOP macro as a complete call wrapper; correct register preservation and ZA ownership depend on the surrounding ABI contract. Arm's [SME calling-convention guide](https://developer.arm.com/documentation/109246/0101/Toolchains-and-model-support/Calling-conventions/Preparation-for-entering-and-exiting-streaming-mode) is the appropriate starting point for assembly.\n\nTreat Z and P values that are live across a streaming-mode change as invalid unless they have been saved according to the ABI. Streaming vector length can differ from the ordinary SVE vector length, so compute loop bounds for the active mode. On Linux, system calls and signals have [documented SME state rules](https://docs.kernel.org/arch/arm64/sme.html); a JIT or runtime that crosses those boundaries must account for them.\n\nGroup enough work per streaming region to justify transitions, but do not manually hold streaming mode across arbitrary framework or library calls. Keep the mode and ZA contracts visible to the compiler or JIT and verify the generated instructions in disassembly.", "tags": [ "SME" ], "golden": true }, { "id": "s06-q03", "slug": "manage-za-tile", "question": "How do I manage ZA tile state when integrating SME routines into a larger application that also uses Neon or SVE?", "summary": "Give each SME routine an explicit ZA ownership contract and keep that contract visible at every declaration and call site. For a self-contained kernel called from ordinary Neon/SVE code, use a normal external interface with __arm_locally_streaming and private __arm_new(\"za\") state. If several streaming functions cooperate on one tile, use shared-ZA attributes such as __arm_inout(\"za\"). Let an ACLE-aware compiler manage mode and ZA transitions; handwritten assembly must implement the matching AAPCS64 rules.", "answer": "Give each SME routine an explicit **ZA ownership contract** and keep that contract visible at every declaration and call site. For a self-contained kernel called from ordinary Neon/SVE code, use a normal external interface with `__arm_locally_streaming` and private `__arm_new(\"za\")` state. If several streaming functions cooperate on one tile, use shared-ZA attributes such as `__arm_inout(\"za\")`. Let an ACLE-aware compiler manage mode and ZA transitions; handwritten assembly must implement the matching AAPCS64 rules.\n\n## Separate Mode from Tile Ownership\n\nPSTATE.SM controls streaming mode; PSTATE.ZA controls whether ZA storage is enabled. They can be controlled independently. A change to streaming mode affects live Z/P vector values, while ZA is a separate state resource that may remain enabled across a mode change. The ordinary SVE vector length and streaming vector length may differ.\n\n| Contract | Meaning | Typical use |\n|---|---|---|\n| `__arm_new(\"za\")` | Create private ZA state for the function | A kernel that does not expose tile contents to its caller |\n| `__arm_in(\"za\")` | Read caller-owned ZA | Store or inspect an accumulated tile |\n| `__arm_out(\"za\")` | Produce caller-visible ZA | Initialize a tile for a shared pipeline |\n| `__arm_inout(\"za\")` | Read and update caller-owned ZA | Accumulate more products |\n| `__arm_preserves(\"za\")` | Preserve caller-owned ZA | A helper with an explicit preservation contract |\n\n`__arm_new(\"za\")` does **not** promise that the tile's numeric contents start at zero. Zero it explicitly with `svzero_za()` before accumulation if the algorithm needs a zero accumulator.\n\n## Two Useful Patterns\n\nAn isolated implementation can hide streaming and ZA details behind an ordinary API:\n\n```c\n#include \n\n__arm_locally_streaming __arm_new(\"za\")\nvoid process_matrix_tile(const float *a, const float *b, float *result)\n{\n svzero_za();\n /* Load vectors, accumulate into ZA, then store the required slices. */\n}\n```\n\nThis is an interface sketch: the algorithm and stores still need implementation. The caller can preprocess with Neon, call `process_matrix_tile`, and postprocess with non-streaming SVE using memory buffers between stages. Check `__ARM_FEATURE_LOCALLY_STREAMING` for compiler support.\n\nFor a pipeline that intentionally passes the same ZA tile between functions, put shared state in the declarations:\n\n```c\nvoid initialize_tile(void) __arm_streaming __arm_out(\"za\");\nvoid accumulate_tile(const float *a, const float *b)\n __arm_streaming __arm_inout(\"za\");\nvoid store_tile(float *dst) __arm_streaming __arm_in(\"za\");\n```\n\nA streaming wrapper with private ZA can call those three functions in sequence. Their declarations are part of the calling convention; do not call one through a function pointer that erases the attributes. The [ACLE ZA examples](https://arm-software.github.io/acle/main/acle.html#using-za-state-in-functions) explain the contract and its compiler handling.\n\n## Calls, Saves, and Performance\n\nAAPCS64 defines a lazy-save protocol for caller-owned ZA when a callee needs private state. It avoids eagerly copying a potentially large tile on every call. The compiler and runtime handle it for correctly annotated C/C++ functions; do not implement a partial `TPIDR2_EL0` protocol in an ordinary inline-assembly block. Arm's [ZA calling-convention guide](https://developer.arm.com/documentation/109246/0101/Toolchains-and-model-support/Calling-conventions/Controlling-the-use-of-ZA-storage) explains when a save is committed.\n\nKeep ordinary Neon/SVE helpers outside a hot streaming loop when possible; repeated mode transitions and any necessary state preservation have a cost. Pass stable data across the boundary through memory or ordinary scalar arguments. A Neon or SVE vector argument can be legal under the ABI in some cases, but do not assume its live register representation or vector length survives a mode change.\n\nFinally, distinguish build support from runtime support. Compile SME source separately, check OS-exposed SME capability before dispatch, and retain a non-SME fallback. SME2 support does not imply ordinary non-streaming SVE2.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s06-q04", "slug": "abi-considerations-calling", "question": "What are the ABI considerations when calling SME-enabled functions from non-SME code in a shared library?", "summary": "Expose an ordinary AArch64 function interface from the shared library, keep SME-specific functions internal, and dispatch to them only after checking OS-exposed SME capability. The key ABI rules are the function's streaming-mode contract, whether it shares ZA with its caller, and the handling of vector state at a mode boundary. These contracts must agree across declarations, definitions, and function pointers.", "answer": "Expose an **ordinary AArch64 function interface** from the shared library, keep SME-specific functions internal, and dispatch to them only after checking OS-exposed SME capability. The key ABI rules are the function's streaming-mode contract, whether it shares ZA with its caller, and the handling of vector state at a mode boundary. These contracts must agree across declarations, definitions, and function pointers.\n\n## Keep the Public Interface Conventional\n\nA public header can use fixed-layout memory and scalar arguments:\n\n```c\n/* matrix_api.h */\n#include \nvoid matrix_multiply(size_t m, size_t n, size_t k,\n const float *a, const float *b, float *out);\n```\n\nThe public function has the normal non-streaming ABI. It can call a private SME implementation built in a separate target-specific translation unit:\n\n```c\n/* SME translation unit: built with -march=armv9.2-a+sme */\n#include \n\n__arm_locally_streaming __arm_new(\"za\")\nvoid matrix_multiply_sme(size_t m, size_t n, size_t k,\n const float *a, const float *b, float *out)\n{\n /* Initialize ZA explicitly, compute tiles, and store results. */\n}\n```\n\nThis is an ABI sketch, not a complete multiplication implementation. `__arm_locally_streaming` keeps the external function type ordinary while running its body in streaming mode; `__arm_new(\"za\")` gives it private ZA state. An application may also give the SME symbol hidden visibility to keep the public ABI small. A non-SME fallback must implement the same numerical contract.\n\nThe dispatcher must use a runtime feature query rather than a compiler target macro. On Linux/Android, the OS-exposed HWCAP2 bits provide an explicit check:\n\n```c\n/* Linux/Android dispatcher source, built for the portable arm64 baseline */\n#include \n#include \n\nstatic int os_exposes_sme(void) {\n return (getauxval(AT_HWCAP2) & HWCAP2_SME) != 0;\n}\n```\n\nACLE also defines `__arm_has_sme()` in `` to report whether the current thread has access to SME; it is a valid runtime query where the toolchain and runtime provide it. It does not establish SME2 support. Choose `HWCAP2_SME2` if the private kernel specifically requires SME2. Keep the feature-specific object isolated so the compiler does not put SME instructions in the public entry point or library initializer. An ordinary arm64 process can load a shared library containing specialized code; it must not **execute** that code on unsupported hardware.\n\n## Know Which Attributes Change the Call Contract\n\n| Contract | Caller-visible meaning |\n|---|---|\n| No streaming attribute | Ordinary non-streaming call |\n| `__arm_streaming` | Streaming interface; the compiler adapts callers when declarations are correct |\n| `__arm_streaming_compatible` | Valid in either mode and preserves the entry mode |\n| `__arm_locally_streaming` | Streaming body, ordinary external interface |\n| `__arm_new(\"za\")` | Private ZA state; does not expose shared ZA in the function type |\n| `__arm_in/out/inout/preserves(\"za\")` | Caller and callee have an explicit shared-ZA contract in the function type |\n\nIf a declaration omits `__arm_streaming` or a shared-ZA attribute that the definition requires, the caller may use the wrong convention. A public C++ header or callback typedef must carry any contract it exposes. The [ACLE attribute rules](https://arm-software.github.io/acle/main/acle.html#sme-keyword-attributes) and [AAPCS64](https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst) are the authoritative details.\n\nKeep scalable SVE types out of an ordinary public interface unless you have explicitly designed and tested that PCS. The ordinary SVE vector length can differ from the streaming vector length, and live Z/P values do not carry unchanged across a mode transition. Pointers to fixed-layout arrays make both the calling convention and fallback simpler.\n\n## ZA, ZT0, and the Dynamic Linker\n\nWhen a caller owns ZA and a callee needs private ZA, AAPCS64 defines a lazy-save protocol. Correctly annotated compiler-generated code and its runtime support handle that protocol; a small handwritten SMSTART/SMSTOP wrapper is **not** a substitute. If you expose a streaming or shared-ZA function as a dynamic symbol, verify that the platform loader, toolchain, and runtime support the variant calling convention. An ordinary public wrapper keeps that concern internal.\n\nSME2's ZT0 has its own sharing attributes. A caller that needs its contents preserved across a call must use the corresponding ZT0 contract or save it explicitly; do not assume ZA's lazy-save handling covers ZT0.\n\nFor validation, build the fallback and SME objects separately, inspect the public header and exported symbols, run on both an SME-capable and a non-SME device, and use a breakpoint or disassembly to confirm that the feature gate selects the intended object. Test callbacks and exception or nonlocal-control-flow paths if your library uses them, since they can cross the declared call boundary.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s06-q05", "slug": "write-custom-sme2", "question": "How do I write a custom SME2 matrix multiply kernel using outer product intrinsics and integrate it into my application?", "summary": "SME2 matrix multiplication works by decomposing C = A × B into vector outer products accumulated in ZA. If the streaming vector length is SVL_bits, ZA contains SVL_bytes × SVL_bytes bytes, where SVL_bytes = SVL_bits / 8. Each of its four FP32 tile views contains L × L elements, where L = SVL_bits / 32. This example preprocesses the left matrix into a column-contiguous layout, then accumulates K outer products per output tile in ZA0.S using svmopa_za32_m. It stores the result from ZA tile slices to memory and runs the kernel in streaming mode with ZA enabled.", "answer": "SME2 matrix multiplication works by decomposing C = A × B into vector outer products accumulated in ZA. If the streaming vector length is `SVL_bits`, ZA contains `SVL_bytes × SVL_bytes` bytes, where `SVL_bytes = SVL_bits / 8`. Each of its four FP32 tile views contains `L × L` elements, where `L = SVL_bits / 32`. This example preprocesses the left matrix into a column-contiguous layout, then accumulates K outer products per output tile in ZA0.S using `svmopa_za32_m`. It stores the result from ZA tile slices to memory and runs the kernel in streaming mode with ZA enabled.\n\n## Architecture Background\n\nZA contains `SVL_bytes²` bytes. For FP32 operations, it has four tile views, ZA0.S–ZA3.S, each containing `L × L` elements. At a 512-bit streaming vector length, ZA contains 4096 bytes and each tile is 16 × 16 FP32 elements. The FMOPA instruction (exposed via `svmopa_za32_m`) computes:\n\n```\nZA0.S[i][j] += A_col[i] * B_row[j] for all valid (i,j) within predicates\n```\n\nWith all lanes active, one instruction performs an `L × L` rank-1 update. After K such accumulations, the tile holds an `L × L` block of the result matrix. Edge predicates restrict the update to valid output elements. In the code below, the variable named `SVL` is the FP32 lane count `L`, obtained with `svcntsw()`, rather than a length in bits or bytes.\n\n## Algorithm Overview\n\nFor matrices A (M×K), B (K×N), C (M×N):\n\n1. **Preprocess A** — rearrange from row-major into column-contiguous blocks so that each column of a tile can be loaded with a single `svld1`\n2. **Tile loop** — iterate over `ceil(M/L) × ceil(N/L)` output tiles, predicating partial edge tiles\n3. **K loop** — for each tile, accumulate K outer products into ZA tile 0\n4. **Store** — extract horizontal tile slices to the result matrix\n\n## Worked Implementation\n\nThe following kernel follows Arm's [SME2 intrinsics walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr/). It teaches packing, tile accumulation, and storage. Pin the example source and compile it with your target toolchain before treating the excerpts as a drop-in library.\n\n### Header and Guards\n\n```c\n// sme2_matmul.h\n#ifndef SME2_MATMUL_H\n#define SME2_MATMUL_H\n\n#include \n\n// Public API — callable from non-streaming code\nvoid sme2_matmul_f32(uint64_t M, uint64_t K, uint64_t N,\n const float *restrict A,\n const float *restrict B,\n float *restrict A_packed,\n float *restrict C);\n\n#endif // SME2_MATMUL_H\n```\n\n### Matrix Preprocessing (Intrinsics Version)\n\nThe left matrix must be transformed so that columns of each `L × L` tile are contiguous in memory. This uses ZA itself as a transposition buffer — data is written as rows and read back as columns:\n\n```c\n// sme2_matmul.c\n#include \"sme2_matmul.h\"\n#include \n\n#ifndef __ARM_FEATURE_SME2\n#error \"Compile this source with SME2 enabled (-march=armv9.3-a+sme2)\"\n#endif\n\nstatic void preprocess_l_intr(\n uint64_t M, uint64_t K, uint64_t SVL,\n const float *restrict a,\n float *restrict a_mod) __arm_streaming __arm_inout(\"za\") {\n\n const uint64_t M_mod = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0));\n\n for (uint64_t row = 0; row < M; row += SVL) {\n svbool_t pMDim = svwhilelt_b32(row, M);\n\n for (uint64_t col = 0; col < K; col += 2 * SVL) {\n svcount_t pKDim = svwhilelt_c32(col, K, 2);\n\n // Load rows of A into ZA tiles (write-as-rows)\n for (uint64_t trow = 0; trow < SVL; trow += 4) {\n svcount_t p0 = svpsel_lane_c32(pKDim, pMDim, trow + 0);\n svcount_t p1 = svpsel_lane_c32(pKDim, pMDim, trow + 1);\n svcount_t p2 = svpsel_lane_c32(pKDim, pMDim, trow + 2);\n svcount_t p3 = svpsel_lane_c32(pKDim, pMDim, trow + 3);\n\n const uint64_t corner = (row + trow) * K + col;\n svfloat32x2_t zp0 = svld1_x2(p0, &a[corner + 0 * K]);\n svfloat32x2_t zp1 = svld1_x2(p1, &a[corner + 1 * K]);\n svfloat32x2_t zp2 = svld1_x2(p2, &a[corner + 2 * K]);\n svfloat32x2_t zp3 = svld1_x2(p3, &a[corner + 3 * K]);\n\n svfloat32x4_t zq0 = svcreate4(svget2(zp0, 0), svget2(zp1, 0),\n svget2(zp2, 0), svget2(zp3, 0));\n svfloat32x4_t zq1 = svcreate4(svget2(zp0, 1), svget2(zp1, 1),\n svget2(zp2, 1), svget2(zp3, 1));\n svwrite_hor_za32_f32_vg4(0, trow, zq0);\n svwrite_hor_za32_f32_vg4(1, trow, zq1);\n }\n\n // Read as columns and store (transposition complete)\n const uint64_t dest_0 = row * K + col * SVL;\n const uint64_t dest_1 = dest_0 + SVL * SVL;\n for (uint64_t tcol = 0; tcol < SVL; tcol += 4) {\n svcount_t p0 = svwhilelt_c32(dest_0 + tcol * SVL, K * M_mod, 4);\n svcount_t p1 = svwhilelt_c32(dest_1 + tcol * SVL, K * M_mod, 4);\n svfloat32x4_t zq0 = svread_ver_za32_f32_vg4(0, tcol);\n svfloat32x4_t zq1 = svread_ver_za32_f32_vg4(1, tcol);\n svst1(p0, &a_mod[dest_0 + tcol * SVL], zq0);\n svst1(p1, &a_mod[dest_1 + tcol * SVL], zq1);\n }\n }\n }\n}\n```\n\n### The Outer Product Kernel\n\nThe core computation iterates over `L × L` output tiles, accumulating K outer products per tile:\n\n```c\nstatic void matmul_intr_impl(\n uint64_t M, uint64_t K, uint64_t N, uint64_t SVL,\n const float *restrict matLeft_mod,\n const float *restrict matRight,\n float *restrict matResult) __arm_streaming __arm_inout(\"za\") {\n\n // Build the result matrix tile by tile\n for (uint64_t row = 0; row < M; row += SVL) {\n svbool_t pMDim = svwhilelt_b32(row, M);\n\n for (uint64_t col = 0; col < N; col += SVL) {\n svbool_t pNDim = svwhilelt_b32(col, N);\n\n // Zero the accumulator tile\n svzero_za();\n\n const uint64_t matLeft_pos = row * K;\n const uint64_t matRight_UL_corner = col;\n\n // Accumulate outer products over K\n for (uint64_t k = 0; k < K; k++) {\n // Load column from preprocessed left matrix\n svfloat32_t zL =\n svld1(pMDim, &matLeft_mod[matLeft_pos + k * SVL]);\n\n // Load row from right matrix (naturally contiguous)\n svfloat32_t zR =\n svld1(pNDim, &matRight[matRight_UL_corner + k * N]);\n\n // Outer product and accumulate into ZA tile 0\n svmopa_za32_m(0, pMDim, pNDim, zL, zR);\n }\n\n // Store ZA tile slices to the result matrix\n const uint64_t result_corner = row * N + col;\n for (uint64_t trow = 0; trow < SVL && row + trow < M; trow += 4) {\n svbool_t p0 = svpsel_lane_b32(pNDim, pMDim, row + trow + 0);\n svbool_t p1 = svpsel_lane_b32(pNDim, pMDim, row + trow + 1);\n svbool_t p2 = svpsel_lane_b32(pNDim, pMDim, row + trow + 2);\n svbool_t p3 = svpsel_lane_b32(pNDim, pMDim, row + trow + 3);\n\n svst1_hor_za32(0, trow + 0, p0,\n &matResult[result_corner + (trow + 0) * N]);\n svst1_hor_za32(0, trow + 1, p1,\n &matResult[result_corner + (trow + 1) * N]);\n svst1_hor_za32(0, trow + 2, p2,\n &matResult[result_corner + (trow + 2) * N]);\n svst1_hor_za32(0, trow + 3, p3,\n &matResult[result_corner + (trow + 3) * N]);\n }\n }\n }\n}\n```\n\n### Top-Level Entry Point\n\nThe public function uses `__arm_locally_streaming` (so callers need not be in streaming mode) and `__arm_new(\"za\")` (to create fresh ZA state without destroying any caller's ZA):\n\n```c\n__arm_new(\"za\") __arm_locally_streaming\nvoid sme2_matmul_f32(\n uint64_t M, uint64_t K, uint64_t N,\n const float *restrict A,\n const float *restrict B,\n float *restrict A_packed,\n float *restrict C) {\n\n uint64_t SVL = svcntsw(); // Number of 32-bit elements per streaming vector\n preprocess_l_intr(M, K, SVL, A, A_packed);\n matmul_intr_impl(M, K, N, SVL, A_packed, B, C);\n}\n```\n\n## Integrating into Your Application\n\n### Workspace Allocation\n\nThe packed buffer size depends on the runtime SVL. Query it once and cache:\n\n```c\n#include \n\n// Call this from streaming-compatible context to get the workspace size\n__arm_locally_streaming\nsize_t sme2_packed_a_size(uint64_t M, uint64_t K) {\n uint64_t SVL = svcntsw();\n uint64_t M_padded = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0));\n return M_padded * K * sizeof(float);\n}\n```\n\n### Runtime Feature Detection\n\nNever assume SME2 is available. Check hardware capability before dispatching:\n\n```c\n#include \n#include \n#include \n\nstatic bool cpu_has_sme2(void) {\n#if defined(__aarch64__) && defined(HWCAP2_SME2)\n return (getauxval(AT_HWCAP2) & HWCAP2_SME2) != 0;\n#else\n return false;\n#endif\n}\n\nvoid application_matmul(uint64_t M, uint64_t K, uint64_t N,\n const float *A, const float *B, float *C) {\n if (cpu_has_sme2()) {\n size_t bytes = sme2_packed_a_size(M, K);\n float *workspace = malloc(bytes);\n if (workspace == NULL) {\n matmul_reference_f32(M, K, N, A, B, C);\n return;\n }\n sme2_matmul_f32(M, K, N, A, B, workspace, C);\n free(workspace);\n } else {\n matmul_reference_f32(M, K, N, A, B, C); // Fallback (Neon, SVE, scalar)\n }\n}\n```\n\n### Build System Integration\n\nIsolate the SME2 kernel in its own translation unit and apply architecture flags only to that file:\n\n```cmake\n# CMakeLists.txt\nadd_library(matmul_sme2 OBJECT sme2_matmul.c)\ntarget_compile_options(matmul_sme2 PRIVATE -O3 -march=armv9.3-a+sme2)\n\nadd_executable(myapp main.c)\ntarget_link_libraries(myapp PRIVATE matmul_sme2)\n# main.c compiled without SME2 flags — stays portable\n```\n\nDirect compilation:\n\n```bash\n# SME2 kernel only\nclang -O3 -march=armv9.3-a+sme2 -c sme2_matmul.c -o sme2_matmul.o\n\n# Application (no SME2 flags needed)\nclang -O3 -march=armv8.2-a main.c sme2_matmul.o -o myapp\n```\n\n## Key ACLE Function Attributes\n\n| Attribute | Purpose | When to Use |\n|-----------|---------|-------------|\n| `__arm_streaming` | Streaming function interface; an ACLE-aware caller inserts a mode transition when needed | Functions whose interface requires streaming mode; shared-ZA requirements apply separately |\n| `__arm_locally_streaming` | Body runs in streaming mode, but callers need not be streaming | Public API entry points (compiler inserts SMSTART/SMSTOP) |\n| `__arm_new(\"za\")` | Creates fresh ZA state; compiler saves/restores any pre-existing ZA | Outermost function that begins using ZA |\n| `__arm_inout(\"za\")` | Function reads and writes caller's ZA state | Inner helper functions that share ZA with their caller |\n| `__arm_in(\"za\")` | Function reads but does not modify ZA | Read-only inspection of ZA |\n| `__arm_out(\"za\")` | Function writes ZA but does not read pre-existing content | Functions that initialize ZA from scratch |\n\n## Key Intrinsics Reference\n\n| Intrinsic | Instruction | Purpose |\n|-----------|-------------|---------|\n| `svmopa_za32_m(tile, pm, pn, zn, zm)` | FMOPA | Outer product accumulate into ZA tile |\n| `svzero_za()` | ZERO | Zero all ZA tile storage |\n| `svld1(pred, ptr)` | LD1W | Predicated contiguous load into SVE vector |\n| `svst1_hor_za32(tile, slice, pred, ptr)` | ST1W (ZA) | Store horizontal ZA tile slice to memory |\n| `svwhilelt_b32(start, limit)` | WHILELT | Generate predicate for loop boundary handling |\n| `svcntsw()` | RDSVL | Query SVL in 32-bit elements (runtime) |\n| `svpsel_lane_b32(pn, pm, idx)` | PSEL | Select predicate lane for tile slice access |\n| `svwrite_hor_za32_f32_vg4(tile, slice, zt)` | MOVA (to ZA) | Write 4 vectors as horizontal tile rows |\n| `svread_ver_za32_f32_vg4(tile, slice)` | MOVA (from ZA) | Read 4 vectors as vertical tile columns |\n\n## Compiler Requirements\n\n| Compiler | Minimum Version | Notes |\n|----------|----------------|-------|\n| Clang/LLVM | 18+ | SME2 intrinsics through ``; check the release for newer optional extensions |\n| GCC | 14+ | SME2 ACLE support added in GCC 14 |\n| Arm Compiler for Embedded 6 | 6.21+ | For bare-metal targets; check the release for the required intrinsics |\n\n## Optimization Considerations\n\n- **Minimize streaming mode transitions** — enter streaming mode once and perform all tile work before returning. Each mode transition has a cost and invalidates live Z/P values across the boundary.\n- **Reuse packed operands** — if the same A matrix is used for multiple B matrices (e.g., batched GEMM), preprocess A once and reuse across calls.\n- **Unroll the K loop** — issue multiple `svmopa_za32_m` calls per iteration to improve instruction-level parallelism and hide load latency.\n- **Use multi-vector SME2 data movement** — grouped loads and ZA slice transfers can reduce the instructions needed for packing and storing tiles, as shown in the preprocessing code. The `svmopa_za32_m` outer-product intrinsic takes two individual vectors, not x2/x4 vector groups; multi-vector arithmetic forms use their own accumulator layouts.\n- **Fuse post-operations** — apply bias addition, ReLU, or quantization while extracting tile slices, before the data leaves registers.\n- **Consider data types** — SME2 supports FP16→FP32 (`svmopa_za32_f16_m`), BF16→FP32, INT8→INT32 and other widening outer products for ML workloads.\n- **Pre-arranged layouts** — in production frameworks, arrange the left operand in column-major order upstream, eliminating the preprocessing step entirely.\n- **Benchmark realistic shapes** — SME2 startup cost and packing overhead mean Neon or SVE may remain faster for small matrices (depending on the shape and selected kernel).\n\n## Resources\n\n- [Outer product approach and transposition](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product/?utm_source=arm-mcp#preprocessing-with-preprocess_l) — why left-matrix preprocessing is needed and how it works\n- [ACLE SME language extensions and intrinsics](https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics) — official ACLE specification for SME", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s06-q06", "slug": "know-if-kleidiai", "question": "How do I know if I should use KleidiAI for my framework integration?", "summary": "Use KleidiAI when Arm CPU inference is important to your users, a profiled hot operator matches a KleidiAI kernel's complete type/shape/quantization contract, and an existing backend does not already give you that path. Prototype one operator and measure end-to-end benefit before committing to a new framework integration.", "answer": "Use KleidiAI when Arm CPU inference is important to your users, a profiled hot operator matches a KleidiAI kernel's **complete** type/shape/quantization contract, and an existing backend does not already give you that path. Prototype one operator and measure end-to-end benefit before committing to a new framework integration.\n\n## Check for an Existing Path First\n\n| Framework path | Where KleidiAI can enter | What to verify |\n|---|---|---|\n| ExecuTorch or LiteRT → XNNPACK | Supported delegated matrix and convolution operators | Backend included, exact operator support, runtime kernel selection |\n| ONNX Runtime → MLAS | Selected GEMM or quantized matrix operations | Build option, MLAS kernel registration, selected function |\n| llama.cpp → GGML CPU | Selected `GGML_OP_MUL_MAT` formats | `GGML_CPU_KLEIDIAI=ON`, model format, active CPU backend, selected kernel |\n\nIf the framework already reaches KleidiAI, first test that route with the model you actually deploy. An installed package or successful inference run does not prove KleidiAI or SME2 executed. Trace the selected microkernel and compare latency with a controlled fallback.\n\n## When a Direct Integration Pays\n\nKleidiAI supplies low-level microkernels and packing functions, not model loading, graph scheduling, or a universal \"best kernel\" policy. A direct integration has to:\n\n1. Map a lowered operator to a kernel with matching input/weight/output types, quantization granularity, group size, layout, shape, and post-operations.\n2. Query OS-exposed CPU features and select a compatible kernel. There is no universal SME2 → I8MM → DotProd ranking independent of operation and shape.\n3. Allocate and manage packing/workspace buffers, including any weight packing that can be reused across inference calls.\n4. Provide a correct fallback for unsupported hardware and operator signatures.\n5. Integrate with your framework's threading and scheduling, then test numerical correctness and performance.\n\nArm's [kernel selection guide](https://learn.arm.com/learning-paths/cross-platform/kleidiai-explainer/page2/) explains how to match a kernel to a workload. The [KleidiAI source inventory](https://gitlab.arm.com/kleidi/kleidiai) is the authority for the exact release you bundle.\n\n## Make the Decision from a Prototype\n\nProfile representative models on the target Arm devices. If matrix multiplication or compatible convolutions account for meaningful absolute runtime, compare one candidate KleidiAI kernel with your existing tuned implementation at real shapes and thread counts. Include weight packing, activation quantization, layout conversions, and the full inference call in the measurement.\n\nKleidiAI is less likely to justify a new integration when the CPU path is minor beside GPU/NPU work, the hot operators have no matching kernels, or one-shot packing costs dominate. Batch-1 matrix-vector operations can still have specialized kernels, so do not reject them from shape alone. If the prototype saves meaningful end-to-end time or energy and the fallback remains maintainable, expand coverage one operator at a time.\n\nThe [ONNX Runtime integration walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/kleidiai_integration/) is a concrete example of the registration, packing, and dispatch work involved. Hardware SME2 support by itself is not the deciding factor: KleidiAI also targets other Arm features, and the best path depends on the workload.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s07-q01", "slug": "quickly-test-whether", "question": "How can I quickly test whether my workload benefits from SME without rewriting my entire application?", "summary": "Start with the library or framework already doing your matrix work. Run the same representative workload with its SME or SME2 kernel path enabled and disabled, confirm which kernel actually ran, and compare end-to-end time. This tests the opportunity without committing to a custom implementation. Compiling the application with +sme2 alone does not turn arbitrary loops into ZA outer-product kernels.", "answer": "Start with the library or framework already doing your matrix work. Run the same representative workload with its SME or SME2 kernel path enabled and disabled, confirm which kernel actually ran, and compare end-to-end time. This tests the opportunity without committing to a custom implementation. Compiling the application with `+sme2` alone does not turn arbitrary loops into ZA outer-product kernels.\n\n## Find the work worth testing\n\nProfile a representative run and identify the expensive operation and its shapes. For a native Linux program, `perf record -g -- ./app` and `perf report` can locate a hot GEMM, convolution, or packing routine. Framework operator profilers can show whether MatMul or Conv dominates an inference. Record the data type, matrix dimensions, quantization format, and thread count; these determine whether an available SME2 kernel can run.\n\nCheck that the **OS exposes** SME or SME2 on the machine you will measure. On Linux, inspect `/proc/cpuinfo` or use `getauxval(AT_HWCAP2)`; a CPU marketing name alone is insufficient. On Android, test on the actual device. An emulator can check correctness, but its timings do not predict hardware speed.\n\n## Use an existing implementation\n\n| Workload | Low-effort experiment |\n|---|---|\n| LiteRT or ExecuTorch model | Use a build whose XNNPACK backend includes KleidiAI, then compare its SME2-enabled and disabled variants. Profile the operators and verify the selected microkernel. |\n| ONNX Runtime model | Use its KleidiAI-enabled MLAS build and an otherwise equivalent baseline. Check operator timing and sampled kernel symbols. |\n| llama.cpp model | Use the upstream KleidiAI build path and compare an equivalent build without it. Check the selected matmul kernel; the `CPU_KLEIDIAI` buffer label alone does not prove SME2 execution. |\n| BLAS application on a supported Apple M-series Mac | Compare Arm Performance Libraries' documented SME SGEMM/DGEMM path with the existing BLAS provider for the same calls. Confirm the provider and implementation used. |\n\nIf no backend covers your operator and format, extract **one** hot kernel and build a small SME prototype. A function-specific target and ACLE attributes keep SME code isolated from the portable application. Include packing, dispatch, and streaming-mode transitions in the timing when they occur in production.\n\n## Make the result meaningful\n\nUse identical inputs, precision, thread count, and core placement. Warm both variants, repeat measurements, and report the median plus variation. Compare the kernel time **and** the full request or inference time. Validate outputs against the same reference; different accumulation orders may require a stated floating-point tolerance.\n\nA fast SME2 kernel is useful only when the framework dispatches to it and its savings exceed packing, mode-transition, and other application costs. If the overall time barely changes, the profile should show whether the operator was a small fraction of runtime, the workload missed kernel coverage, or another stage became dominant.\n\n## Resources\n\n- [Arm's ONNX Runtime profiling walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) shows an operator-level and hardware-counter comparison.\n- [Arm's llama.cpp SME2 performance walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_llama_cpp_sme2/run_llm/) provides a concrete model benchmark.\n- [Arm Performance Libraries 26.01 update](https://developer.arm.com/community/arm-community-blogs/b/servers-and-cloud-computing-blog/posts/what-s-new-in-arm-performance-libraries-26-01) identifies the macOS SME GEMM support.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s07-q02", "slug": "emulator-simulator-test", "question": "Is there an emulator or simulator I can use to test SME code if I don't have access to SME-capable hardware?", "summary": "Yes. QEMU's AArch64 emulation can run SME code for functional testing, including different streaming vector lengths. Arm Fixed Virtual Platforms (FVPs) are another route when you need a complete simulated system or bare-metal environment. Neither predicts the speed of a physical SME processor.", "answer": "Yes. QEMU's AArch64 emulation can run SME code for functional testing, including different streaming vector lengths. Arm Fixed Virtual Platforms (FVPs) are another route when you need a complete simulated system or bare-metal environment. Neither predicts the speed of a physical SME processor.\n\n## Start with QEMU user mode\n\nBuild an AArch64 Linux test binary with a compiler that supports the SME intrinsics you use, then run it through a recent `qemu-aarch64`. If the binary is dynamically linked, provide an AArch64 sysroot with `-L` or use a compatible static build. For example:\n\n```sh\nclang --target=aarch64-linux-gnu -O2 -march=armv9-a+sme2 test.c -o test\nqemu-aarch64 -cpu max,sme=on,sme-default-vector-length=32 -L /path/to/aarch64-sysroot ./test\n```\n\nThe sysroot path and link command depend on your toolchain. QEMU's `sme-default-vector-length` value is **bytes**, so 32 selects a 256-bit default if that length is enabled. Try 16, 32, and 64 bytes with dimensions that are not neat tile multiples. QEMU's `max` CPU may enable SME subfeatures such as SME2; check the reported capabilities of the QEMU version you use.\n\nA tiny test can check that a locally streaming function runs and sees a streaming vector length:\n\n```c\n#include \n#include \n#include \n\n#if defined(__ARM_FEATURE_LOCALLY_STREAMING)\n__arm_locally_streaming\n#endif\nstatic unsigned words_per_streaming_vector(void) {\n return svcntsw();\n}\n\nint main(void) {\n printf(\"%u 32-bit lanes\\n\", words_per_streaming_vector());\n}\n```\n\nHere `svcntsw()` counts 32-bit lanes in the streaming vector. The compiler handles entry and exit from streaming mode for the annotated function. This is a feature and ABI smoke test, not a speed test. If your compiler lacks `__arm_locally_streaming`, use an explicit streaming function interface supported by that toolchain.\n\n## Use a full-system platform when the OS matters\n\nQEMU system mode and Arm FVPs can test boot, feature exposure, signals, and context switching with an appropriate guest image. Their setup depends on the exact machine model and image, so follow that platform's instructions rather than copying a generic bare-metal command. Check that both the simulated CPU and guest kernel expose SME to user space.\n\nA native SME-capable Apple M-series Mac is useful for macOS application testing. Its OS APIs and toolchain differ from Linux; do not use Linux HWCAP instructions as an Apple feature check.\n\nUse emulation to find invalid instructions, ABI mistakes, tail bugs, and vector-length assumptions. Use real hardware for latency, throughput, power, cache behavior, and any Neon/SVE/SME comparison.\n\n## Resources\n\n- [QEMU Arm CPU feature documentation](https://www.qemu.org/docs/master/system/arm/cpu-features.html) defines SME CPU properties and the user-mode vector-length setting.\n- [Arm's SME2 environment check](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment/) walks through compiler and runtime checks.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s07-q03", "slug": "best-way-prototype", "question": "What is the best way to prototype and test SME kernels for my candidate workloads?", "summary": "Prototype one hot operation at a time. Write a scalar reference, implement the same operation with SME intrinsics, test awkward dimensions and vector lengths, then measure on real hardware. Keep packing and streaming-mode costs visible: a fast inner instruction does not guarantee a fast application.", "answer": "Prototype one hot operation at a time. Write a scalar reference, implement the same operation with SME intrinsics, test awkward dimensions and vector lengths, then measure on real hardware. Keep packing and streaming-mode costs visible: a fast inner instruction does not guarantee a fast application.\n\n## Choose a candidate and establish correctness\n\nProfile the application first. Record the operation, input layout, data type, quantization parameters, and typical dimensions. Matrix multiplication and convolution lowered to matrix multiplication are natural candidates; tiny or memory-bound operations may not repay the setup cost.\n\nUse a simple scalar implementation as the reference. For FP32 matrix multiplication, for example:\n\n```c\nfor (size_t m = 0; m < M; ++m)\n for (size_t n = 0; n < N; ++n) {\n float sum = 0.0f;\n for (size_t k = 0; k < K; ++k)\n sum += A[m*K + k] * B[k*N + n];\n C[m*N + n] = sum;\n }\n```\n\nTest tails such as M=17 and N=19, K boundaries, zero dimensions where permitted, and nonzero initial C if the kernel accumulates. Compare floating-point results with an explained tolerance because accumulation order can differ.\n\n## Build the smallest SME experiment\n\nArm's SIMD Loops project has comparable scalar, Neon, SVE, and SME examples. It is a useful starting point for understanding build targets and checking output, but adapt its loop to your real dimensions and data layout.\n\nAn SME outer-product kernel repeatedly loads one vector from each operand, accumulates their pairwise products in ZA, and stores the output tile after the K loop. ACLE expresses the central step as:\n\n```c\n// Inside a streaming function with ZA enabled; predicates and vectors\n// must match the chosen tile and packed input layout.\nsvmopa_za32_m(0, row_predicate, column_predicate,\n lhs_vector, rhs_vector);\n```\n\nThis is an illustrative inner step, not a complete matrix-multiply function. A complete kernel must choose tile dimensions from `svcntsw()`, pack or gather the left-hand columns, handle tails with predicates, initialize ZA when appropriate, and store each output slice. Use `__arm_streaming` and a suitable ZA interface such as `__arm_new(\"za\")` for a self-contained leaf kernel; explicitly zero the accumulator before use. Build that source for SME with `-march=armv9-a+sme` or for SME2 with `-march=armv9-a+sme2`, according to the instructions used. Keep the rest of the application on a portable target and dispatch only to supported hardware.\n\n## Test, then measure\n\nQEMU user mode or an Arm FVP can catch instruction, ABI, predication, and vector-length bugs. Try more than one streaming vector length. Their run time is not a hardware performance prediction.\n\nOn an SME-capable device, benchmark both kernels with identical inputs and thread placement. Warm up, repeat, report median and variation, and separately record packing, kernel, and full-application time. Check the binary or sampled instructions to confirm that the SME path ran. If the prototype loses, inspect matrix size, layout conversion, cache traffic, mode transitions, and ZA saves before changing the arithmetic.\n\n## Resources\n\n- [Arm's SME2 matrix multiplication walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/) provides a complete implementation to study.\n- [Arm SIMD Loops](https://learn.arm.com/learning-paths/cross-platform/simd-loops/) provides comparable loop implementations.\n- [ACLE SME intrinsics and attributes](https://arm-software.github.io/acle/main/acle.html#sme-keyword-attributes) defines the function interface.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s07-q04", "slug": "profiling-tools-measure", "question": "What profiling tools can I use to measure how much time my workload spends in matrix operations that SME could accelerate?", "summary": "Use a profiler that attributes time to functions or model operators first. A counter cannot tell you how much time could be accelerated before an SME kernel exists. Once you have an SME-enabled implementation, compare the same workload and confirm which kernel ran.", "answer": "Use a profiler that attributes time to functions or model operators first. A counter cannot tell you how much time *could* be accelerated before an SME kernel exists. Once you have an SME-enabled implementation, compare the same workload and confirm which kernel ran.\n\n## Native Linux applications\n\nperf provides a practical first pass:\n\n```sh\nperf record -g -- ./your_workload\nperf report\nperf list | grep -Ei 'sme|mopa|matrix'\n```\n\nLook for GEMM, convolution, and packing routines in the hot call paths. Available PMU event names vary by processor and kernel. If SME-related events exist, they can confirm instruction activity, but they do not by themselves measure saved time. Arm Streamline provides timeline and call-path views plus supported hardware counters, which help relate the matrix work to thread scheduling and memory stalls. Arm Performix and Linaro MAP can provide source-level hotspot views on supported systems.\n\nFor a BLAS application, call logging can identify how often SGEMM or DGEMM runs and with which dimensions. Check that an interposition tool actually intercepts the BLAS provider your executable uses; a preload library is not a universal substitute for relinking.\n\n## Framework models\n\n| Framework | Useful measurement |\n|---|---|\n| ExecuTorch | ETDump operator timings and the XNNPACK/KleidiAI profiling workflow |\n| ONNX Runtime | Built-in JSON profiling for per-operator latency; Streamline or sampling for native kernels |\n| LiteRT | Benchmark tool and XNNPACK operator trace, where available |\n| llama.cpp | Built-in prompt processing and token-generation timings, plus sampled kernel symbols |\n\nAn operator name or XNNPACK `P` prefix can identify a packed path, but does not alone prove SME2 execution. Use a kernel symbol, sampled instruction, or supported hardware event for that claim.\n\n## Compare before and after\n\nProfile identical inputs, model, precision, thread count, and core placement with SME2 enabled and disabled in the backend. Repeat after warm-up and compare end-to-end time as well as the matrix operator. Packing and streaming-mode transitions count if they occur during real use. A faster GEMM may expose a new bottleneck in memory movement or other operators.\n\n## Resources\n\n- [Arm's ONNX Runtime profiling example](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) shows operator profiling and Streamline together.\n- [Arm's ExecuTorch profiling guide](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/01-overview/) shows how to compare operator timing.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s07-q05", "slug": "compare-performance-workload", "question": "How do I compare the performance of my workload running with Neon, SVE, and SME to determine which gives the best results?", "summary": "Implement the same operation with the same input and output contract, verify correctness, and time each variant on one processor that supports all the extensions being compared. Report both kernel and end-to-end results. The fastest choice depends on operation, shape, layout, vector length, and implementation quality; SME does not automatically win because it has ZA.", "answer": "Implement the same operation with the same input and output contract, verify correctness, and time each variant on **one processor that supports all the extensions being compared**. Report both kernel and end-to-end results. The fastest choice depends on operation, shape, layout, vector length, and implementation quality; SME does not automatically win because it has ZA.\n\n## Make the variants comparable\n\nNeon is the AArch64 Advanced SIMD path; SVE supplies scalable vector operations and predication; SME adds streaming mode and ZA outer-product accumulation. For a matrix multiplication, each version should calculate the same output with the same precision and include equivalent packing or conversion costs. A compiler flag alone does not create a competitive SME kernel. Use explicit intrinsics, an established library, or verified disassembly to know what ran.\n\nFor example, compile source files that actually contain each implementation with appropriate targets:\n\n```sh\nclang -O3 -march=armv8-a+simd -c neon.c -o neon.o\nclang -O3 -march=armv8-a+sve -c sve.c -o sve.o\nclang -O3 -march=armv9-a+sme -c sme.c -o sme.o\n```\n\nUse SME2 when the kernel uses SME2-specific instructions. Function-specific targets can keep all variants in one dispatching application. Run only supported variants, and inspect disassembly or sampled symbols to confirm the intended path. Compare each output with a scalar reference using a stated tolerance for floating-point reorderings.\n\n## Control the measurement\n\nUse representative small, medium, and large inputs. Keep core placement, thread count, input data, memory layout, and frequency/thermal state as consistent as possible. Warm up each variant and repeat enough times to report a median and variation. On Linux, `perf stat -r 10 -- ./bench` can add cycles and cache events; discover model-specific counters with `perf list`. Measure elapsed time directly as the primary result.\n\nReport a ratio such as median_neon_time / median_sme_time, and state whether it covers the inner kernel or the whole operation. A scalar, unoptimized \"vanilla\" baseline is useful for validation but is a weak performance comparator. For an application decision, also compare an optimized existing backend.\n\n## Explain the result\n\nA small tile may favor Neon because packing and streaming entry have little work to amortize. SVE may be effective for predicated loops that do not map well to ZA. SME can win on sufficiently large matrix work when operands arrive in a useful layout and the output stays in ZA through the reduction. The actual crossover is device and implementation specific. If SME loses, profile packing, memory stalls, mode transitions, and fallback dispatch before drawing an architectural conclusion.\n\nEmulators can validate each variant's behavior across vector lengths; use real hardware for timing.\n\n## Resources\n\n- [Arm SIMD Loops](https://learn.arm.com/learning-paths/cross-platform/simd-loops/) provides comparable kernels.\n- [Arm's Neon, SVE, and SME matrix-multiplication comparison](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared) explains different mappings of the same work.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s07-q06", "slug": "check-whether-existing", "question": "How can I check whether existing SME2 kernels support my model's operators, tensor shapes, data types, and quantization formats?", "summary": "Check the framework's dispatch path for your exact model, not just the KleidiAI kernel catalog. A kernel can exist in KleidiAI but be unavailable through your LiteRT, ExecuTorch, ONNX Runtime, or llama.cpp build. Eligibility depends on the operator, input and output types, weight format, quantization parameters, layout, shape, backend build, and device features.", "answer": "Check the **framework's dispatch path for your exact model**, not just the KleidiAI kernel catalog. A kernel can exist in KleidiAI but be unavailable through your LiteRT, ExecuTorch, ONNX Runtime, or llama.cpp build. Eligibility depends on the operator, input and output types, weight format, quantization parameters, layout, shape, backend build, and device features.\n\n## Build an operator inventory\n\nExport the model's expensive nodes. For each, record the operator and effective matrix dimensions, activation and weight types, layout, quantization symmetry and zero points, per-tensor/channel/block scaling, and any fused activation. Dynamic shapes need representative runtime dimensions. A model viewer or a framework profiler is often quicker than reading graph source.\n\nFor ONNX, shape inference can fill in known static dimensions:\n\n```python\nimport onnx\nfrom onnx import shape_inference\n\nmodel = shape_inference.infer_shapes(onnx.load(\"model.onnx\"))\nfor node in model.graph.node:\n print(node.name, node.op_type, list(node.input), list(node.output))\n```\n\nThis lists nodes, but it does not prove that an execution provider delegates them or reveal every runtime shape.\n\n## Match the actual integration\n\nRead the supported-operator table for the **framework version you build**, then follow its backend to the KleidiAI kernel it calls. For example, XNNPACK is the relevant path for many LiteRT and ExecuTorch CPU operators; MLAS is the path for ONNX Runtime; GGML's type and operation dispatch matters for llama.cpp. \"INT4\" is not enough: block size, signedness, scale layout, and activation type must match. A kernel's packing function and tail handling also matter.\n\nDo not assume that a matrix dimension must be a multiple of a tile size. Many kernels handle tails, while some small shapes select a different implementation for performance. Check the particular dispatch condition.\n\n## Verify on the device\n\nRun a representative input on SME2 hardware with the intended build. Use framework operator profiling to find which nodes were delegated and sampled native symbols or instruction disassembly to see which microkernel ran. In XNNPACK traces, a `P` node label indicates a packed path; it is not by itself proof of SME2. If a node misses, inspect the backend's eligibility check, its exact tensor contract, and the device feature result.\n\nKeep a small coverage sheet:\n\n| Node | Shape | Types and quantization | Eligible in this build? | Kernel observed? | Reason for fallback |\n|---|---|---|---|---|---|\n| encoder MatMul | 128×768×768 | FP32 activations, INT4 weights with block scales | Check backend | Check trace | Record actual condition |\n\nThis separates static eligibility from measured dispatch, the distinction that matters when deciding whether to change a model or a build.\n\n## Resources\n\n- [LiteRT's KleidiAI operator configurations](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-model/) gives one concrete support table.\n- [KleidiAI source](https://github.com/ARM-software/kleidiai) defines kernel and packing contracts.\n- [Arm's llama.cpp integration walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_llama_cpp_sme2/kleidiai_integration/) explains that backend's selection path.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q01", "slug": "tile-matrix-operations", "question": "How do I tile my matrix operations to make optimal use of the ZA tile storage in SME2?", "summary": "Choose an output microtile that fits the ZA accumulators, keep it there through as much of the K reduction as practical, and store it once. Pack operands so each reduction step can load the A column values and B row values contiguously. The best tile shape depends on your data type, shape, cache, and hardware; \"use every ZA tile\" is a starting hypothesis, not a rule.", "answer": "Choose an output microtile that fits the ZA accumulators, keep it there through as much of the K reduction as practical, and store it once. Pack operands so each reduction step can load the A column values and B row values contiguously. The best tile shape depends on your data type, shape, cache, and hardware; \"use every ZA tile\" is a starting hypothesis, not a rule.\n\n## Derive the tile size\n\nZA is a square storage array whose side is the streaming vector length in bytes: `SVL_bytes = SVL_bits / 8`. It contains `SVL_bytes²` bytes. Its typed tile views share this storage. For 32-bit accumulation there are four ZA.S tiles, each containing `L × L` elements, where `L = SVL_bits / 32`. At a 512-bit SVL, ZA contains 4096 bytes and each tile is 16×16 32-bit elements. You can map the four tiles to a 2×2 output block of 32×32 elements or a 1×4 block of 16×64. The former reuses both operand panels; the latter can suit a narrow M dimension and wide N dimension. Query `svcntsw()` in streaming mode to get the number of 32-bit lanes. Do not bake 16 into the kernel.\n\n| 32-bit tile mapping | Output block at SVL=512 | Useful case |\n|---|---:|---|\n| 2×2 ZA tiles | 32×32 | Comparable reuse on both sides |\n| 1×4 ZA tiles | 16×64 | Narrow rows, wide output |\n\nA single output element is the sum of products across K. For a 2×2 tile arrangement, each K step loads two A vectors and two B vectors, then issues four outer products to update ZA0–ZA3. This is the core mapping:\n\n```text\nfor each output microtile:\n initialize its ZA accumulators\n for each k in the reduction:\n load A vectors for the output rows\n load B vectors for the output columns\n accumulate A0×B0, A0×B1, A1×B0, A1×B1 in ZA\n store the completed output microtile\n```\n\nA real implementation needs predicates for partial edge tiles and an explicit choice of whether C is overwritten or accumulated. If K is split into cache-sized blocks, carry partial sums across those blocks in ZA or load/add them back; zeroing ZA at each K block and overwriting C loses earlier products.\n\n## Pack and block around that kernel\n\nFor row-major A, the column values needed at one K step are strided. Pack panels so those values become consecutive, while B row values are already consecutive or can be packed into a panel that matches the selected N tile. Pack invariant weights once when possible. If packing dominates a small operation, a different algorithm or tile may win.\n\nChoose outer M, N, and K cache blocks so packed panels are reused without overflowing the useful cache. The cache sizes and optimal block order are implementation-specific; benchmark realistic dimensions rather than assuming a fixed 2×2 arrangement. Initialize ZA before accumulation, keep streaming-mode transitions outside the hot tile loop where practical, and store only valid rows and columns with predicates.\n\nThe following shows the four accumulator updates. It is an **inner-loop sketch**, not a complete runnable matmul: the vectors and predicates must come from correctly packed panels, and the surrounding function must have a valid streaming/ZA interface.\n\n```c\nsvmopa_za32_m(0, pm0, pn0, a0, b0);\nsvmopa_za32_m(1, pm0, pn1, a0, b1);\nsvmopa_za32_m(2, pm1, pn0, a1, b0);\nsvmopa_za32_m(3, pm1, pn1, a1, b1);\n```\n\nA wrapper can use `__arm_locally_streaming` and `__arm_new(\"za\")` where supported. The latter gives the function private ZA state; it does **not** initialize numeric accumulator contents, so zero the tiles explicitly. A portable caller should select this compiled kernel only after a runtime feature check.\n\n## Resources\n\n- [Arm's SME2 matrix multiplication walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr/) gives a complete intrinsics example with packing and tails.\n- [Arm's KleidiAI microkernel explanation](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/kai_sme2_matmul_ukernel_explained/kai_matmul_kernel_overview/) shows production tiling choices.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q02", "slug": "best-practices-data", "question": "What are the best practices for data layout and memory access patterns when targeting SME2 outer product instructions?", "summary": "Arrange each operand in the order the microkernel consumes it. For matrix multiplication, one outer-product step needs values from a column of A and a row of B. In ordinary row-major storage, B's row is contiguous but A's column is strided. Packing an A panel into column-contiguous vectors often removes expensive gathers from the reduction loop. Production kernels may pack both sides to improve reuse and match their exact instruction layout.", "answer": "Arrange each operand in the order the microkernel consumes it. For matrix multiplication, one outer-product step needs values from a column of A and a row of B. In ordinary row-major storage, B's row is contiguous but A's column is strided. Packing an A panel into column-contiguous vectors often removes expensive gathers from the reduction loop. Production kernels may pack both sides to improve reuse and match their exact instruction layout.\n\n## Follow the data through one tile\n\nFor a 32-bit accumulator tile, the conceptual inner loop is:\n\n```text\nload consecutive A values for the tile's output rows at k\nload consecutive B values for the tile's output columns at k\nouter-product accumulate into ZA\nrepeat for k, then store valid output rows and columns\n```\n\nA 2×2 arrangement of ZA.S tiles can reuse two A vectors and two B vectors across four outer products. The point of packing is to make those loads simple and contiguous. A layout that looks tidy as a matrix may be poor for the actual instruction stream.\n\nFor example, an A panel can be stored as `packed_A[k][row_lane]` and a B panel as `packed_B[k][column_lane]`. This is an illustrative layout; use the selected kernel's documented pack functions and strides for real integration. The packed format must define edge padding and the order of low-precision groups.\n\n## Account for cache and packing cost\n\nBlock M, N, and K so packed panels are reused in cache. If weights are constant, cache their packed representation rather than repacking at every inference. Keep the accumulator in ZA across the reduction; when you split K into blocks, preserve partial sums. Align panels as required by the load and pack routines, and allocate large temporary panels on the heap. An aligned allocation API may require the size to be a multiple of the alignment.\n\nFor BF16 and INT8 outer products, one instruction combines multiple narrow products into a wider accumulator. Their K grouping and lane order differ from FP32. Check the exact instruction and microkernel contract before packing; a generic \"INT8 layout\" is not enough. Quantized data also needs correct scales, zero points, and padding values, not just packed weights.\n\nUse predicates or padded input panels for partial M and N tiles. Derive tile sizes from the streaming vector length, for example with `svcntsw()` for 32-bit lanes while streaming. Avoid calling ordinary non-streaming helpers inside a tight streaming loop when each call forces a mode transition.\n\n## Check the result\n\nBenchmark a realistic matrix size with packing included. If speed falls short, profile strided loads, packed-panel cache misses, excess repacking, mode transitions, and output stores. A fast outer-product loop can still lose overall when its input format is expensive to create.\n\n## Resources\n\n- [Arm's matrix preprocessing walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product/) shows why and how A is transposed for outer products.\n- [Arm's KleidiAI packing explanation](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/kai_sme2_matmul_ukernel_explained/kai_matmul_kernel_overview/) connects packed panels to a production microkernel.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q03", "slug": "benchmark-sme2-matrix", "question": "How do I benchmark my SME2 matrix multiplication against a vanilla implementation to measure the actual speedup?", "summary": "Run both implementations on the same physical SME2 device with identical matrices and output semantics. Validate the results first, then compare repeated elapsed-time measurements. Report whether the measured interval includes packing, allocation, mode transitions, and stores. A scalar triple loop is a useful correctness reference, but an optimized Neon, SVE, or BLAS implementation is the meaningful performance baseline.", "answer": "Run both implementations on the same physical SME2 device with identical matrices and output semantics. Validate the results first, then compare repeated elapsed-time measurements. Report whether the measured interval includes packing, allocation, mode transitions, and stores. A scalar triple loop is a useful correctness reference, but an optimized Neon, SVE, or BLAS implementation is the meaningful performance baseline.\n\n## Set up a fair comparison\n\nUse the same M, N, K, input values, output type, thread count, and core placement. Initialize C the same way for both variants; distinguish C = A×B from C += A×B. If one kernel accepts packed weights, time packing separately and include it in an end-to-end result when weights are repacked for each call. If weights stay constant across many calls, report the amortized setup case too.\n\nCheck output with both absolute and relative tolerances appropriate for the data type and reduction length. NaNs, infinities, and near-zero expected values need explicit handling; a fixed absolute tolerance alone can hide important errors or reject harmless accumulation differences. Exercise partial tiles, odd K values, and more than one streaming vector length if possible.\n\n## Collect timings\n\nThis C driver is complete apart from your `sme2_matmul` implementation, which must use the same overwrite contract and signature. It checks output, warms both paths, times them in alternating order, and reports a median. Choose error tolerances appropriate to your algorithm and K length.\n\n```c\n#define _POSIX_C_SOURCE 200809L\n#include \n#include \n#include \n#include \n#include \n\ntypedef void (*matmul_fn)(int, int, int, const float *,\n const float *, float *);\nextern void sme2_matmul(int, int, int, const float *,\n const float *, float *);\n\nstatic void scalar_matmul(int M, int K, int N,\n const float *A, const float *B, float *C) {\n for (int m = 0; m < M; ++m)\n for (int n = 0; n < N; ++n) {\n float sum = 0.0f;\n for (int k = 0; k < K; ++k)\n sum += A[m*K + k] * B[k*N + n];\n C[m*N + n] = sum;\n }\n}\n\nstatic uint64_t now_ns(void) {\n struct timespec t;\n clock_gettime(CLOCK_MONOTONIC, &t);\n return (uint64_t)t.tv_sec * 1000000000ull + (uint64_t)t.tv_nsec;\n}\n\nstatic uint64_t time_one(matmul_fn fn, int M, int K, int N,\n const float *A, const float *B, float *C) {\n uint64_t start = now_ns();\n fn(M, K, N, A, B, C);\n return now_ns() - start;\n}\n\nstatic int cmp_u64(const void *a, const void *b) {\n uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b;\n return (x > y) - (x < y);\n}\n\nint main(void) {\n enum { M = 257, K = 255, N = 259, RUNS = 21 };\n float *A = malloc((size_t)M*K*sizeof *A);\n float *B = malloc((size_t)K*N*sizeof *B);\n float *ref = malloc((size_t)M*N*sizeof *ref);\n float *got = malloc((size_t)M*N*sizeof *got);\n if (!A || !B || !ref || !got) return 1;\n for (size_t i = 0; i < (size_t)M*K; ++i)\n A[i] = (float)(i % 13) / 13.0f;\n for (size_t i = 0; i < (size_t)K*N; ++i)\n B[i] = (float)(i % 17) / 17.0f;\n\n scalar_matmul(M, K, N, A, B, ref);\n sme2_matmul(M, K, N, A, B, got);\n for (size_t i = 0; i < (size_t)M*N; ++i) {\n float tolerance = 1e-4f + 1e-4f * fabsf(ref[i]);\n if (!isfinite(got[i]) || fabsf(ref[i] - got[i]) > tolerance) {\n fprintf(stderr, \"Mismatch at %zu: %g vs %g\\n\",\n i, ref[i], got[i]);\n return 2;\n }\n }\n for (int i = 0; i < 10; ++i) {\n scalar_matmul(M, K, N, A, B, ref);\n sme2_matmul(M, K, N, A, B, got);\n }\n uint64_t base[RUNS], sme[RUNS];\n for (int i = 0; i < RUNS; ++i) {\n if (i & 1) {\n sme[i] = time_one(sme2_matmul, M, K, N, A, B, got);\n base[i] = time_one(scalar_matmul, M, K, N, A, B, ref);\n } else {\n base[i] = time_one(scalar_matmul, M, K, N, A, B, ref);\n sme[i] = time_one(sme2_matmul, M, K, N, A, B, got);\n }\n }\n qsort(base, RUNS, sizeof base[0], cmp_u64);\n qsort(sme, RUNS, sizeof sme[0], cmp_u64);\n printf(\"median: scalar=%llu ns SME2=%llu ns ratio=%.2fx\\n\",\n (unsigned long long)base[RUNS/2],\n (unsigned long long)sme[RUNS/2],\n (double)base[RUNS/2] / (double)sme[RUNS/2]);\n free(A); free(B); free(ref); free(got);\n return 0;\n}\n```\n\nCompile the driver for a portable AArch64 target and link it with a separately compiled SME2 implementation. Replace the scalar comparator with an optimized Neon, SVE, or BLAS implementation before making a competitive speedup claim. The driver times only the function call, so state where packing and allocation happen in your implementation. For FP32 GEMM, 2×M×N×K divided by elapsed seconds estimates FLOP/s.\n\nSweep sizes below and above the expected cache and tile boundaries, including awkward dimensions. On Linux, taskset can pin a process and `perf stat` can collect cycles and cache events when available. Watch frequency and thermal behavior, especially on phones. An emulator is useful for correctness but cannot supply a physical-hardware speedup.\n\n## Interpret the result\n\nIf the SME2 inner loop wins but the full call loses, profile packing, memory traffic, allocation, mode switches, and edge handling. If a \"vanilla\" result is surprisingly slow, inspect its generated instructions and compare an established optimized library. Include a full-application or model measurement before claiming a user-visible gain.\n\n## Resources\n\n- [Arm's SME2 matrix multiplication benchmark walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/) offers a complete project to run and adapt.\n- [Arm's matrix-multiplication comparison](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared) explains why the baseline implementation matters.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q04", "slug": "common-performance-pitfalls", "question": "What are common performance pitfalls when using SME, such as excessive streaming mode transitions or ZA spills?", "summary": "The recurring costs are entering and leaving streaming mode too often, saving live ZA state across incompatible calls, packing operands repeatedly, and using only a small part of each output tile. Diagnose them in a profile of the full operation before changing the kernel.", "answer": "The recurring costs are entering and leaving streaming mode too often, saving live ZA state across incompatible calls, packing operands repeatedly, and using only a small part of each output tile. Diagnose them in a profile of the full operation before changing the kernel.\n\n## Too many mode transitions\n\nA streaming function that calls ordinary non-streaming helpers in its inner loop can make the compiler insert repeated SMSTOP and SMSTART transitions. Mode changes also invalidate the Z and P register contents across the boundary, so the generated code may need extra saves and reloads. Move logging, allocation, and callbacks out of the hot loop. Give a cohesive SME kernel a streaming interface, then inspect disassembly around its calls. `__arm_locally_streaming` is useful for a non-streaming public caller when the compiler supports it; it does not make arbitrary calls inside the function free.\n\n```c\n// Illustrative interfaces: the caller can remain non-streaming.\n__arm_locally_streaming __arm_new(\"za\")\nvoid compute_tiles(const float *packed_a, const float *packed_b,\n float *out, size_t tile_count) {\n // Initialize ZA before accumulating.\n // Process multiple tiles here; avoid logging inside the tile loop.\n}\n```\n\n`__arm_new(\"za\")` gives this function private ZA state, but does not zero its accumulator values. The body still needs explicit initialization. Where cooperating functions should share ZA, declare their interfaces accordingly rather than giving each helper a fresh ZA lifetime.\n\n## ZA save traffic\n\nZA storage grows with the square of streaming vector length: at 512-bit SVL it is 4 KiB, and at 2048 bits it is 64 KiB. The SME ABI's lazy-save protocol can require memory traffic when a function with live ZA calls code that does not share that state, especially a callee creating private ZA. It does **not** mean every call necessarily copies the whole array. Keep the hot reduction within a clear ZA lifetime, use a sharing interface such as `__arm_inout(\"za\")` for appropriate internal helpers, and inspect actual generated save/restore paths before blaming them.\n\n## Work that cannot amortize setup\n\nSmall matrices, partial tiles, and narrow reductions may spend more time on predicates, packing, ZA initialization, and dispatch than on arithmetic. Keep a measured Neon/SVE fallback for shapes where it wins. There is no universal M, N, or K threshold.\n\nFor row-major matrix multiplication, loading A columns directly can produce strided traffic. Pack panels in the microkernel's consumption order, reuse packed invariant weights, and tune cache blocks on the target device. Preserve partial sums correctly when K is split into blocks. Measure the complete call as well as the inner loop; a high FMOPA rate can coexist with poor end-to-end latency.\n\nFinally, do not assume ordinary SVE vector length equals SME streaming vector length. Derive streaming tile dimensions with a streaming query such as `svcntsw()`, and test non-multiple dimensions.\n\n## Resources\n\n- [ACLE SME function attributes](https://arm-software.github.io/acle/main/acle.html#sme-keyword-attributes) defines the streaming and ZA contracts.\n- [Arm's streaming-mode walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode/) explains the boundary behavior.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q05", "slug": "profiling-data-identify", "question": "How do I use profiling data to identify remaining bottlenecks after converting my workload to use SME2?", "summary": "Compare profiles of the same workload with SME2 enabled and disabled, then rank the time that remains in the enabled run. Faster matrix code may expose packing, memory movement, elementwise operators, synchronization, or other work. There is no fixed \"after SME2\" time breakdown; measure your model and device.", "answer": "Compare profiles of the same workload with SME2 enabled and disabled, then rank the time that remains in the enabled run. Faster matrix code may expose packing, memory movement, elementwise operators, synchronization, or other work. There is no fixed \"after SME2\" time breakdown; measure your model and device.\n\n## Rebuild the hotspot map\n\nHold inputs, precision, threads, core placement, and warm-up procedure constant. Record repeated end-to-end times and per-operator or function timings. Use `perf record -g` for native Linux call paths, Arm Streamline for a timeline and available hardware counters, ETDump for ExecuTorch operators, or ONNX Runtime profiling for model nodes. Note which kernel actually executed; a backend build flag or packed-node label alone does not prove SME2 dispatch.\n\nCreate a before/after table with **absolute time**, not percentages alone:\n\n| Stage | SME2 disabled | SME2 enabled | Next question |\n|---|---:|---:|---|\n| Matrix kernel | Measured | Measured | Did the expected microkernel run? |\n| Packing / conversion | Measured | Measured | Is it repeated unnecessarily? |\n| Other operators | Measured | Measured | Did this become the largest cost? |\n| Total request | Measured | Measured | Did the user-visible path improve? |\n\nA stage's percentage can rise while its time stays flat because another stage got faster. Optimize the largest *absolute* remaining cost on the critical path.\n\n## Classify the new hotspot\n\nIf packing dominates, cache invariant packed weights or change the data layout at a producer/consumer boundary. If cache misses and memory stalls rise with size, test panel reuse and memory bandwidth; a high miss count alone is not proof of a bandwidth limit. If small shapes suffer, inspect mode transitions, ZA state handling, and dispatch thresholds. If multiple threads stop scaling, inspect scheduling, shared cache, bandwidth, and synchronization.\n\nSample hot symbols and inspect disassembly where needed to confirm the SME2 instruction path and look for unexpected transitions or spills. Counting SMSTART instructions in the entire binary is only a static clue; the hot execution path matters. Sweep representative shapes and thread counts to see whether the diagnosis holds beyond one test case.\n\n## Resources\n\n- [Arm's ExecuTorch operator profiling guide](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/01-overview/) shows an SME2 on/off workflow.\n- [Arm's ONNX Runtime profiling example](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) combines operator and hardware views.\n- [Arm Performix microarchitecture guide](https://learn.arm.com/learning-paths/servers-and-cloud-computing/performix-microarchitecture/) helps classify CPU bottlenecks.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q06", "slug": "kleidiai-make-easier", "question": "Does using KleidiAI make it easier to optimize my workload?", "summary": "Yes, when a supported framework already calls a KleidiAI microkernel for your operator and data format. KleidiAI supplies specialized packing and compute routines for Arm CPUs, including SME2 variants, so you can avoid writing those low-level kernels yourself. Integration and performance are still conditional: the framework must include the relevant path, its dispatcher must select it, and the cost of packing must be worthwhile.", "answer": "Yes, when a supported framework already calls a KleidiAI microkernel for your operator and data format. KleidiAI supplies specialized packing and compute routines for Arm CPUs, including SME2 variants, so you can avoid writing those low-level kernels yourself. Integration and performance are still conditional: the framework must include the relevant path, its dispatcher must select it, and the cost of packing must be worthwhile.\n\n## If you use an existing framework\n\nXNNPACK integrations can expose KleidiAI kernels to LiteRT and ExecuTorch; MLAS can expose them to ONNX Runtime; GGML can expose them to llama.cpp. Each framework enables a **subset** of the KleidiAI catalog. Check your version's build configuration and operator coverage, then profile a representative run. A CPU feature check says a kernel *could* run, while sampled kernel symbols or instructions show what actually ran.\n\nThere is no universal selection order such as \"SME2, then I8MM, then Dot Product.\" The dispatcher considers operation, tensor type, shape, hardware features, and available implementations. A model may mix KleidiAI and other kernels.\n\n## If you build your own runtime\n\nKleidiAI gives you small C APIs for the operation's pack and matmul steps. You still own feature detection, kernel choice, allocation, threading, lifetime of packed weights, quantization metadata, and the fallback path. Follow the example for the exact kernel family you need, including its packing format. The library itself does not schedule a model or automatically choose the best kernel for an arbitrary operation.\n\n```text\ncheck feature and tensor contract\n→ choose matching pack and compute functions\n→ pack weights and activations as required\n→ run the microkernel\n→ handle unsupported cases with another implementation\n```\n\nFor constant weights, packing once and reusing the result can matter as much as the inner-kernel speed. For small operations or unsupported quantization formats, an existing backend may already be faster or simpler. Measure end-to-end latency and verify output before claiming a gain. KleidiAI targets selected AI operators on Arm CPUs; it does not accelerate every model node, training step, or GPU operation.\n\n## Resources\n\n- [KleidiAI repository and API overview](https://github.com/ARM-software/kleidiai) explains the pack/compute interface and its integration responsibilities.\n- [Arm's KleidiAI integration walkthrough](https://learn.arm.com/learning-paths/cross-platform/kleidiai-explainer/page2/) shows how a framework selects a microkernel.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s08-q07", "slug": "sme2-support-int2", "question": "Does SME2 support int2 quantization and how can I access it without writing my own code?", "summary": "SME2 has a way to decode packed 2-bit indices, but it does not provide a general 2-bit multiply-accumulate instruction. LUTI2 uses each 2-bit index to select one of four values from a table, producing wider elements that a later instruction can multiply. Whether you can use this without writing a kernel depends on your framework, model format, and its current backend integration. Do not infer INT2 support from SME2 hardware alone.", "answer": "SME2 has a way to **decode packed 2-bit indices**, but it does not provide a general 2-bit multiply-accumulate instruction. LUTI2 uses each 2-bit index to select one of four values from a table, producing wider elements that a later instruction can multiply. Whether you can use this without writing a kernel depends on your framework, model format, and its current backend integration. Do not infer INT2 support from SME2 hardware alone.\n\n## What the instruction does\n\nSME2's ZT0 is a fixed 512-bit lookup-table register. A LUTI2 instruction reads packed 2-bit indices from a Z register and uses them to select table values. For a signed two-bit codebook such as −2, −1, 0, 1, the kernel can expand the weights to 8-bit values and feed those values to an INT8 dot-product or outer-product instruction. It must still apply the model's scales, zero points, and packing rules correctly.\n\n```text\npacked 2-bit indices → LUTI2 table lookup → wider values → matrix arithmetic\n```\n\nThe table lookup is decompression, not direct INT2 arithmetic. Arm's [LUTI2 walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/luti/02_luti_and_sme2/) shows ZT0 loading, packed indices, and the following matrix instruction in one worked example.\n\n## Find a no-code path for your model\n\nCheck the **exact quantization format** and backend, then verify runtime kernel selection. KleidiAI and its LiteRT/XNNPACK, ExecuTorch, ONNX Runtime, and llama.cpp integrations have format-specific kernels; an INT4 path does not imply a matching INT2 path. In particular, a GGUF format name containing \"2\" does not establish that its quantization scheme matches a LUTI2 kernel.\n\nAs of September 2026, [AngelSlim's Hy-MT1.5 documentation](https://github.com/Tencent/AngelSlim/blob/main/docs/source/models/Hy-MT1.5/hy-mt1.5.md) describes a 2-bit SEQ SME2 route. Its separate 1.25-bit STQ1_0 llama.cpp work is a different format and should not be presented as an interchangeable 2-bit backend. Follow the project's own model and runtime instructions for the SEQ path, then profile the actual device to confirm the selected kernel.\n\nIf your framework has no kernel for your operator and quantization contract, you will need a custom kernel or a different supported format. The [Arm LUTI Learning Path](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/luti/) provides a functional example for building and checking such a kernel; emulation can check correctness, while speed requires physical SME2 hardware.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s09-q01", "slug": "inspect-contents-za", "question": "How do I inspect the contents of the ZA tile array during debugging of an SME application?", "summary": "Stop inside the SME kernel while ZA is active, inspect SVCR first, then read the ZA register or a typed tile view. On Linux, an SME-aware GDB or LLDB obtains this state through the kernel's SME ptrace register sets. When SVCR.ZA is clear, the debugger may display zeroes for ZA; those zeroes are not the previous accumulator values.", "answer": "Stop inside the SME kernel while ZA is active, inspect SVCR first, then read the ZA register or a typed tile view. On Linux, an SME-aware GDB or LLDB obtains this state through the kernel's SME ptrace register sets. When SVCR.ZA is clear, the debugger may display zeroes for ZA; those zeroes are not the previous accumulator values.\n\n## In GDB\n\nSet a breakpoint after the kernel has initialized and updated a tile:\n\n```gdb\n(gdb) break my_sme_kernel\n(gdb) run\n(gdb) info registers svcr svg\n(gdb) print $za\n(gdb) print $za0s\n```\n\nSVCR bit 0 reports streaming mode and bit 1 reports active ZA. SVG reports the streaming vector length in 64-bit granules, so `SVG=8` means a 64-byte SVL. GDB also provides typed tile and slice pseudo-registers when the target advertises SME. For a 32-bit tile, inspect `$za0s`; use `info all-registers` to discover the slice names exposed by your GDB version. The full ZA array can be large: at a 64-byte SVL it holds 4096 bytes.\n\nStop at several points in the K loop to check whether accumulation changes the expected tile. If a tile unexpectedly becomes zero, inspect the transition and call immediately before that point. Avoid using GDB's injected `call` command within live SME state; GDB documents limits in its SME-aware call handling.\n\n## In LLDB\n\n```text\n(lldb) register read svcr svg\n(lldb) register read za\n(lldb) register read zt0\n```\n\n`zt0` applies to SME2. LLDB presents ZA as a byte vector by default and supports formatting options, but does not infer the matrix element type. Use the program's tile mapping to interpret it. `svcr` and `svg` are read-only in LLDB.\n\nIf ZA is missing, check the CPU and OS exposure, debugger version, local versus remote stub, and whether you stopped after ZA was enabled. GDB documents that a remote gdbserver may show the wrong ZA layout if the target changes SVL during the session. For automated numeric comparison, store a logical tile to an ordinary memory buffer within the kernel and compare it with a scalar result.\n\n## Resources\n\n- [GDB's AArch64 SME register guide](https://www.sourceware.org/gdb/current/onlinedocs/gdb.html/AArch64.html) defines ZA tile views and debugger limitations.\n- [LLDB's AArch64 Linux guide](https://lldb.llvm.org/use/aarch64-linux.html) shows its SME register presentation.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s09-q02", "slug": "hardware-debug-features", "question": "What hardware debug features does Arm provide specifically for SME, and how do I use them with GDB or LLDB?", "summary": "SME adds architectural state for a debugger to expose: streaming-mode status, the streaming vector length, ZA, and, with SME2, ZT0. Ordinary breakpoints and memory watchpoints still work. On Linux, GDB and LLDB get SME register state through kernel ptrace interfaces; you do not need a special \"SME breakpoint\" facility.", "answer": "SME adds architectural state for a debugger to expose: streaming-mode status, the streaming vector length, ZA, and, with SME2, ZT0. Ordinary breakpoints and memory watchpoints still work. On Linux, GDB and LLDB get SME register state through kernel ptrace interfaces; you do not need a special \"SME breakpoint\" facility.\n\n## What to inspect\n\n| State | Why it matters |\n|---|---|\n| SVCR.SM | Shows whether the thread is in streaming mode |\n| SVCR.ZA | Shows whether ZA and ZT0 state is active |\n| SVG | Gives streaming vector length in 64-bit granules |\n| ZA and tile views | Hold matrix accumulators |\n| ZT0 | Holds the SME2 lookup table |\n| Streaming Z and P registers | Show vectors and predicates used by the current instruction |\n\nOn Linux, the kernel exposes streaming SVE state through `NT_ARM_SSVE`, ZA through `NT_ARM_ZA`, and ZT0 through `NT_ARM_ZT` when available. Whether a debugger can display them depends on the kernel, hardware, and debugger or remote stub. Do not infer support from a fixed minimum version without testing that combination.\n\n## GDB and LLDB workflow\n\n```gdb\n(gdb) info registers svcr svg\n(gdb) print $za0s\n(gdb) print $zt0\n(gdb) x/8i $pc\n```\n\nThe tile view `$za0s` is useful for 32-bit accumulator elements. If SVCR.ZA is clear, GDB displays inactive ZA as zero; it does not reveal a saved earlier value. GDB supports ZA slice pseudo-registers and documents their naming and limits.\n\n```text\n(lldb) register read svcr svg za zt0\n(lldb) disassemble --pc\n```\n\nLLDB shows ZA as bytes unless you choose a format. It treats `svcr` and `svg` as read-only. When debugging a memory error, set a watchpoint on the input or output buffer, then inspect the faulting load/store and its predicate. A hardware watchpoint does not directly watch an element inside ZA.\n\nFor bare-metal work, an SME-aware debug probe and monitor can expose the same architectural state, but exact commands depend on the probe and target. For remote GDB, check whether the stub advertises the optional SME target feature. Avoid debugger-injected function calls while ZA is live unless the debugger documents correct ABI handling.\n\n## Resources\n\n- [Linux SME ptrace documentation](https://www.kernel.org/doc/html/latest/arch/arm64/sme.html) defines the register sets.\n- [GDB AArch64 SME guide](https://www.sourceware.org/gdb/current/onlinedocs/gdb.html/AArch64.html) describes tile views and limitations.\n- [LLDB AArch64 Linux guide](https://lldb.llvm.org/use/aarch64-linux.html) describes register display and state changes.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s09-q03", "slug": "dump-za-tile", "question": "How can I dump ZA tile contents to a memory buffer at runtime to compare against a reference implementation?", "summary": "Store the logical tile slices you want to compare, rather than dumping all raw ZA storage. A horizontal-store ACLE intrinsic writes one row of a typed tile to memory. Loop over its rows in streaming mode, with ZA active, then compare that row-major buffer with the corresponding region of your reference output.", "answer": "Store the **logical tile slices** you want to compare, rather than dumping all raw ZA storage. A horizontal-store ACLE intrinsic writes one row of a typed tile to memory. Loop over its rows in streaming mode, with ZA active, then compare that row-major buffer with the corresponding region of your reference output.\n\n## Store one 32-bit tile\n\nThis helper illustrates the interface and buffer contract. It assumes that the caller has active ZA and that the buffer has room for `side * side` 32-bit elements, where `side` is the number of 32-bit lanes in the streaming vector.\n\n```c\n#include \n#include \n#include \n#include \n\nvoid dump_za0s(uint32_t *dst) __arm_streaming __arm_in(\"za\") {\n size_t side = svcntsw();\n svbool_t all = svptrue_b32();\n for (size_t row = 0; row < side; ++row)\n svst1_hor_za32(0, row, all, dst + row * side);\n}\n```\n\nThe buffer holds one ZA0.S tile, not the entire ZA array. For a 512-bit SVL, side=16, so this example writes 16×16 32-bit values (1024 bytes). Four 32-bit tile views share the 4096-byte ZA storage. Adapt the tile number and element width to the accumulator you actually use.\n\nAllocate the buffer before entering a streaming hot loop when possible. A private-ZA wrapper can perform the calculation, dump the tile, and return to a non-streaming caller. If it uses `__arm_new(\"za\")`, initialize ZA explicitly before accumulation; that attribute does not zero numeric contents.\n\n## Compare with the reference\n\nMap each stored tile row and column to the output coordinates chosen by your microkernel. Compare only active M and N lanes at edge tiles. For floating-point results, report the first mismatch, maximum absolute and relative error, and the inputs and K position that produced it. For integer accumulation, compare exact values unless the algorithm specifies saturation or rounding.\n\nA raw ZA dump is useful for a debugger or context-state investigation, but typed tiles are interleaved within raw ZA storage. Its bytes cannot be treated as one row-major FP32 matrix without decoding the tile mapping. If you need an entire raw snapshot, use the architecture's ZA array-vector store or the corresponding ACLE operation and size the buffer as streaming-vector-length-in-bytes squared.\n\n## Resources\n\n- [Arm's ZA storage guide](https://developer.arm.com/documentation/109974/0100/ZA-storage) explains tile and slice mapping.\n- [ACLE SME intrinsics](https://arm-software.github.io/acle/main/acle.html) defines the store operations and function attributes.", "tags": [ "SME" ], "golden": true }, { "id": "s09-q04", "slug": "recommended-approach-writing", "question": "What is the recommended approach for writing a tiled reference implementation to validate SME results step-by-step?", "summary": "Use two references. First write a plain scalar implementation for the mathematical result. Then write a scalar tiled version with the same M/N tiles and K-block order as the SME kernel. Compare their final outputs, and add checkpoints in the SME kernel only where you need to locate a mismatch. The tiled reference should mirror the logical work, not ZA's physical register interleaving.", "answer": "Use two references. First write a plain scalar implementation for the mathematical result. Then write a scalar **tiled** version with the same M/N tiles and K-block order as the SME kernel. Compare their final outputs, and add checkpoints in the SME kernel only where you need to locate a mismatch. The tiled reference should mirror the logical work, not ZA's physical register interleaving.\n\n## Start with a scalar result\n\nFor row-major FP32 matrices and overwrite semantics:\n\n```c\nfor (size_t m = 0; m < M; ++m)\n for (size_t n = 0; n < N; ++n) {\n float sum = 0.0f;\n for (size_t k = 0; k < K; ++k)\n sum += A[m*K + k] * B[k*N + n];\n C_ref[m*N + n] = sum;\n }\n```\n\nIf your real operation adds to an existing C, includes bias, or uses quantization, put those semantics in the reference too.\n\n## Mirror the kernel's tiling\n\nThe tiled reference below uses scalar arithmetic but makes the M/N and K boundaries visible. It illustrates one output element at a time; a real harness can keep a small local tile array to compare all elements after each K block.\n\n```c\nfor (size_t m0 = 0; m0 < M; m0 += MR)\n for (size_t n0 = 0; n0 < N; n0 += NR)\n for (size_t m = m0; m < M && m < m0 + MR; ++m)\n for (size_t n = n0; n < N && n < n0 + NR; ++n) {\n float sum = 0.0f;\n for (size_t k0 = 0; k0 < K; k0 += KC)\n for (size_t k = k0; k < K && k < k0 + KC; ++k)\n sum += A[m*K + k] * B[k*N + n];\n C_tiled[m*N + n] = sum;\n }\n```\n\nCompare `C_tiled` with `C_ref` before examining SME. Then test the packed A and B panels separately by decoding a few indices back to source coordinates. In the SME kernel, optionally store the accumulator tile after each K block to a debug buffer. Map each tile lane to its global output coordinate and compare it with the tiled reference at the same K checkpoint. Keep the normal fast path free of debug stores.\n\n## Build a useful test matrix\n\nInclude dimensions of 1, a full tile, one past a tile, and values just before and after K-block boundaries. Try more than one supported streaming vector length. Check zero padding, inactive predicates, and C overwrite versus accumulation. For FP32, use a tolerance based on the result magnitude and K length; report absolute and relative error, not a single unexplained threshold. For INT8/INT4 kernels, also check scales, zero points, unpack order, and overflow behavior.\n\nA mismatch in both scalar references points to the reference or semantics. A mismatch only after packing points to layout. A mismatch at the first K checkpoint points to initialization, operand loading, or predicate mapping; one appearing later suggests K-block carry or accumulation order. This structure narrows the bug without emulating ZA's raw byte layout.\n\n## Resources\n\n- [Arm's SME2 matrix multiplication example](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/) shows the tiled implementation to validate.\n- [Arm's ZA storage guide](https://developer.arm.com/documentation/109974/0100/ZA-storage) explains how logical tile slices map to ZA.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s09-q05", "slug": "debug-issues-streaming", "question": "How do I debug issues with streaming mode transitions, such as unexpected loss of ZA state?", "summary": "First distinguish leaving streaming mode from disabling ZA. SMSTOP SM clears streaming mode but can leave ZA active; SMSTOP ZA disables ZA, and later re-enabling ZA initializes its state rather than restoring your accumulator. A broad SMSTOP affects both. Check the actual instruction and SVCR bits at the point where the value changes.", "answer": "First distinguish **leaving streaming mode** from **disabling ZA**. `SMSTOP SM` clears streaming mode but can leave ZA active; `SMSTOP ZA` disables ZA, and later re-enabling ZA initializes its state rather than restoring your accumulator. A broad SMSTOP affects both. Check the actual instruction and SVCR bits at the point where the value changes.\n\n## Trace the failing boundary\n\nStop immediately before and after the suspicious call or mode transition. In GDB, `info registers svcr` shows bit 0 (SM) and bit 1 (ZA); in LLDB use `register read svcr`. Step through SMSTART and SMSTOP in disassembly. If ZA stays active but values change, inspect a callee that shares ZA, incorrect tile indices, or handwritten assembly. If ZA becomes inactive, find who disabled it and whether the calling contract allowed that.\n\nWrite down each function's ACLE interface:\n\n| Interface | Expected effect |\n|---|---|\n| `__arm_streaming` | Function body runs in streaming mode |\n| `__arm_locally_streaming` | Compiler manages a local streaming region for a non-streaming interface |\n| `__arm_inout(\"za\")` | Callee shares and modifies caller's ZA |\n| `__arm_new(\"za\")` | Callee gets private ZA; caller's live ZA must be preserved by the ABI |\n\nDo not insert manual SMSTART or SMSTOP into a function whose streaming state the compiler manages unless the assembly contract is fully accounted for. For a self-contained kernel with `__arm_new(\"za\")`, explicitly initialize its accumulators before use. If hand-written assembly crosses a function boundary with live ZA, follow the SME ABI's lazy-save protocol rather than assuming a normal AArch64 call preserves the matrix register.\n\n## Make the failure reproducible\n\nIn a debug build, store a small logical tile or checksum to ordinary memory before and after the boundary, and log only after leaving the hot streaming region. Avoid calling `printf` inside the active kernel just to inspect state: the debugger or compiler may need mode transitions for the call. Test both immediate and repeated calls, including signals or context switches when the issue appears only on a device. A supported OS should preserve architectural state across ordinary scheduling; do not treat every context switch as proof of an OS bug.\n\nIf a remote debugger changes or misreports SVL, repeat locally or check the kernel's ptrace state. GDB documents limitations when gdbserver and GDB disagree after an SVL change.\n\n## Resources\n\n- [ACLE SME function attributes](https://arm-software.github.io/acle/main/acle.html#sme-keyword-attributes) defines the interface contracts.\n- [Arm's streaming-mode guide](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode/) illustrates state transitions.\n- [GDB's SME debugging notes](https://www.sourceware.org/gdb/current/onlinedocs/gdb.html/AArch64.html) describe remote and function-call limitations.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s09-q06", "slug": "diagnose-sme2-enabled", "question": "How can I diagnose why my SME2-enabled application behaves differently or performs differently on a real device compared with my expected baseline or reference implementation?", "summary": "Separate three questions: did the intended SME2 path run, did it produce the right values, and did it improve the full workload? A build flag or an SME2 instruction somewhere in the binary answers none of these by itself.", "answer": "Separate three questions: did the intended SME2 path run, did it produce the right values, and did it improve the full workload? A build flag or an SME2 instruction somewhere in the binary answers none of these by itself.\n\n## Confirm the executed path\n\nCheck that the OS exposes SME2 to the process. On Linux, use `getauxval(AT_HWCAP2)` with `HWCAP2_SME2`; on macOS, use the documented feature sysctl. Record the device, OS, compiler, framework version, and streaming vector length. Inspect runtime logs and sample the hot symbols to see the selected kernel. Disassembly of that kernel can show SME2 instructions, but disassembly of an unused function is not execution proof.\n\nKeep feature-targeted code isolated so a portable binary does not execute SME2 before the runtime check. If the application falls back, inspect the operator's data type, quantization scheme, shape, backend configuration, and feature detection.\n\n## Localize a correctness difference\n\nCompare a scalar reference and the device result for the same inputs. Start with a full tile, then non-multiple M/N/K dimensions and different vector lengths. If output diverges only at edges, inspect predicates and padded panels. If it diverges after a K block, inspect accumulator initialization and partial-sum carry. Decode a few packed elements back to source coordinates. For floating point, report absolute and relative error, NaNs, and the accumulation order; a bitwise mismatch can be legitimate while a large numeric error is not.\n\nAudit every streaming and ZA function interface. `__arm_new(\"za\")` supplies private ZA but does not zero its numeric contents. A mismatch between `__arm_inout(\"za\")` declarations across translation units or a manual mode transition can corrupt state. Inspect SVCR and generated SMSTART/SMSTOP around the first failing call. Debuggers and emulators can help localize this even when they cannot predict speed.\n\n## Explain a performance difference\n\nRun repeated native-device measurements with the same input, threads, affinity, warm-up, and thermal conditions as the baseline. Report median and variation. Time packing, the matrix kernel, other operators, and the full inference or request separately. If only the inner loop improves, packing or another stage may dominate. If the expected kernel ran but is slow, inspect tile utilization, memory stalls, mode transitions, and ZA save traffic. Compare with a genuinely optimized baseline, not solely a scalar reference.\n\nDo not assume the simulator's time, a different device's speedup, or a fixed \"SME2 gain\" applies to this hardware. A short table of observed path, correctness, kernel time, and end-to-end time makes the difference actionable.\n\n## Resources\n\n- [Arm's SME2 debugging walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/9-debugging/) covers assembly and correctness checks.\n- [Linux SME userspace documentation](https://www.kernel.org/doc/html/latest/arch/arm64/sme.html) covers feature exposure and state.\n- [Arm's ONNX Runtime profiling example](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) shows end-to-end and operator comparisons.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s10-q01", "slug": "detect-at-runtime", "question": "How do I detect at runtime whether the current processor supports SME and SME2 features?", "summary": "Use the operating system's process-visible feature interface. On Linux and Android, test AT_HWCAP2 for HWCAP2_SME or HWCAP2_SME2; on macOS, query Apple's hw.optional.arm.FEAT_SME and FEAT_SME2 selectors. A compiler macro says what the binary was built to use, not what the current device can run.", "answer": "Use the operating system's **process-visible feature interface**. On Linux and Android, test `AT_HWCAP2` for `HWCAP2_SME` or `HWCAP2_SME2`; on macOS, query Apple's `hw.optional.arm.FEAT_SME` and FEAT_SME2 selectors. A compiler macro says what the binary was built to use, not what the current device can run.\n\n## Linux and Android\n\n```c\n#include \n#include \n#include \n#include \n\nint main(void) {\n unsigned long caps = getauxval(AT_HWCAP2);\n bool sme = (caps & HWCAP2_SME) != 0;\n bool sme2 = (caps & HWCAP2_SME2) != 0;\n printf(\"SME=%d SME2=%d\\n\", sme, sme2);\n return 0;\n}\n```\n\nBuild this detection source for a **portable AArch64 target**. Current system headers define these bits; if you must support older headers, verify any local fallback values against the kernel UAPI before adding them. As a shell smoke test, `/proc/cpuinfo` commonly lists `sme` and `sme2` in the Features line. Use HWCAP in application code because it represents what the kernel makes usable to the process, including its SME state-management interfaces.\n\nIf the code needs an optional SME subfeature, check its own HWCAP bit too. SME2 implies SME, but it does not imply every optional arithmetic format or ordinary non-streaming SVE2. Keep SME functions in a separate translation unit or otherwise isolate their target attributes; call them only after dispatch chooses a supported path.\n\n## macOS\n\n```sh\nsysctl hw.optional.arm.FEAT_SME\nsysctl hw.optional.arm.FEAT_SME2\n```\n\nIn C or C++, query the same selectors with `sysctlbyname`, checking the return value and returned size. A result of 1 indicates that macOS exposes the feature. Do not use Linux HWCAP interfaces on macOS.\n\nFor bare-metal or privileged software, architectural ID registers can describe the processor's implemented features. They do not replace the OS-visible test in a user-space application. Do not probe by executing an SME instruction and waiting for SIGILL; Linux does not guarantee reliable feature discovery that way.\n\n## Resources\n\n- [Linux arm64 ELF HWCAP documentation](https://docs.kernel.org/arch/arm64/elf_hwcaps.html) defines process-visible feature tests.\n- [Linux SME userspace documentation](https://docs.kernel.org/arch/arm64/sme.html) explains the SME interfaces attached to those flags.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s10-q02", "slug": "verify-application-actually", "question": "How do I verify that my application is actually using SME instructions and not falling back to Neon or scalar code?", "summary": "Collect evidence at three levels: the device exposes SME, the runtime chooses an SME implementation, and that implementation executes SME instructions during your representative workload. An SME instruction present in a binary proves only that it was compiled.", "answer": "Collect evidence at three levels: the device exposes SME, the runtime chooses an SME implementation, and that implementation **executes SME instructions** during your representative workload. An SME instruction present in a binary proves only that it was compiled.\n\n## Check availability and dispatch\n\nOn Linux or Android, check `getauxval(AT_HWCAP2)` for SME/SME2; on macOS, check the corresponding feature sysctl selectors. Then inspect the framework's operator profile or backend diagnostics. For a model, note the exact operator, data types, quantization format, shape, and selected kernel. A packed XNNPACK node label or llama.cpp `CPU_KLEIDIAI` model buffer can show part of the path but does not by itself prove SME2.\n\nIf the operator falls back, inspect its eligibility conditions and the build configuration. A device with SME2 can still run Neon for an unsupported shape or format. Modern libraries may use different kernel families for prefill and decode, or for full and partial tiles.\n\n## Prove the instruction ran\n\nSample the hot call path on the device, for example:\n\n```sh\nperf record -g -- ./your_application\nperf report\nperf annotate\n```\n\nUse simpleperf on Android when appropriate. A sample on a named SME kernel is stronger evidence than a build flag; disassemble that function and look for FMOPA, SMOPA, ZA accesses, or other SME instructions. A breakpoint at the kernel entry, followed by stepping into its arithmetic, is useful when sampling misses a short call. SMSTART alone shows a mode change, not that matrix arithmetic ran. Hardware event names and their availability depend on the core, so discover them with `perf list` rather than copying a generic SME event.\n\nFor a multiversioned library, inspect both the selected SME path and the fallback path. Record how many calls or what fraction of time uses each. A trace build can identify a kernel, but turn verbose logging off for timing because I/O distorts the result.\n\n## Resources\n\n- [Arm's ONNX Runtime profiling walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) shows operator and hardware evidence.\n- [GDB's AArch64 SME guide](https://www.sourceware.org/gdb/current/onlinedocs/gdb.html/AArch64.html) helps when stepping through a selected kernel.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s10-q04", "slug": "if-want-prototype", "question": "If I want to prototype on macOS, what hardware and software is required for full SME support, and how do I check that it's enabled?", "summary": "Use a Mac whose CPU and macOS release expose SME or SME2, plus a compiler and SDK that accept the SME intrinsics you need. Apple M4 and M5 family Macs are documented SME2 development targets; do not infer support merely from \"Apple silicon\" or a model name. Check the feature selectors on the exact machine before building an SME-only binary.", "answer": "Use a Mac whose CPU and macOS release **expose** SME or SME2, plus a compiler and SDK that accept the SME intrinsics you need. Apple M4 and M5 family Macs are documented SME2 development targets; do not infer support merely from \"Apple silicon\" or a model name. Check the feature selectors on the exact machine before building an SME-only binary.\n\n## Check the machine\n\n```sh\nsysctl hw.optional.arm.FEAT_SME\nsysctl hw.optional.arm.FEAT_SME2\n```\n\nA value of 1 means the feature is exposed to applications. An unknown selector or zero means you should not dispatch to that path. M1–M3 Macs do not provide architectural SME. A MacBook Neo uses an A-series processor, so test its selectors rather than treating it as an M-series configuration.\n\nYou can make the same check in an application with `sysctlbyname`:\n\n```c\n#include \n#include \n\nstatic bool has_feature(const char *name) {\n int value = 0;\n size_t size = sizeof(value);\n return sysctlbyname(name, &value, &size, 0, 0) == 0\n && size == sizeof(value) && value == 1;\n}\n\n/* has_feature(\"hw.optional.arm.FEAT_SME2\") */\n```\n\nA compiler target macro such as `__ARM_FEATURE_SME2` describes the source being compiled; it is not a runtime feature check.\n\n## Build and test\n\nUse a current Xcode command-line toolchain or another macOS AArch64 compiler with SME ACLE support. Compile a small source containing the required intrinsic and attribute first; toolchain support varies by version and intrinsic family. Keep this source in a feature-targeted object or function, while the dispatching code remains compatible with older Macs. If a build flag such as `-march=armv9-a+sme2` is accepted by the chosen compiler, apply it to the SME2 source, not indiscriminately to the whole app.\n\nRun a correctness test with odd matrix dimensions and inspect the selected function's instructions or profile samples. For performance, measure the same work through an optimized baseline with the same thread count and thermal conditions. On macOS use `sysctlbyname` for feature detection; Linux `getauxval(AT_HWCAP2)` is not a macOS API.\n\n## Resources\n\n- [Arm's macOS SME2 setup walkthrough](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/) identifies a working toolchain route.\n- [Arm's SME2 environment check](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment/) shows a small capability test.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s10-q05", "slug": "map-model-operator", "question": "How can I map a model operator to the XNNPACK or KleidiAI microkernel that is actually selected at runtime?", "summary": "Trace one node through framework delegation, backend operator creation, kernel configuration, and executed function. A static support table can tell you what is eligible, but shape, data type, build options, hardware features, and runtime heuristics determine what actually runs.", "answer": "Trace one node through **framework delegation, backend operator creation, kernel configuration, and executed function**. A static support table can tell you what is eligible, but shape, data type, build options, hardware features, and runtime heuristics determine what actually runs.\n\n## Follow one operator\n\n1. Capture the model operator name, input/output shapes, layout, and quantization parameters in a framework profile.\n2. Check whether the operator was delegated to XNNPACK or stayed in the framework's own CPU path. ONNX Runtime's MLAS and llama.cpp's GGML paths are separate from XNNPACK; do not draw them as XNNPACK stages.\n3. If delegated, inspect the backend's operator type and kernel-selection trace for the exact build. One model node may become several backend operations, and a fused backend operation may cover several model nodes.\n4. Sample native call stacks or set a debugger breakpoint at the selected `xnn_*` or `kai_*` symbol. Resolve stripped addresses against the matching unstripped binary or symbol file.\n5. Disassemble the hot leaf function to distinguish SME2, SME, Neon, and scalar instructions. A `P` label in an XNNPACK operator name indicates a packed path, not necessarily an SME2 instruction.\n\nFor example, a Linear layer may use a matrix-multiply kernel for prompt prefill and a vector-matrix path for single-token decode. Record both phases rather than assigning the layer one permanent microkernel.\n\n| Evidence | What it establishes |\n|---|---|\n| Framework operator profile | Which model node takes time |\n| Delegation or XNNPACK trace | Which backend operation and candidate kernel were configured |\n| Sampled native symbol or breakpoint | Which implementation executed |\n| Instruction view | Which architecture instructions it used |\n\nIf the expected kernel is absent, check the supported type and quantization contract, the actual runtime shape, the build's included microkernels, the device feature result, and the selected fallback. A model can mix SME2 and Neon paths.\n\n## Resources\n\n- [Arm's LiteRT/XNNPACK tracing example](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/4-benchmark/) explains packed node labels.\n- [Arm's ExecuTorch profiling workflow](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/01-overview/) connects model operators to timings.\n- [Arm's llama.cpp KleidiAI trace](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_llama_cpp_sme2/kleidiai_integration/) shows the separate GGML route.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s10-q06", "slug": "enable-xnnpack-kernel", "question": "How do I enable XNNPACK kernel tracing or debug logging to verify whether an SME2, Neon, or scalar kernel was selected?", "summary": "Build a separate diagnostic binary with XNNPACK logging enabled, run one representative inference, and inspect the logged microkernel name. Logging is compiled into XNNPACK; setting an environment variable cannot restore messages removed from a release build. Then confirm that the named function executes with a breakpoint or profiler, because registration logs and runtime execution are different evidence.", "answer": "Build a **separate diagnostic binary** with XNNPACK logging enabled, run one representative inference, and inspect the logged microkernel name. Logging is compiled into XNNPACK; setting an environment variable cannot restore messages removed from a release build. Then confirm that the named function executes with a breakpoint or profiler, because registration logs and runtime execution are different evidence.\n\n## Enable logging in the build you actually ship through\n\nIn XNNPACK's CMake build, the logging target defines `XNN_LOG_LEVEL=5` for Debug and 0 for other configurations. A standalone debug build can start with:\n\n```sh\ncmake -S XNNPACK -B build-xnn-trace -DCMAKE_BUILD_TYPE=Debug\ncmake --build build-xnn-trace -j\n```\n\nFor ExecuTorch or LiteRT, XNNPACK is embedded in a larger build. Follow that project's profiling preset or propagate its documented XNNPACK logging option through the dependency build; rebuilding only the top-level executable may leave the bundled XNNPACK library unchanged. Confirm the final binary contains the logging configuration. XNNPACK build options and log routing differ among CMake, Bazel, Android, and Apple builds, so do not assume one `-D` flag works everywhere.\n\nCapture output from a short run, including Android logcat when that build uses platform logging. Look for the full microkernel symbol, for example a name containing `sme2`, `neon`, or `scalar`. Check `sme2` before `neon`: an SME2 symbol may contain both strings. A `kai_` prefix indicates a KleidiAI function family, but the exact architecture is in the full variant name.\n\n## Confirm the chosen path\n\nA message that a kernel was registered or configured may precede a different operator path at execution time. Set a breakpoint on the logged symbol or sample the hot native call stack. Inspect its instructions for SME2 outer products or other SME2 operations. If the expected name never appears, check the device's OS-exposed SME2 bit, the build's compiled kernel set, the operator/data-format contract, and shape-specific fallback. Use a quiet optimized build for actual latency comparisons; debug logging and Debug optimization settings alter performance.\n\n## Resources\n\n- [XNNPACK CMake configuration](https://github.com/google/XNNPACK/blob/master/CMakeLists.txt) shows its logging definition.\n- [Arm's ExecuTorch SME2 profiling setup](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline/) provides a trace-build workflow.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s10-q07", "slug": "android-studio-cpu", "question": "Does the Android Studio CPU workflow support SME?", "summary": "Yes for building, deploying, debugging, and general CPU profiling of an Android app that contains SME code. Android Studio does not have a dedicated view that proves SME instructions executed or displays ZA as a matrix. Use a physical SME-capable device and pair the IDE's workflow with runtime feature checks and a native profiler or debugger.", "answer": "Yes for building, deploying, debugging, and general CPU profiling of an Android app that contains SME code. Android Studio does not have a dedicated view that proves SME instructions executed or displays ZA as a matrix. Use a physical SME-capable device and pair the IDE's workflow with runtime feature checks and a native profiler or debugger.\n\n## Practical workflow\n\n1. Install an Android NDK whose Clang supports the SME intrinsics or assembly you use. Build an arm64-v8a native library with SME code isolated behind runtime dispatch.\n2. On the device, check `sme` or `sme2` in `/proc/cpuinfo` as a smoke test. Production native code should read the Linux/Android `AT_HWCAP2` flags and retain a compatible fallback.\n3. Use Android Studio's CPU Profiler to find expensive app and native functions. For instruction-level evidence, use simpleperf, a debugger breakpoint, or an SME-aware profiler, then inspect the selected native kernel's instructions.\n4. Compare equivalent runs with the accelerated path enabled and disabled, including full app latency. A function named \"SME\" in a stack trace or an arm64-v8a build does not establish that SME arithmetic ran.\n\nA standard Android virtual device should not be used for hardware speed measurements. LLDB's ability to show SME registers depends on the debugger, OS image, and target support; test it on the actual device.\n\n## Resources\n\n- [Arm's Android SME2 environment guide](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/) covers NDK setup.\n- [Arm's Android Studio profiling guide](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/profiling-ml-on-arm/app-profiling-android-studio/) covers the IDE's CPU view.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s11-q01", "slug": "take-existing-onnx", "question": "How do I take an existing ONNX or LiteRT model and optimize it for edge inference on Arm with SME2?", "summary": "Keep the model in its existing format and first try an SME2-capable CPU runtime. ONNX Runtime can reach KleidiAI through MLAS; LiteRT can reach it through XNNPACK. Then profile the model, verify which operators actually use SME2 kernels, and change precision or graph structure only where the measured result justifies it. SME2 is a runtime implementation choice, not an ONNX or .tflite export format.", "answer": "Keep the model in its existing format and first try an SME2-capable **CPU runtime**. ONNX Runtime can reach KleidiAI through MLAS; LiteRT can reach it through XNNPACK. Then profile the model, verify which operators actually use SME2 kernels, and change precision or graph structure only where the measured result justifies it. SME2 is a runtime implementation choice, not an ONNX or `.tflite` export format.\n\n## Establish a device baseline\n\nConfirm the target OS exposes SME2 and run the unmodified model on the physical device. Record accuracy, cold-start and steady-state latency, memory use, thread count, and representative input shapes. Keep the same measurements for every later variant. A supported CPU does not imply every model operator has an SME2 kernel.\n\n## Use the matching runtime path\n\n| Model | CPU path to inspect | First practical step |\n|---|---|---|\n| ONNX | ONNX Runtime CPU execution provider → MLAS → eligible KleidiAI kernel | Build or obtain the exact ONNX Runtime version with KleidiAI support, then run its performance test and operator profiler. |\n| LiteRT (`.tflite`) | LiteRT XNNPACK delegate → eligible KleidiAI kernel | Build the LiteRT benchmark tool with the documented XNNPACK/KleidiAI options, then compare its enabled and baseline builds. |\n\nFor an Android ONNX Runtime build, Arm's [KleidiAI build walkthrough](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/build_ort/) documents the versioned build command and `onnxruntime_USE_KLEIDIAI=ON` option. Follow its Android SDK and NDK path parameters rather than treating the SDK path as the NDK path. For LiteRT, Arm's [benchmark-tool build step](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/3-build-tool/) documents the matching Bazel options. These flags compile candidate kernels; they do not force each operator to use SME2.\n\nIn a LiteRT app, enable the XNNPACK delegate and check the return from `ModifyGraphWithDelegate`. For ONNX Runtime, use the CPU execution provider and an optimized session graph. If deployment uses a packaged runtime, inspect that binary's build options instead of assuming it matches a source tutorial.\n\n## Optimize where profiling points\n\nList the expensive MatMul, FullyConnected, Conv, and related operators with actual shapes, types, and quantization parameters. Check the framework's support table for that **full tuple**, then inspect operator timing and sampled native symbols. If a hot operator misses the SME2 path, find the eligibility condition before rewriting the model.\n\nTry FP32 or FP16 as a correctness baseline, then compare INT8 or INT4 only where the runtime has a matching kernel and the model meets its accuracy target. Quantization may introduce conversion and packing work or change operator coverage. Simplify redundant transposes and fuse operations only when the runtime supports the fused form and the model still produces acceptable outputs. Keep a portable CPU fallback for devices without SME2.\n\nThe result to report is the model's accuracy and end-to-end latency on the device, plus the operators whose implementation changed. A faster isolated microkernel is useful evidence but not the final deployment result.\n\n## Resources\n\n- [Arm's ONNX Runtime KleidiAI build and profile path](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/build_ort/) gives a concrete Android route.\n- [Arm's LiteRT operator configurations](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-model/) helps match model formats to available kernels.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s11-q02", "slug": "quantization-strategies-int4", "question": "What quantization strategies (INT4, INT8, FP16) work best with SME2 for edge AI model deployment?", "summary": "Start with the model's accuracy and memory target, then check actual backend kernel coverage for the chosen format. INT4 weights can cut weight storage for large models; INT8 can work well for vision and other quantized models; FP16 is a useful lower-conversion path when quantization hurts accuracy or coverage. None is inherently the fastest SME2 choice for every operator, shape, or device.", "answer": "Start with the model's accuracy and memory target, then check **actual backend kernel coverage** for the chosen format. INT4 weights can cut weight storage for large models; INT8 can work well for vision and other quantized models; FP16 is a useful lower-conversion path when quantization hurts accuracy or coverage. None is inherently the fastest SME2 choice for every operator, shape, or device.\n\n| Strategy | Typical reason to try it | Main check |\n|---|---|---|\n| INT4 weights, wider activations | Weight memory and bandwidth dominate, often in an LLM | Does the backend support this exact block size, scale layout, and activation type? |\n| INT8 weights and activations | Calibrated integer inference is available | Are all expensive operators delegated, and what is quantize/dequantize cost? |\n| FP16 | Quantization accuracy or conversion cost is troublesome | Does the runtime have an FP16 SME path for the operator, and what precision does it accumulate in? |\n\nSME2's LUTI4 can expand packed four-bit indices to wider values that an integer matrix instruction consumes. That is one possible kernel design, not a guarantee that every INT4 model format uses LUTI4 or even SME2. An INT4 GGUF Q4_0 path, an ONNX blockwise quantization scheme, and a LiteRT per-channel format have different contracts. Do not treat the bit width as the whole compatibility test.\n\nFor INT8, calibration data should represent deployment inputs. Per-channel weight scales can protect accuracy, but the runtime must support that scheme for the operator. Dynamic activation quantization avoids a fixed calibration range but adds per-inference work. FP16 reduces weight storage relative to FP32; its accuracy and accumulator behavior depend on the selected kernel. Test sensitive layers separately and consider a mixed-precision model if a few quantized layers cause the error.\n\nMeasure each candidate on the same device and inputs. Record model size, peak memory, quality metric, cold and warm latency, and which microkernels executed. Include packing and conversion in end-to-end time. Use a non-quantized baseline and do not extrapolate speedup from another model or a single matrix kernel.\n\n## Resources\n\n- [Arm's KleidiAI quantization explanation](https://learn.arm.com/learning-paths/cross-platform/kleidiai-explainer/page2/) explains kernel format choices.\n- [Arm's LiteRT SME2 supported configurations](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-model/) provides one concrete backend coverage table.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s11-q03", "slug": "profile-existing-model", "question": "How do I profile my existing model's inference performance on Arm and identify which operators would benefit from SME acceleration?", "summary": "First profile the current model to find expensive operators. Then check whether those operators have an SME or SME2 kernel for their exact types, shapes, and quantization scheme. Finally compare the same model with that backend path enabled and disabled on a physical device. Before the conversion, no profiler can directly measure \"time SME would save.\"", "answer": "First profile the **current** model to find expensive operators. Then check whether those operators have an SME or SME2 kernel for their exact types, shapes, and quantization scheme. Finally compare the same model with that backend path enabled and disabled on a physical device. Before the conversion, no profiler can directly measure \"time SME would save.\"\n\n## Get an operator breakdown\n\nUse the profiler built into the runtime: ExecuTorch ETDump, ONNX Runtime JSON profiling, LiteRT's benchmark/operator profile, or the equivalent for your framework. Record per-node latency, input shapes, data types, and thread count for representative inferences. Group related operations such as GEMM, convolution, packing, layout conversion, normalization, and data movement, but keep their **absolute time** visible. Percentages alone can mislead after one stage gets faster.\n\nA matrix-heavy operator is a candidate, but its backend may lack a kernel for the model's format. Check the exact framework version's support table and whether delegation or fusion changes the operator shape. Small matrix-vector and edge-tile cases can favor a different implementation.\n\n## Compare an SME-enabled path\n\nOn Linux/Android, confirm the OS exposes SME2 through `AT_HWCAP2`; on macOS use feature sysctl. Build equivalent enabled and disabled runtime variants where the project supports that comparison. Run identical inputs, model, precision, threads, affinity, and warm-up. Profile operators and end-to-end latency in both runs. Sample native symbols or instructions to confirm the SME kernel actually executed; a packed-node label alone is insufficient.\n\nIf the matrix node improves but inference does not, inspect packing, tensor conversions, memory movement, and other operators. If the expected node does not improve, inspect eligibility, selected fallback, shape, and thermal or frequency effects. Report median latency and variation over repeated runs rather than one speedup number.\n\n## Resources\n\n- [Arm's ExecuTorch profiling workflow](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/01-overview/) shows per-operator comparison.\n- [Arm's ONNX Runtime profiling example](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/performance_onnxruntime_kleidiai_sme2/profiling_example/) shows how to pair operator timing with hardware evidence.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s11-q04", "slug": "workflow-converting-pytorch", "question": "Is there a workflow for converting a PyTorch model to ExecuTorch with KleidiAI SME2 kernels for on-device inference?", "summary": "Yes, for models and operators that ExecuTorch's XNNPACK backend can lower and that its KleidiAI integration covers. The route is PyTorch export, optional quantization, XNNPACK partitioning, a .pte program, and an ExecuTorch runner built for the target. XNNPACK selects a suitable microkernel at runtime; exporting a .pte file alone does not prove SME2 acceleration.", "answer": "Yes, for models and operators that ExecuTorch's XNNPACK backend can lower and that its KleidiAI integration covers. The route is PyTorch export, optional quantization, XNNPACK partitioning, a `.pte` program, and an ExecuTorch runner built for the target. XNNPACK selects a suitable microkernel at runtime; exporting a `.pte` file alone does not prove SME2 acceleration.\n\n## Prepare and export\n\nStart with an evaluated PyTorch model and representative example inputs. Check `torch.export` support for its operators and dynamic shapes, then compare exported output with eager PyTorch. For a minimal supported model, the structure is:\n\n```python\nimport torch\nfrom executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner\nfrom executorch.exir import to_edge_transform_and_lower\n\nmodel = model.eval() # your prepared model\nexample_inputs = (sample_input,) # representative shape and dtype\nexported = torch.export.export(model, example_inputs)\nedge = to_edge_transform_and_lower(\n exported, partitioner=[XnnpackPartitioner()]\n)\nprogram = edge.to_executorch()\nwith open(\"model.pte\", \"wb\") as out:\n out.write(program.buffer)\n```\n\nThis is an **export sketch**: define `model` and `sample_input`, and use APIs from a matching ExecuTorch revision. Real models may need operator replacements or shape constraints before export succeeds. If quantization is needed, follow ExecuTorch's PT2E or current quantization recipe for the model family, calibrate with representative data, and recheck accuracy. Choose an INT4 or INT8 format only after checking the XNNPACK/KleidiAI kernel contract.\n\n## Build and deploy the matching runner\n\nBuild ExecuTorch with XNNPACK and its KleidiAI dependency for the target AArch64 platform. Pin the exporter and runner to the **same ExecuTorch revision**. Arm's [ExecuTorch profiling setup](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/02-setup-and-pipeline/) supplies versioned Android runner steps and SME2-on/off builds; use its current commands rather than assuming a CMake flag from a different release still works.\n\nFor that documented profiling kit, after cloning the [kit repository](https://github.com/ArmDeveloperEcosystem/sme-executorch-profiling) and selecting a compatible ExecuTorch revision, the setup and build steps are:\n\n```sh\nbash model_profiling/scripts/setup_repo.sh\nbash model_profiling/scripts/build_runners.sh\n```\n\nThe script creates Android SME2-on and SME2-off runners under `executorch/cmake-out/` when `ANDROID_NDK` is set, and analogous macOS runners for local development. It enables the XNNPACK/KleidiAI dependency in the on variant and provides a comparable off variant. Build the runners once per ExecuTorch revision, then use the same pair for each exported model. This gives a concrete starting point without embedding a long build command whose flags may drift between releases.\n\nPush the `.pte` model and runner to the device, or package them in an app. Confirm that the OS exposes SME2. Run a correctness comparison against PyTorch or an unaccelerated backend with identical inputs. Enable ETDump/operator tracing in a diagnostic build to find delegated operators, then sample the native kernel path. An ETDump timing entry or XNNPACK `P` label does not alone identify an SME2 instruction.\n\n## Decide whether it helped\n\nCompare SME2-on and SME2-off builds with the same model, quantization, inputs, thread count, and affinity. Measure end-to-end latency and per-operator time, including packing and data movement. Use a quiet timing build; verbose kernel logging changes latency. Unsupported operations may remain in portable ExecuTorch or another XNNPACK path, so report coverage rather than claiming the whole model uses SME2.\n\n## Resources\n\n- [ExecuTorch XNNPACK backend documentation](https://docs.pytorch.org/executorch/stable/backends/xnnpack/xnnpack-arch-internals.html) describes partitioning and supported execution.\n- [Arm's SME2 ExecuTorch profiling workflow](https://learn.arm.com/learning-paths/cross-platform/sme-executorch-profiling/01-overview/) gives a complete deploy-and-compare route.", "tags": [ "SME", "SME2" ], "golden": true }, { "id": "s11-q05", "slug": "model-architecture-choices", "question": "What model architecture choices (layer types, tensor shapes) are most friendly to SME2 acceleration?", "summary": "Favor architectures whose expensive work becomes regular matrix operations with reusable weights, then confirm the target runtime has SME2 kernels for those operators and formats. Dense projections, sufficiently large matrix multiplies, and some convolutions are common opportunities. SME2 hardware alone does not make every Linear, attention, or Conv layer fast.", "answer": "Favor architectures whose expensive work becomes **regular matrix operations with reusable weights**, then confirm the target runtime has SME2 kernels for those operators and formats. Dense projections, sufficiently large matrix multiplies, and some convolutions are common opportunities. SME2 hardware alone does not make every Linear, attention, or Conv layer fast.\n\n## Layer and shape choices\n\n| Choice | Why it can help | Check before redesigning |\n|---|---|---|\n| Wide dense/Linear or MLP projections | Many products reuse packed weights and fill output tiles | Does the backend select an SME2 GEMM or GEMV kernel for the actual batch and sequence length? |\n| Batched MatMul and attention projections | Larger M and N dimensions can amortize setup | Does the runtime delegate this operator and layout? |\n| Pointwise or dense convolution | May lower to a matrix-like kernel | Are channel counts, stride, and quantization supported? |\n| Fused bias/activation | Can avoid extra tensor passes | Does the backend offer the fused variant? |\n\nFor matrix multiplication, M is the number of output rows, N the output width, and K the reduction dimension. Larger regular M and N often use ZA tiles better, but tile dimensions depend on streaming vector length and the chosen microkernel. Avoid hard-coded rules such as \"all dimensions must be multiples of 64 or 128.\" Kernels can handle tails; changing model dimensions may cost accuracy or increase total work. Measure the actual shape distribution, including batch size and single-token decode. Decode can use specialized GEMV kernels even though M=1 does not fill a GEMM tile.\n\nPrefer stable input layouts and constant weights that a runtime can pack once. Remove repeated transposes or conversions when profiling shows they dominate. For CNNs, a channel-last layout can fit some XNNPACK paths, but evaluate the whole graph before changing layout. Depthwise convolutions, softmax, normalization, and gathers may remain on different kernels; their time can limit end-to-end benefit.\n\n## Choose data formats with the backend\n\nINT4 weight paths can reduce bandwidth for large models, but the block size and scales must match a kernel. INT8 may be easier to deploy for a calibrated vision model; FP16 may preserve accuracy with less conversion. SME2's LUTI4 can expand four-bit indices for some kernels, but it does not mean every INT4 scheme is accelerated. Keep accuracy and memory measurements alongside latency.\n\nStart with the existing architecture. Profile it, identify a costly unsupported or poorly tiled operator, and change one design choice at a time. Report accuracy, model size, operator coverage, and end-to-end time on the target device. This avoids optimizing tile alignment at the expense of model quality or total latency.\n\n## Resources\n\n- [Arm's LiteRT SME2 operator coverage](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/litert-sme/2-build-model/) shows concrete supported formats.\n- [Arm's special matrix-shape guidance](https://learn.arm.com/learning-paths/cross-platform/multiplying-matrices-with-sme2/10-going-further/) explains why one tile size does not fit all workloads.", "tags": [ "SME", "SME2" ], "golden": true } ] }