Understand why LUTI helps low-bit matrix multiplication

What you’ll learn

Large language model (LLM) inference on the CPU is now practical on mobile and edge devices, due in large part to the rise of low-bit AI models.

Low-bit AI models store weights in a compact packed format that must be efficiently expanded before matrix multiplication. You’ll see how 2-bit and 4-bit weights are stored, why conventional unpacking adds cycles, and how lookup-table instructions (LUTIs) remove the unpacking step.

By the end, you should be able to understand the following:

• Why packed low-bit indices are efficient for storage and memory traffic
• How the indices are laid out in memory
• How to use LUTIs to efficiently expand indices

How sub-byte weights are stored

LLM inference on mobile and edge devices is often limited by memory capacity and bandwidth. During inference, model weights must be transferred from memory to the CPU, contributing to latency and energy use.

Quantization reduces this traffic by storing weights in lower-precision formats. Weight-only quantization maps each 32-bit floating-point (fp32) weight to a compact logical code and stores shared metadata, such as a scale or zero point, for each block.

Physical packing is the storage layout that places several low-bit codes into each byte. If you ignore the metadata, four 2-bit codes or two 4-bit codes can be stored in one byte.

Packing low-bit weight codes into a scalable vector register

This approach trades reconstruction accuracy for lower memory use. Its value also depends on decoding the packed codes efficiently. LUTI achieves that by expanding low-bit codes directly into arithmetic-ready vector values.

Understand the LUTI operation

Matrix multiplication kernels don’t usually operate on packed 2-bit or 4-bit codes. Before arithmetic, the codes needs to be decoded into values that the computation can consume.

Conceptually, the operation is:

    

        
        
index = get_lut_index(packed_code);
expanded_value = lookup_table[index];

    

Armv9-A LUTIs perform lookup-table operations that map low-bit indices to expanded values. LUTI2 and LUTI4 operate on 2-bit and 4-bit indices respectively.

• LUTI2 uses each 2-bit index to select one of four lookup-table values.
• LUTI4 uses each 4-bit index to select one of 16 lookup-table values.

The lookup table defines the expanded value that’s associated with each code according to the quantization scheme.

Lookup table for 2-bit codes

For example, a 2-bit lookup table might contain:

From packed 2-bit codes to 8-bit values

The key benefit of LUTI is that the matrix multiplication kernel can load weights in their compact form. A source vector of packed weights therefore carries more values per memory load than a vector containing already expanded 8-bit, 16-bit, or 32-bit values.

For 2-bit codes, LUTI uses the packed indices in a source vector to select lookup-table entries and writes the resulting values to destination vector registers.

The following diagram shows how 2-bit codes expand into 8-bit values:

LUTI maps packed 2-bit codes to lookup-table indices and writes the selected 8-bit values to a destination vector

Identify LUTI responsibilities

Consider the following when using LUTIs:

• The lookup table defines the meaning of each packed code.
• LUTI2 and LUTI4 expand indices. They don’t calculate quantization metadata.
• The following operations remain separate:
  • Scaling
  • Zero-point correction
  • Bias
  • Activation
  • Clamping
  • Requantization
• Expansion happens in the vector path, close to the arithmetic that consumes the values.

What you’ve learned and what’s next

You’ve learned how LUTI uses packed low-bit codes as indices and expands them into values for subsequent arithmetic.

Next, you’ll learn how LUTI operates with ZT0, Z registers, and ZA in SME2.