Litespark-Inference

Litespark-Inference is an open-source CPU inference runtime for BitNet b1.58 ternary-weight Large Language Models (LLMs).

A single pip install reads your CPU’s feature flags and compiles the right C++ kernel for it using Neon and SDOT on Arm. This saves you from needing to manually pick the right C++ kernel for best performance.

You can install Litespark-Inference on Linux and macOS.

Before you begin

Before installing Litespark-Inference, make sure your local machine has the following:

• Python 3.10 or later
• A C++ toolchain such as clang or g++
• About 5 GB of free disk

To check the Python version on your machine, run python3 --version.

The first run downloads the BitNet-2B model from Hugging Face.

Install Litespark-Inference into a clean virtual environment so the install doesn’t conflict with anything else on your machine.

If you don’t have Python virtual environment support installed, run:

    

        
        
sudo apt update
sudo apt install python3-venv -y

    

Create a Python virtual environment:

    

        
        
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools

    

Install Litespark-Inference on Arm Linux

The released package builds the correct kernel for your CPU automatically. It uses Neon and SDOT on Arm.

Supported CPUs include:

• AWS Graviton 2, 3, or 4
• Ampere family processors
• Neoverse N1, N2, V1, or V2
• Raspberry Pi 5

For Ubuntu or Debian distributions, install the C++ toolchain, then the Litespark-Inference Python package:

    

        
        
sudo apt-get update
sudo apt-get install -y build-essential clang ninja-build git python3-pip
pip install litespark-inference

    

For Red Hat, Fedora, or RHEL, install the toolchain and package with dnf instead:

    

        
        
sudo dnf install -y gcc-c++ clang ninja-build git python3-pip
pip install litespark-inference

    

Confirm the package imports and list the installed version of Litespark-Inference:

    

        
        
python3 -c "import litespark_inference; print(litespark_inference.__version__)"

    

The output is similar to:

    

        
        1.0.3

        
    

Inspect which kernel was built for your CPU:

    

        
        
python -m litespark_inference.torchless info

    

The output ends with one of the following, depending on your CPU:

    

        
        litespark_inference.torchless
  platform : Linux aarch64
  python   : 3.12.3
  kernel   : /home/ubuntu/.venv/lib/python3.12/site-packages/litespark_inference/torchless/_matmul_lut_neon.cpython-312-aarch64-linux-gnu.so
  OpenMP   : True  (max_threads=8)
  Accelerate: False

        
    

Install Litespark-Inference on Apple silicon macOS

Apple’s CPUs have Neon SDOT, and Litespark-Inference uses it directly.

Litespark uses OpenMP for multi-threading inside the kernel. However, Apple’s toolchain doesn’t ship a built-in OpenMP runtime. To address this, you’ll need to install libomp.

Install Xcode command-line tools, libomp, and the Litespark-Inference Python package:

    

        
        
xcode-select --install
brew install libomp
pip install litespark-inference

    

Verify that the installation was successful:

    

        
        
python -m litespark_inference.torchless info

    

The output is similar to:

    

        
        litespark_inference.torchless
  platform : Darwin arm64
  python   : 3.14.5
  kernel   : .venv/lib/python3.14/site-packages/litespark_inference/torchless/_matmul_lut_neon.cpython-314-darwin.so
  OpenMP   : True  (max_threads=12)
  Accelerate: True

        
    

Troubleshoot missing OpenMP support

If you see OpenMP : False, the build didn’t find Homebrew’s libomp. The most common cause is that Homebrew is installed under /opt/homebrew, which is the Apple silicon default, but pip install ran in an environment that hides the directory.

To fix this, re-run pip install litespark-inference from a normal shell.

(Optional) Install Litespark-Inference from source

To modify the runtime or kernels, install Litespark-Inference from source rather than from PyPI:

    

        
        
git clone https://github.com/Mindbeam-AI/Litespark-Inference.git
cd Litespark-Inference
pip install -e .

    

Verify the Litespark-Inference installation

To verify that Litespark-Inference works as expected, run the following command:

    

        
        
litespark-inference generate "Hello, world!" --max-tokens 16

    

The same command works on both Linux and macOS.

The output is similar to:

    

        
        Prompt (36 tokens): 'System: You are Litespark, a helpful AI assistant running locally. Provide accurate, concise, and practical answers.<|eot_id|>User: Hello, world!<|eot_id|>Assistant: '
Prefill...
  36 tokens in 0.40s (90.09 tok/s)
Generate:
Hello! How can I assist you today?

Generated 9 tokens in 0.29s (31.58 tok/s)

--- output ---
Hello! How can I assist you today?

        
    

Next steps

You’re now ready to run BitNet-2B.

To learn more, see the Run and benchmark BitNet-2B inference on Arm CPUs with Litespark-Inference Learning Path.