Run and benchmark BitNet-2B inference on Arm CPUs with Litespark-Inference
Introduction
Understand Litespark-Inference and BitNet b1.58
Run BitNet-2B with Litespark-Inference
Benchmark Litespark-Inference on Arm
Next Steps
Run and benchmark BitNet-2B inference on Arm CPUs with Litespark-Inference
Who is this for?
This is an introductory topic for developers who want to run a large language model (LLM) on a CPU without a GPU using Litespark-Inference.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Run BitNet-2B from the Litespark-Inference CLI or from Python using the high-level BitNet API.
- Pick the right embed dtype (BF16, INT8, or INT4) for the memory versus quality trade-off you want.
- Compare Litespark-Inference with PyTorch Transformers for memory, time to first token, and throughput.
- Measure energy per token on Apple silicon.
Prerequisites
Before starting, you will need the following:
- An Arm Linux machine or macOS machine with Apple Silicon, running Python 3.10 or later. The machine can be anything from a Raspberry Pi 5 to an Arm server.
- Litespark-Inference installed by following the instructions in the Litespark-Inference install guide .
- About 5 GB of free disk for the BitNet-2B model that downloads on first run.
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll run BitNet-2B on Arm CPUs with Litespark-Inference from the command line and Python. You’ll verify the Arm Neon kernel and compare BF16 and INT4 embedding memory use. Then, you’ll benchmark the torchless runtime against PyTorch for memory, time to first token, and throughput. On Apple silicon, you’ll also measure energy per token.
Frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You can add
--max-tokens to litespark-inference generate. For example, use --max-tokens 64 to limit the response to 64 new tokens.Start with BF16 as the reference by adding
--embed-dtype bf16. You can use INT8 or INT4 to reduce memory use, but validate output quality for your workload.Run
python -m litespark_inference.torchless info. On Arm, confirm that the kernel name contains neon and that OpenMP is True. The generation command also identifies the torchless runtime at startup.Add
--pytorch to litespark-benchmark. The command runs the torchless and PyTorch passes, then reports memory use, time to first token, prompt prefill throughput, and token-generation throughput for comparison.Refresh your
sudo credentials with sudo -v, then add --power to litespark-benchmark. If the results report available: false, your system doesn’t expose a supported energy counter.