Run a local LLM chatbot on a Raspberry Pi 5
Introduction
Run a Large Language Model (LLM) chatbot on a Raspberry Pi 5
Next Steps
Run a local LLM chatbot on a Raspberry Pi 5
Who is this for?
This is an introductory topic for anyone interested in running a local Large Language Model on a Raspberry Pi 5.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Install the Python version of llama.cpp on your Raspberry Pi 5.
- Download an LLM from Hugging Face.
- Assess LLM memory size and performance.
- Run the LLM on your Raspberry Pi 5 using Python bindings for llama.cpp.
Prerequisites
Before starting, you will need the following:
- A Raspberry Pi 5 running Raspberry Pi OS.
Summary
This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
You’ll run a local chatbot on a Raspberry Pi 5 with 64-bit Raspberry Pi OS. First, you’ll install the Python bindings for
llama.cpp, choose a model that fits available memory, and download it from Hugging Face. Then, you’ll load the model, generate responses, and verify interactive inference directly on the Arm Linux device.Frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
The 8 GB RAM Raspberry Pi 5 model is preferred for exploring an LLM. Smaller memory configurations might restrict the model that you can run.
Check your system information to confirm the operating system and kernel report a 64-bit build. If not, reinstall Raspberry Pi OS using the 64-bit image.
Yes. The instructions apply to other Arm Linux systems with minor modifications where Raspberry Pi–specific steps appear.
Compare the model’s size to your available memory and leave headroom for the operating system and Python runtime. Select a model that fits comfortably within your RAM, then assess performance.
A Python script loads the chosen model and generates text responses in your terminal. If it starts and responds to prompts, your setup is working.