Run a PyTorch LLM chatbot with KleidiAI on Arm
Introduction
Run a Large Language model (LLM) chatbot on Arm servers
Chatbot with Streamlit Frontend
Next Steps
Run a PyTorch LLM chatbot with KleidiAI on Arm
Review frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
Use an Arm server with Ubuntu 24.04 LTS, at least 16 cores, 64 GB of memory, and 50 GB of storage. You can use the tested AWS Graviton4
r8g.4xlarge setup as reference.Download Meta Llama 3.1 from its Meta repository on Hugging Face.
Run
source torch_env/bin/activate before you install the frontend dependencies.Install
httpx==0.27.2, then restart the frontend.Start the Torchchat backend and Streamlit frontend. Open the web interface, enter a prompt, and check that you receive a model response.