Run AI models with Docker Model Runner
Introduction
Run AI models using Docker Model Runner
Run a containerized AI chat app with Docker Compose
Next Steps
Run AI models with Docker Model Runner
Review frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
No. Docker Model Runner uses
llama.cpp, so you don’t need to download, build, or install LLM frameworks.The example supports local AI models such as Llama 3.2 or Gemma 3. Use a model available through Docker Model Runner in your environment.
Docker Compose brings up a Flask-based web frontend and a backend that serves AI responses through Docker Model Runner. You can enter a prompt and receive a generated reply.
Open the chat interface and send a prompt; a response indicates the backend is reachable and the model is active. If there’s no response, check the status of the container and logs in Docker Desktop.
No. The models run locally without cloud dependencies.