Run AI models with Docker Model Runner
Introduction
Run AI models using Docker Model Runner
Run a containerized AI chat app with Docker Compose
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Introductory
Reading time:  45 min
Last updated:  29 Jul 2026
Skill level:
  Introductory
Reading time:
  45 min
Last updated:
  29 Jul 2026
Author:Jason Andrews, Arm
Arm IP:Neoverse Cortex-A
Tags: Containers and Virtualization Windows macOS Docker Python LLM
Author:
  • Jason Andrews, Arm
Arm IP:
Neoverse Cortex-A
Tags:
Containers and Virtualization Windows macOS Docker Python LLM

Who is this for?

This is for software developers and AI enthusiasts who want to run pre-trained AI models locally using Docker Model Runner.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Run AI models locally using Docker Model Runner.
  • Build containerized applications that integrate Large Language Models (LLMs).

Prerequisites

Before starting, you will need the following:

  • Docker Desktop (version 4.40 or later) installed on a system with at least 16GB of RAM (recommended).
  • Basic understanding of Docker CLI and concepts.
  • Familiarity with LLM concepts.

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll run pretrained LLMs locally with Docker Model Runner, then compose a Flask chat application around them. You’ll use the Model Runner extension with llama.cpp, clone the example repository, start the frontend and backend with Docker Compose, choose a supported model such as Llama 3.2 or Gemma 3, and verify the chat UI and container status.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
Do I need to install or build any ML frameworks to run a model with Docker Model Runner?
No. Docker Model Runner uses llama.cpp, so you don’t need to download, build, or install LLM frameworks.
Which model should I choose for the example chat application?
The example supports local AI models such as Llama 3.2 or Gemma 3. Use a model available through Docker Model Runner in your environment.
What should I expect after starting the Docker Compose project?
Docker Compose brings up a Flask-based web frontend and a backend that serves AI responses through Docker Model Runner. You can enter a prompt and receive a generated reply.
How can I confirm that the model and services are running correctly?
Open the chat interface and send a prompt; a response indicates the backend is reachable and the model is active. If there’s no response, check the status of the container and logs in Docker Desktop.
Does local inference with Docker Model Runner require a cloud service?
No. The models run locally without cloud dependencies.
Next