# Run AI models using Docker Model Runner

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/laptops-and-desktops/docker-models/)
- [Run AI models using Docker Model Runner](https://learn.arm.com/learning-paths/laptops-and-desktops/docker-models/models/)
- [Run a containerized AI chat app with Docker Compose](https://learn.arm.com/learning-paths/laptops-and-desktops/docker-models/compose/)
- [Next Steps](https://learn.arm.com/learning-paths/laptops-and-desktops/docker-models/_next-steps/)

## Simplified Local LLM Inference
Docker Model Runner is an official Docker extension that allows you to run Large Language Models (LLMs) directly on your local computer. It provides a convenient way to deploy and use AI models across different environments, including Arm-based systems, without complex framework setup or cloud dependencies.

Docker uses [llama.cpp](https://github.com/ggml-org/llama.cpp), an open source C/C++ project developed by Georgi Gerganov that enables efficient LLM inference on a variety of hardware, but you do not need to download, build, or install any LLM frameworks.

Docker Model Runner provides an easy-to-use CLI interface that is familiar to Docker users.

## Before you begin
Verify Docker is running with:
```
docker version
```
You should see your Docker version shown in the output.

Confirm that Docker Desktop is version 4.40 or above, for example:
```
__output__ Server: Docker Desktop 4.41.2 (191736)
```
Make sure the Docker Model Runner is enabled:
```
docker model --help
```
You should see this output:
```
__output__ Usage:  docker model COMMAND
__output__ 
__output__ Docker Model Runner
__output__ 
__output__ Commands:
__output__  inspect     Display detailed information on one model
__output__  list        List the available models that can be run with the Docker Model Runner
__output__  logs        Fetch the Docker Model Runner logs
__output__  pull        Download a model
__output__  push        Upload a model
__output__  rm          Remove models downloaded from Docker Hub
__output__  run         Run a model with the Docker Model Runner
__output__  status      Check if the Docker Model Runner is running
__output__  tag         Tag a model
__output__  version     Show the Docker Model Runner version
```
If Docker Model Runner is not enabled, enable it by following the [Docker Model Runner documentation](https://docs.docker.com/model-runner/).

You should also see the **Models** tab and icon appear in your Docker Desktop sidebar.

![Docker Models UI](https://via.placeholder.com/150 "Docker Models UI")

## Run your first AI model with Docker Model Runner
Docker Model Runner is an extension for Docker Desktop that simplifies running AI models locally.

Docker Model Runner automatically selects compatible model versions and optimizes performance for the Arm architecture.

You can try Model Runner by downloading and running a model from Docker Hub.

The example below uses the [SmolLM2 model](https://hub.docker.com/r/ai/smollm2), a compact LLM with ~360 million parameters, designed for efficient on-device inference while performing a wide range of language tasks. You can explore further models in [Docker Hub](https://hub.docker.com/u/ai).

1. Download the model
```
docker model pull ai/smollm2
```
2. Run the model interactively
For a simple chat interface, run the model:
```
docker model run ai/smollm2
```
Enter a prompt at the CLI:
```
write a simple hello world program in C++
```
You see the output from the SmolLM2 model:
```
__output__ #include <iostream>

__output__ int main() {
    std::cout << "Hello, World!" << std::endl;
    return 0;
}
```
To exit the chat, use the `"/bye"` command.

3. View downloaded models
You can print the list of models on your computer using:
```
docker model list
```
Your list will be different based on the models you have downloaded.
```
__output__ MODEL NAME   PARAMETERS  QUANTIZATION    ARCHITECTURE  MODEL ID      CREATED       SIZE
__output__ ai/gemma3    3.88 B      IQ2_XXS/Q4_K_M  gemma3        0b329b335467  2 months ago  2.31 GiB
__output__ ai/phi4      14.66 B     IQ2_XXS/Q4_K_M  phi3          03c0bc8e0f5a  2 months ago  8.43 GiB
__output__ ai/smollm2   361.82 M    IQ2_XXS/Q4_K_M  llama         354bf30d0aa3  2 months ago  256.35 MiB
__output__ ai/llama3.2  3.21 B      IQ2_XXS/Q4_K_M  llama         436bb282b419  2 months ago  1.87 GiB
```

## Use the OpenAI endpoint to call the model
Docker Model Runner exposes a REST endpoint compatible with OpenAI’s API spec.

From your host computer, you can access the model using the OpenAI endpoint and a TCP port.

First, enable the TCP port to connect with the model:
```
docker desktop enable model-runner --tcp 12434
```
Next, use a text editor to save the code below in a file named `curl-test.sh`:
```sh
#!/bin/sh

curl http://localhost:12434/engines/llama.cpp/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Please write a hello world program in Java."
      }
    ]
  }'
```
Run the shell script:
```
bash ./curl-test.sh | jq
```
If you don’t have `jq` installed, you can eliminate piping the output.

The output, including the performance information, is shown below:
```
__output__ {
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Here's a simple \"Hello World\" program in Java:\n\n```java\npublic class HelloWorld {\n    public static void main(String[] args) {\n        System.out.println(\"Hello, World!\");\n    }\n}\n```\n\nThis program declares a `HelloWorld` class, defines a `main` method that contains the program's execution, and then uses `System.out.println` to print \"Hello, World!\" to the console."
      }
    }
  ],
  "created": 1748622685,
  "model": "ai/smollm2",
  "system_fingerprint": "b1-a0f7016",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 101,
    "prompt_tokens": 28,
    "total_tokens": 129
  },
  "id": "chatcmpl-uZGBuFoS2ERodT4KilStxDwhySLQBTN9",
  "timings": {
    "prompt_n": 28,
    "prompt_ms": 32.349,
    "prompt_per_token_ms": 1.1553214285714284,
    "prompt_per_second": 865.5599863983431,
    "predicted_n": 101,
    "predicted_ms": 469.524,
    "predicted_per_token_ms": 4.648752475247525,
    "predicted_per_second": 215.11147459980745
  }
}
```
You now have a fully functioning OpenAI-compatible inference endpoint running locally.

## What you’ve learned
In this section, you learned:
- How to verify and use Docker Model Runner on Docker Desktop
- How to run a model interactively from the CLI
- How to connect to a model using a local OpenAI-compatible API

In the next section, you’ll use Docker Compose to deploy a web-based AI chat interface powered by Docker Model Runner.