# Deploy Phi-4-mini model with ONNX Runtime on Azure Cobalt 100

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/)
- [Demo](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/_demo/)
- [Build ONNX Runtime and set up the Phi-4-mini Model](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/setup/)
- [Run the Chatbot Server](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/chatbot/)
- [Interact with the Phi-4-mini Chatbot](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/analysis/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/_next-steps/)

## About this Learning Path

| Skill level:       | Advanced           |
|---------------------|-------------------|
| Reading time:       | 30 min            |
| Last updated:       | 02 Oct 2026       |

| Author:             | Nobel Chowdary Mandepudi, Arm |
|---------------------|-------------------------------|
| Arm IP:             | [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors)         |
| Tags:               | ML, Microsoft Azure Cobalt, Linux, Python, ONNX Runtime        |

## Who is this for?
This is an advanced topic for developers, ML engineers, and cloud practitioners looking to deploy Microsoft's Phi Models on Arm-based servers using ONNX Runtime.

## What will you learn?
Upon completion of this Learning Path, you will be able to:
- Quantize and run the Phi-4-mini model with ONNX Runtime on Azure.
- Analyze performance on Arm Neoverse N2-based Azure VMs powered by Cobalt 100.

## Prerequisites
Before starting, you will need the following:
- A Microsoft Azure [Arm-based instance](https://learn.arm.com/learning-paths/servers-and-cloud-computing/csp/)
- Basic understanding of Python and machine learning concepts
- Familiarity with ONNX Runtime and Azure cloud services
- Knowledge of large language model (LLM) fundamentals

## Summary
You’ll build ONNX Runtime on an Arm-based Azure server powered by Cobalt 100 and use it to run a Phi-4-mini chatbot. First, you’ll prepare Ubuntu 24.04 LTS, quantize and convert the model, and start the Python chatbot with `onnxruntime_genai`. You’ll then send text prompts and inspect the terminal metrics, including tokens per second and time to first token, to validate inference and assess basic performance on the Arm CPU.

## Frequently asked questions

<details>
<summary>What do I pass to the `--model_path` argument in the chatbot script?</summary>
Use the directory containing `genai_config.json` and `model.onnx` for the downloaded INT4-quantized Phi-4-mini model. With the provided download command, pass `cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4` as the model path.
</details>

<details>
<summary>Which execution provider should I choose when running the chatbot?</summary>
If you’re unsure, keep the default execution provider, `follow_config`, so that the script uses providers defined in the model config. Override only if you need to explicitly set a different provider, as the script clears providers when you do.
</details>

<details>
<summary>How do I know that the chatbot server started correctly?</summary>
You should see the model load without errors and a prompt to enter text. If you start the script with `--timings` or `-g`, you’ll also see tokens per second and time to first token after sending a prompt.
</details>

<details>
<summary>What Azure VM configuration should I use?</summary>
Use the tested configuration as your reference: an Azure `Dpls_v6` Cobalt 100 VM with 32 cores, 64 GB of RAM, and 32 GB of disk space.
</details>

<details>
<summary>How can I limit the number of tokens in a chatbot response?</summary>
Set `--max_length` (or `-l`) when you run `phi4.py`. This limit includes both the prompt and generated tokens, so it isn’t a limit on the response alone. If you omit the option, the script uses a maximum length of 2048 tokens.
</details>