# Deploy Phi-4-mini model with ONNX Runtime on Azure Cobalt 100

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/)
- [Demo](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/_demo/)
- [Build ONNX Runtime and set up the Phi-4-mini Model](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/setup/)
- [Run the Chatbot Server](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/chatbot/)
- [Interact with the Phi-4-mini Chatbot](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/analysis/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/onnx/_next-steps/)

## About this Learning Path

| Skill level: | Advanced |
|--------------|----------|
| Reading time:| 30 min   |
| Last updated:| 02 Oct 2026 |

| Author:                                  | Nobel Chowdary Mandepudi, Arm          |
|------------------------------------------|----------------------------------------|
| Arm IP:                                  | [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors) |
| Tags:                                    | ML, Microsoft Azure Cobalt, Linux, Python, ONNX Runtime |

## Who is this for?
This is an advanced topic for developers, ML engineers, and cloud practitioners looking to deploy Microsoft's Phi Models on Arm-based servers using ONNX Runtime.

## What will you learn?
After completing this Learning Path, you'll be able to perform the following tasks:
- Quantize and run the Phi-4-mini model with ONNX Runtime on Azure.
- Analyze performance on Arm Neoverse N2-based Azure VMs powered by Cobalt 100.

## Prerequisites
Before starting, you'll need the following prerequisites:
- A Microsoft Azure [Arm-based instance](https://learn.arm.com/learning-paths/servers-and-cloud-computing/csp/)
- Basic understanding of Python and machine learning concepts
- Familiarity with ONNX Runtime and Azure cloud services
- Knowledge of large language model (LLM) fundamentals

## Summary
You’ll build ONNX Runtime on an Arm-based Azure server powered by Cobalt 100 and use it to run a Phi-4-mini chatbot. First, you’ll prepare Ubuntu 24.04 LTS, quantize and convert the model, and start the Python chatbot with `onnxruntime_genai`. You’ll then send text prompts and inspect the terminal metrics, including tokens per second and time to first token, to validate inference and assess basic performance on the Arm CPU.

## Frequently asked questions

### What do I pass to the `--model_path` argument in the chatbot script?
Use the directory containing `genai_config.json` and `model.onnx` for the downloaded INT4-quantized Phi-4-mini model. With the provided download command, pass `cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4` as the model path.

### Which execution provider should I choose when running the chatbot?
If you’re unsure, keep the default execution provider, `follow_config`, so that the script uses providers defined in the model config. Override only if you need to explicitly set a different provider, as the script clears providers when you do.

### How do I know that the chatbot server started correctly?
You should see the model load without errors and a prompt to enter text. If you start the script with `--timings` or `-g`, you’ll also see tokens per second and time to first token after sending a prompt.

### What Azure VM configuration should I use?
Use the tested configuration as your reference: an Azure `Dpls_v6` Cobalt 100 VM with 32 cores, 64 GB of RAM, and 32 GB of disk space.

### How can I limit the number of tokens in a chatbot response?
Set `--max_length` (or `-l`) when you run `phi4.py`. This limit includes both the prompt and generated tokens, so it isn’t a limit on the response alone. If you omit the option, the script uses a maximum length of 2048 tokens.