Deploy Phi-4-mini model with ONNX Runtime on Azure Cobalt 100
Introduction
Demo
Build ONNX Runtime and set up the Phi-4-mini Model
Run the Chatbot Server
Interact with the Phi-4-mini Chatbot
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Advanced
Reading time:  30 min
Last updated:  02 Oct 2026
Skill level:
  Advanced
Reading time:
  30 min
Last updated:
  02 Oct 2026
Author:Nobel Chowdary Mandepudi, Arm
Arm IP:Neoverse
Tags: ML Microsoft Azure Cobalt Linux Python ONNX Runtime
Author:
  • Nobel Chowdary Mandepudi, Arm
Arm IP:
Neoverse
Tags:
ML Microsoft Azure Cobalt Linux Python ONNX Runtime

Who is this for?

This is an advanced topic for developers, ML engineers, and cloud practitioners looking to deploy Microsoft's Phi Models on Arm-based servers using ONNX Runtime.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Quantize and run the Phi-4-mini model with ONNX Runtime on Azure.
  • Analyze performance on Arm Neoverse N2-based Azure VMs powered by Cobalt 100.

Prerequisites

Before starting, you will need the following:

  • A Microsoft Azure Arm-based instance
  • Basic understanding of Python and machine learning concepts
  • Familiarity with ONNX Runtime and Azure cloud services
  • Knowledge of large language model (LLM) fundamentals

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll build ONNX Runtime on an Arm-based Azure server powered by Cobalt 100 and use it to run a Phi-4-mini chatbot. First, you’ll prepare Ubuntu 24.04 LTS, quantize and convert the model, and start the Python chatbot with onnxruntime_genai. You’ll then send text prompts and inspect the terminal metrics, including tokens per second and time to first token, to validate inference and assess basic performance on the Arm CPU.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
What do I pass to the `--model_path` argument in the chatbot script?
Use the directory containing genai_config.json and model.onnx for the downloaded INT4-quantized Phi-4-mini model. With the provided download command, pass cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 as the model path.
Which execution provider should I choose when running the chatbot?
If you’re unsure, keep the default execution provider, follow_config, so that the script uses providers defined in the model config. Override only if you need to explicitly set a different provider, as the script clears providers when you do.
How do I know that the chatbot server started correctly?
You should see the model load without errors and a prompt to enter text. If you start the script with --timings or -g, you’ll also see tokens per second and time to first token after sending a prompt.
What Azure VM configuration should I use?
Use the tested configuration as your reference: an Azure Dpls_v6 Cobalt 100 VM with 32 cores, 64 GB of RAM, and 32 GB of disk space.
How can I limit the number of tokens in a chatbot response?
Set --max_length (or -l) when you run phi4.py. This limit includes both the prompt and generated tokens, so it isn’t a limit on the response alone. If you omit the option, the script uses a maximum length of 2048 tokens.
Next