Deploy Phi-4-mini model with ONNX Runtime on Azure Cobalt 100
Introduction
Demo
Build ONNX Runtime and set up the Phi-4-mini Model
Run the Chatbot Server
Interact with the Phi-4-mini Chatbot
Next Steps
Deploy Phi-4-mini model with ONNX Runtime on Azure Cobalt 100
Review frequently asked questions
These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.
Use the directory containing
genai_config.json and model.onnx for the downloaded INT4-quantized Phi-4-mini model. With the provided download command, pass cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 as the model path.If you’re unsure, keep the default execution provider,
follow_config, so that the script uses providers defined in the model config. Override only if you need to explicitly set a different provider, as the script clears providers when you do.You should see the model load without errors and a prompt to enter text. If you start the script with
--timings or -g, you’ll also see tokens per second and time to first token after sending a prompt.Use the tested configuration as your reference: an Azure
Dpls_v6 Cobalt 100 VM with 32 cores, 64 GB of RAM, and 32 GB of disk space.Set
--max_length (or -l) when you run phi4.py. This limit includes both the prompt and generated tokens, so it isn’t a limit on the response alone. If you omit the option, the script uses a maximum length of 2048 tokens.