Explore related resources

Find more information about the topics in this Learning Path:

Give feedback

How would you rate this Learning Path?

Review frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
What do I pass to the `--model_path` argument in the chatbot script?
Use the directory containing genai_config.json and model.onnx for the downloaded INT4-quantized Phi-4-mini model. With the provided download command, pass cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 as the model path.
Which execution provider should I choose when running the chatbot?
If you’re unsure, keep the default execution provider, follow_config, so that the script uses providers defined in the model config. Override only if you need to explicitly set a different provider, as the script clears providers when you do.
How do I know that the chatbot server started correctly?
You should see the model load without errors and a prompt to enter text. If you start the script with --timings or -g, you’ll also see tokens per second and time to first token after sending a prompt.
What Azure VM configuration should I use?
Use the tested configuration as your reference: an Azure Dpls_v6 Cobalt 100 VM with 32 cores, 64 GB of RAM, and 32 GB of disk space.
How can I limit the number of tokens in a chatbot response?
Set --max_length (or -l) when you run phi4.py. This limit includes both the prompt and generated tokens, so it isn’t a limit on the response alone. If you omit the option, the script uses a maximum length of 2048 tokens.

Share

Tell others about this Learning Path.

Return to Servers and Cloud Computing