# Deploy a LLM-based vision chatbot with PyTorch and Hugging Face transformers on Google Axion processors

## In this learning path

- [Introduction](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-vision/)
- [Set up an LLM based-Vision Chatbot](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-vision/vision_chatbot/)
- [Deploy Vision Chatbot LLM backend server](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-vision/backend/)
- [Deploy Vision Chatbot LLM frontend server](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-vision/frontend/)
- [Inference with Vision Chatbot](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-vision/conclusion/)
- [Next Steps](https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama-vision/_next-steps/)

## About this Learning Path

| Skill level:      | Advanced       |
|-------------------|----------------|
| Reading time:     | 45 min         |
| Last updated:     | 30 Sep 2026    |
| Author:           | Nobel Chowdary Mandepudi, Arm |
| Arm IP:           | [Neoverse](https://support.arm.com/?tab=compute-ip&Product%20Type=Infrastructure%20Processors) |
| Tags:             | ML, Google Axion, Linux, Python, PyTorch, Streamlit |

## Who is this for?

This Learning Path is for software developers and ML engineers who are interested in deploying a production-ready vision chatbot for their application with optimized performance on the Arm Architecture.

## What will you learn?

After completing this Learning Path, you'll be able to perform the following tasks:

- Build a frontend with Streamlit to input images and prompts.
- Build the backend to download a Llama 3.2-Vision model, quantize it, and run it using PyTorch and Hugging Face Transformers.
- Monitor and analyze inference on Arm CPUs.

## Prerequisites

Before starting, you'll need the following prerequisites:

- A Google Cloud Axion compute instance or [any Arm-based instance](https://learn.arm.com/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider with at least 32 cores
- Familiarity with REST APIs and web services
- A basic understanding of Python and ML concepts
- A basic understanding of Streamlit
- A basic understanding of LLM fundamentals

## Summary

You’ll build and deploy a vision-enabled chatbot with PyTorch, Transformers, and Streamlit on an Arm-based instance powered by Google Axion. First, you’ll run a Flask backend that downloads and serves a quantized Llama 3.2-Vision model, then create a Streamlit frontend for image uploads and prompts. You’ll configure firewall access, start both services on Ubuntu, and verify image-plus-text responses in the web app.

## Frequently asked questions

### What result should I expect when both the backend and frontend are running?

Open the browser to the app. You’ll see the title **LLM Vision Chatbot on Arm** with controls to upload an image and enter a prompt. After submitting, the page will display a generated text response that uses the image as context.

### Which address should I use to open the web app?

Use `http://[your instance ip]:8501` in your browser. If the page doesn’t load, allow inbound TCP traffic to port `8501` in your instance’s security rules.

### How do I run the backend and frontend at the same time?

Start the backend script in one terminal with the virtual environment activated. Then, open a new terminal, activate the same environment, and start the Streamlit frontend.

### What should I check if the frontend can't reach the backend?

Confirm that `backend.py` is running on port `5000` and that `frontend.py` uses `http://localhost:5000/v1/chat/completions`. For remote browser access, open the Streamlit frontend on port `8501`. The frontend connects to the backend locally.

### How do I know that the model download and 4-bit quantization completed?

Start `backend.py` and wait for the Flask output showing that the server is running on port `5000`. The `backend.py` startup loads the Llama 3.2 Vision model and performs quantization before serving requests.