Deploy a LLM-based Vision Chatbot with PyTorch and Hugging Face Transformers on Google Axion processors
Introduction
Set up an LLM based-Vision Chatbot
Deploy Vision Chatbot LLM backend server
Deploy Vision Chatbot LLM frontend server
Inference with Vision Chatbot
Next Steps
Deploy a LLM-based Vision Chatbot with PyTorch and Hugging Face Transformers on Google Axion processors
Who is this for?
This Learning Path is for software developers and ML engineers who are interested in deploying a production-ready vision chatbot for their application with optimized performance on the Arm Architecture.
What will you learn?
Upon completion of this Learning Path, you will be able to:
- Build a frontend with Streamlit to input images and prompts
- Build the backend to download a Llama 3.2-Vision model, quantize it, and run it using PyTorch and Hugging Face Transformers
- Monitor and analyze inference on Arm CPUs
Prerequisites
Before starting, you will need the following:
- A Google Cloud Axion compute instance or any Arm-based instance from a cloud service provider with at least 32 cores.
- Familiarity with REST APIs and web services.
- A basic understanding of Python and ML concepts.
- A basic understanding of Streamlit.
- A basic understanding of LLM fundamentals.