Build an Android chat app with Llama, ExecuTorch, and XNNPACK
Introduction
Create a development environment
ExecuTorch Setup
Understanding Llama models
Prepare Llama models for ExecuTorch
Run Benchmark on Android phone
Build and Run Android chat app
Review
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

Build the Android Archive (AAR)

Note

You can use the Android demo application included in ExecuTorch repository LlamaDemo to demonstrate local inference with ExecuTorch.

  1. Open a terminal window and navigate to the root directory of the executorch repository.

  2. Set the following environment variables:

        

        
        
            export ANDROID_NDK=~/Library/Android/sdk/ndk/25.0.8775105
export ANDROID_ABI=arm64-v8a
Note

<path_to_android_ndk> is the root for the NDK, which is usually under ~/Library/Android/sdk/ndk/XX.Y.ZZZZZ for macOS, and contains NOTICE and README.md. Make sure you can confirm <path_to_android_ndk>/build/cmake/android.toolchain.cmake is available for CMake to cross-compile.

  1. (Optional) If you need to use tiktoken as the tokenizer (for LLaMA 3), set EXECUTORCH_USE_TIKTOKEN=ON and CMake uses it as the tokenizer. If you run other models like LLaMA 2, skip this step.

        

        
        
            export EXECUTORCH_USE_TIKTOKEN=ON # Only for LLaMA3

  2. Run the following commands to set up the required JNI library:

        

        
        
            pushd extension/android
./gradlew build
popd
pushd examples/demo-apps/android/LlamaDemo
./gradlew :app:setup
popd
Note

This is running the shell script setup.sh which configures and builds the required core ExecuTorch, Llama 2, and Android libraries.

Getting models

Make sure the exported model and tokenizer are copied to the Android phone:

Option 1: using adb

  1. Check if the files are available on the phone:
    

        
        
            adb shell "ls -la /data/local/tmp/llama/"
  1. If not, copy them:
    

        
        
            adb shell mkdir -p /data/local/tmp/llama
adb push <model.pte> /data/local/tmp/llama/
adb push <tokenizer.bin> /data/local/tmp/llama/

Option 2: Using Android Studio

  1. Use Android Studio’s device explorer to look for the model files.

Image Alt Text:Device ExplorerFigure 1. Android Studio Device Explorer

  1. Upload the files.

If the files are not on the device, use the device explorer to copy them.

Image Alt Text:Files UploadFigure 2. Android Studio upload files using Device Explorer

Build the Android Package Kit

Option 1: Using Android Studio

This is the recommended option.

  1. Open Android Studio and select “Open an existing Android Studio project” and navigate to open examples/demo-apps/android/LlamaDemo.

  2. Run the app (^R). This builds and launches the app on the phone.

Option 2: Command line

Without Android Studio UI, you can run gradle directly to build the app. You need to set up the Android SDK path and invoke gradle.

    

        
        
            export ANDROID_HOME=<path_to_android_sdk_home>
pushd examples/demo-apps/android/LlamaDemo
./gradlew :app:installDebug
popd

You should now see a running app on your phone that looks like this:

Image Alt Text:First example prompt Image Alt Text:Second example prompt

Back
Next