Measure on target hardware

This step measures PyTorch eager and FP32 ExecuTorch with XNNPACK on the same Raspberry Pi using the same weights, image, preprocessing, warm-up count, and run count.

Module 1 used two warm-up runs and 20 measured runs for its slower transformer paths. MobileNetV2 completes each forward pass much more quickly, so ten warm-up runs and 50 measured runs remain practical and provide a larger latency sample.

Prepare the Raspberry Pi

If you connect over SSH, forward the benchmark dashboard and web application ports:

    

        
        
ssh -L 8080:localhost:8080 -L 8092:localhost:8092 <user>@<pi>

    

Use the Pi environment created by the Module 1 Raspberry Pi setup and install the Module 2 requirements. If .cpu_lab_venv does not exist, follow that setup page through creating and activating the environment, then return here. You do not need to complete the rest of Module 1.

    

        
        
cd ~/executorch_on_arm
source .cpu_lab_venv/bin/activate
python -m pip install -r module-2-cnn-inference-with-quantization/requirements.txt

    

Keep this SSH session open. The remaining Raspberry Pi commands assume that .cpu_lab_venv is active.

On the development machine, copy the MobileNetV2 artifacts to the Pi. Substitute the username and IP address for your Raspberry Pi:

    

        
        
ssh <user>@<pi> "mkdir -p ~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts"
scp -r module-2-cnn-inference-with-quantization/artifacts/mobilenetv2 \
  <user>@<pi>:~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts/

    

Run the FP32 PyTorch benchmark

    

        
        
cd ~/executorch_on_arm

module-2-cnn-inference-with-quantization/scripts/pi/benchmark_mobilenetv2.py \
  --runtime pytorch \
  --spec ~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts/mobilenetv2/model_spec.json \
  --image ~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts/mobilenetv2/sample.jpg \
  --runs 50 \
  --warmups 10 \
  --out ~/executorch_on_arm/module-2-cnn-inference-with-quantization/results/mobilenetv2_pytorch.json

    

This run records the classification output and latency for the full PyTorch stack.

Run FP32 ExecuTorch with XNNPACK

The Python script drives the benchmark. When it calls the ExecuTorch model, the .pte program executes through the native C++ ExecuTorch runtime and the XNNPACK backend.

    

        
        
module-2-cnn-inference-with-quantization/scripts/pi/benchmark_mobilenetv2.py \
  --runtime executorch \
  --backend xnnpack \
  --spec ~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts/mobilenetv2/model_spec.json \
  --pte ~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts/mobilenetv2/mobilenetv2_xnnpack.pte \
  --image ~/executorch_on_arm/module-2-cnn-inference-with-quantization/artifacts/mobilenetv2/sample.jpg \
  --runs 50 \
  --warmups 10 \
  --out ~/executorch_on_arm/module-2-cnn-inference-with-quantization/results/mobilenetv2_xnnpack.json

    

The script prepares the input before selecting the runtime, then passes the selected forward function to the same benchmark helper:

    

        
        
spec = json.loads(args.spec.read_text())
input_tensor = preprocess_image(image, spec)

if args.runtime == "pytorch":
    module = load_pytorch_model(args.spec, spec)

    def forward():
        return module(input_tensor)
else:
    module = load_executorch_program(args.pte)

    def forward():
        return module.forward((input_tensor,))[0]

times_ms, stats = benchmark(
    forward,
    warmups=args.warmups,
    runs=args.runs,
)

    

Both commands pass the same model specification, image, warm-up count, and run count. The image is preprocessed before the runtime branch, so both branches receive the same tensor. The PyTorch model and .pte artifact were created from the same saved weights. Only the model loader and forward() call change.