DistilBERT benchmark on Raspberry Pi

Run the following on the Raspberry Pi from the repo root:

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/benchmark_model.py \
  --runtime pytorch \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/model_spec.json \
  --runs 20 \
  --warmups 2 \
  --out ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/distilbert_pytorch.json

    

This benchmarks full PyTorch eager execution for DistilBERT, writes JSON and CSV results to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results, and plots to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/plots.

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/benchmark_model.py \
  --runtime executorch \
  --backend portable \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/model_spec.json \
  --pte ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/distilbert_fillmask_portable.pte \
  --runs 1 \
  --warmups 1 \
  --out ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/distilbert_portable.json

    

This will run the portable ExecuTorch DistilBERT path once. As with GPT-2, this will confirm the portable path’s functional execution without spending a long time on repeated portable runs.

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/benchmark_model.py \
  --runtime executorch \
  --backend xnnpack \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/model_spec.json \
  --pte ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/distilbert_fillmask_xnnpack.pte \
  --runs 20 \
  --warmups 2 \
  --out ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/distilbert_xnnpack.json

    

This benchmarks the DistilBERT ExecuTorch with the XNNPACK .pte artifact, writes JSON and CSV results to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results, and plots to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/plots.