GPT-2 benchmark on Raspberry Pi

You will run three benchmark tests: PyTorch eager, portable ExecuTorch, and ExecuTorch with XNNPACK. Within each test, a measured run is one timed call to the model. The PyTorch and XNNPACK tests use repeated measured runs so that one unusually fast or slow call does not dominate the result. The portable test uses one warmup pass and one measured run because that path can be slow and is included as a deployable baseline.

All three tests use FP32 to ensure a fair comparison. A typical ExecuTorch deployment would, however, use quantization such as INT8. This is covered in Module 2.

The benchmark_model.py script is used for all three tests. It reads the model_spec.json, prepares the correct inputs, builds a forward() function for the selected runtime, measures that function, then writes the result files. The same script will be reused later for the DistilBERT model.

The first test is the PyTorch eager execution. For GPT-2, the script loads the Hugging Face model, applies the same LastTokenLogits wrapper used during export, and uses the example prompt from the spec:

    

        
        
module, inputs = prepare_model(spec)

def forward() -> object:
    with torch.inference_mode():
        return module(*inputs)

    

The ExecuTorch tests are different. They load a .pte artifact with the ExecuTorch Python bindings, create the same fixed-shape input tensors, and call the model through the ExecuTorch runtime:

    

        
        
model = load_executorch_program(args.pte)
inputs = example_inputs(spec)

def forward() -> object:
    with torch.inference_mode():
        return model.forward(inputs)

    

The timing loop performs warmup passes first, then records each measured run in milliseconds:

    

        
        
for _ in range(warmups):
    fn()

times = np.empty(runs, dtype=np.float64)
for i in range(runs):
    start = time.perf_counter()
    fn()
    times[i] = (time.perf_counter() - start) * 1000.0

    

The PyTorch and XNNPACK tests each use two warmup passes, giving the runtime a chance to pay one-time costs before timing starts. These costs can include loading code paths, allocating internal buffers, and filling CPU caches. Each test then records 20 measured runs, giving you a small latency distribution for comparing the mean, median, and spread.

For each test, the script writes one summary JSON file and one CSV file with a row for each measured run. For the PyTorch and XNNPACK tests, it also writes two plots: a latency histogram and a run-by-run time series. The JSON summaries provide the mean, median, minimum, maximum, and percentile latency used later in the results dashboard.

Run on the Raspberry Pi from the repo root:

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/benchmark_model.py \
  --runtime pytorch \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/gpt2/model_spec.json \
  --runs 20 \
  --warmups 2 \
  --out ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/gpt2_pytorch.json

    

This completes the PyTorch eager test for GPT-2. It writes the JSON summary and 20-row CSV file to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results, and the two plots to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/plots.

Next, you will test the portable kernel ExecuTorch path. Do not be surprised if the portable result is slower than PyTorch eager, which can often dispatch large matrix operations to highly optimized CPU kernels. The portable path uses the Portable Kernel Library, which prioritizes broad operator coverage and correctness. It does not delegate supported subgraphs to XNNPACK.

The portable test uses --warmups 1 --runs 1 to avoid a long wait. If you want to measure a fuller portable latency distribution, increase --runs, but expect the test to take much longer to complete.

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/benchmark_model.py \
  --runtime executorch \
  --backend portable \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/gpt2/model_spec.json \
  --pte ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/gpt2/gpt2_portable.pte \
  --runs 1 \
  --warmups 1 \
  --out ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/gpt2_portable.json

    

This completes the portable ExecuTorch test with one measured run. It writes a JSON summary and a one-row CSV file to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results. The dashboard table includes this result, but it does not show a histogram or time series for a single measurement.

The increased latency does not make the portable path useless. It gives you a deployable baseline and a compact .pte artifact. On smaller targets, deployability and runtime footprint may matter as much as raw latency.

Execute the XNNPACK test, using two warmup passes and 20 measured runs:

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/benchmark_model.py \
  --runtime executorch \
  --backend xnnpack \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/gpt2/model_spec.json \
  --pte ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/gpt2/gpt2_xnnpack.pte \
  --runs 20 \
  --warmups 2 \
  --out ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/gpt2_xnnpack.json

    

This completes the XNNPACK test using the GPT-2 .pte artifact. It writes the JSON summary and 20-row CSV file to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results, and the two plots to ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/results/plots.