Convert the FP32 workflow into an INT8 deployment path

The FP32 ExecuTorch artifact improves latency, but quantization can reduce it further.

Quantization happens as part of the ahead-of-time workflow, before lowering and delegation. This module uses a direct PyTorch 2 Export (PT2E) quantization path: post-training quantization with observers, a calibration dataset, conversion, and XNNPACK lowering. torchao is a PyTorch library that provides model-optimization tools, including quantization, and this module uses its PT2E prepare_pt2e() and convert_pt2e() APIs.

Understand post-training quantization

Post-training quantization (PTQ) starts with a trained FP32 model. Preparation adds observers to the graph. A calibration pass then runs example images through the prepared model so the observers can record activation ranges. Conversion uses those ranges to select parameters such as scales and zero-points and replaces supported FP32 computation with quantized forms.

This module uses static W8A8 quantization: supported quantized operations use 8-bit weights and activations, with activation scales fixed during inference. The final classifier head remains FP32 by default, as explained on the next page.

This module uses XNNPACKQuantizer, so the quantization configuration is designed for the backend that will run the model. This is preferable to applying a generic INT8 transformation and expecting the backend to accept every resulting pattern. The converted graph must still contain forms that XNNPACK can lower and execute efficiently.

The three expected benefits are:

• a smaller model artifact
• lower memory bandwidth requirements
• lower latency where optimized integer kernels are available

The cost is numerical approximation. You will evaluate whether that cost significantly impairs accuracy, or whether the model still functions correctly for the intended use-case.

The complete preparation path is:

    

        
        
FP32 PyTorch model
  -> torch.export
  -> prepare_pt2e with XNNPACKQuantizer
  -> calibration images
  -> convert_pt2e
  -> torch.export quantized model
  -> lower to ExecuTorch with XNNPACK
  -> INT8 .pte artifact