Compare FP32 and INT8 cutouts

The INT8 artifact is smaller, which brings benefits for storage, memory, and access to optimized integer kernels, but quantization reduces numerical precision, which can reduce accuracy and can change mask boundaries, selected classes, or missing regions. Validate that the smaller INT8 artifact still produces a useful cutout.

Run the same validation path with the INT8 model and the same sample image:

    

        
        
module-3-executorch-on-android-semantic-segmentation/scripts/aot/validate_cutout.py \
  --spec module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/model_spec.json \
  --pte module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/lraspp_mobilenetv3_int8_xnnpack.pte \
  --image module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/sample.jpg \
  --out-dir module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/validation_int8

    

The selected class should remain dog. The reported forward time will vary by development machine:

    

        
        
Selected class: dog (12)
ExecuTorch forward: ... ms

    

The produced mask.png, overlay.png, and cutout.png can be found in module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/validation_int8/.

Measure mask agreement

Visual inspection can miss small differences between the masks. You can use the compare_segmentation_outputs.py script to run both models on the sample image and compare their selected foreground pixels:

    

        
        
module-3-executorch-on-android-semantic-segmentation/scripts/aot/compare_segmentation_outputs.py \
  --spec module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/model_spec.json \
  --fp32-pte module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/lraspp_mobilenetv3_xnnpack.pte \
  --int8-pte module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/lraspp_mobilenetv3_int8_xnnpack.pte \
  --image module-3-executorch-on-android-semantic-segmentation/artifacts/lraspp_mobilenetv3/sample.jpg

    

The output should be similar to:

    

        
        
FP32 selected class: dog (12)
INT8 selected class: dog (12)
Pixel agreement: 98.90%
Foreground IoU: 97.87%
Foreground Dice: 98.92%

    

Pixel agreement is the percentage of image pixels for which both cutout masks make the same foreground or background choice. Intersection over union (IoU) and Dice measure the overlap between their foreground regions. These values measure how much quantization changed the FP32 result; they do not measure accuracy against labeled ground truth.

Compare FP32 and INT8 cutout results

You can now open the mask, cutout, and overlay generated by the INT8 model and visually inspect them, as well as consider the Pixel agreement, IoU, and Dice values. Record your conclusions to the questions below: