GPT-2 lowering

The preparation step produced an exported PyTorch program. Lowering converts that graph into the representation used by ExecuTorch and, when selected, prepares supported graph regions for XNNPACK. The script then packages the resulting program as a deployable .pte artifact.

The lower_to_executorch.py script reads model_spec.json, finds the exported .pt2 program, lowers it through ExecuTorch, writes the .pte, and saves a small JSON metadata file containing the output path, file size, backend name, and delegation summary.

You will run the script twice:

• once with --backend portable, which creates a generic ExecuTorch baseline without backend delegation

• once with --backend xnnpack, which asks ExecuTorch to partition supported parts of the graph for the XNNPACK CPU backend

The core lowering logic looks like this:

    

        
        
exported_program = load(exported_path)
program, delegation = lower(exported_program, backend=args.backend)
write_pte(program, out)

metadata = {
    "model": spec["name"],
    "model_id": spec["model_id"],
    "backend": args.backend,
    "exported_program": str(exported_path),
    "pte": str(out),
    "pte_size_mb": out.stat().st_size / 1_000_000.0,
    "seq_len": spec["seq_len"],
    "delegation": delegation,
}
metadata_path = out.with_suffix(".json")
metadata_path.write_text(json.dumps(metadata, indent=2) + "\n")

    

Inside the lower() helper, the backend choice controls whether ExecuTorch uses no partitioner or the XNNPACK partitioner:

    

        
        
if backend == "portable":
    edge = to_edge_transform_and_lower(exported_program)
    return edge.to_executorch(), None

edge = to_edge_transform_and_lower(
    exported_program,
    partitioner=[XnnpackPartitioner()],
)
return edge.to_executorch(), inspect_delegation(edge)

    

The script builds the delegation summary by walking the lowered graph, counting calls to the ExecuTorch delegate operator, and recording the remaining call_function targets as fallback operators:

    

        
        
graph = edge.exported_program().graph_module.graph
delegate_op = torch.ops.higher_order.executorch_call_delegate

delegate_calls = 0
fallback_targets = []
for node in graph.nodes:
    if node.op != "call_function":
        continue
    if node.target == delegate_op:
        delegate_calls += 1
    else:
        fallback_targets.append(str(node.target))

    

The printed counts here tell you how many delegate boundaries and fallback operators were found. Model Explorer is introduced in the next section, which allows you to visualize the graphs to see where those boundaries sit in the model. Delegated regions will appear as LoweredBackendModule blocks. Each one represents a subgraph that will run through XNNPACK.

Run the following commands on the development machine:

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/aot/lower_to_executorch.py \
  --spec module-1-transformer-inference-with-xnnpack/artifacts/gpt2/model_spec.json \
  --backend portable

    

This creates gpt2_portable.pte and gpt2_portable.json.

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/aot/lower_to_executorch.py \
  --spec module-1-transformer-inference-with-xnnpack/artifacts/gpt2/model_spec.json \
  --backend xnnpack

    

This creates gpt2_xnnpack.pte and gpt2_xnnpack.json. The XNNPACK metadata includes the delegation summary printed by the script.

Example artifact sizes are recorded below:

Before moving on, confirm that the lowering step produced:

• the portable .pte
• the XNNPACK .pte
• the XNNPACK metadata JSON

In the reference run, the XNNPACK artifact is about 154 MB larger than the portable artifact. That increase comes from backend-specific lowered data and metadata stored inside the .pte. This is normal. Delegation can improve runtime performance, but it can also make the deployable artifact larger.

The XNNPACK reference run reported:

• XNNPACK delegate call sites: 75
• fallback call_function operators: 746

Exact delegation counts can vary when the exported graph or software versions differ. A small change in the fallback count does not by itself indicate a problem.

The fallback list was dominated by graph allocation and indexing operations, including alloc (311), getitem (172), expand_copy (49), and native_layer_norm (25). Mask and elementwise operators included logical_not, mul.Scalar, scalar_tensor, and where (24 each).

When ExecuTorch lowers with XNNPACK, it partitions the graph. Supported subgraphs become XNNPACK delegate regions. The remaining operators stay on the default ExecuTorch path, backed by the Portable Kernel Library.

The delegate call count tells you how many separate XNNPACK regions were created. The fallback operator list shows what did not make it into those regions. Many fallback operators perform relatively low-cost work, such as tensor allocation, indexing, shape handling, and mask processing. A smaller number of delegated operators may execute the matrix operations that dominate transformer inference.

More delegation is usually good, but the shape of the delegation can also matter. A few large delegated regions are typically better than many small regions split up by fallback operators. In the example above, 75 delegate call sites means XNNPACK found useful regions, but it also means the graph is not one continuous delegated block. The more that execution has to keep moving between XNNPACK and the default path, the more that graph fragmentation can eat into the benefit. A developer could identify unsupported operators and replace them with supported equivalents, reducing this fragmentation. Alternatively, they could consider another backend or a model with better operator coverage. Whether this is necessary depends on whether the performance is sufficient for the use case.

In Module 2 you can use ETDump and ETRecord to map runtime events back to operators and delegate regions, helping you identify where execution time is spent.