Export, lower, and benchmark DistilBERT with ExecuTorch and XNNPACK

DistilBERT

DistilBERT is a compact transformer encoder created by distilling BERT into a smaller model. It is bidirectional rather than autoregressive: instead of generating one token at a time, it reads the whole input sequence and produces contextual predictions from that sequence.

In this module, DistilBERT is used as a fill-mask model. The application places one [MASK] token in a sentence, runs one forward pass, and uses the model’s vocabulary logits to suggest likely replacements. That gives you a second transformer workload to compare with GPT-2: encoder inference with one fixed request/response pass rather than decoder generation with a repeated next-token loop.