Run the DistilBERT autocomplete demo

The DistilBERT demo turns the fill-mask model into a small autocomplete application. Unlike the GPT-2 demo, it does not generate a sequence token by token. It inserts one [MASK] token, runs one model forward pass, and turns the top vocabulary logits into suggestions.

Run the following on the Raspberry Pi:

    

        
        
module-1-transformer-inference-with-xnnpack/scripts/pi/serve_distilbert_autocomplete.py \
  --spec ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/model_spec.json \
  --pte ~/executorch_on_arm/module-1-transformer-inference-with-xnnpack/artifacts/distilbert/distilbert_fillmask_xnnpack.pte \
  --host 127.0.0.1 \
  --port 8091

    

Then open:

    

        
        
http://localhost:8091

    

Try out the application by typing a short sentence. The demo inserts one mask token, runs the ExecuTorch with XNNPACK .pte artifact, and refreshes the ranked suggestions as the sentence changes.

The .pte provides the model execution, but the product behavior comes from the surrounding application code: where the mask is placed, how suggestions are filtered, and how the ranked tokens are presented to the user.

Understand the code

As with the GPT-2 example, the serving script is mostly HTTP server and page-handling code. The fill-mask runtime logic is in module-1-transformer-inference-with-xnnpack/scripts/pi/core/distilbert_autocomplete_core.py.

The core loop differs from GPT-2. The app appends the mask token, prepares fixed-shape inputs, calls the ExecuTorch model once, then ranks the returned logits:

    

        
        
masked_prompt = self.mask_at_end(text)
encoded = self.tokenizer(
    masked_prompt,
    return_tensors="pt",
    padding="max_length",
    truncation=True,
    max_length=int(self.spec["seq_len"]),
)

logits = self.model.forward((input_ids, attention_mask))[0]
scores = torch.softmax(logits[0], dim=-1)
top = torch.topk(scores, candidate_count)

    

The app then decodes candidate token IDs back into words and filters out special tokens and word pieces:

    

        
        
token_piece = self.tokenizer.convert_ids_to_tokens([token_id])[0]
token = self.tokenizer.decode([token_id]).strip()
if not is_word_suggestion(token, token_piece, self.tokenizer.all_special_tokens):
    continue

    

You are now ready to summarize and compare what you have learned.