Benchmark and evaluate the quantized models

Benchmark performance using llama-bench

Use the llama-bench tool to measure model performance, including inference speed and memory usage.

Run basic benchmarks

Benchmark multiple model versions to compare performance:

    

        
        
# Benchmark the full-precision model
bin/llama-bench -m models/afm-4-5b/afm-4-5B-F16.gguf

# Benchmark the 8-bit quantized model
bin/llama-bench -m models/afm-4-5b/afm-4-5B-Q8_0.gguf

# Benchmark the 4-bit quantized model
bin/llama-bench -m models/afm-4-5b/afm-4-5B-Q4_0.gguf

    

Typical results on a 16 vCPU instance:

• F16 model: ~25 tokens/second, ~9GB memory usage
• Q8_0 model: ~40 tokens/second, ~5GB memory usage
• Q4_0 model: ~60 tokens/second, ~3GB memory usage

Your actual results might vary depending on your specific instance configuration and system load.

Run advanced benchmarks

Use this command to benchmark performance across prompt sizes and thread counts:

    

        
        
bin/llama-bench -m models/afm-4-5b/afm-4-5B-Q4_0.gguf \
  -p 128,256,512 \
  -n 128 \
  -t 4,8,16

    

This command does the following:

• Loads the 4-bit model and runs inference benchmarks
• -p: evaluates prompt lengths of 128, 256, and 512 tokens
• -n: generates 128 tokens
• -t: runs inference using 4, 8, and 16 threads

Here’s an example of how performance scales across threads and prompt sizes (pp = prompt processing, tg = text generation):

Even with just four threads, the Q4_0 model achieves comfortable generation speeds. On larger instances, you can run multiple concurrent model processes to support parallel workloads.

To benchmark batch inference, use llama-batched-bench .

Evaluate model quality using llama-perplexity

Use the llama-perplexity tool to measure how well each model predicts the next token in a sequence. Perplexity is a measure of how well a language model predicts text. It gives you insight into the model’s confidence and predictive ability, representing the average number of possible next tokens the model considers when predicting each word:

• A lower perplexity score indicates the model is more confident in its predictions and generally performs better on the given text.
• For example, a perplexity of 2.0 means the model typically considers ~2 tokens per step when making each prediction, while a perplexity of 10.0 means it considers 10 possible tokens on average, indicating more uncertainty.

The llama-perplexity tool evaluates the model’s quality on text datasets by calculating perplexity scores. Lower perplexity indicates better quality.

Download a test dataset

Use the following script to download and extract the Wikitext-2 dataset:

    

        
        
sh scripts/get-wikitext-2.sh

    

This script downloads and extracts the dataset to a local folder named wikitext-2-raw.

Run a perplexity evaluation

Run the llama-perplexity tool to evaluate how well each model predicts the Wikitext-2 test set:

    

        
        
bin/llama-perplexity -m models/afm-4-5b/afm-4-5B-F16.gguf -f wikitext-2-raw/wiki.test.raw
bin/llama-perplexity -m models/afm-4-5b/afm-4-5B-Q8_0.gguf -f wikitext-2-raw/wiki.test.raw
bin/llama-perplexity -m models/afm-4-5b/afm-4-5B-Q4_0.gguf -f wikitext-2-raw/wiki.test.raw

    

Run the evaluation as a background script

Running a full perplexity evaluation on all three models takes about 3 hours. To avoid SSH timeouts and keep the process running after logout, wrap the commands in a shell script and run it in the background.

Create a script named ppl.sh:

For example:

    

        
        
#!/bin/bash
# ppl.sh
bin/llama-perplexity -m models/afm-4-5b/afm-4-5B-F16.gguf -f wikitext-2-raw/wiki.test.raw
bin/llama-perplexity -m models/afm-4-5b/afm-4-5B-Q8_0.gguf -f wikitext-2-raw/wiki.test.raw
bin/llama-perplexity -m models/afm-4-5b/afm-4-5B-Q4_0.gguf -f wikitext-2-raw/wiki.test.raw

    

    

        
        
 nohup sh ppl.sh >& ppl.sh.log &
 tail -f ppl.sh.log

    

We can see that 8-bit quantization introduces negligible degradation. The 4-bit model does suffer more, but may still serve its purpose for simpler use cases. As always, you should run your own tests and make up your own mind.

When you have finished your benchmarking and evaluation, make sure to terminate your AWS EC2 instance in the AWS Management Console to avoid incurring unnecessary charges for unused compute resources.