All projects

On-Device LLM Summarizer

Apple draws its on-device LLM line at the A17 Pro with 8 GB. This one runs below it, on an A16 with 6 GB.

GitHub (opens in a new tab)
  • 0.29 → 0.46ROUGE-L, base to the shipped 4-bit model
  • 880 MB4-bit model on disk, down from 3.1 GB
  • 44.4 tok/ssustained decode on an iPhone 14 Pro
THE PROBLEM

Apple drew the on-device line above this phone.

Apple Intelligence runs its on-device language model only on the A17 Pro and newer, which ship with 8 GB of RAM. The iPhone 14 Pro sits one generation below that line, an A16 with 6 GB, and gets nothing. The hardware gap is real, but the line is drawn for a general assistant.

A model that has to do everything needs the headroom Apple says it needs. A model that has to do one thing does not. Narrowing the task is the whole experiment.

THE APPROACH

Apple’s own recipe, one size down.

Apple’s published on-device approach is a small base model, a task-specific LoRA adapter, and aggressive quantization. This project runs that recipe end to end on its own model: QLoRA fine-tuning of Qwen2.5-1.5B on DialogSum, merged to fp16, then quantized to 4-bit MLX at group size 64, which lands at 4.5 effective bits per weight.

Licensing decided as much as size did. Qwen2.5-1.5B is Apache 2.0 where the 3B is not, and DialogSum is MIT where the more common SAMSum forbids commercial use. Training ran on a single RTX 3080 with about 9 GB of usable VRAM, which is what made 4-bit QLoRA the only way in rather than a nice-to-have.

The app is three Swift files on MLX Swift, running on the phone’s GPU through Metal. It rebuilds the eval’s exact prompt and decoding, greedy at temperature zero with a 96-token cap, so the published ROUGE numbers describe the model on the phone and not a friendlier lab configuration.

The summarizer on an iPhone 14 Pro: a generated summary of a doctor visit, above live statistics reading 44.4 tokens per second, 2047 ms to first token and 831 MB of memory.
On the iPhone 14 ProThe shipped 4-bit model, summarizing on the phone. 44.4 tokens per second after 2.0 seconds to first token, and nothing leaves the device. Every number on the screen is the app reporting on itself.
Grouped bars of ROUGE-1, ROUGE-2 and ROUGE-L for three models. Base Qwen2.5-1.5B scores .3706, .1498 and .2889; the fp16 fine-tune .5581, .3101 and .4808 with 95 percent confidence intervals; the shipped 4-bit model .5433, .2905 and .4622.
Base, fine-tune, and the model that actually ships. Quantizing to 4-bit gives back under two points of the fine-tune's gain.
THE IMPACT

It runs, and the cost of running it is measured.

On an iPhone 14 Pro the model is 880 MB on disk, peaks at 1.05 GB of memory against a 5.25 GB ceiling, and decodes at 44.4 tokens per second after 2.0 seconds to first token. After the first download it never touches the network.

Quantization costs about 1.5 to 2 ROUGE points against the fp16 fine-tune, and the loss is almost entirely precision: recall barely moves, and there are no repetition loops, truncation spikes, or empty outputs. The gain over the base model deserves the same honesty. The base model summarizes fine, it just writes around 68 tokens where the human references average 27.8, so its precision collapses. What the fine-tune actually learned is length and register, which for a task-scoped summarizer is exactly the job.

It began as a way to put the Generative AI with Large Language Models coursework into practice on hardware I already owned. Core ML and the Neural Engine were scoped as a stretch and left unattempted, so this is an MLX and Metal result.

  • Python
  • PyTorch
  • QLoRA
  • MLX
  • Swift
  • SwiftUI