10 Small Language Models on an iPhone: 4K/8K Memory and Agent Accuracy

We wanted a harder answer to a simple question: which small language model is actually suitable for a useful local agent on a phone?

Our first comparison covered five compact GGUF models. We have now expanded it with five international candidates: Meta's Llama 3.2 1B and 3B, Hugging Face's SmolLM2 1.7B and SmolLM3 3B, and Liquid AI's LFM2.5 1.2B. The completed study contains 10 models, 20 context configurations, 60 successful iPhone launches, and 1,920 deterministic routing evaluations.

The resource tests ran on an iPhone 15 Pro Max. Every model used GGUF Q4_K_M, full Metal offload, and both 4K and 8K contexts. Capability was measured with the same 96 Chinese and English agent requests, covering structured parsing, Skill and Action selection, Mode classification, missing slots, and planning.

This is not a general leaderboard. It asks whether a model can fit a phone, respond quickly, and make reliable structured decisions for an agent that may touch reminders, mail drafts, or memory.

The 10 tested models

For reproducibility, the added weights were pinned by repository commit, SHA-256, license, and prompt profile. Llama 3 used its header-token template, SmolLM3 ran with /no_think, and LFM2.5 used its required start-of-text token.

Why Qwen3.5 when the mainline has reached Qwen3.8?

The newest Qwen mainline is already beyond 3.5, but its public focus is on much larger cloud and workstation models. Those releases are not in the same deployment class as a 1B-4B model that must live inside an 8 GB phone.

Our target needs a practical 4K default, optional 8K capacity, predictable memory, and stable structured output. Under those constraints, Qwen3.5-2B remains a relevant phone-sized candidate. Qwen3.6, Qwen3.7, and Qwen3.8 do not currently provide a clear official open 1B/2B-class replacement for this exact role.

Test protocol

flowchart LR
    A[Pin model, commit and SHA-256] --> B[Apply the native prompt profile]
    B --> C[Run 4K and 8K on iPhone]
    C --> D[Record load, TTFT, process and Metal memory]
    D --> E[Run the same 96 bilingual routing cases]
    E --> F[Apply resource and capability gates]

The phone was an iPhone 15 Pro Max with A17 Pro, 8 GB physical memory, iOS 26.6.1, SwiftLlama, and llama.cpp b8668. Each model/context pair was launched independently three times. All 60 runs loaded successfully.

The phone prompt was intentionally short, so these numbers measure loading, context allocation, short-prompt TTFT, basic generation, and memory footprint. They are not a full 4K/8K prefill stress test. The deterministic 96-case capability suite ran with the same model files on a desktop Metal evaluator to keep comparisons repeatable.

iPhone 15 Pro Max: 4K and 8K resources

Values are P50 across three independent launches. Each paired value is 4K / 8K.

LFM2.5 was the resource winner: its 8K Metal allocation stayed at 931.4 MiB. Qwen3.5-2B added only 52 MiB of Metal memory when moving from 4K to 8K. SmolLM2 added 768 MiB, while Llama 3.2 3B reached a Serious thermal state in all six phone samples.

The 96-case Agent capability test

The dataset contains 48 Chinese and 48 English requests across 12 Skills, split evenly between core and challenge cases. Parse means the full two-stage protocol succeeded. Every score is calculated over all 96 cases. Paired values are 4K / 8K.

The added candidates did not become more accurate at 8K because these short routing requests fit comfortably inside 4K. Here, 8K tests deployment cost and compatibility, not extra intelligence.

What the international candidates changed

LFM2.5-1.2B is the strongest new resource candidate. It is the lightest model on the phone and reached 80.2% Skill accuracy. Its 38.5% Action accuracy, however, is not safe for full execution. It is worth studying as a low-risk first-stage Skill shortlist model.

Llama 3.2 1B has the best new Skill score at 86.5%. But 53.1% Parse and 42.7% Action show a clear boundary: recognizing intent is easier than producing a complete execution contract. It is another Skill-only research candidate, subject to license review.

Llama 3.2 3B produced valid structure but often chose the wrong semantics. Parse reached 94.8%, while Skill remained at 51.0%. It also used 3071.3 MiB of Metal memory at 8K and repeatedly reached Serious thermal state.

SmolLM2-1.7B does not fit this bilingual routing task. Its 21.9% Skill and 13.5% Action scores were the lowest, and its 8K context cost was unusually high.

SmolLM3-3B improved substantially over SmolLM2, but 56.2% Skill and 45.8% Action remain below the execution threshold. It may still be useful for English-focused tool-use research.

Final decision

The international expansion changes the map, but not the default:

1. Qwen3.5-2B Q4_K_M remains the default local Agent router. It is still the only model here with 100% Parse, 95.8% Skill and Action, 96.9% Mode, and less than 2 GiB of 8K Metal memory.
2. Use 4K by default and switch to 8K only when required. Qwen3.5's measured 8K Metal increment was only 52 MiB.
3. Keep LFM2.5-1.2B and Llama 3.2 1B in a Skill-only research track. Their resource profiles are excellent, but they should not directly generate execution parameters.
4. Do not equate successful loading with production readiness. Stable Action, Mode, missing-slot, bilingual, thermal, and recovery behavior are separate gates.

Limits and next tests

The next resource round should add approximately 3.5K and 7.5K prefill workloads, 20-turn sustained sessions, generation throughput, battery drop, thermal curves, and Jetsam recovery. Multimodal candidates such as Gemma 3n E2B and MiniCPM-V need a separate vision-language protocol for screen understanding, OCR, images, and privacy boundaries.

The larger lesson is that phone-local model selection is a systems problem. Model size matters, but prompt protocol, structured reliability, context memory, language consistency, heat, and the runtime's deterministic safeguards matter just as much.