How We Benchmark On-Device Models for a Phone Agent

This benchmark did not begin as an attempt to build another model leaderboard.

We had a much narrower problem. A local Agent on a phone needs one default routing model. That model cannot merely hold a conversation. Within a short response window, it has to understand the request, select the right capability, extract execution parameters, and stop to ask a question when essential information is missing. It also has to fit on the device without turning an 8K context into an unacceptable memory or thermal cost.

The question we set out to answer was:

In our bilingual, structured, on-device Agent workflow, which model can make routing and execution decisions most reliably at a practical resource cost?

That qualification matters. We are not measuring general intelligence, writing quality, mathematics, or open-domain knowledge. We are answering a product engineering question.

Start with the Agent's actual job

Before an Agent can act on a request such as “Wake me at seven tomorrow,” it has at least two decisions to make.

First, it selects the right Skill from the capabilities currently available: alarms, calendars, mail, memory, and so on. It then selects an Action inside that Skill, extracts known parameters, identifies missing ones, and decides whether to act, plan, or ask the user for clarification.

flowchart LR
    A[User request] --> B[Retrieve candidate Skills]
    B --> C[Stage 1: select Skill]
    C --> D[Stage 2: Action and parameters]
    D --> E{Enough information?}
    E -->|Yes| F[Act or plan]
    E -->|No| G[Structured clarification]
    F --> H[Swift Runtime validation]
    G --> H

We do not ask the model for a plausible paragraph and grade it by feel. The evaluator requires an explicit structure: Skill, Action, Mode, known Slots, missing Slots, and objectives. The Swift Runtime then checks that the answer belongs to the retrieved candidate set and conforms to the current schema.

That distinction matters for an Agent with access to system capabilities. “Roughly understood the request” is not a useful execution guarantee. Choosing the wrong Action and omitting one required parameter are different failures, and both need to be visible.

What is in the 96-case suite

The model comparison uses the v3-96-staged-skill-action dataset:

Core cases cover common phrasing around alarms, calendars, mail drafts, and other routine requests. Challenge cases add nearby Skill distractors, missing required information, cross-capability planning, and more ambiguous wording.

We deliberately keep short requests in the set. Real phone commands are often only a few words long. A model that works only when every request is written like a test question is not particularly useful in a product.

Each case stores the expected Skill, Action, Mode, and relevant structural constraints. Published accuracy uses all 96 cases as the denominator. Parse failures are not quietly removed.

The 96 cases and their expected routes

The expected result below is the routing contract, not the final prose answer. A case passes each scored field only when the model selects the listed Skill, Action, and Mode. Extra assertion shows whether that case also checks missing-slot emptiness or requires a plan.

Three concrete scoring examples

• Complete command：Set an alarm for 7 AM tomorrow.

builtin.alarm.manage → create_alarm, Mode act, missing_slots must be empty.

• Incomplete command：Create an email draft for me.

builtin.mail.draft → create_draft, Mode clarify, missing_slots must be non-empty. The mail schema requires recipient, subject, and body.

• Multi-step command：First inspect tomorrow's calendar, then create reminders for unresolved items.

builtin.task.plan_recover → orchestrate, Mode act, and requires_plan = true.

One current limitation is visible here: the v3 suite scores missing slots as empty versus non-empty. It does not yet award separate points for naming every missing field correctly. Exact slot names and values belong in the next contract-test version.

Core: 48 routine cases

Challenge: 48 boundary cases

What we score

We do not collapse the result into one weighted score that lets strengths hide failures.

A high Parse rate does not imply semantic correctness. A model can emit perfectly valid JSON while consistently selecting the wrong Skill. The opposite failure is also possible: the model appears to understand the request, but its output drifts far enough that the Runtime cannot use it.

For a model that may drive execution, Skill, Action, Mode, and missing-slot behavior matter most. A model with strong Skill selection but weak Action accuracy may still be interesting as a first-stage shortlist model, but it should not receive direct authority to produce execution parameters.

Fair does not mean one literal prompt wrapper

All 10 models use GGUF Q4_K_M, the same evaluator, dataset, 4K/8K settings, and fixed decoding:

temperature = 0
top_p = 1
top_k = 1
seed = 42
candidate_limit = 3

We do not force the same ChatML string onto every model. Llama 3 uses its header tokens, SmolLM3 explicitly disables extended thinking, and LFM2.5 receives its required start-of-text token. The task, information, and output contract stay the same; the outer template follows the model's native format.

That is our definition of a fair comparison: keep the work and rules constant without knowingly giving a model the wrong input format.

For every weight, we also record the publisher, repository, commit, filename, byte count, SHA-256, license, and Prompt Profile. A model name alone is not enough to reproduce a result. Two conversions or quantizations carrying the same name may behave differently.

Why capability and phone measurements are separate

This is the easiest part of the study to misread.

The 96 capability cases run with the same model files in a desktop Metal evaluator. That environment makes it practical to execute the suite in bulk, retain every raw answer, and repeat comparisons under stable conditions.

Resource measurements come from an iPhone 15 Pro Max with A17 Pro, 8 GB of physical memory, SwiftLlama, and llama.cpp b8668. Every model is tested at 4K and 8K, with three independent launches for each combination, for 60 launches in total.

The phone run records:

• model load time;
• TTFT for a short prompt;
• process memory;
• Metal memory;
• thermal state;
• whether basic generation completed.

flowchart TB
    A[Pin model file and SHA-256] --> B1[Desktop Metal capability run]
    A --> B2[iPhone resource run]
    B1 --> C1[96 bilingual requests]
    C1 --> D1[Parse Skill Action Mode]
    B2 --> C2[Three independent 4K and 8K launches]
    C2 --> D2[Load TTFT Memory Thermal]
    D1 --> E[Selection decision]
    D2 --> E

We keep these evidence tracks separate. Desktop capability accuracy is not presented as iPhone end-to-end task success. Successful loading on the phone does not prove that the model understood the 96 requests.

What 4K and 8K mean in this round

The phone prompt and generation are intentionally short. The 4K/8K measurements answer three questions:

1. Can the model load with that context configuration on the target device?
2. How much extra memory does the larger context reserve?
3. Are startup, first-token latency, and thermal state acceptable for a short request?

They do not prove that speed remains unchanged after filling 8K tokens, and they do not measure long-prompt prefill throughput. That requires real workloads around 3.5K and 7.5K tokens, plus prefill speed, generation throughput, energy use, and sustained thermal curves.

In this round, 8K is a capacity and resource test, not an “8K long-context certification.”

How one result becomes evidence

Each capability run retains the raw model output and its SHA-256, not just the final percentage. The summary JSON records:

• dataset version and digest;
• model file SHA-256;
• Prompt configuration SHA-256;
• Prompt Profile;
• context size and candidate limit;
• expected and actual values for each case;
• TTFT, end-to-end latency, and token counts;
• whether format repair was needed.

On the phone side, we retain the raw text from every independent launch, CSV summaries, and per-model checksum files. The full evidence directory has a separate SHA256SUMS.txt.

This does not eliminate every possible human error. It does let us answer three basic questions later: which exact file was tested, which configuration was used, and which raw samples produced the percentage in the table.

What this benchmark does not prove

The current evidence supports choosing a router for this particular on-device Agent. It does not directly support claims that:

• one model has greater general intelligence;
• one model is better across every language and Tool domain;
• long 8K prefill performance has been validated;
• real alarm, calendar, or mail tasks achieve a stated end-to-end success rate;
• this is an academic or industry-wide ranking.

Of the 96 current cases, the 48 core cases have participated in development feedback and are not a fully independent final holdout. Important slices such as clarification and planning are still small. Each phone configuration has only three launches, without randomized model order, controlled cool-down, battery measurement, or a sustained 20-turn session.

Those limitations are not a polite paragraph added at the end. They are part of the result.

What we are adding next

The next version has a stricter target:

1. Build an independent holdout with at least 300 cases, isolated from tuning.
2. Have two non-authors review every case against a versioned rubric.
3. Run at least three trials per case and report stable passes, unstable cases, and the 95% Wilson lower bound.
4. Add real 3.5K/7.5K prefill, sustained sessions, energy use, thermal curves, and Jetsam recovery.
5. Validate parameters, Tool order, approvals, clarification, and final environment state in isolated Tool fixtures.
6. Finish with a small set of high-risk end-to-end checks against real system capabilities on the phone.

A 300-case tuning/holdout draft and the evaluation harness already exist, but the cases have not completed independent two-person review. We therefore do not present them as formal results.

Back to the original question

This method does not try to identify “the world's smartest small model.” It does something narrower and more useful: under a defined device, Runtime, and Agent contract, it identifies the best default for the current workflow and makes the reasons, failures, and evidence boundaries inspectable.

For the full 10-model results, including 4K/8K iPhone resources and model-by-model findings, read 10 Small Language Models on an iPhone.