After Choosing Qwen3.5-2B: What 4K and 8K Really Cost on an iPhone

Our ten-model comparison ended with a practical choice: Qwen3.5-2B Q4_K_M was the only candidate that cleared every resource and structured Agent-routing gate we had set.

That result answered which model to use. It did not yet answer how we should use its context window.

In the first iPhone matrix, moving Qwen3.5-2B from a 4K to an 8K context added only 52 MiB of Metal memory. The short-prompt TTFT barely changed. It would be easy to read those numbers and conclude that 8K is almost free.

It is not. Reserving an 8K context and actually processing nearly 8K tokens are different workloads. After selecting the model, we ran a second iPhone test to measure that difference directly.

Why this follow-up came after model selection

The original ten-model comparison evaluated two separate requirements:

Those results made Qwen3.5-2B the default candidate for our phone-local Agent router. The 4K/8K phone runs in that comparison deliberately used an 85-token prompt. They measured loading, context allocation, short-request latency, memory, and basic generation, not the cost of filling the context.

The follow-up asks a narrower production question:

If the Runtime gives the selected model a nearly full 4K or 8K prompt, how long does prefill take, how fast does generation remain, and what happens to memory and thermal state?

Test setup

We used the same model file and inference stack as the main comparison:

The app used the model's own tokenizer after loading. It calibrated the complete rendered prompt, including the system and chat wrappers, to an exact token count:

• 4K context: 3,584 prompt tokens;
• 8K context: 7,680 prompt tokens;
• output headroom: 512 tokens in both tiers;
• measured output: 63 generated tokens in every run.

Execution order alternated between 4K and 8K to reduce a simple order bias. llama.cpp's native counters measured prompt evaluation and generation. TTFT was also measured from the Swift inference boundary to the first non-empty streamed piece.

flowchart LR
    A[Load the selected model] --> B[Calibrate rendered prompt with model tokenizer]
    B --> C1[3,584-token prompt in 4K context]
    B --> C2[7,680-token prompt in 8K context]
    C1 --> D[Prefill, generate 63 tokens, record memory and thermal state]
    C2 --> D
    D --> E[Compare capacity cost with active-use cost]

The measured 4K and 8K cost

The table reports P50 across three independent launches. The min/P50/max column exposes the small sample instead of hiding it behind one number.

Three observations matter more than the individual numbers.

Available capacity is cheap

In the short-prompt allocation test, the 8K configuration used only 52 MiB more Metal memory than 4K. The same gap remained after generation in this run: 1821.4 versus 1769.4 MiB. The model file is unchanged, and an 8 GB phone still has a workable memory envelope.

This supports making 8K available. It does not support filling it by default.

Active context is a compute and latency cost

The 7,680-token P50 prefill took 30.583 seconds, 2.29 times the 13.375 seconds needed for 3,584 tokens. Prefill throughput also fell from 268.0 to 251.1 tokens per second.

Decode was almost unchanged at 19.9 versus 19.7 tokens per second. In other words, the long wait comes before the answer starts. Once generation begins, the model emits tokens at roughly the same speed.

This is why a short-prompt TTFT of 0.274 seconds and a near-full 8K TTFT of 30.602 seconds can both be correct. They describe different active prompt lengths under the same context ceiling.

Thermal pressure is the harder limit

Both nearly full tiers produced thermal pressure. Two of the three 3,584-token runs ended in Serious; all three 7,680-token runs did. Thirty seconds of cooling between launches did not prevent that result.

The memory numbers alone would make 8K look easy. The thermal and latency results show why context policy has to be controlled by the Runtime rather than treated as an unlimited text buffer.

The deployment policy we are using

The resulting policy separates a hard capability ceiling from a normal working budget:

1. Enable an 8K context ceiling on devices with at least 8 GB of physical memory.
2. Keep ordinary active prompts near 2K-4K.
3. Count the fully rendered prompt with the model tokenizer before inference.
4. Compact old conversation history before removing current instructions or relevant evidence.
5. Rerank and trim retrieved passages and Tool output instead of appending everything.
6. Expand toward 8K only when the task benefits from the extra evidence enough to justify a roughly 20-31 second prefill.
7. Preserve at least 512 tokens for output and recovery instructions.

flowchart TD
    A[Assemble system, history, evidence and Tool output] --> B[Count rendered tokens]
    B --> C{Within 2K-4K working budget?}
    C -->|Yes| D[Run normal inference]
    C -->|No| E[Compact history and rerank evidence]
    E --> F{Fits below 8K with output reserve?}
    F -->|Yes, task justifies delay| G[Run long-context inference]
    F -->|No| H[Split the task or request narrower scope]

The important choice is not “4K or 8K” as a permanent global switch. It is “8K available, 2K-4K active by default.”

That gives complex tasks room for local documents, retrieved evidence, and Tool results without imposing a 30-second first-token delay on routine requests.

What this test does not prove

This was a deterministic synthetic compute workload. It measures how the selected model processes known token counts on one target phone. It does not establish:

• semantic accuracy over natural multi-document content at 7.5K;
• battery drain across a sustained work session;
• behavior over 20 conversational turns;
• complete thermal recovery time;
• Jetsam behavior under simultaneous app pressure;
• the same performance on lower-memory devices.

Each tier also has only three samples. The min/P50/max range and thermal counts are included so that this limit remains visible.

Those are follow-up tests, not details that should be inferred from the present data.

Final decision

The deeper test does not reverse the model selection. Qwen3.5-2B Q4_K_M remains our default local routing model because it combines the strongest structured Agent result in this comparison with less than 2 GiB of Metal allocation at 8K.

It does change how we deploy it:

• 8K is the supported ceiling on 8 GB devices.
• 2K-4K is the normal active-prompt target.
• Near-full 8K is an explicit slow path, not the default.

The distinction between reserved capacity and active computation is the most useful result of this follow-up. On a phone, a context window is not merely memory. It is also first-token latency, heat, energy, and the amount of irrelevant material the model must process before it can act.

For the complete evaluation design and all 96 expected Agent routes, see How We Benchmark On-Device Models for a Phone Agent.