Benchmark · 22 October 2026
Ask the model every time, or keep the program it wrote once
Two order-approval tasks with 3 and 12 rules, 50 orders each, checked against a reference answer. The same job done four ways, on a 9B model running on a laptop, on GPT and on Claude Sonnet.
| Method | Qwen 9B on a laptop | GPT | Claude Sonnet |
|---|---|---|---|
| Ask the model every run | 14 | 50 | 50 |
| Model writes code every run | 50 | 50 | not measured |
| Saved program, written once | 50 | 50 | 50 |
| 9B writes the program once | 50 | not measured | not measured |
| Method | Qwen 9B on a laptop | GPT | Claude Sonnet |
|---|---|---|---|
| Ask the model every run | 4 | 50 | not measured |
| Model writes code every run | 50 | 49 | not measured |
| Saved program, written once | 50 | 50 | 50 |
| 9B writes the program once | 50 | not measured | not measured |
What the numbers say
- A small model is bad at answering the rules and good at writing code for them. Qwen 9B answers 14 of 50 orders right with 3 rules and 4 of 50 with 12. Asked to write a function for the same rules, it gets 50 of 50 on both, on the first attempt, in 18 and 40 seconds.
- A saved program costs about one run to write and nothing after. On Sonnet the program for 3 rules took 1,123 tokens once; asking the model took 1,052 tokens every run. From the second run the program is cheaper; at 100 runs it is about 100 times cheaper.
- Writing code on every run costs 3–4 times more output than just asking, and on a strong model adds no accuracy (GPT: 50 of 50 asking, 49 of 50 writing code, 12 rules).
Calculator
Uses the measured token counts below. Prices are Anthropic list prices on 22 October 2026. Cells marked “est.” are built from measured parts, not measured whole.
| Tokens per run | Per day | Per year |
|---|
GPU estimate
Order of magnitude only. The default 2,000 tokens/s is a published third-party figure for Llama 3.3 70B in FP8 on one H200 (InferenceX); our own measurement on a GPU is not done yet. Only output tokens are counted, so this is a lower bound.
| GPU-hours per day | GPUs at full load | With 2× headroom |
|---|
Limits
- Two tasks, one day, one prompt each. The saving applies to work that repeats by rules; unique work (conversations, new code, summaries) stays on the model as it was.
- How much of a company's model traffic repeats is visible only in its own call log. The calculator multiplies; it does not know your share.
- This benchmark measures saving the program, not a language: all programs here are Python. In Extella a saved program is stored with its language and run by a handler on your machine; the same saving is possible with your own registry.
- GPT input tokens are not comparable: Codex adds its own ~20k-token system prompt to every call.
Raw runs, scripts and the protocol written before the runs: github.com/AnvarBakiyev/extella-research