1 Pick a task
Loading tasks…
Or watch a recorded run replay at 4x speed (no model calls, free): Receipts → Nano, squeezed to 42x cheaper · Your own task: 100 SMS, no answers → Nano, 17x · Receipts → use Nano, 36x cheaper · Receipt photos → Super VL beats Ultra, no OCR · Support triage → turn thinking off · Contract clauses → no single model, a chain: 2.5x · SMS → the answer key audits itself · Contract clauses → keep Ultra (before the chain)
2 The ladder
| Rung model and thinking mode |
Price per 1M tokens in / out |
Chosen prompt tuning / validation |
Held out score, 95% CI, vs Ultra |
US$ per 1k calls | p50 latency | Status |
|---|
3 Prompt rewrites
Ultra rewrites the instructions and picks examples from the tuning split; it only ever sees tuning rows. Each candidate is scored on the tuning rows it did not use as examples and on the validation rows. The candidate with the best validation score is the only one scored on the held-out split, so the gap between the three scores shows how much the rewrite overfit.