Median completion time for one fixed no-tool task through each vendor's subscription CLI by model, reasoning effort, and mode; detail columns add stream token rates and the retired agentic-suite metrics
Loading results…

E2E is the median wall-clock time to complete one fixed no-tool task through each vendor's subscription CLI — ready session to final result, process startup excluded, thinking time included. The task (longform-scheduler-v1) is a frozen ~600-word specification of a deterministic job scheduler answered entirely in one response: plan, full implementation, table-driven tests, and a design note. A structural contract rejects runs that skip the format; correctness is deliberately not graded, output length varies by model and is part of what the clock measures, and runs that touch tools or web search, hit an API retry, or get served by a different model are dropped. Every run is bounded by a 30-minute cap and the CLI's 64,000-output-token ceiling; a configuration that exhausts either bound on every attempt reads “no run finished” rather than hiding. Claude Code's measured window also includes the CLI's small concurrent Haiku utility call and result wrap-up (sub-second), which the Codex turn boundary does not have. The vs button on any measured row sets it as a baseline: every other row then also shows its median as a percentage faster or slower than that pick. Detail columns add stream rates from the separate copy-passage protocol and the retired agentic-suite metrics: token units are vendor-specific (that passage is 1,692 Claude tokens but 1,026 GPT tokens), so tokens-per-second ranks models only within one vendor, and OpenAI's subscription stream is delivery-paced (~55 tok/s normal, ~84 fast) — transport policy, not decoder speed — while Claude Code passes model speed through.