Median completion time for one fixed no-tool task through each vendor's subscription CLI by model, reasoning effort, and mode; detail columns add stream token rates and the retired agentic-suite metrics
Loading results…

E2E is the median wall-clock time to complete one fixed no-tool task through each vendor's subscription CLI — ready session to final result, process startup excluded, thinking time included. Every valid run must reproduce the same fixed passage byte-for-byte (SHA-verified), and runs that touch tools or web search, hit an API retry, or get served by a different model are dropped from the timer, so cross-vendor seconds time identical visible work; correctness beyond the exact-output match is not graded. Claude Code's measured window also includes the CLI's small concurrent Haiku utility call and result wrap-up (sub-second), which the Codex turn boundary does not have. Detail columns add stream rates and the retired agentic-suite metrics: token units are vendor-specific (this passage is 1,692 Claude tokens but 1,026 GPT tokens), so tokens-per-second ranks models only within one vendor, and OpenAI's subscription stream is delivery-paced (~55 tok/s normal, ~84 fast) — transport policy, not decoder speed — while Claude Code passes model speed through. Agentic E2E and Passes come from the earlier hidden-verifier repair-task protocol, which timed different tool-using work per model.