Codex vs Claude Code: Both Passed Every Test. Only One Survived a Big Order.

On Cognition’s FrontierCode 2.0 benchmark, Opus 5.5 scores 47.3% and GPT-6 Astra 44.0%. Three points apart sounds like a tie, so if Codex and Claude Code both pass your tests, either one should be fine. I gave them the same job and they both passed every test I wrote. Then I hid one more test, and they split.

📺 Watch the video: Codex vs Claude Code: Both Passed Every Test. Only One Survived a Big Order.

The two sides

On one side is Claude Code 2.1.322 in -p mode, signed in with my Max plan, running Opus 5.5 and Fable 5.2 at medium effort. On the other is OpenAI’s Codex CLI 0.169.0 with GPT-6 Astra, deployed in Microsoft Foundry on my Azure free credit, at medium (Codex’s default) and high. The free credit can’t bill my card. Both ran on the same Mac mini, in the same repo, with the same prompt and web search off, one run at a time.

The job

The repo is the shop till from my earlier videos. The prompt points at four rule docs: offers, returns, weighed fruit and a voucher. After each run, a hidden check the agents never see rings up 36 receipts and compares each one with the right answer.

Test 1: the written spec

Claude Code passed 15 of 15 runs (10 with Opus, 5 with Fable) and Codex passed 10 of 10, with all 36 checks right every time. Codex was quicker. A typical run took 4:19 for Codex at medium and 7:36 for Claude Code with Opus. Every run ended by saying it was done and that its own tests passed.

Test 2: the order nobody wrote down

None of the rule docs say how big an order can get. So the hidden test adds four big carts: a party order of 175 items, a cafe order, a cart of 300 items and one of 200 rows. Each has to ring up the right total in under 2 seconds.

Written spec Big orders
Claude Code (Opus 5.5 and Fable 5.2, medium) 15 of 15 14 of 15 (Opus 9 of 10, Fable 5 of 5)
Codex (GPT-6 Astra, medium and high) 10 of 10 5 of 10 (medium 3 of 7, high 2 of 3)

Every till that failed missed all four carts, and most were still running when the check stopped them at 10 seconds. More effort didn’t save Codex, since at high it got two runs of three through.

I rebuilt the slow tills from their diffs to see why. A hundred bakery items took a third of a second, and a hundred bottles of oat milk took a twentieth. With both in one cart, the till was still going at ten seconds. Oat milk shares no offer with the bread, but the slow tills search the whole cart as one puzzle and remember every item’s count at every step, so each bottle multiplies the ways to group the bread. The fast tills only remember the items the later offers use, and they rang up every big cart in under a tenth of a second.

Who tried a big cart

Claude’s first drafts were just as slow, which I didn’t expect. In 9 of its 10 Opus runs, Claude Code made up a big cart of its own, watched its first version run out of memory or take seconds, and rewrote the search before it said done. The one Opus run that never tried a big cart is the one that failed. Codex ran a big cart in 1 run of 10.

The model alone doesn’t explain it. In my effort post, Opus at medium ran through my own bare setup on the Agent SDK, and there it tried a big cart in only 2 runs of 11.

Across all 80 runs I have on this till, the ones that tried a big order on their own held 46 times in 50. The ones that didn’t held 7 times in 30. That’s a pattern in the logs, and it doesn’t prove the check is what saved them.

One line in AGENTS.md

Every run starts from nothing, so the check has to happen every time or be written down where the agent will read it. Codex reads AGENTS.md before it starts, so I committed one line there:

Before you say a change is done, run it on an input much bigger than any example in the docs and make sure it still finishes quickly.

With it, Codex at medium held in all 5 runs, against 3 of 7 without it, and every run tried inputs far bigger than my carts before it said done. It cost time. A typical run went from about 4 minutes to about 12 and a half, which is slower than Claude Code. Five runs is a small sample, and all five held.

Test 3: a bug they both miss

Cognition’s FrontierCode 2.0 post includes a real bug in Mock Service Worker, a library that fakes network requests in tests. After a reset, the library has to start a generator over and close the old one, so the cleanup in its finally block runs. Codex failed all 3 runs, at medium, at high, and at high with its network on. Every run restarted the generator and dropped the old one without closing it, and its own tests passed.

Claude made the same mistake in 20 of its 21 runs over my last two videos. The only fix in all 24 runs is one extra line from an Opus run, with the comment “Close the abandoned generator so its finally blocks run.” A bigger input wouldn’t have shown this one, so if you want cleanup checked, it has to be in your test.

What I took from it

  1. Put the big-input line in AGENTS.md for Codex, or in CLAUDE.md for Claude Code.
  2. Write a hidden test for the case you didn’t write down, like size or cleanup.
  3. Run both agents on it before you trust either one.

This is one till job and one bug, with 15 and 10 runs on the till, so I wouldn’t rank the two agents on it. I still don’t know what makes Claude Code try a big cart when the same model in my bare SDK setup mostly didn’t. If you know which part of Claude Code does that, tell me in the video’s comments.

Run both on your own task with which-claude

which-claude 0.7.0 gives the same task and your own hidden check to Claude Code and to Codex, each as it ships, side by side, and marks the runs that said done and failed it. example/staircase/c4 is the till from this post with the big-order check. On that till, one run each, Claude Code passed in 6:26 ($1.14) and Codex failed 4 of 4 big orders after saying it was done.

which-claude "your task" --check "npm test" --agents claude-code,codex --models opus --effort medium

Code & video

If this was useful, subscribe to The Coderlog on YouTube for the next test.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top