Claude Code vs Codex Built the Same API. I Load Tested Both Until One Broke.

Your coding agent’s API can pass every test you wrote and still fall over on its first busy day. I had Claude Code and Codex build the same shop API three times each. All six builds passed my 45 tests, and all six broke at 200 requests a second. Then I gave each agent its own load test report and one more try, and one of them went at least twice as far in every build.

📺 Watch the video: Claude Code vs Codex Built the Same API. I Load Tested Both Until One Broke.

Why I tried this

In February, Codeflash ran its optimizer on two pull requests written with Claude Code and found functions up to 446 times slower than necessary. That was single functions on a profiler. I wanted a whole API under real traffic, for both agents.

The two sides

On one side is Claude Code 2.1.322 in -p mode with Opus 5.5, signed in with my Max plan. On the other is OpenAI’s Codex CLI 0.169.0 with GPT-6 Astra in Microsoft Foundry, paid from my Azure free credit. Both ran at their default effort, medium, from the same prompt and repo, one build at a time on my Mac mini with web search off. Codex had its sandbox network on so npm and the local Postgres worked, and Claude Code had a fixed list of allowed commands.

The job is a shop’s order API in Node and PostgreSQL 17 with six endpoints: product search, a product page, place order, get order, customer history and a daily top-sellers report. The spec is four short files, and none of them says how much traffic to expect.

The load test

The server is two Azure Standard_D2as_v5 machines (2 vCPUs, 8 GiB each) on one private network. One runs the API and Postgres with 100,000 products and 2,000,000 past orders, and the other runs k6 v1.3.0.

k6 sends requests at a steady rate, so requests a second means how many customers walk in each second. The rate doubles every minute, from 50 up to 3,200. A step breaks when p95 (the time 95 of every 100 requests beat) passes 500 ms, or more than 1% of requests fail or get dropped. The traffic is 35% product pages, 20% search, 20% new orders, 10% order lookups, 10% customer history and 5% daily reports.

Test 1: the written tests

All six builds passed 45 of 45, each in 66 to 106 seconds. They even planned for two customers buying the last unit at once: every build locks the products before it takes the stock.

Test 2: the crowd

Every build held 100 requests a second and broke at 200. In build 1, Claude Code’s p95 went 65, 213 and 583 ms at 50, 100 and 200 requests a second, and Codex’s went 74, 390 and 754 ms.

They broke in the same place, too. A search for a word like “mug” made the database read the names of all 100,000 products, because a normal index can’t help when the word can sit anywhere in the name. The other slow query was the daily report, which added up a whole day of orders on every call. Each took about a tenth of a second (83 to 170 ms on average across the six builds), and two CPUs only work on a couple at once.

A real shop with a queue out the door hires more staff. My server stays at two CPUs, so the only way to serve more people is to make each one cheaper to serve.

Test 3: the second chance

Each agent got its build back with its own k6 report (the steps and each endpoint’s p95) and one ask: hold as much traffic as you can on the same server, with the tables unchanged and all 45 tests still green.

Codex took 14 to 17 minutes per build ($4.13 to $5.08 on Azure). Claude Code took 16 to 32 minutes, and my allowed list refused 11 to 15 of its commands each time. All six still passed 45 of 45. None of them could test on the real server, and every one said only my k6 run there would give the real number, which was fair, since my Mac mini is far faster than that server. So I reran all six.

Codex (GPT-6 Astra) Claude Code (Opus 5.5)
Written tests 45/45, 45/45, 45/45 45/45, 45/45, 45/45
First break 200, 200, 200 200, 200, 200
Held after the fix 400, 400, 400 800, 1,600, 3,200

The 3,200 build never broke. That’s the top of my ladder, and its p95 there was 9 ms. Codex was the most consistent thing in the whole test, with 400 three times out of three.

Where the gap comes from

Every Codex build made the same queries faster. It added a trigram index, which cuts each product name into three-letter pieces so a search for a word inside a name can use an index, and it cut the trips to the database for a new order.

Claude Code did that kind of work too, and then it took jobs away from the database. All three of its builds stopped adding up a whole day of orders on every report: two kept the answer in memory, and one kept a running tally per day in a table of its own. Two builds also moved the product search into the server’s memory and only read price and stock from the database. Codex’s report still went to the database about 2,400 times in each run.

Codex kept the report on the database on purpose. One of its builds wrote:

Responses always read committed database data; there are no response caches or precomputed reports to become stale.

Claude Code’s builds added caches, two of them with triggers to keep the caches right, and each one named its catch, like stale reports if someone writes orders straight into the database.

The stock held up in all six. The popular products sold out in every second-chance run, the builds turned away the orders they couldn’t fill, and not one sold a unit it didn’t have.

What I took from it

  1. Load test on a server the size of the real one, because your laptop is faster.
  2. Give the agent the load test report.
  3. Ask what the database does on every request that it could do once, and what could go stale if it stops.

This is one API on one server size with three builds each, so I wouldn’t rank the two agents on it. Azure cost $16.62 in all, and Claude Code ran on my Max plan. I still don’t know whether Codex would cache too if I told it a stale report is fine. If you’d make Codex’s choice for a real shop, tell me why in the video’s comments.

Load test your own agent’s build

The spec, the 45 tests, the data generator, the deploy script and the k6 script are in agent-load-test, with both prompts the agents got in PROMPTS.md. Give your agent the build prompt, run npm test until it passes, put the build on a server the size of production with load/deploy.sh, then from a second machine:

k6 run -e TARGET=http://<api-server>:8080 -e OUT=summary.json load/load.js

load/stock.sql counts oversold units afterwards, and load/top.sql shows which queries took the database’s time.

Code & video

If this was useful, subscribe to The Coderlog on YouTube for the next test.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top