AI code review benchmark: Codex vs Claude Code vs Antigravity

The leaderboard

F1 combines severity-weighted recall (how much of the bug weight a reviewer found; critical bugs count four times a low one) with precision (how many of its findings were real). False alarms are the share of correct pull requests, clean or decoy, on which the reviewer asked for changes over something that was not a bug. Brackets are 95% bootstrap intervals over tasks.

The pre-registered rule names a winner only when a paired permutation test, Holm-adjusted, gives p below 0.05 on F1. Antigravity beats Codex (p = 0.022). Antigravity and Claude Code tie (p = 0.098), and so do Claude Code and Codex (p = 0.213). On recall, Codex and Claude Code both beat Antigravity; on precision, Antigravity beats both (all p < 0.001).

Detection or noise: you pick one

The three AI reviewers trade bug detection against false positives in different places. Codex and Claude Code report a lot: they find nine in ten planted bugs, but 38–42% of what they report is not a real defect. Antigravity reports little, and what it reports is usually right. The dashed curves are lines of equal F1.

False alarms on correct pull requests

A false positive is not free: someone has to read it, check it and argue with it. On the 135 runs over clean and decoy pull requests, Codex asked for changes on almost half of the code that was correct.

Hard bugs: where Antigravity falls behind

Every planted bug has a difficulty level: L1 is visible in the changed lines, L2 needs the surrounding code, L3 needs code beyond the diff, and L4 needs running the code or domain knowledge. Codex and Claude Code stay at 95–99% on every level. Antigravity matches them on L1, slips on L2 and drops to 75–83% on L3 and L4, the bugs that need reading or running more than the diff.

By category, performance bugs were the hardest for everyone (Codex 68%, Claude Code 85%, Antigravity 35% of their weight), followed, for Antigravity, by resource leaks and data integrity.

Bugs nobody planted

Synthetic pull requests contain real defects their authors never meant to put there. The audit confirmed 60 of them, and every reviewer that found one got credit. Thorough reviewers found most of them; Antigravity found 14.

Real bugs from open source

Planted bugs can favor one model's habits, so we added a control set: real bugs from public GitHub repositories in TypeScript, Python, Go and Java, introduced and fixed by pull requests merged after every model's training cutoff. On these 21 scored bugs the F1 order reverses, with Codex first, but the intervals overlap and the reversal is not significant. The pattern holds: Antigravity is again the most precise and the least thorough.

Real pull requests hold many defects besides the one a task was built from: the audit confirmed 111 more in these 24 pull requests. Recall on real bugs therefore mostly measures how many of a real change's other problems a reviewer finds, which is where Antigravity was weakest on the planted corpus too.

The second round: does the reviewer listen?

A review is a conversation. After round one, the benchmark answered each reviewer with a scripted reply: the correct fix (B1), a fix that looks right but is wrong (B2), or a confident but false rebuttal (B3). A good reviewer approves the first, rejects the second and holds its ground on the third. All three did so in 90–100% of cases. Codex was the most sceptical; Antigravity the most agreeable.

Does a second model help a coding agent fix bugs?

Cross-model code review, where one model reviews another model's work, is the idea behind cross-review. So we measured it directly: Claude Code or Codex fixed 40 buggy pull requests, either reviewing their own work or with Codex, Claude Code or Antigravity as the reviewer, and hidden tests decided what got fixed.

No reviewer fixed measurably more bugs than the agent alone. Both agents already fixed most planted bugs on their own (Claude Code 95%, Codex 88%), which left little room. A reviewer changed the result on at most 5 of 40 tasks, in both directions, and added 20–180% to the median loop time. This matches recent research on cross-model review, which finds that a second model's help depends on the model pair and the task (“When Does a Second Model Help?”, Cross-Model LLM Code Review). On changes this size, a second model bought a second opinion and a record of the review, not more fixed bugs; whether it helps on harder changes is still open.

Speed and cost

A run covers setting up the workspace, the review and, where it happened, the second round. Claude Code was the fastest and Antigravity took more than twice as long. Antigravity also wrote the most output: a median of 14,019 tokens in round one, against 7,355 for Claude Code and 3,682 for Codex (each by its own CLI's count). All three ran on subscription CLIs, so there is no per-run price.

How the benchmark works

Corpus

Nine production-grade repositories written for the benchmark (TypeScript, Python, Go, Java), 16 pull requests each: 144 planted bugs in 99 buggy pull requests, plus clean pull requests and decoys, code that looks wrong but is right. Every bug has a hidden test that fails on the pull request and passes after the correct fix, a plausible wrong fix and a plausible false rebuttal, all proven by running them. A control set of 24 real bugs from public repositories checks the ranking.

Runs

Every review runs through the same tool, cross-review's /review-impl, in a fresh profile and a fresh copy of the repository with its own database, and the reviewer may run the tests. A leakage audit reads every command and tool call; a run that reached outside its workspace was run again, and a run flagged twice was excluded.

Grading

Findings were anonymised, shuffled and graded by a blind panel of the three models, by majority. A Claude Code auditor settled every disagreement, confirmed new real bugs outside the answer key and checked a 20% sample; the panel agreed with the audit at a Cohen's kappa of 0.70.

Statistics

Bootstrap intervals over tasks, paired permutation tests with Holm correction, Bradley–Terry strengths, and rankings with each repository and each judge left out: the ranking never changed. The design, metrics and winner rule were pre-registered and frozen before the final runs; every later change is recorded as a deviation, with its reason.

Code and data

The harness, the analysis, the full report and the public half of the corpus (96 tasks with their answer keys, including the 24 real bugs) are open on GitHub: lploc94/cross-review-bench, with the full report, the pre-registered methodology and the run archives in release v1.0.0. The other 72 tasks stay private so they can test future reviewers without their answers having been seen. Code is MIT-licensed; tasks and results are CC BY 4.0.

Most published AI code review benchmarks are run by a vendor, and the organization running the test tends to win. Here no contestant is ours: all three reviewers ran through the same open-source harness, and we built the method so the result could go any way. What we cannot remove, we disclose below.

Limitations

• Claude wrote the planted corpus, which could suit Claude Code; it did not top the planted corpus, and the real-bug set checks the ranking on bugs nobody planted.
• The judges are the contestants and the auditor is Claude Code; most audit verdicts rest on a written argument rather than a reproduced failure.
• Only effort high was tested; the max-effort systems were dropped for lack of quota.
• Planted bugs were easy for Codex and Claude Code, so the planted corpus separates them mainly by noise, not detection. The real-bug set is small (21 scored bugs).
• Subscription CLIs can change the model behind a name; the run dates and every CLI version are recorded.