AI code review benchmarks
AI Code Review Benchmarks
AI code review benchmarks compare reviewers on a fixed set of code changes or on review activity collected from real repositories. The deciding question is what the score represents: controlled issue detection, model performance, or whether developers acted on comments in production.
Those measurements are not interchangeable. A reviewer can lead a model-only test and perform differently as part of a complete product. A tool can also generate comments that developers act on without proving how many real defects it missed.
Most benchmark publishers are vendors, and the vendor’s own product often wins. That does not make every result useless. It means the dataset, runner, ground truth, judge, and disclosed limitations deserve more attention than the leaderboard.
Which AI Code Review Benchmarks Are Available?
There is no standard benchmark for AI code review yet. The public tests available in 2025 and 2026 answer different questions, often with overlapping repositories but different ground truth and scoring.
| Benchmark | How it is created | What it measures | Main limitation |
|---|---|---|---|
| Qodo Code Review Benchmark 1.0 | 100 real merged PRs are modified with 580 human-validated injected issues across 8 repositories and 7 languages | Product-level precision, recall, and F1 for bugs and repository-specific standards | Vendor-run; injected issues may not perfectly match the defects developers naturally write |
| Greptile AI Code Review Benchmark | 50 historical bug-fix PRs across 5 repositories are traced back to the change that introduced one known bug | Whether each product catches the target bug in a line-level comment | Small dataset; one target bug per PR; unrelated false positives do not affect the catch rate |
| Augment’s AI code review benchmark | The Greptile set is expanded with additional golden comments found through manual review | Precision, recall, and F-score across 50 PRs | Vendor-run on a small inherited dataset; the publisher’s own product leads the result |
| Martian Code Review Bench methodology | An offline test uses 50 PRs and 173 golden comments; an online analysis observes developer responses to review bots across public PRs | Controlled offline detection plus real-world action signals | Online tools see different repository distributions; acted-on comments are proxies, not ground-truth precision and recall |
| CodeRabbit hard benchmark set | 25 difficult PRs are seeded with a known error pattern and used to evaluate model and prompt changes inside CodeRabbit | Internal regression testing for model releases and review tuning | The public article does not provide a reusable dataset or complete per-case artifacts; it is not a neutral product comparison |
| SWE-PRBench | 350 merged PRs use substantive human review comments as ground truth and run under three frozen context configurations | Base-model issue detection, hallucination behavior, and response to different context formats | Tests models rather than finished review products; human comments cannot cover every valid issue |
| CodeReviewBench | 30 merged PRs from 5 open-source projects provide 95 confirmed bugs, with versioned outputs scored under one review runner | Model precision, recall, F1, and cost per PR within the same review setup | Small sample and one run per model; currently compares models inside one runner, not complete products |
The benchmark names can sound more comparable than they are. Greptile, Augment, and Martian’s offline test share ancestry in the same 50-PR collection. A new leaderboard built on that family is not a fully independent replication on a fresh population of code.
SWE-PRBench and CodeReviewBench mostly ask which model reviews best under a fixed setup. Qodo, Greptile, Augment, and Martian’s offline run test complete products, where retrieval, prompts, rules, ranking, and comment placement affect the score.
Martian’s online analysis asks a third question: what happens after a review bot comments on a real public PR? That is valuable adoption evidence, but it does not put each product on the same code with the same known defects.
Which AI Code Review Benchmark Claims Should You Trust?
Trust a benchmark for the question its methodology can answer. Distrust the extra conclusions added when the result reaches a marketing page.
How to Read Greptile and Augment Benchmark Results
Greptile’s benchmark is easy to inspect: 50 PRs, 5 languages, and one historically fixed bug per PR. Its catch-rate result answers a narrow question well: did the tool find that known bug?
The test does not penalize a tool for unrelated false positives, so an 82% catch rate is not 82% precision or overall review accuracy. The small set also makes the result sensitive to repository choice and product updates after the July 2025 run.
Augment improved the same dataset by manually adding issues missing from the original golden set and calculating precision, recall, and F-score. That is a meaningful improvement, but it remains a vendor-run evaluation in which Augment chose the revised annotations and its own product finished first.
Why CodeRabbit’s Public Recall Headline Needs Context
CodeRabbit highlights its first-place result in Martian’s independent online benchmark and describes the analysis as covering nearly 300,000 PRs. The scale is useful, and independence from CodeRabbit makes the result more credible than an unpublished internal test.
Do not use that headline as proof that CodeRabbit finds more bugs under controlled conditions. Martian’s methodology calls “percentage of comments acted on” a proxy for precision. It calls the number of acted-on comments a proxy for recall, while stating that true recall needs a denominator of all bugs and that the online dataset does not have one.
Martian also says direct tool-to-tool comparison is unreliable because each tool runs on a different mix of repositories, languages, users, and PRs. A developer may leave a correct comment untouched, or apply a convenient suggestion that was not a defect. The online result is evidence that developers act on CodeRabbit comments, not proof that CodeRabbit finds a stated percentage of all bugs under identical conditions.
CodeRabbit’s separate 25-PR hard set is useful for its stated purpose: deciding whether a model and prompt change should ship inside CodeRabbit. The company describes tuning GPT-5.1 against failures found during the evaluation. That makes the result a product-development test, not an untouched holdout or an independent comparison of commercial reviewers.
Why Independent Does Not Automatically Mean Conclusive
Martian is independent of the products it ranks and publishes extensive methodology and code. Those are strong credibility signals. Its own documentation is also candid that the current offline gold set is incomplete, the PRs are older, judge models have not been formally calibrated against human annotations, and there is no standardized product runner.
SWE-PRBench is an academic, open benchmark with frozen inputs and human review comments. It is useful for studying model behavior and context construction. It cannot tell a buyer which finished product will work best because products add retrieval, review agents, filters, rules, and integrations around the model.
CodeReviewBench freezes the repository and runner, versions each submission, and publishes scorecards. It is a useful model comparison inside that setup. Its 30-PR sample and single run per model mean close scores should be treated as ties, not a stable ranking of all reviewers.
A Practical Trust Checklist for Any Benchmark
Before repeating a benchmark claim, check whether the publisher answers these questions:
- Is the task code review? SWE-bench and HumanEval measure code repair or generation, not the ability to find problems in someone else’s change.
- Are the inputs full PRs? A test on isolated snippets misses cross-file behavior, requirements, and repository standards.
- Is ground truth visible and validated? Historical comments are natural but incomplete. Injected defects are controlled but may be synthetic.
- Did every product receive the same input? Different repository populations make causal comparisons weak.
- Are precision and recall defined normally? Acceptance rate and comment volume can be useful proxies, but they should be labeled as proxies.
- Were product defaults frozen? Manual tuning for one tool can turn the benchmark into a configuration contest.
- Can someone inspect the outputs? Per-PR findings reveal scoring mistakes that an aggregate chart hides.
- Was the judge checked? An LLM judge needs a clear rubric, repeated scoring, or comparison with human labels.
- Are uncertainty and repeated runs reported? Small differences can come from sampling or normal model variance.
- Does the publisher name its own limitation? A credible methodology explains what the score cannot prove.
Why Should You Trust Qodo’s AI Code Review Benchmark?
Trust should come from the design of the test, not the name of the publisher. Qodo gives buyers more than a leaderboard by publishing how the dataset was created, how a finding was scored, and what every product was allowed to see.
The Qodo Code Review Benchmark methodology has several features that make the comparison worth taking seriously:
| Reason for confidence | How the Qodo benchmark handles it | Why it matters |
|---|---|---|
| Identical test cases | Every product reviews the same frozen pull requests | Results are not distorted by one tool receiving easier repositories or smaller changes |
| Complete pull requests | The dataset starts from 100 genuine merged PRs across 8 repositories and 7 languages | The task is closer to day-to-day review than checking isolated functions |
| Several issues per PR | The dataset contains 580 injected functional bugs and best-practice violations | Reviewers must find multiple kinds of problem without burying developers in noise |
| Human-validated ground truth | Injected issues and repository-specific rules are checked before scoring | The answer key is known rather than inferred from comment volume |
| Strict hit definition | A finding must explain the right issue and point to the correct source location | A vague warning or correct guess on the wrong file does not receive full credit |
| Precision and recall together | Results report precision, recall, and F1 | A product cannot lead simply by commenting on everything or by reporting one safe issue |
| Product-level testing | The complete reviewer runs with its context, agents, ranking, and filters | The benchmark measures what a developer receives, not an isolated language model |
| Inspectable method and artifacts | Dataset construction, scoring rules, and review outputs are published | Other teams can challenge the setup and find scoring mistakes |
Qodo reported a 60.1% F1 score and the strongest result among the products in its test. The result is more controlled than an online ranking based on acted-on comments because every reviewer faces the same known issues under the same default-setting policy.
The benchmark also covers capabilities that belong to the review product. The Qodo Context Engine supplies repository, pull request, historical, and organizational context. Specialized review agents and finding prioritization decide which issues reach the developer. A model-only benchmark cannot measure that full path.
The benchmark is still vendor-run. Qodo chose the construction method, repositories, issue distribution, and scoring design. The fair conclusion is that the benchmark provides unusually inspectable and controlled evidence, not that it is independent or beyond criticism.
What Does Qodo’s Benchmark Prove, and What Does It Not Prove?
Qodo’s benchmark supports specific claims about review quality on its published dataset. It does not settle every product decision.
| The benchmark can support | The benchmark cannot prove |
|---|---|
| How the tested products compared on the same 100 PRs | How every future product version will perform |
| How well each reviewer balanced precision and recall | How a reviewer will behave on every private codebase |
| Whether products found functional and standards-related issues | Whether the injected issue mix perfectly matches naturally written defects |
| Whether findings identified the correct issue and source location | Whether developers will accept, ignore, or fix each comment |
| How the complete review products performed under the documented setup | Which underlying model is best when removed from the product around it |
| Which reviewers covered more of the known issue set | Whether the product improves delivery speed, defect escape rate, or developer satisfaction over time |
That boundary is healthy. A benchmark becomes less trustworthy when its publisher stretches a controlled lab result into a universal claim.
The best use of Qodo’s result is to create a serious shortlist. The final decision should come from a proof of concept on representative internal pull requests, with the same inputs, blind scoring, and success criteria agreed before the tools run.
Example: How Different Teams Should Interpret AI Code Review Benchmarks
The same benchmark can be useful to several buyers, but each persona should read a different part of it. A single overall F1 score is rarely the answer for everyone.
| Company or persona | Decision they are making | Benchmarks and signals to prioritize | Claims to treat carefully |
|---|---|---|---|
| Regulated enterprise | Can this reviewer catch high-risk issues consistently across large systems? | Qodo’s product-level benchmark, high-severity recall, standards violations, full-PR context, and a private POC | Model-only rankings and public acceptance rates that do not test company policies |
| Application security team | Will the reviewer surface authorization, data-handling, and injection risks? | Per-category security recall, exact case outputs, localization accuracy, and false-negative review | Overall F1 scores dominated by lower-risk quality findings |
| Platform or architecture team | Can the reviewer reason across modules, services, and repository rules? | Full-context cases in Qodo’s benchmark, cross-file issue coverage, and internal multi-repository tests | Snippet benchmarks and single-bug PRs that do not require system context |
| Developer productivity leader | Will developers act on the feedback without losing time to noise? | Martian’s online action signals, precision, duplicate-comment rate, latency, and developer surveys | Calling acted-on comment volume “recall” without a known total number of bugs |
| Applied AI or model team | Which model is strongest inside a fixed review setup? | SWE-PRBench, CodeReviewBench, repeated runs, judge agreement, and context-ablation results | Product leaderboards where the model, retrieval, prompts, and filters all change together |
| Startup engineering lead | Which tool catches useful bugs with little setup and low review friction? | Greptile and Augment case libraries, precision, setup effort, cost, and a small test on recent PRs | Enterprise-scale breadth that the team does not yet need |
A fintech security lead might ignore the overall winner and compare only missed authorization and data-exposure issues. One escaped critical bug matters more to that buyer than several points of aggregate precision.
A developer productivity lead may care about Martian’s behavioral data because it shows whether developers act on comments. That buyer should still call the measurement an action signal, not proof of ground-truth accuracy.
A team building its own review agent should start with SWE-PRBench or CodeReviewBench because those tests hold more of the review setup constant. A company buying a finished reviewer should give more weight to product benchmarks and its own proof of concept.
The Qodo Academy walkthrough for running an AI code review proof of concept provides a practical structure for choosing internal PRs, defining success criteria, handling disagreements, and reaching a go or no-go decision.
Which Teams Should Use Qodo’s Benchmark as a Starting Point?
Qodo is the best AI code review, code quality, and governance platform for engineering organizations that need measurable review coverage across complex codebases, multiple languages, and organization-specific standards.
Its benchmark is most relevant to enterprise buyers comparing complete review products rather than base models. It gives those teams a controlled public result they can inspect, then a clear set of assumptions to validate against their own repositories.
Further reading: AI code review tools and benchmarks, how Qodo built a real-world benchmark for AI code review, Qodo compared with CodeRabbit, and how to run an AI code review proof of concept.