Service eligibility and regional restrictions
PuppyIP serves only compliant overseas businesses and their authorized personnel. Proxy services are not available in mainland China. The service may only be used for lawful business activities outside mainland China. Use of this service within mainland China is prohibited.
Hosting a proxy IP or server overseas does not change these restrictions. The service must not be provided to end users in mainland China through relaying, forwarding, sharing or resale. Before use, read the Terms of Service.
Key Takeaways
- GitHub announced the ReviewBench research preview on October 5, 2026. Its full corpus has 219 public PRs from 187 repositories and 19 languages; the 25-PR test set is a subset.
- Grounded precision assesses only findings matched to the reference set; grounded recall measures coverage of known valid issues. With the same data and scope, grounded recall supports cross-reviewer coverage comparisons.
- Augmented recall adds each reviewer’s new valid findings to its denominator, so it is not directly comparable across reviewers. The 96.6% figure is agreement between benchmark labels and human judgment, not reviewer accuracy.
- Offline results cannot appear on the leaderboard. Official submission includes portal onboarding, a 25-PR test, three fresh rounds on all 219 PRs and maintainer review. Tuning and tests are self-funded, and final submission does not make reviewer inference free.
What does ReviewBench compare?
ReviewBench compares AI code-review agents: systems that inspect code changes and report problems. GitHub announced it as a research preview on October 5, 2026. This guide explains that existing benchmark using its current public methodology and participation documentation.
The full corpus contains 219 public pull requests (PRs), drawn from 187 repositories and 19 languages. GitHub separately analyzed 103.9 million PRs to study workload distributions; that is not the number of PRs evaluated by the benchmark.
Its labeled reference set, or gold set, combines findings from human reviews, author follow-up changes, deterministic tools and multiple models, with semantic deduplication and a shared labeling rubric. A finding is a specific review observation; a true positive is one judged to identify a real, relevant issue worth addressing.
Grounded metrics: how many known issues did the reviewer find?
Grounded precision considers only candidate findings matched to the reference set. It measures the proportion matched to known true positives. Unmatched findings enter neither its numerator nor its denominator, so a high score does not establish that every review comment is valid.
Grounded recall measures coverage of known issues. Its numerator is the number of reference true positives covered by at least one candidate finding; its denominator is all reference true positives for that PR. With the same dataset version and filters, reviewers face the same known issues, making this a useful coverage comparison.
Ask two separate questions: how many matched comments identify valid issues, and how many valid reference issues were found? Precision and recall answer those questions respectively; assess both.
Augmented metrics: why do new findings change the denominator?
An unmatched comment may identify a problem missing from the reference set. Augmented metrics send it to the same judge model for classification. Augmented precision uses all candidate findings as its denominator and counts both matched true positives and newly classified true positives in its numerator.
Augmented recall adds new true positives to both its numerator and denominator. Each reviewer can discover a different number of new issues, so its denominator changes. Treat it as a diagnostic of an individual reviewer’s discoveries rather than a direct ranking of coverage against shared known issues.
Hypothetical example: two reviewers each find four of ten known valid issues, giving both 40% grounded recall. If they also find one and four valid new issues respectively, their augmented recall becomes 5/11 and 8/14. The different ratios do not change the fact that both covered four shared known issues. These are illustrative numbers, not measured results.
What does the 96.6% figure actually mean?
The official methodology uses a published rubric and a classifier to label validity, severity and category. Current documentation identifies Claude Sonnet 5 as the classifier and the official leaderboard judge. Senior engineers independently review reference labels and investigate disagreements.
The reported 96.6% is agreement between the benchmark’s true-positive/false-positive labels and human relabeling. It is not a reviewer’s bug-finding accuracy or a security guarantee. The reference set can miss issues, and automated classification and semantic matching can make mistakes.
An offline score describes performance on particular samples under a particular judging method. A team’s code, review standards and remediation process can differ, so practical benefits need separate validation.
How can you make two evaluation results comparable?
Hold shared conditions constant: dataset and reference-set versions, PR base/head snapshots, judge and matcher, severity and category filters, and aggregation. Record each reviewer’s image version and configuration separately and keep them stable across its repeated rounds. Different reviewers do not need to use the same image.
Fβ combines precision and recall: β below 1 emphasizes avoiding false alarms, while β above 1 emphasizes avoiding missed issues. Macro averaging gives each PR equal weight; micro averaging pools findings before calculating ratios. Check these choices before reading the scores.
The public leaderboard captured for this guide shows grounded F1 for high-severity issues. It is not an all-severity conclusion. Select the scope and trade-offs that fit your team, then compare corresponding results; a current rank cannot replace validation on your own code.
How do offline tuning and official submission differ?
You can evaluate the public dataset on your own infrastructure without registering. Local results support development and tuning but cannot be published directly to the leaderboard. The official local container-check script checks that the interface runs and its output format is valid; it does not score or submit results.
For official participation, follow the self-service portal linked from the official README and sign in with GitHub. Register an image version pinned by content digest, configuration labels and the names of required secrets. The portal opens an onboarding PR, which needs validation and a maintainer merge; you do not open a separate onboarding PR yourself.
After onboarding, the portal test covers 25 PRs, a subset of the full 219-PR corpus. Tune on your own infrastructure and select a configuration. The official final then measures three fresh rounds on all 219 PRs; you cannot select a best tuning run as the final result.
Final results still require maintainer review. The original announcement says results are published for a first entry or when they outperform that reviewer’s current leaderboard score. Onboarding approval, a successful test and publication are separate stages.
Who pays for models and compute?
Participants fund reviewer inference, judge inference and compute for tuning, tests, local experiments and repeated full-set scoring. ReviewBench covers judge inference only for the official final submission; the submitted reviewer still uses the participant’s model credentials and budget. A public benchmark does not make the entire evaluation free.
Do not put key values in images, public PRs, issues or logs. The official README recommends a dedicated key with a spending cap that you can revoke. Check model access eligibility, supported regions and budgets with the services you use; an accessible leaderboard does not establish access for every account.
Sources
Frequently Asked Questions
Can I add new PRs to the public benchmark corpus now?
The current README says the corpus is not accepting new PR contributions. That is separate from registering your reviewer. Check the project’s contribution guidance for later changes.
Why might review durations from two reports be incomparable?
Check what is timed. The methodology measures the successful candidate-container invocation for each PR, excluding preparation, judge calls and failed retries. This is not total elapsed time from starting an evaluation to receiving results. Older results may use a fallback calculation or lack data; missing duration is unknown.
Does a finding without a reference-set match count as invalid?
No. It may be a valid new issue or a false positive and needs further judging. Grounded metrics exclude unmatched findings rather than marking them wrong; augmented metrics classify them separately.