False Positive Rates in AI Code Review Comments
Missing context and weak prompts drive AI review noise, and teams can fix both.

Those causes are addressable. Teams can reduce noise without abandoning AI review by improving what reviewers can see, narrowing what they are asked to judge, and designing a process that preserves developers’ critical engagement.
Why AI code review produces more false positives than traditional static analysis
Traditional static analysis and LLM-based review perform structurally different tasks: traditional static analysis checks code against defined rules, while LLM-based review infers what code intends to do and what might go wrong. A deterministic check operates within a narrow domain where the condition for a finding can be specified in advance. An AI reviewer looks for more kinds of possible defects, including problems that take more than pattern matching to spot. Looking for more kinds of defects gives it more chances to flag something that looks wrong but is safe.
A model asked to infer intent has to judge possibilities that a fixed rule would never consider. Its ability to raise those concerns is also the source of its noise, because a pattern can look unsafe without being unsafe in the application where it appears. Although some false positives are avoidable, evaluating AI review solely against the precision of narrow static checks overlooks the different work the reviewer is attempting. Teams need to distinguish uncertainty that comes with that broader task from uncertainty the review system could remove.
How limited codebase context turns probabilistic reasoning into noise
The largest removable source of uncertainty is missing codebase context. Most AI reviewers operate on the diff alone, which makes them strongest on defects visible within the changed code and weakest when correctness depends on something outside it. A shared utility illustrates the problem clearly. A change may be safe because of how other code calls the utility, yet look dangerous when those callers are invisible. The reviewer then flags a possible problem that an engineer who knows the surrounding code would rule out.
Full-codebase indexing addresses that blind spot by making external code available to the review process. The distinction between diff-only review and indexed review is therefore more than a convenience, especially when behavior depends on downstream consumers. However, adding context is not automatically beneficial. Functional specifications can increase bug detection while also increasing false positives when a model overinterprets stylistic elements. The same limitation can produce missed bugs, because understanding specification intent is different from treating every detail as an implementation requirement.
The Carnegie Mellon practitioner study places this problem within the organization of review itself. It synthesized practitioner discourse from thousands of engineering blogs and Reddit threads. Its finding reaches beyond the model alone. Review effectiveness depends on how teams structure context and expertise around the process.
How weak prompt design amplifies the context problem
Missing context limits what the reviewer can know, while weak prompts distort what it does with the information available. A generic request to review code leaves the model without a clear distinction between a defect and a preference. It may flag a team convention as a violation or treat a documented trade-off as an oversight. Plausibility becomes the threshold for producing a comment. Even with enough repository context, the model still needs instructions that spell out what to flag.
Ericsson researchers found that useful review with simple prompts such as “Please generate a code review for the following code” needs care in data selection, prompt design, and context extraction. Context supplies evidence, while the prompt directs how that evidence should be used and which kinds of conclusions belong in the output.
Prompt design therefore deserves treatment as an engineering discipline. The same diff can produce qualitatively different signal-to-noise ratios under a bare instruction and a well-specified request. That difference separates two sources of error. Some uncertainty belongs to probabilistic reasoning, but another share of production noise comes from how the system was instructed to behave.
What happens to developer behavior when false positive rates are high
Repeated false positives change how developers allocate attention. A reviewer who keeps seeing incorrect AI comments may start skimming them and later stop reading them at all. Once comments are known for being wrong, reviewers may pass over useful findings because they expect the next warning to be noise too. This “cry wolf” dynamic helps explain why poor signal can lead teams to abandon AI review.
Developers who accept AI feedback uncritically surrender judgment to output that can be wrong, while developers who dismiss it entirely lose access to useful findings. Calibrated skepticism avoids both extremes by keeping human evaluation active. That requires more than a tool that produces plausible comments. It requires reviewers who still check whether each concern applies to the code.
The Carnegie Mellon study describes a related feedback loop involving review depth and surface plausibility. Under pressure from high volumes of AI-generated code, shallow review can allow more defects through and undermine the ability to perform skilled reviews later. This is not simply another name for lost trust. It identifies how reduced engagement can weaken the expertise that future reviews depend on, making process quality a continuing influence on the value of the tool.
A model asked to infer intent has to judge possibilities that a fixed rule would never consider.
What independent benchmarks reveal about precision across tools
Benchmark results show that precision varies widely across AI review tools, but those results need to be read with care. Published comparisons differ in what they test and who runs them, and many are vendor-run. There is no independent, peer-reviewed benchmark of commercial AI code review tools in the evidence described here. Differences between vendor-reported and externally measured results also caution against attributing performance to model choice alone.
The Macroscope benchmark used confirmed runtime bugs across multiple repositories and languages. Its results illustrate that substantial gaps in bug detection can persist even among the strongest performers in a comparison. False positives and missed defects can coexist. A reviewer can write incorrect comments and still miss real problems, so the number of comments says little about coverage.
Tools can improve apparent precision by withholding more speculative findings, but that restraint can also reduce recall. GitHub Copilot Review illustrates the pattern of prioritizing quality even when the result is silence. Silence can be useful when the alternative is an unfounded warning, but it does not establish that the code is free of defects. Precision has to be understood together with what a reviewer leaves unreported.
How hybrid SAST-plus-LLM pipelines reduce false positives at the triage layer
Hybrid pipelines give the model a smaller task than looking for possible problems across all the code under review. A deterministic SAST tool first identifies candidate findings under defined rules. An LLM then evaluates those findings against codebase context to assess whether they are exploitable. The division of labor is the advantage.
That architecture does not require the initial SAST output to be free of noise. A deterministic match can still produce a false positive when the flagged pattern is not exploitable in context. The model’s task is to distinguish those cases. SAST-Genius combined Semgrep with a fine-tuned LLM and achieved a precision of 89.5% in its experiments. The framework reduced false positives substantially compared with Semgrep alone, demonstrating the value of contextual triage within that evaluation.
Semgrep Assistant applies a version of this approach by analyzing findings, assessing exploitability, and prioritizing them by severity and exploitability. It also auto-triages known false positive patterns. An emerging alternative adds a second LLM to critique the first model’s review, validate findings, and identify missed issues. Both approaches constrain review through an additional judgment stage, with the SAST hybrid beginning with a deterministic candidate set.
What teams can do at the configuration and process level to reduce noise without suppressing useful signal
Teams that retain their current tooling can act on the same mechanisms through configuration and process design. Giving the reviewer access to the full repository is the highest-leverage decision. For large or interconnected codebases, this is a prerequisite for useful interpretation. The aim is to expose the dependencies that determine correctness. More information helps only when the review process can connect it to the finding being assessed.
Prompt specificity provides a second form of control by narrowing the categories the model is asked to flag. Teams can ask the reviewer to focus on concerns that matter to them, such as security patterns, null safety, and error handling, and leave out general comments on style. This reduces the scope for preferences to masquerade as defects. A bounded review deliberately limits coverage to findings the team is prepared to evaluate and act on.
Teams can also measure the suggestion dismissal rate as the fraction of AI comments developers explicitly dismiss or ignore. That measure provides an early warning of noise before disengagement becomes habitual. A high rate should lead teams to check context configuration and prompt design before deciding that the tool cannot get better. It connects developer behavior to the technical choices producing the comments.
Hunk-level review lets developers check, accept, or reject AI-written code in smaller pieces instead of whole files, so they can more easily keep useful changes and reject problems next to them. Improving comment precision remains a separate task from this control over reviewing generated code. The Carnegie Mellon study supports the broader principle that review structure and human expertise influence whether AI assistance strengthens or weakens the process.
Where the false positive problem sits in the broader trajectory of AI code review
Hybrid architectures and better context handling are cutting false positives, but these gains alone may not win back developers’ trust. Developer surveys show that developers trust AI accuracy less and worry that AI-generated code can look correct even when it cannot be relied on. The burden is also changing. AI-generated code makes up a growing share of the code under review and has more defects per pull request than human-written code. Review systems must therefore contend with noise while also catching defects in code that can be harder to assess.
This creates a gap between improving a tool and improving its use. Developers who have learned to ignore comments may not change that behavior simply because precision improves. Conversely, plausible output can still encourage shallow acceptance even when the reviewer remains imperfect. Better prompts, repository context, and hybrid triage address the quality of findings, while review structure preserves the judgment needed to use them.
The Carnegie Mellon study’s organizing claim is that AI does not determine the direction of its effect on review quality by itself. Human expertise and the structure of review influence that outcome. For teams, the practical response is deliberate engineering of both the system and the process around it.
Sources
- 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse
Supplied the practitioner-discourse findings on review structure, expertise, and the shallow-review feedback loop.
- From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests
- Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry
Informed the discussion of using LLMs to triage static-analysis findings and cut false positives.
- An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors
Supported the points about simple prompts and the need for careful context extraction and prompt design in LLM code review.
- 1 LLM-Driven SAST-Genius: A Hybrid Static Analysis Framework for
Provided the SAST-Genius hybrid Semgrep-plus-LLM results and the false-positive reduction over Semgrep alone.