Measuring AI-generated code across open-source repositories: the method, published before the results
We are running a security and defect analysis across open-source repositories with significant AI-generated contribution. This is the methodology, published first — including what would prove us wrong.
Almost every claim you can read about the quality of AI-generated code comes from one of two places: a vendor selling a scanner, or a survey asking developers how they feel. Both are useful. Neither is a measurement of code.
We are running the analysis. This post is the method, published before we have the results, because a methodology released after the numbers is a methodology you cannot check.
Why publish the method first
If we publish findings and the method together, you have no way to know whether we chose the method to fit the findings. Pre-registration is standard practice in fields where that temptation is understood to be real, and it costs us nothing except the option to quietly change our minds later. That option is exactly what we want to give up.
There is a second reason, and it is commercial rather than scientific. This is the same method we run inside a client engagement. If you are considering the diagnostic, the honest way to show you what you are buying is to show you the procedure, in public, applied to code neither of us controls.
The question
Does code developed with heavy AI assistance carry a measurably different defect and vulnerability profile from code in comparable repositories developed without it?
Not worse. Different, and measurably. We are interested in where the difference lands — which classes, which components, which parts of the review process stop catching things.
What is already established, and what is not
Three findings frame the question and none of them answer it.
- Developers do not trust the output they are shipping. 46% actively distrust the accuracy of AI-generated code while 84% use it or plan to, and 66% regularly encounter solutions that are “almost right, but not quite” — Stack Overflow Developer Survey 2025 (2025).
- AI-generated code passes a security review 56% of the time, with cross-site scripting passing at 15% — Veracode GenAI Code Security Report (28 Jul 2026). Note the interest: Veracode sells security testing. So do we. Treat both of us accordingly.
- In a randomised controlled trial, experienced developers using AI tools were 19% slower while believing they had been 20% faster — METR (10 Jul 2025).
What none of these tells you is what happens to a real codebase over time — where the defects concentrate, whether they escape to production at a different rate, and whether existing review and scanning practice catches them. That is a longitudinal question about repositories, not a cross-sectional question about snippets or a question about developer perception.
The hard part, stated honestly
There is no reliable way to label a line of code as AI-generated. Anyone who tells you otherwise is selling something.
Commit metadata does not record it. Stylistic classifiers are unreliable and get less reliable as models improve. Self-reporting is inconsistent even within a single project. This is the central methodological weakness of every study in this area including ours, and pretending otherwise would invalidate the result before it exists.
What we can do is work with repository-level exposure rather than line-level attribution, using signals that are documented rather than inferred:
- Repositories whose contribution guidelines or governance documents explicitly permit, require disclosure of, or encourage AI assistance.
- Repositories with a documented, dated policy change on AI assistance — these are the most valuable in the sample, because the project supplies its own before-and-after boundary.
- Commits or pull requests that carry explicit AI-assistance trailers or bot co-author attribution.
Category 2 is the strongest design available: it is a natural experiment inside a single codebase, with the same maintainers, the same review culture and the same domain on both sides of a known date. Where we have enough of those, that is the primary analysis. Everything else is supporting.
Sample frame
- Repositories with a minimum contribution volume and history sufficient for before-and-after comparison, so that a handful of commits cannot move the result.
- Spread across languages, weighted toward those where the published pass rates are worst at the web layer.
- Excluding repositories where we or anyone we have worked for is a significant contributor.
- Excluding forks, mirrors, vendored trees and generated code, all of which would inflate counts without representing authored work.
The exact repository list is fixed and recorded before analysis begins, and it is published with the results whether or not the findings are interesting. A sample chosen after seeing the data is not a sample.
What gets measured
| Measure | Why it is in |
|---|---|
| Vulnerability findings by CWE class, normalised per thousand lines changed | The class distribution is the finding, not the raw count |
| Fix latency — introduction to remediation | Measures whether the review layer is still catching things |
| Reverted and hot-fixed commit rate | The closest open-source proxy for defect escape |
| Test coverage delta on changed paths | Tests the assumption that coverage follows authorship |
| Review depth — comments and iterations per PR, and time to approval | Direct measure of whether review adapted |
| Dependency additions per change | A known and under-examined AI failure mode |
Static analysis produces the raw candidate findings. Every finding that enters the published result is verified by one of our engineers. Unverified scanner output is not a finding — it is a scanner’s opinion, and reporting it as a measurement is the specific practice this study exists to argue against. We use AI tooling for triage and first-pass analysis on a corpus this size. Nothing reaches the result on its say-so.
What would prove us wrong
We expect a measurable difference concentrated in injection, cross-site scripting, and authorisation handling, with review depth roughly unchanged across the policy boundary.
The result that contradicts us: no significant difference in class distribution, and fix latency flat or improved. If that is what the data says, that is what gets published, in the same detail, with the same prominence. A finding that the concern is overstated is a real finding, and it is worth more to you than a comfortable one is to us.
We are also expecting the sample to be smaller than we would like. Repositories with explicit, dated AI policy changes and enough volume to compare are not common. If the usable sample is too small to support the comparison, the honest output is a null result and a description of why the study could not be done — not a weaker analysis of a bigger sample.
Limitations, before anyone else has to point them out
- Repository-level exposure is not line-level attribution. Every conclusion inherits that.
- Open-source review culture differs from commercial review culture. This does not transfer directly to your codebase, and we will not claim it does.
- Projects that publish an AI policy are self-selected: they are more deliberate than average about tooling, which likely biases the sample toward better outcomes than the median commercial team.
- Static analysis has its own class-specific blind spots, which pull independently of the effect being measured.
Timing
The repository list and the analysis scripts are fixed before collection begins. Results follow, with the full method, the sample, the analysis code, and the raw findings, so anyone can rerun it and disagree with us using our own data.
If you want to argue with the design before we run it — which is the most useful moment to argue with it — email us. Method criticism received before collection is worth considerably more than method criticism received afterwards, and we would rather fix it now.