The five questions to ask before your team adopts an AI coding assistant
Not whether to adopt — that decision is usually already made. These are the five measurements to take first, so you can tell later what actually changed.
Most teams we speak to did not decide to adopt AI coding assistants. Adoption happened, from the bottom up, and leadership ratified it afterwards. That is not a criticism — it is how most useful tooling arrives.
The cost of arriving that way is that nobody took a baseline. A year later the throughput numbers are up, everyone has an opinion about quality, and there is no measurement that can settle it.
These are the five questions worth answering. If you are adopting now, answer them before you roll out. If you adopted eighteen months ago, answer them anyway — most are recoverable from data you already have.
1. What is your defect escape rate, and when did you last look at it?
Escape rate — defects that reach production per unit of delivery — is the only quality metric that survives contact with an engineering leadership meeting. It is also the one most teams cannot produce on request.
Take it now, segmented by severity and by component, and keep at least six months of history from before the rollout. If your defect tracker has that history, you have a baseline whether or not anyone planned one. That single dataset is worth more than every survey on this topic put together, because it is about your software.
The specific trap: throughput improvements and escape-rate degradation cancel out in a “velocity” chart. You ship more, more breaks, and the ratio hides both. DORA’s 2026 research found higher AI adoption correlating with increased delivery throughput and increased delivery instability at the same time — DORA (10 Mar 2026, published by a research programme housed at a cloud vendor). Faster and more fragile is a coherent outcome. Measure them separately or you will not see it.
2. Where does your test coverage sit relative to the code the assistant actually touches?
Coverage percentage as a single number has always been a poor metric, and assistance makes it worse: total coverage can hold steady or rise while coverage on newly-changed paths falls, because generated code arrives faster than tests for it.
The question is not “what is our coverage”. It is “what is our coverage on the paths that changed this quarter”. Those are different numbers and they diverge quietly.
Practical version: instrument coverage per changed file over a release window and compare it to the repository average. If changed-path coverage is materially below the average, generated code is outrunning the tests, and that gap is where the escapes come from.
3. Has your code review process changed to match what it is now reviewing?
Review was designed around an assumption that is no longer reliably true: the author can explain the code. That assumption is what made “looks fine to me” a defensible approval.
45.2% of developers report that debugging AI-generated code takes them longer than debugging their own, and 66% regularly encounter output that is “almost right, but not quite” — Stack Overflow Developer Survey 2025 (2025). “Almost right” is precisely the defect class that passes review, because review is optimised to catch code that looks wrong.
Two things worth checking in your own data:
- Time-to-approval. If PR size went up and approval time went down, review is not keeping pace. That is measurable this afternoon.
- Comment density per changed line. A sustained fall means review became a rubber stamp without anyone deciding it should.
Neither number is damning on its own. Both moving together, across the adoption boundary, is a finding.
4. Which vulnerability classes does your scanning actually catch — and are they the ones at risk?
Security scanning is usually configured once and then inherited. The classes it is tuned for are not necessarily the classes where generated code fails.
AI-generated code passes a security review 56% of the time, and cross-site scripting passes at 15% — Veracode GenAI Code Security Report (28 Jul 2026, published by a vendor that sells security testing). Weight your review toward the classes with the worst published pass rates rather than running a generic scan and reading the summary count.
This matters more than it did two years ago. Vulnerability exploitation is now the top initial access vector at 31% of breaches, surpassing stolen credentials for the first time in 19 years — Verizon DBIR (19 May 2026). The path from “a class your scanner does not cover” to “an incident” got shorter.
5. Who signs off on the release, and what evidence do they hold?
The other four questions are measurements. This one is organisational, and in our experience it is the one that actually predicts outcomes.
Ask who approves a release, then ask what they look at before approving. If the honest answer is “the pipeline was green”, you have automated the sign-off without automating the judgment behind it. Green means the checks that exist passed. It says nothing about whether the checks that exist are the right ones — and the checks were designed for a different authorship model.
Somebody has to hold a position on whether the release is safe, and be able to say why. That role existed implicitly when a human wrote every line and could explain it. It does not survive the transition automatically, and most teams have not noticed it went missing.
What to do with the answers
If you can answer all five with data, you do not need us, and we would tell you so.
If you cannot answer three of them — which is the common case, and not a sign of a badly-run team — you have an unmeasured change in your delivery risk. That is worth two weeks of somebody’s attention before it is worth a remediation project.
None of this is an argument against using AI. It is an argument that the verification layer which used to be implicit now has to be deliberate, and that somebody has to be accountable for it.
The AI Code Assurance Diagnostic answers all five in two weeks, for a fixed fee agreed before any work starts. Or take the questions and run them yourself — they work just as well without us, and we would rather you measured than not.