AI Coding Tools ROI: Measuring Real Business Value
A year-one ROI model for AI developer tools: the cost categories vendors leave out, a go/no-go framework, and the conditions that should change the decision.
AI developer tooling budgets fail in a predictable direction. The seat price looks like the decision, and since GitHub moved Copilot to usage-based billing in June 2026 it is not even a fixed number. The lines that dominate year-one spend are the ones no vendor model contains: metered usage stacked on top of the seat, the productivity dip while everyone learns the tools, reviewer capacity to absorb a larger pull-request queue, and security controls retrofitted after the first incident.
The workable default for an engineering leader is a narrow one: standardize on three or four tools, fund review and security capacity before seats, and hold the program to business outcomes instead of activity counts. The cost model, ROI formulas, and go/no-go framework below are built around that default. Every figure in them is either a published list price, a result from named research, or an assumption labelled as one.
The Real Cost Structure
Model Inputs and List Prices
Take a 200-developer organization as the worked example. Every row below is either a published list price or an assumption stated as one, and the assumptions are the rows that decide the answer.
| Input | Value used | Basis |
|---|---|---|
| Copilot Business seat | $19 per user per month | GitHub list price |
| Copilot Enterprise seat | $39 per user per month | GitHub list price |
| AI credit | $0.01 | GitHub billing documentation |
| Included allowance, standard | 1,900 credits per Business user per month; 3,900 for Enterprise | GitHub billing documentation |
| Static analysis | SonarQube Cloud Team from $34 per month, billed per instance per year by lines of code | Sonar list price |
| Median developer wage | $135,980 per year | US Bureau of Labor Statistics, 2025 |
| Loading multiplier | 1.5 | assumption, no published figure |
| Working days per year | 220 | assumption, no published figure |
| Dip duration | reader-set | no published figure |
| Reviewer days per generated change | reader-set | no published figure |
The US Bureau of Labor Statistics puts the 2025 median annual wage for software developers at $135,980, with a median hourly wage of $65.38. That is base wage only, excluding benefits, taxes and overhead, and at eight hours it works out to roughly $523 per day. The loaded engineering day everything else is priced against comes from the two assumption rows sitting next to it: $135,980 × 1.5 ÷ 220 = $927. Set the multiplier at 1.3 and the day costs about $803; set it at 1.8 and it costs about $1,113. Every people-cost line in the model scales linearly with that number, so choose it deliberately rather than inheriting it.
The seat line is now a floor
GitHub moved Copilot to usage-based billing on 1 June 2026. The announcement prices Business at $19 per user per month “including $19 in monthly AI Credits” and Enterprise at $39 per user per month “including $39 in monthly AI Credits”, with credits “consumed based on token usage, including input, output, and cached tokens, according to the published API rates for each model”. GitHub’s billing documentation sets one AI credit at $0.01 and the standard included allowance at 1,900 credits per Business user per month, which is the same $19 the seat already costs. Past that allowance, usage bills at published per-credit rates unless an administrator disables paid usage or sets a budget.
| Line | Arithmetic | Result |
|---|---|---|
| Seat floor, 200 Business seats for a year | 200 × $19 × 12 | $45,600 |
| Credits included at that floor | 200 × 1,900 × 12 | 4,560,000 credits |
| Cash value of those credits | 4,560,000 × $0.01 | $45,600 |
| Overage if consumption runs at twice the allowance | 4,560,000 × $0.01 | $45,600 |
| Modeled year at twice the allowance | $45,600 + $45,600 | $91,200 |
The ceiling is whatever the administrator configures. With no budget set there is no ceiling, which makes credits consumed per seat per month the first thing to instrument, ahead of anything about the code itself. GitHub also documents user, organization, cost-centre and enterprise-level limits, and they are worth setting on day one rather than after the first invoice.
A dated step sits in the same line. GitHub is running a promotional allowance from 1 June to 1 September 2026 at 3,000 credits per Business user and 7,000 per Enterprise user, after which included usage returns to the standard amounts. A team that sized its budget inside that window loses 1,100 of 3,000 included credits per user, about 37% of the allowance, at an unchanged seat price.
Static analysis scales with the codebase
Sonar prices SonarQube Cloud Team from $34 per month and states that both Cloud and Server editions are billed per instance per year based on lines of code. That puts the checking tool on the same axis as the generating tool. As generated volume grows, the line that inspects it grows with it, and neither headcount nor seat count predicts the increase.
The lines with no list price
Three inputs carry no published price, and inventing one is where a cost model stops being checkable.
Unsanctioned tool spend cannot be sourced. Nobody publishes what teams put on expense cards, so any figure in that row is manufactured. The consequence is measurable even when the spend is not. IBM’s 2026 Cost of a Data Breach study, drawn from 602 breached organizations, puts the global average breach at $4.99 million and AI-enabled breaches at $6 million, roughly $1 million above average, and reports that more than 20% of organizations had a breach targeting AI models or applications. That figure belongs in the risk line.
The productivity dip has a measured magnitude and no measured duration. METR’s randomized trial put 16 experienced open-source developers through 246 issues in repositories they had contributed to for around five years, and found them 19% slower with AI tools allowed, after forecasting a 24% speedup and, having finished, still estimating a 20% speedup. METR limits the claim to that population and now labels the result historical because the tools have moved since. No study reports how long such a dip lasts, so duration stays a knob the reader sets. It is an expensive knob: at $927 per loaded day, one week across 200 developers is 200 × 5 × $927 = $927,000 of capacity. Work that takes 19% longer leaves 1 ÷ 1.19 of the original throughput, about 84%, so a slowdown sustained through that week leaves roughly $148,000 of it producing nothing. Whether that runs for two weeks or for six is an estimate the reader owns.
Review capacity is the third. LinearB’s 2026 benchmarks, from 8.1 million pull requests across 4,800 teams in 42 countries, give anchors rather than a cost: AI-generated pull requests wait more than 16 hours before a reviewer picks them up against roughly 200 minutes for unassisted ones, and merge at 32.7% against about 84.5%. Turning that into reviewer days needs the reader’s own pull request volume and the share of it that reaches a reviewer at all. LinearB sells engineering analytics, which is worth knowing when reading its benchmarks.
Put the model back together and the shape matters more than the total. Two lines have list prices and one of those is uncapped by default. One has a measured magnitude and no duration. One has no published figure at any point. A year-one number produced from that mix is a model output carrying the reader’s own assumptions, and presenting it as a forecast is how the overrun story writes itself.
The Hidden Cost Categories
Three categories rarely appear in a tooling business case and none of them can be costed from published research. No study allocates engineer-days to cleaning up generated code, remediating generated vulnerabilities, or repairing brittle generated tests. What the research does establish is direction and mechanism, which is enough to plan against.
Maintainability drifts toward duplication. GitClear’s 2026 maintainability research, based on 623 million analyzed changes between 2023 and 2026, reports block duplication up 81% since 2023, from 40.3 to 73.0 duplicated lines per million changed lines, with commits containing a duplicated block rising roughly tenfold over two years. Moved lines, the signal for reshaping code rather than adding to it, fell from 21% of changed lines in 2022 to 13% in 2023 and 3.8% in 2026 to date, while copy-paste rose from 9.4% in 2022 to 15.7% in the first half of 2026. Cross-file function calls dropped 35%, refactoring activity 70%, long-term legacy maintenance 74%. Error-masking constructs rose 47% and two-week churn 15%. GitClear sells code-quality tooling, and the data is correlational across a period when AI adoption climbed, so the reports do not isolate AI as the cause. Read it as a measurement of what codebases now look like rather than proof of why.
Security remediation concentrates by language and vulnerability class. Veracode’s Spring 2026 GenAI code security update tested more than 150 models on 80 coding tasks across four languages and four vulnerability classes, and found that in 45% of cases the model introduces a known security flaw, unchanged from its 2025 result. The load is not spread evenly:
| Dimension | Pass rate |
|---|---|
| Java | 29% |
| JavaScript | 57% |
| C# | 58% |
| Python | 62% |
| Cross-site scripting (CWE-80) | 15% |
| Log injection (CWE-117) | 13% |
| SQL injection (CWE-89) | 82% |
| Insecure cryptography (CWE-327) | 86% |
A Java shop and a Python shop should not budget the same remediation line, and a codebase whose exposure sits in cross-site scripting is in a different position from one whose exposure sits in SQL injection. Veracode publishes this as vendor security research, and it measures generated code before review rather than production incidents. That distinction is the whole argument for funding review capacity first: review is the only thing standing between a 45% flaw rate in generation and a flaw rate in production.
Organizational friction has no measured price. Change management, tool evaluation cycles, vendor negotiation and the effect of review load on senior engineers are all real and all recurring. No published source puts a number on any of them. Carry them as named line items with locally estimated hours rather than importing a figure from somewhere else, and revisit the estimate each quarter against what the calendar actually shows.
Measuring Real Business Value
The Metrics That Matter
Three questions carry most of the first-year signal: what happens to delivery, where the gain actually lands, and what happens to quality. Published research answers all three, and disagrees with itself twice: once on throughput, once on quality.
Delivery
DORA’s 2024 report associated a 25% increase in AI adoption with an estimated 1.5% decrease in delivery throughput and an estimated 7.2% decrease in delivery stability. The 2025 report, from the same research programme and nearly 5,000 respondents, reversed the first half: it reports a positive relationship between AI adoption and both delivery throughput and product performance, while stating that adoption continues to have a negative relationship with delivery stability. One programme, two years, opposite signs on throughput and the same sign on stability. Anyone quoting a single throughput figure is quoting whichever year suits them.
Faros AI’s analysis of more than 10,000 developers across 1,255 teams shows the level at which that disagreement resolves. High-adoption teams completed 21% more tasks and merged 98% more pull requests, while pull request review time rose 91%, average pull request size rose 154%, and bugs per developer rose 9%. At company level, Faros found no significant correlation between AI adoption and improvement in throughput, DORA metrics or quality. That is the sourced version of the productivity paradox: the team-level gain is large and real, and it does not aggregate.
Review capacity
| Signal | Unassisted | AI-generated |
|---|---|---|
| Review pickup time | about 200 minutes | more than 16 hours |
| Agentic pull request pickup | 201 minutes | 1,055 minutes |
| Merge rate | about 84.5% | 32.7% |
| Size at the 75th percentile | 157 lines | more than 400 lines |
| Refactoring share at the 75th percentile | about 37% | near zero |
Those are LinearB’s 2026 figures across 8.1 million pull requests. The same dataset puts regular AI use at 88.3% of developers, up from just under 72% in early 2024, so the queue described here is the common case rather than an early-adopter artifact. A merge rate of 32.7% means roughly two thirds of AI-generated pull requests are opened and never ship. It does not say how much reviewer time they consumed: a pull request can be closed or abandoned without anyone reading it. Review cost has to come from recorded review events rather than from the merge outcome.
Where the measured gain lands
DORA 2024 recorded three positive effects per 25% increase in adoption, all three of them respondent ratings: documentation quality up 7.5%, code quality up 3.4%, and code review speed up 3.1%. Documentation is the standout by more than double, and it is the only one of the three large enough to plan around.
The largest randomized evidence on individual output comes from Cui and colleagues in Management Science, pooling three field experiments at Microsoft, Accenture and an anonymous Fortune 100 company across 4,867 developers: a 26.08% increase in completed tasks with a standard error of 10.3%, alongside higher adoption and greater gains among less experienced developers. The standard error deserves as much attention as the point estimate, because the interval it implies is wide.
No published source supports a headcount figure. There is no study reporting a technical writing or QA reduction attributable to AI tooling, and none reporting a test coverage delta attributable to generated tests. Faros points the other way on the QA question, with bugs per developer up 9%. A savings column here would be manufactured, so the honest version of this row is a direction: documentation is where the reported quality gain is largest, and less experienced developers gain more than experienced ones.
Quality
The delivery, defect and maintainability signals point the same way. Faros measured bugs per developer up 9% on high-adoption teams. DORA found delivery stability down in 2024 and confirmed the negative relationship in 2025. GitClear’s maintainability proxies move with them. Stack Overflow’s 2025 survey, with 33,662 responses in its AI section, has 66% of developers citing “AI solutions that are almost right, but not quite” and 45.2% citing that debugging AI-generated code is more time-consuming. The result running the other way is the rating quoted above: DORA 2024 associates each 25% increase in adoption with code quality up 3.4%. A rating developers give their own output is a weaker instrument than a defect count, and it is still a published result in the opposite direction. What settles the disagreement locally is defect data from the teams whose ratings improved, not the ratings themselves.
On security, Veracode’s Spring 2026 testing found a known flaw introduced in 45% of cases across more than 150 models. That is a property of generated code before review, not an incident rate, and the distance between the two is exactly the review capacity the table above describes.
Developer sentiment is the one satisfaction measure with a source behind it. Stack Overflow 2025 reports 84% using or planning to use AI tools, up from 76% in 2024, and 51% of professional developers using them daily, while trust in accuracy splits 3.1% highly trusting, 29.6% somewhat trusting, 26.1% somewhat distrusting and 19.6% highly distrusting. Positive sentiment fell from over 70% in 2023 and 2024 to 60% in 2025. Adoption and confidence moved in opposite directions. Nothing published connects AI coding tool adoption to end-customer satisfaction or churn, so a customer-side row in this table would be invention.
The three results that disagree
| Study | Setting | Result |
|---|---|---|
| GitHub, 2023 | Controlled task, recruited developers | 55% faster than the control group |
| Cui and colleagues, Management Science | Three field experiments, 4,867 developers | 26.08% more completed tasks, standard error 10.3% |
| METR, 2025 | Real issues, 16 experienced maintainers, familiar repositories | 19% slower |
All three are defensible. They differ on task realism, on how well the developer already knows the codebase, and on what counts as output. METR’s participants expected 24% faster beforehand and still estimated 20% faster afterwards, while economics experts had predicted 39% faster and machine learning experts 38%. The gap between belief and measurement is the practical finding, and it has a direct operational consequence: a program measured by developer self-report will report a success it did not have.
ROI Calculation Framework
The following framework supports honest ROI assessment:
class AIToolROICalculator {
calculateTrueROI(period: "quarterly" | "annual"): ROIAnalysis {
const costs = {
direct: {
licensing: this.getLicensingCosts(period),
infrastructure: this.getInfrastructureCosts(period),
support: this.getSupportCosts(period)
},
indirect: {
training: this.getTrainingInvestment(period),
productivityLoss: this.getProductivityImpact(period),
securityIncidents: this.getSecurityCosts(period),
technicalDebt: this.getTechnicalDebtCost(period)
},
opportunity: {
delayedRevenue: this.getRevenueDelay(period),
attrition: this.getAttritionCost(period),
competitiveLoss: this.getCompetitiveImpact(period)
}
};
const benefits = {
productivity: {
documentationSavings: this.getDocumentationROI(period),
testingSavings: this.getTestingROI(period),
juniorAcceleration: this.getJuniorProductivityGain(period)
},
quality: {
// Note: Most quality metrics got worse
testCoverage: this.getTestCoverageValue(period),
documentationQuality: this.getDocQualityValue(period)
},
strategic: {
futureReadiness: this.getStrategicValue(period),
talentAttraction: this.getTalentValue(period),
learningInvestment: this.getLearningROI(period)
}
};
const totalCosts = this.sumAllCosts(costs);
const totalBenefits = this.sumAllBenefits(benefits);
return {
roi: ((totalBenefits - totalCosts) / totalCosts) * 100,
paybackPeriod: totalCosts / (totalBenefits / 12), // Months
breakEven: this.calculateBreakEven(costs, benefits),
recommendation: this.generateRecommendation(totalCosts, totalBenefits)
};
}
}
// Model inputs. Two rows are list prices, two are stated assumptions,
// and two have no published figure at all.
const modelInputs = {
seats: 200,
seatFloorPerYear: 45_600, // 200 x $19 x 12, GitHub Business list price
includedCreditsPerSeatMonth: 1_900, // GitHub billing docs, 1 credit = $0.01
creditPriceUsd: 0.01,
loadedEngineeringDay: 927, // $135,980 x 1.5 / 220, BLS wage + stated loading
dipWeeks: null, // no published duration; set locally
reviewerDaysPerGeneratedChange: null // no published figure; derive from your PR data
};
Strategic Planning Framework
The Adoption Maturity Model
A maturity model keeps the investment question tied to the stage the organization is actually in, rather than the stage the roadmap claims:
| Level | Characteristics | Focus areas | Timeframe | Investment | Risk |
|---|---|---|---|---|---|
| 1. Experimental | Individual tool adoption; no governance framework; shadow AI prevalent; metrics undefined | Establish governance; define success metrics; run controlled pilots; build security controls | Months 0-6 | Low | Medium |
| 2. Controlled | Formal pilot programs; basic governance in place; security controls active; metrics being collected | Expand to early adopters; refine security controls; build training programs; address bottlenecks | Months 6-12 | Medium | High |
| 3. Scaled | Organization-wide deployment; mature governance; integrated workflows; clear ROI tracking | Optimize tool selection; advanced training; workflow integration; continuous improvement | Months 12-24 | High | Medium |
| 4. Optimized | AI-first workflows; custom tools/models; measurable business value; industry leadership | Custom model training; advanced automation; industry collaboration; next-gen capabilities | Year 2+ | Very High | Low to Medium |
| 5. Transformative | AI defines development; autonomous systems; new business models; competitive advantage | Business model innovation; autonomous development; AI-native products; market disruption | Year 3+ | Transformative | Varies |
Decision Framework for Tool Investment
class AIToolInvestmentDecision {
evaluateTool(tool: AITool): InvestmentRecommendation {
const scores = {
problemSolutionFit: this.assessProblemFit(tool),
organizationalReadiness: this.assessReadiness(tool),
financialViability: this.assessFinancials(tool),
riskProfile: this.assessRisk(tool),
strategicAlignment: this.assessStrategy(tool)
};
const criteria = {
mustHave: [
scores.problemSolutionFit > 7,
scores.organizationalReadiness > 6,
scores.financialViability > 5
],
shouldHave: [
scores.riskProfile < 7,
scores.strategicAlignment > 6
],
niceToHave: [
"Vendor stability",
"Community support",
"Integration ecosystem"
]
};
if (!criteria.mustHave.every(c => c)) {
return {
recommendation: "REJECT",
reasoning: "Failed mandatory criteria",
alternativeAction: "Address gaps first"
};
}
const weightedScore = this.calculateWeightedScore(scores);
return {
recommendation: weightedScore > 70 ? "ADOPT" :
weightedScore > 50 ? "PILOT" : "DEFER",
investmentLevel: this.calculateInvestment(tool),
timeframe: this.estimateTimeframe(tool),
successCriteria: this.defineSuccess(tool)
};
}
}
Preparing for the Next Wave
Capability forecasts age badly, and a roadmap pinned to a specific quarter tends to be wrong in both directions at once: too optimistic about autonomy, too pessimistic about how quickly a narrow capability becomes ordinary. The preparation that survives a wrong forecast is the preparation that pays off without it. Comprehensive test coverage, documented business logic, modular boundaries, and observability are each worth funding on their own merits, and each is also the precondition for whatever arrives next.
Preparation Strategy
class FuturePreparationStrategy {
private initiatives = {
technical: {
infrastructure: [
"Upgrade to AI-ready development environments",
"Implement comprehensive observability",
"Build vector databases for code",
"Establish formal specification practices"
],
architecture: [
"Modularize monoliths for AI interaction",
"Implement comprehensive API layers",
"Standardize on AI-friendly patterns",
"Build abstraction layers for AI tools"
],
data: [
"Create comprehensive test suites",
"Document all business logic",
"Build training data pipelines",
"Establish data governance"
]
},
organizational: {
skills: [
"Train developers in AI collaboration",
"Build AI security expertise",
"Develop prompt engineering skills",
"Create AI ethics guidelines"
],
processes: [
"Redesign code review for AI scale",
"Implement AI-aware CI/CD",
"Build AI governance frameworks",
"Establish success metrics"
],
culture: [
"Embrace experimentation mindset",
"Build trust in AI tools",
"Encourage continuous learning",
"Reward AI innovation"
]
},
strategic: {
partnerships: [
"Engage with AI tool vendors",
"Join industry consortiums",
"Partner with universities",
"Build vendor relationships"
],
investments: [
"Allocate R&D budget for AI",
"Fund training programs",
"Invest in infrastructure",
"Budget for experimentation"
],
governance: [
"Establish AI steering committee",
"Define clear policies",
"Build risk frameworks",
"Create success metrics"
]
}
};
getQuarterlyPlan(quarter: string): ActionPlan {
return {
priorities: this.selectPriorities(quarter),
budget: this.allocateBudget(quarter),
resources: this.assignResources(quarter),
milestones: this.defineMilestones(quarter),
risks: this.identifyRisks(quarter),
contingencies: this.planContingencies(quarter)
};
}
}
Making the Strategic Decision
The Go/No-Go Framework
Business case
The model returns a number only for the lines that have one. The rest stay named and open, because closing them with an invented figure is what makes a business case unfalsifiable.
| Line | Basis |
|---|---|
| Seat floor | Seats × $19 × 12 at GitHub Business list price |
| Metered credit overage | Your own consumption at $0.01 per credit, uncapped unless a budget is set |
| Static analysis | Quoted per instance per year against your line count |
| Productivity dip | $927 per loaded day × developers × the dip duration you set × the capacity share the slowdown removes; about 16% at METR’s 19% |
| Review capacity | $927 per loaded day × the reviewer days your own pull request data implies |
| Documentation gain | Directional only: DORA recorded a respondent-reported documentation quality gain of 7.5% per 25% increase in adoption |
| Headcount reduction | Not modellable: no published source supports a writing or QA saving |
| Customer impact | Not modellable: no published source links tool adoption to customer satisfaction or churn |
Strategic value
| Dimension | Rating |
|---|---|
| Future readiness | HIGH |
| Talent attraction | MEDIUM |
| Competitive necessity | HIGH |
| Learning investment | CRITICAL |
Decision criteria (weighted)
| Criterion | Weight | Score (/10) | Rationale |
|---|---|---|---|
| Financial | 0.3 | 3 | Two lines have list prices; the largest lines are reader-set and the seat line is uncapped by default |
| Strategic | 0.3 | 8 | Regular AI use sits at 88.3% of developers, so the learning curve is being climbed either way |
| Risk | 0.2 | 4 | A known flaw in 45% of generated samples, and stability down in both DORA years |
| Organizational | 0.2 | 6 | 30% of developers report little or no trust in generated code |
Those weights and scores give 0.3 × 3 + 0.3 × 8 + 0.2 × 4 + 0.2 × 6 = 5.3 out of 10. On the framework above that is a pilot rather than an adoption, and the weights belong to whoever signs the budget.
Recommendation: continue, narrowed
A boundary score argues for tightening scope:
- Reduce tool sprawl: standardize on three or four tools
- Set credit budgets and spending limits before the first metered billing cycle
- Fund review capacity ahead of the next tranche of seats
- Focus on documentation, the one area where the reported quality gain is largest
- Measure business outcomes rather than activity
What to measure
Targets are local. What is not local is the set of published baselines a target can be checked against, so measure the signals that have one:
| Signal | Measurement | Why this one |
|---|---|---|
| Review pickup time | Median hours from open to first review, split by whether the change was AI-assisted | The published gap between the two is the largest single delivery effect on record |
| Merge rate | Share of opened pull requests that merge, split the same way | Generated volume that never merges is cost without output |
| Delivery stability | Change failure rate and time to restore | The one DORA finding that pointed the same way in 2024 and 2025 |
| Credit consumption | Credits per seat per month against the included allowance | The seat line is metered now, and this is the only early warning |
| Security findings by class | Findings per release, grouped by vulnerability class | Veracode pass rates run from 13% to 86% by class, so an aggregate hides the exposure |
| Trust | Periodic developer survey, identical wording each time | Published baselines differ by more than 15 points on wording alone |
Exit criteria
Set each threshold before the program starts rather than after the first bad quarter:
- A security incident attributable to generated code
- A sustained decline in delivery throughput against the pre-adoption baseline
- Attrition concentrated among the engineers carrying review load
- Credit consumption growing faster than any measured delivery signal
Wind-down plan
| Step | Action |
|---|---|
| Gradual wind-down | 6-month phase out |
| Knowledge retention | Document all learnings |
| Tool consolidation | Keep high-value tools only |
| Team transition | Retrain on alternative approaches |
Lessons for Leaders
Key Lessons for Early-Stage Adoptions
Looking back at the beginning of an AI adoption journey:
- Start with problems, not tools - It is easy to get excited about capabilities before understanding the actual constraints
- Budget a floor with a meter on top - Copilot seats include a credit allowance and bill overage at $0.01 per credit with no ceiling unless an administrator sets one, so the fixed-cost assumption underneath any budget multiple is the part that fails first
- Security first, adoption second - Retrofitting security is exponentially harder, and Veracode finds a known flaw in 45% of generated samples before review ever starts
- Measure business value from day one - Activity metrics mislead, and METR’s participants believed they were 20% faster while measuring 19% slower
- Accept the productivity paradox - Faros measured 98% more merged pull requests on high-adoption teams and no significant company-level correlation with throughput, DORA metrics or quality
The Hard Truths
The uncomfortable parts of a first year, which no business case tends to state up front:
- The largest cost lines have no list price - Dip duration and reviewer days dominate the model, and neither has a published figure
- Senior developers remain skeptical - METR measured experienced maintainers 19% slower on repositories they knew well
- Security risks are measurable before they are visible - A known flaw in 45% of generated samples, and AI-enabled breaches averaging $6 million against a $4.99 million global average
- Stability degrades - The one DORA finding that held in both 2024 and 2025
- Review capacity is the binding constraint - AI pull requests wait over 16 hours for pickup and merge at 32.7% against about 84.5%
The Strategic Imperatives
Despite the costs, the argument for continuing is not sentimental. Competitors are climbing the same learning curve, and developers increasingly expect these tools in the stack. The gap between this year’s tooling and next year’s is wide enough that sitting out means starting the learning curve from zero later, at a point where the organization has less slack to absorb it. What year one actually buys is the knowledge of which use cases pay; the financial return arrives once that knowledge exists.
Year 2 Roadmap
Tool consolidation
| Action | Criterion |
|---|---|
| Keep | Usage shows up in metrics you already publish, and the spend is capped or predictable |
| Consolidate | Two tools cover the same workflow step |
| Re-evaluate | Pricing model changed since procurement, starting with anything metered |
The saving from consolidation depends entirely on what is being consolidated, and no published figure applies to it. Compute it from the contracts on the table rather than from a benchmark.
Investment
Sequencing matters more than amounts here, because the amounts are local and the ordering is not:
| Area | Line item | Sequencing |
|---|---|---|
| Security | Scanning tuned to the vulnerability classes your language mix fails most | Before expanding seats |
| Security | Training aimed at those same classes | Before expanding seats |
| Process improvement | Review automation on the largest generated changes | Before expanding seats |
| Process improvement | Credit budgets and spending limits per team | With the first metered billing cycle |
| Process improvement | Instrumentation on pickup time and merge rate | With the first metered billing cycle |
Published baselines to measure against
Targets for a single organization are forecasts. Baselines are measurements, and these are the ones a second-year program can position itself against:
| Signal | Published baseline |
|---|---|
| Adoption | 90% use AI at work (DORA 2025); 88.3% use it regularly (LinearB 2026); 84% use or plan to use it (Stack Overflow 2025) |
| Trust | 30% report little or no trust, down from 39% (DORA 2025 and 2024); 3.1% highly trust and 19.6% highly distrust (Stack Overflow 2025) |
| Review pickup | More than 16 hours for AI pull requests against about 200 minutes unassisted (LinearB 2026) |
| Merge rate | 32.7% for AI pull requests against about 84.5% unassisted (LinearB 2026) |
| Stability | Delivery stability down 7.2% per 25% increase in adoption (DORA 2024), with the negative relationship confirmed in 2025 |
The trust row is the one to read carefully. DORA and Stack Overflow ask the same question differently and land more than 15 points apart, so a trust target is only meaningful next to the exact wording that produced it.
When This Default Holds
Continuing with a narrowed toolset, funded review capacity, and outcome-based measurement is the right call for an organization that can carry unmodellable cost lines through a first year and already has security controls worth extending. The tools improve, the pricing settles, and the workflows mature, but on a multi-year clock rather than a quarterly one, so the budget has to be sized for the clock it actually runs on.
Three situations should override it. If review capacity cannot grow, adding generation capacity only lengthens a queue that already sits at more than 16 hours before pickup, and a merge rate near a third means most of what is opened never ships. If the codebase has no meaningful test coverage, nothing catches what the tools get wrong, and with a known flaw appearing in 45% of generated samples that gap widens with every accepted suggestion. And if an incident is already attributable to generated code, pause expansion and fix the controls before buying another seat.
Series Conclusion
The four parts cover the productivity paradox, the security surface, the implementation patterns, and the cost model. They point at the same conclusion from different angles: the capability is real, the costs that decide the outcome are the ones with no list price, and the organizations that come out ahead are the ones that budgeted for both.
References
- GitHub Copilot is moving to usage-based billing - The GitHub Blog - GitHub’s announcement of the 1 June 2026 billing change: Business at $19 per user per month including $19 in monthly AI Credits, Enterprise at $39 including $39, with credits consumed on input, output and cached tokens at published API rates.
- Usage-based billing for organizations and enterprises - GitHub Docs - The mechanics behind the seat price: one AI credit equals $0.01, standard included allowances of 1,900 credits per Business user and 3,900 per Enterprise user, overage rates, and the budget and spending-limit controls that cap them.
- SonarQube plans and pricing - Sonar - Sonar’s list prices, including SonarQube Cloud Team from $34 per month and the statement that Cloud and Server editions are billed per instance per year based on lines of code.
- Software Developers - O*NET OnLine - Wage data for SOC 15-1252 sourced from the Bureau of Labor Statistics Occupational Employment and Wage Statistics programme: $135,980 median annual wage and $65.38 median hourly wage for 2025.
- Announcing the 2024 DORA report - Google Cloud Blog - The 2024 findings quoted throughout: per 25% increase in AI adoption, documentation quality up 7.5%, code quality up 3.4%, review speed up 3.1%, delivery throughput down 1.5% and delivery stability down 7.2%, with 39% reporting little or no trust in generated code.
- Announcing the 2025 DORA report - Google Cloud Blog - Nearly 5,000 respondents: 90% use AI at work, more than 80% believe it raised their productivity, 30% report little or no trust, and the throughput relationship turns positive while the stability relationship stays negative.
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - METR - Randomized trial with 16 experienced maintainers across 246 issues in repositories they knew well, measuring a 19% slowdown against a forecast 24% speedup. METR limits the result to that population and now marks it historical.
- The Effects of Generative AI on High-Skilled Work - Management Science - Cui and colleagues pool three field experiments at Microsoft, Accenture and an anonymous Fortune 100 company across 4,867 developers, reporting a 26.08% increase in completed tasks with a 10.3% standard error and larger gains among less experienced developers.
- The AI Productivity Paradox Research Report - Faros AI - More than 10,000 developers across 1,255 teams: high-adoption teams complete 21% more tasks and merge 98% more pull requests while review time rises 91%, pull request size rises 154% and bugs per developer rise 9%, with no significant correlation at company level.
- 8 million pull requests reveal where engineering productivity breaks down - LinearB - Benchmarks from 8.1 million pull requests across 4,800 teams in 42 countries covering review pickup time, merge rates, pull request size and refactoring share. Published by an engineering analytics vendor.
- Spring 2026 GenAI Code Security Update - Veracode - More than 150 models tested on 80 coding tasks across four languages and four vulnerability classes, with a known flaw introduced in 45% of cases and pass rates broken out by language and CWE. Vendor security research measuring generated code before review.
- The Maintainability Gap: 2026 AI Code Quality Research - GitClear - 623 million analyzed changes between 2023 and 2026 showing duplication, copy-paste and churn rising while refactoring and cross-file calls fall. Correlational and published by a code-quality vendor.
- IBM study: one in four malicious breaches are AI-enabled - IBM Newsroom - Cost of a Data Breach 2026, based on 602 breached organizations: $4.99 million global average, $6 million for AI-enabled breaches, and more than 20% of organizations reporting a breach targeting AI models or applications.
- AI - 2025 Stack Overflow Developer Survey - Survey section with 33,662 responses covering adoption, daily use, the trust split from highly trusting to highly distrusting, and the frustration categories for almost-right answers and debugging generated code.
- Research: Quantifying GitHub Copilot’s Impact on Developer Productivity - GitHub Blog - GitHub’s 2023 controlled study, in which developers using Copilot completed the assigned task 55% faster than the control group. The lab-task end of the evidence range.
AI Tools for Developers
A comprehensive guide to AI-powered development tools, from code completion to intelligent debugging, exploring how AI transforms the developer workflow.