Randomized trial finds AI does not improve productivity in firms, indicating operator skill and task types matter.
Nine in ten firms report that AI has changed nothing about their productivity. The same survey of nearly 6,000 executives found that the executives who use AI use it about 1.5 hours a week. Two results from the controlled literature sharpen the picture: consultants using AI on tasks outside its capability frontier performed 19 percentage points worse than colleagues using no AI at all, and experienced developers on real work were measured 19% slower with AI while believing they were 20% faster. The public debate treats all of this as a dispute about whether AI works. It is not. Across every controlled study we could find the deciding variable is the operator, not the model, and the direction of the effect flips with the task. Where the task arrives already specified, AI compresses the skill distribution and the least skilled gain most: novice support agents gained 34% while experts gained nothing, and professional writers got 40% faster with inequality between them falling. Where the operator must supply the specification, the decomposition and the verification, AI amplifies the distribution instead — which is where the two results above come from. Both camps are therefore right about different task classes, and averaging them produces a number that describes nobody. We show why the gap stays invisible: self-report is uncalibrated in both directions, the successful case is a tail that population surveys average away, and AI implementation is a textbook credence good whose canonical example in economics is car repair, so rational buyers discount every claim including the true ones. We then instrument one case from the upper tail — a production European AI product, 5,869 commits over 145 working days, live in 27 languages — and argue that the usual way of sizing such a case is wrong. Asking how many developers it would have taken measures only tool construction, the smallest of three terms; the larger two are the coordination and authorisation never incurred, most of which would never have passed a 2021 budget committee, and the work the tools then perform indefinitely at zero marginal headcount. Costed against COCOMO II and published translation and technical-writing throughputs, the countable part comes to roughly 80 to 310 full-time roles in 2021 terms, centred near 150, with the largest terms left deliberately unpriced. The sharper result is a schedule rather than a headcount: the delivery sits at 14 to 18% of COCOMO II's nominal duration for a system this size, three to five times beyond the model's own compression limits, so in 2021 this outcome was not purchasable in this time at any headcount. That points at the effect no survey can reach — the largest consequence of competent AI use is not that known work goes faster but that the cost of finding out collapses. This operation built a complete social platform in two days purely to decide whether it wanted one, then repurposed it six weeks later. No productivity instrument measures the projects an organisation now dares to attempt. Practical conclusions: measure bounded and unbounded work separately, buy implementation on outcome-contingent liability rather than credentials, count the experiments you can now afford rather than only the backlog you can now clear, and staff the competence as a profession with a real learning curve.' tags: ['productivity', 'adoption', 'economics', 'skills', 'agents', 'measurement'] status: 'published' pdf: '/papers/frits-tr-2026-08-bimodal-ai-outcomes-and-the-operator-skill-gap.pdf Technical report FRITS-TR-2026-08' authors: - name: 'Frits Lyneborg' affiliation: 'FRITS AI ApS' date: 2026-08-11 abstract: 'Nine in ten firms report that AI has changed nothing about their productivity. The same survey of nearly 6,000 executives found that the executives who use AI use it about 1.5 hours a week. Two results from the controlled literature sharpen the picture: consultants using AI on tasks outside its capability frontier performed 19 percentage points worse than colleagues using no AI at all, and experienced developers on real work were measured 19% slower with AI while believing they were 20% faster. The public debate treats all of this as a dispute about whether AI works. It is not. Across every controlled study we could find the deciding variable is the operator, not the model, and the direction of the effect flips with the task. Where the task arrives already specified, AI compresses the skill distribution and the least skilled gain most: novice support agents gained 34% while experts gained nothing, and professional writers got 40% faster with inequality between them falling. Where the operator must supply the specification, the decomposition and the verification, AI amplifies the distribution instead — which is where the two results above come from. Both camps are therefore right about different task classes, and averaging them produces a number that describes nobody. We show why the gap stays invisible: self-report is uncalibrated in both directions, the successful case is a tail that population surveys average away, and AI implementation is a textbook credence good whose canonical example in economics is car repair, so rational buyers discount every claim including the true ones. We then instrument one case from the upper tail — a production European AI product, 5,869 commits over 145 working days, live in 27 languages — and argue that the usual way of sizing such a case is wrong. Asking how many developers it would have taken measures only tool construction, the smallest of three terms; the larger two are the coordination and authorisation never incurred, most of which would never have passed a 2021 budget committee, and the work the tools then perform indefinitely at zero marginal headcount. Costed against COCOMO II and published translation and technical-writing throughputs, the countable part comes to roughly 80 to 310 full-time roles in 2021 terms, centred near 150, with the largest terms left deliberately unpriced. The sharper result is a schedule rather than a headcount: the delivery sits at 14 to 18% of COCOMO II''s nominal duration for a system this size, three to five times beyond the model''s own compression limits, so in 2021 this outcome was not purchasable in this time at any headcount. That points at the effect no survey can reach — the largest consequence of competent AI use is not that known work goes faster but that the cost of finding out collapses. This operation built a complete social platform in two days purely to decide whether it wanted one, then repurposed it six weeks later. No productivity instrument measures the projects an organisation now dares to attempt. Practical conclusions: measure bounded and unbounded work separately, buy implementation on outcome-contingent liability rather than credentials, count the experiments you can now afford rather than only the backlog you can now clear, and staff the competence as a profession with a real learning curve.' tags: ['productivity', 'adoption', 'economics', 'skills', 'agents', 'measurement'] status: 'published' pdf: '/papers/frits-tr-2026-08-bimodal-ai-outcomes-and-the-operator-skill-gap.pdf, FRITS AI ApS. Self-published and not peer-reviewed. Web version: https://frits.ai/research/bimodal-ai-outcomes-and-the-operator-skill-gap/
No takes yet. Share an insight, caveat, or question.
Frits Lyneborg (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: