Harvard Study: AI Coding Agents Shift Bottleneck to Review

Introduction

On October 9, 2026, Ars Technica's Kyle Orland reported on a Harvard University working paper by Fiona Chen and James Stratton titled "Artificial Intelligence in the Firm: Bottlenecks in Software Production." The paper is a job market paper, with a current version dated August 4, 2026 (the initial version is dated January 7, 2026). It is a working paper and has not been peer-reviewed or published in a journal.

The headline finding is a mismatch. AI coding agents increase how much code engineers produce, but firm-level output, measured through resolved Jira issues and epics, shows small positive effects that are not statistically significant. The authors argue that code review is a bottleneck: more code arrives at review, and each unit of it takes more review effort.

Data and Method

The authors use a proprietary dataset from Jellyfish, an engineering analytics platform. According to the paper, it covers about 300 million work events and about 700,000 workers at 718 firms (the abstract's figure; the introduction says approximately 700), from January 2021 to March 2026. It combines five data sources:

• HR information
• Version control (GitHub)
• Issue management (Jira)
• Scheduling (Google Calendar)
• Generative AI usage for GitHub Copilot, Cursor and Claude Code

The method is a staggered difference-in-differences design. It exploits differences in when firms adopted AI coding assistants versus AI coding agents, so the treatment is firm-level adoption rather than individual use. The authors say this is probably why their estimates sit on the lower end of the prior literature.

Key Findings

More code, more commits, more pull requests

For assistants, just the commits effect is statistically significant. For agents, all three effects are significant.

Firm output barely moves

Jira issue resolution and Jira epic resolution show small positive but statistically insignificant effects for both assistants and agents. For agents, the estimates rule out an output increase larger than 12% of the baseline mean. A machine-learning prediction of issue length showed no change, so the result is not explained by teams shifting toward bigger or more complex issues.

Employment effects are not detectable

The authors cannot attribute significant employment changes to AI. For agents, they find a precise zero on overall employment (measured with LinkedIn data), ruling out a decline greater than 2.9%, and a noisier zero on engineering employment (active Jellyfish workers), ruling out a decline greater than 13.8%. For assistants, the engineering-employment result is ambiguous because of differential pre-trends.

The review bottleneck

After firms adopt agents, the average time to review a pull request rises 49%. The share of pull requests with changes requested nearly doubles, comments per pull request rise 35%, and the share of workers performing code reviews rises 14%. The authors report that the bottleneck persists after firms adopt AI code review tools, and that human reviewers remain central.

The Authors' Model

The paper models software production as two complementary stages: code writing and code review. AI enters through the writing stage in two ways. It raises coder productivity, which increases the volume of code reaching review, and it changes the quality of the draft code entering review, which changes review effort per unit of code. Higher productivity can reduce demand for coding labor, while quality changes can raise demand for review labor.

Engineers spend 30-40% of their time writing code, according to the authors' estimate imputed from Jellyfish data and validated with a survey of 100 Prolific engineers. Our reading is that when writing is a minority share of the work week, speeding it up has limited reach on its own, in the manner of Amdahl's law. That framing is an Evermx interpretation, not a claim from the paper.

Practical Implications

For engineering managers, the paper suggests that counting commits or pull requests after an AI rollout can overstate progress. Measures closer to completed work, such as resolved issues and epics, tell a different story in this dataset. Review capacity, review load per engineer and the rate of changes requested are the numbers the paper points to as worth watching.

One caution is editorial and not a source claim: the findings are firm-level averages, and individual teams can differ.

Strengths and Limitations

The study's strengths are its scale, its combination of version control, issue tracking, HR and calendar data, and its explicit attempt to rule out large effects instead of reporting point estimates alone. Its limits are that it is a non-peer-reviewed working paper, relies on one vendor's data, and measures adoption at the firm level. Jellyfish reviewed the manuscript for proprietary-information disclosure, not scientific content, and the conclusions are the authors' own. Funding includes the Stripe Economics of AI Fellowship and the Schmidt Sciences AI at Work Program.

Outlook

The model implies two levers: reduce the review effort each unit of AI-written code requires, or expand review capacity. The paper's observation that review remains a constraint even with AI code review tools in place suggests that tooling alone has not closed the gap in this sample. Later versions of the paper and independent replications will show whether the pattern holds as agent tooling changes.

Conclusion

This is a carefully scoped economics paper with a clear message: agents raise code volume, pull requests and review burden, while firm-level output gains are small and statistically insignificant. It suits engineering leaders, platform teams and researchers tracking AI productivity claims. We rate it 4 out of 5 for evidence quality and relevance, tempered by its working-paper status.

Editor's Verdict

Harvard Study: AI Coding Agents Shift Bottleneck to Review earns a solid recommendation within the research space.

The strongest case for paying attention: a dataset covering roughly 700,000 workers and five data sources, from January 2021 to March 2026, gives the estimates broad coverage. That alone raises the bar for what readers should expect in this space. Reinforcing that, the staggered difference-in-differences design uses variation in when firms adopted tools, not just simple before-and-after comparisons — practical value rather than just headline appeal. The broader signal worth registering is straightforward: the gap between code volume and finished work is the central result — agents raised pull requests by 23%, yet Jira issue and epic resolution showed no statistically significant gain. On the other side of the ledger, one constraint is real rather than a marketing footnote: the paper is a non-peer-reviewed working paper, so its estimates have not been through journal review. It should factor into any serious decision. Layered on top of that, a firm-level treatment definition means the effects cannot be read as the gain any single engineer would see — which narrows the set of teams for whom this is an obvious yes.

For engineering managers, platform teams, and researchers tracking AI productivity claims, this paper is worth reading closely, not just bookmarking. For everyone else, the safer posture is to watch for later versions of the paper and independent replications before drawing firm conclusions.