The strongest argument for AI code review is not that it comments faster. It is that it may see a connection across services that no individual reviewer holds in working memory. Datadog tested that proposition against the hardest available evidence: pull requests that had already passed human review and contributed to real incidents.
Its AI Development Experience team integrated OpenAI Codex into one of the company’s busiest repositories, where the agent reviews every pull request, compares the stated intent with the code, and runs tests.
The deployment shifts evaluation away from generic productivity claims. Datadog asked whether the system could have prevented failures that mattered to customers.
System Context Separates Review From an Advanced Linter
Traditional static analysis is effective at deterministic rules, syntax problems, and known patterns. Earlier AI review tools often added little more, generating noisy suggestions without understanding how a change affected the wider architecture.
Datadog needed reasoning across dependencies. A local edit can alter assumptions in another service, leave cross-service coupling untested, or interact with a module the developer never touched directly. These risks become harder for human reviewers to track as a codebase and organisation grow.
The AI agent can spend consistent attention on that surrounding context. It does not become tired by routine pull requests or limit its review to the visible difference. Human engineers still contribute knowledge of intent, architecture, and acceptable trade-offs that a broad scan cannot guarantee.
Historical Incident Replay Creates a Credible Test
The team reconstructed earlier pull requests known to have caused production incidents and ran the agent against them. This “incident replay harness” asked a concrete counterfactual question: would the reviewer have identified the risk before the code shipped?
The agent surfaced more than ten cases, approximately 22 percent of the incidents examined, where its feedback might have prevented the error. These changes had already escaped human review, making the result more relevant than a collection of synthetic bugs.
The percentage also provides useful restraint. The tool did not catch everything, and a warning is not the same as prevention; a developer must understand and act on it. Replay can reveal false negatives, false positives, and the types of failure where AI adds the most value.
Historical testing should be refreshed as architecture changes. An agent tuned to yesterday’s incidents may overlook new services, dependencies, and deployment patterns. Recent near-misses can supplement confirmed outages and keep evaluation connected to emerging risk.
Human Review Moves Toward Architecture and Judgement
Datadog expanded the system to more than 1,000 engineers. Users reported that it found missing coverage and interactions outside the immediate change. Brad Carter, who leads AI DevX, described preventing incidents as more compelling at Datadog’s scale than general efficiency.
When automation carries part of the cross-codebase search, human reviewers can focus on design, maintainability, product intent, and architectural choices. That is augmentation only if teams continue to challenge automated comments. Excessive low-quality feedback would create alert fatigue and weaken the gate.
Organisations adopting a similar approach should build evaluation from their own incident history. The valuable error patterns, services, and risk tolerance differ by company. Metrics should include accepted findings, prevented regressions, false-positive burden, and production outcomes—not only review time.
Security and access also matter because the reviewer needs broad code context and may execute tests. Permissions, data handling, generated changes, and audit logs must be governed like any developer with substantial repository access.
Teams should preserve human override and make automated comments easy to rate. That feedback helps distinguish findings engineers repeatedly value from patterns that create noise, allowing the review policy to improve without silently suppressing difficult warnings.
Datadog’s experiment reframes code review as a reliability system. AI earns a place not by producing the most comments, but by preventing a class of failures that scale has made difficult for people to see alone.
