The first wave of AI in software development lived inside the editor. Coding assistants completed lines, answered questions and drafted functions while a developer watched. By 2025 that wave had reached saturation: DORA’s survey of nearly 5,000 professionals found 90% using AI at work, for a median of two hours a day (Google Cloud, DORA 2025).

The second wave is different in kind. A background agent is given a task, works unattended in a sandbox with access to the repository, the build and the tests, and comes back with a pull request. Nobody watches it type. And the organisations getting the most out of it are not pointing one agent at one repository. They are pointing one agent at hundreds of services at once, using their service catalog to decide where it runs.

That combination, unattended and fleet-wide, is where the strongest public results sit: Google halving migration time, Spotify merging more than 1,500 agent-generated pull requests, Airbnb finishing an eighteen-month migration in six weeks, Meta landing a quarter of its repair agent’s fixes. None of these is a story about an agent inventing a product feature. All are about work that is bounded, mechanically verifiable and repeated across a large codebase, which is exactly the work engineers like least.

This article walks through the kinds of agents these organisations are building, what each achieved, the architecture they converged on, what the evidence supports and what it does not, and where the practice is heading.

What people are building

Migration and upgrade agents

Framework and dependency migrations are the best-documented use case, because they are large, dull, and easy to verify.

Google built an LLM-based migration toolkit and used it across Ads, Search, Workspace and YouTube. One project changed 32-bit identifiers to 64-bit across a codebase of more than 500 million lines, work that “if done manually was expected to require hundreds of software engineering years” (How is Google using AI for internal code migrations). The toolkit finds candidate locations with code search, asks the model for edits, and then runs a fixed validation chain: did the model finish, did it only change whitespace, does the file still parse, was the change necessary at all, does it build, do the tests pass. Anything that fails goes to a human. A year-long follow-up study covering 39 migrations and 595 submitted changes found that 74% of the changes and 69% of the individual edits were generated by the model, and developers estimated a 50% reduction in total migration time (Migrating Code At Scale With LLMs At Google).

Spotify already had a Fleet Management platform that applied scripted code transformations across thousands of repositories and opened pull requests automatically; since mid-2024 around half of all Spotify’s pull requests have come from it. Its limit was that every transformation had to be written as a program, and the Maven dependency updater alone had grown to 20,000 lines to handle corner cases. In 2025 the team replaced the transformation script with a prompt and a coding agent, internally called Honk, and left everything else (targeting, sandboxing, pull requests, review, merge) unchanged. Within months more than 1,500 agent-generated pull requests had been merged, covering language modernisation, upgrades with breaking changes and framework migrations, with reported time savings of 60 to 90% compared with writing the changes by hand (1,500+ PRs Later). By mid-2026 a Java migration across Spotify’s backend services took three days, and the platform team’s summary was that “what used to be hundreds of teams doing migrations for their components, taking weeks and weeks or months, now can be done by a single engineer in a few days” (Coding Is No Longer the Constraint).

Airbnb migrated nearly 3,500 React test files from Enzyme to React Testing Library. The manual estimate was 1.5 years of engineering time; the LLM-driven pipeline finished in six weeks. The team’s approach is instructive because it was deliberately unsophisticated: each file moved through a state machine of validation and refactor steps, failures were simply retried with richer context (up to 50 related files, good examples, component-specific rules), and after four days of prompt tuning 97% of files were converted. The remaining hundred or so were fixed by hand, because pushing automation to 100% was not worth it (Accelerating Large-Scale Test Migration with LLMs).

Brex runs Micronaut across more than 500 Kotlin microservices. A manual attempt to move 50 of them from Micronaut 3 to 4 consumed three engineers for a full quarter and stalled product work. The team then built agent automations for the migration and reports a 32-fold acceleration (Agent automations for big migrations).

PagerDuty offers the small-scale version: a product manager, not an engineer, migrated the Rundeck codebase (5,100 tests) from Grails 6 to 7 in nine days against a consultant quote of six months, capturing 44 reusable migration patterns on the way (How AI tools accelerated a 6-month framework migration to 9 days).

Vendors have productised the pattern. Amazon reports migrating tens of thousands of internal applications from Java 8 and 11 to Java 17 with its Q Developer transformation agent, claiming 4,500 developer-years saved, the average upgrade dropping from about 50 developer-days to a few hours, and 79% of the generated changes shipped without edits (AWS). Microsoft ships GitHub Copilot app modernization for .NET, which assesses an application, generates an upgrade plan, applies transformations step by step and learns from the developer’s corrections (Microsoft). These are vendor figures rather than independent studies, but they describe the same shape of work as the engineering reports above.

One academic result keeps the enthusiasm honest. A study presented at ICSE 2026 ran GitHub Copilot’s agent mode on a Python library migration across ten client applications and found that the agent migrated 100% of the API usages in the median case, but the applications passed only about 40% of their tests afterwards (Using Copilot Agent Mode to Automate Library Migration). A migrated diff is not a migrated application. Every successful programme above treats the test suite as the gate.

Dependency and vulnerability agents

Renovate and Dependabot have opened version-bump pull requests for years. What agents add is the part scripts could not do: fixing whatever breaks after the bump. JPMorgan’s AI research group published a three-agent framework (a summary agent that reads migration documentation, a control agent, and a code agent) that locates library usages in Java codebases and applies the recommended changes, reaching 71% precision on their benchmark while using fewer tokens than competing approaches (LLM Agents for Automated Dependency Upgrades).

On the vulnerability side, agents are moving from suggesting fixes to finding flaws. GitHub’s Copilot Autofix attaches a proposed fix to CodeQL code-scanning alerts (GitHub). Google’s Project Zero used an agent it calls Big Sleep to find a previously unknown, exploitable memory-safety bug in SQLite, the first public case of an AI agent finding a real vulnerability in widely used production software (Project Zero). In 2026 Anthropic launched Claude Code Security and then Claude Security, which scan a codebase by reasoning about data flow rather than matching patterns, verify their own findings to cut false positives, and propose patches for human review; Anthropic reported more than 500 vulnerabilities found in production open-source codebases during the preview (TechInformed).

Test generation and test-failure repair agents

Meta was early and rigorous here. TestGen-LLM generates candidate unit tests and then applies hard filters: a test is only kept if it builds, passes reliably on repeated runs, and measurably increases coverage. In deployment on Instagram and Facebook, 75% of generated tests built correctly, 57% passed reliably, 25% increased coverage, and 73% of the improvements that survived the filters were accepted by engineers (Automated Unit Test Improvement using LLMs at Meta). A later system used mutation testing to aim the generated tests at faults that matter (Mutation-Guided LLM-based Test Generation at Meta).

Meta then went from writing tests to fixing the code that fails them. Its Engineering Agent picks up a failing test from a rule-based triage bot, sets up the environment, reasons with a ReAct loop over fifteen tools from reading files to running builds, refines its patch using static-analysis and test feedback, and passes the result to a separate model acting as judge before a human sees it. Over three months in production, 80% of the generated fixes were reviewed and 25.5% of all generated fixes were landed. Two production lessons stand out: engineers valued partially correct fixes as starting points, and flaky tests forced the team to give the agent a fully isolated environment (Agentic Program Repair from Test Failures at Scale).

Google measured the same task on 178 bugs from its internal tracker with a deliberately minimal agent called Passerine and found a sharp split: plausible patches for 73% of machine-reported bugs (sanitizer and test-order findings) but only 25.6% of human-reported ones, of which 17.9% were semantically correct on inspection (Evaluating Agent-based Program Repair at Google). The difference is not the model; it is the information in the report.

Bug reproduction agents

The challenge for repair agents described above can be overcome by reproducing a bug first, so the agent can extract all the information it needs to create a fix. A bug that can be reproduced by a failing test is a bug an agent can fix and a human can verify.

Google’s BRT Agent takes a human-written bug report, gathers context, writes a candidate bug-reproduction test, runs it, and iterates. On 80 human-reported bugs it produced plausible reproduction tests for 28%, against 10% for the previous state of the art. More importantly, feeding those tests to the repair agent produced 30% more bugs with plausible fixes, and using the tests to rank candidate fixes selected a correct one from a pool of 20 in 70% of cases (Agentic Bug Reproduction for Effective Automated Program Repair at Google).

Academic work is converging on the same design: Issue2Test generates reproducing tests from issue reports (arXiv); AEGIS, presented in the FSE 2025 industry track, adds structured context summarisation and a state machine to guide script refinement (FSE 2025); and researchers working with Mozilla combined LLMs with search-based test generation to reproduce issues in Firefox-scale code (arXiv). The practical picture for a team adopting this: expect the agent to reproduce a minority of human-reported bugs in its first year and the large majority of machine-reported failures, and expect the reproduction rate to rise as reports get more structured.

Code review agents

Review agents appear in two roles. The first is reviewer of human pull requests: a bot that comments on every pull request against the team’s standards and security checklist, never blocking on its own. Anthropic’s Claude Code GitHub Action provides this pattern out of the box, triggered by mentioning the agent on a pull request or issue (GitHub), and GitHub offers the equivalent with Copilot code review.

The second role is judge of other agents. Spotify’s harness evaluates every agent-produced diff with a judge model before the pull request is opened; Meta’s Engineering Agent does the same before a human sees a patch; Google’s toolkit asks the model whether its own change was necessary at all. The judge exists to protect human reviewers, because review is where agent programmes stall. Faros AI’s telemetry across more than 10,000 developers found that teams with high AI adoption merged 98% more pull requests but spent 91% longer reviewing them, with pull requests 154% larger (Faros AI).

Incident investigation agents

The newest category, and the one where the vendors are ahead of the engineering blogs. Datadog’s Bits AI SRE behaves like an on-call engineer: when an alert fires it reads runbooks and telemetry, forms hypotheses, tests them against the data, and posts a conclusion to the incident channel before the human responder has logged in; Datadog claims teams cut time to resolution by up to 95% (Help Net Security). Microsoft launched Azure SRE Agent in preview to do the same for Azure workloads (Microsoft). Elastic shipped an agentic Kubernetes investigation workflow that runs diagnostics the moment an alert fires, and exposed the same capabilities as skills that engineers can use from Claude, Cursor or any MCP-compatible client (Elastic).

The common design across all three is that the agent investigates and reports; it does not remediate. That is what makes it low-risk to adopt, and no independent evaluation of any of them has been published yet.

Documentation and knowledge agents

Once a background agent exists, engineers find uses for it that nobody planned. Spotify exposed Honk through its chat tool and found engineers using it to turn a Slack discussion into an architecture decision record committed to the repository, and product managers using it to propose small changes without cloning or building anything. Honk v2 adds shared agent sessions and team projects, with the platform team describing a future in which “agents collaborate with multiple developers and teams, not just one person at a terminal” (Spotify). The broader pattern is an agent that keeps documentation consistent with the code and answers “how does this service work” from the repository rather than from a stale wiki.

Quality and health sentinels

Continuous assessment of services against quality dimensions (code health, security posture, reliability readiness) is where the ambition runs ahead of the published evidence. Scorecards in internal developer portals are an established practice; the agentic addition is explaining findings, drafting the mechanical fixes (a missing CODEOWNERS file, an unpinned action, scanning switched off) as pull requests, and reasoning across signals that scripts cannot combine. Scheduled security scans with human-reviewed patches, as in Claude Security, are the closest productised example. For something like disaster-recovery readiness there is, at the time of writing, no published account of an agent doing the work, which makes it a candidate for deterministic checks with a model writing the narrative rather than a proven agent pattern.

How the leaders wire them together

Read the Google, Spotify and Meta reports side by side and the same architecture appears, whatever the agents are called.

A catalog decides where agents run. Spotify’s agents are targeted through Backstage and Fleet Management; Google’s through its code index and code search. Fleet-wide work is impossible without a system of record that knows which services exist, who owns them, how they build and how they deploy.

Deterministic tooling goes first, the model handles the residual. Google interleaves AST-based transformations with model edits; Meta’s Java-to-Kotlin conversion wraps a deterministic converter in model-assisted pre- and post-processing (Meta); Spotify keeps Renovate-style bumps as scripts and gives the agent the breakages.

The agent works in an isolated sandbox. Meta learned this from flaky tests polluting results; Anthropic describes the same approach of ephemeral, network-restricted environments for its own coding agent (Claude Code sandboxing). Isolation is a quality control as much as a security control.

A verification chain sits between the agent and any human. Build, tests, analyzers, a scan, and a judge model. Google’s six checks and Meta’s judge are the reference implementations. Anything that fails goes to a human with the agent’s diagnosis attached.

The pull request is the unit of work, and a human merges it. Spotify, Meta and Google all stop at the pull request. Amazon’s claim that 79% of its upgrade pull requests shipped unchanged is the closest anyone has come to publishing a number that would justify going further.

The harness is pluggable. Spotify built a small internal CLI that can delegate to any agent or model, run linting through the Model Context Protocol, judge diffs and capture traces, and has swapped components several times in a year. Anthropic’s own guidance points the same way: prefer simple, composable workflows with a model step over open-ended autonomous agents wherever the task allows (Building effective agents). The cautionary evidence is consistent: a Berkeley study of multi-agent systems attributed most failures to specification and coordination problems rather than model limits (Why Do Multi-Agent LLM Systems Fail), and Anthropic’s multi-agent research system consumed about fifteen times the tokens of a single conversation and only paid off for open-ended research tasks (Anthropic). Fleet maintenance is not that kind of task, and the successful programmes chain single-purpose agents through artifacts, not through conversations between agents.

Everything is visible. Spotify’s engineers watch agent activity on a dashboard nicknamed the Goose Farm, where every migration shows how many pull requests exist, how many merged and which need attention. Visibility turns a clever tool into an organisational capability.

What the evidence actually says

The results that hold up are the ones with published numbers from the teams that did the work: Google, Spotify, Airbnb and Meta above. All are self-reported, none has a control group, and all concern maintenance, migration, testing and repair rather than feature development.

The vendor figures (Amazon’s 4,500 developer-years, Datadog’s 95%) describe real capabilities but should be read as marketing until someone outside the vendor measures them.

Two caveats deserve a place in every leadership conversation on this topic. First, developers overestimate their own speed-up. METR’s randomised controlled trial gave 16 experienced open-source maintainers 246 real tasks and found they took 19% longer with early-2025 AI tools, while believing afterwards they had been 20% faster (METR). The setting differs from background agents, but the lesson transfers: measure outcomes, not sentiment. Second, the gains move the bottleneck. DORA found in 2024 that heavier AI adoption correlated with slightly lower throughput and a 7.2% drop in delivery stability, and concluded in 2025 that AI amplifies whatever engineering fundamentals an organisation already has (DORA). Faros’s 98% more pull requests against 91% longer reviews is the same finding in telemetry. The organisations above succeeded because they built for the review queue, not in spite of it.

Ten lessons from those who have done it

1. The pipeline is the product; the prompt is the cheap part. Spotify’s biggest win came from changing the smallest piece: it swapped the scripted transformation step of its existing Fleet Management platform for a prompt and an agent, and kept targeting, sandboxing, pull-request creation, review and merge exactly as they were. Google’s migration toolkit and Airbnb’s file-by-file state machine have the same shape. Organisations that start by writing prompts and hope the surrounding machinery will follow tend to stall; those that build the harness first can swap prompts, and models, at will.

2. The test suite is the gate, not the diff. When researchers ran GitHub Copilot’s agent mode on a library migration, it rewrote every API usage in the median case and the applications then passed only about 40% of their tests. A diff that looks complete is not a migrated application. Google only submits a change that has passed parsing, building and testing; Meta only keeps a generated test that builds, passes repeatedly and adds coverage. Whatever the agent produces, nothing reaches a human until the existing checks have passed.

3. Retry with richer context; do not chase the last few percent. Airbnb’s migration reached 75% of files in four hours, 97% after four days of adding examples and related files to the prompt, and stopped there. The remaining hundred files were fixed by hand because automating them would have cost more than doing them. Plan for a “needs a human” bucket from the first day, and treat the files in it as the natural end of automation rather than as failures.

4. Scripts first, model second. Google interleaves abstract-syntax-tree transformations with model edits; Meta wraps a deterministic Java-to-Kotlin converter in model-assisted pre- and post-processing; Spotify still uses Renovate-style scripts for plain version bumps and gives the agent only the breakages. Deterministic tools are fast, cheap and correct by construction for the cases they cover. The model earns its cost on the long tail the rules never anticipated.

5. Machine-reported failures are far easier than human-reported bugs. Google’s repair agent produced plausible patches for 73% of bugs found by tools such as sanitizers and test runners, but for only 25.6% of bugs reported by people, because human reports leave out what the agent needs to reproduce the problem. Route test failures, crashes and scanner findings to agents first, and invest in bug reproduction, which raised Google’s fix rate on human-reported bugs by 30%, to reach the harder half.

6. Isolate the agent. Meta had to give its repair agent a fully isolated environment after flaky tests and shared state corrupted its results, and Anthropic runs its own coding agent in ephemeral, network-restricted sandboxes. Isolation is usually presented as a security measure. In practice it is also what makes the agent’s results trustworthy, because a green test in a shared environment proves very little.

7. Protect the reviewers. Telemetry from Faros AI across more than 10,000 developers shows that teams with high AI adoption merge 98% more pull requests and spend 91% longer reviewing them, with pull requests 154% larger. The review queue is where agent programmes die. Every mature programme responds the same way: small pull requests, build and test evidence attached, a model acting as judge before a human looks, and pull requests batched per team rather than streamed.

8. Keep the harness model-agnostic. Spotify built a small internal command-line tool that can hand a task to any agent or model, and has swapped components several times within a year as better options appeared. Models improve every few months and prices move with them. Tying the platform to one model, or one vendor’s agent, means rebuilding the platform each time the market moves.

9. Measure outcomes, not adoption. In METR’s controlled trial, experienced developers took 19% longer with AI tools while believing they had been 20% faster, and DORA found in 2024 that heavier AI use went with slightly lower throughput and a 7.2% drop in delivery stability. Usage counts and satisfaction surveys say nothing about whether the organisation is delivering better. Lead time, deployment frequency, change failure rate, time to restore and merge-without-edit rates do.

10. Agent incidents come from privilege plus untrusted content. In 2025 a prompt injection hidden in a public GitHub issue made a coding agent leak data from a private repository (Invariant Labs); a compromised pull request shipped destructive instructions in a widely used coding extension; and an agent deleted a production database during a code freeze. In each case the agent could read content it should have treated as data and could reach systems the task did not require. The OWASP Top 10 for Agentic Applications now catalogues these patterns (OWASP). The design response is least privilege per task, credentials scoped to one repository, no production access for code-changing agents, and read-only operations agents that investigate but never act.

Where this is heading

Autonomy is being earned one change class at a time. Every serious programme still ends at a pull request that a human merges, and the leaders describe that as a design choice. The direction of travel is visible in the metrics they publish: merge-without-edit rates, post-merge incidents, reviewer minutes. When a class of change (a patch-level dependency bump with green tests, say) reaches an accuracy a human reviewer no longer improves on, the merge gate for that class will move. Nobody has published that step yet.

Agents are becoming users of the developer platform. Spotify’s platform team now states that “developer experience is for agents, too”: the same golden paths, build containers, standards and documentation that make a new engineer productive make an agent productive, and improvements to one serve the other. Agents that read a service catalog, follow a repository’s CLAUDE.md, and open pull requests through the normal pipeline are simply another class of contributor.

Security is becoming a discipline of its own. The 2025 incidents, from the GitHub prompt-injection leak to a coding assistant extension shipping destructive instructions and an agent deleting a production database during a code freeze, have produced the first standards. Expect agent identity, per-repository credentials, sandboxing and audit trails to become as unremarkable as branch protection.

And some things remain unproven. No public study shows agents fixing more than half of human-reported bugs. No one has published an agent that assesses disaster-recovery readiness. Autonomous incident remediation exists only as vendor demonstrations. Feature development at fleet scale is not in any of the engineering reports. The organisations that are winning with agents are the ones that put them on the work that is boring, verifiable and everywhere, and let the humans keep the rest.

Sources

Migrations and upgrades

Tests, repair and reproduction

Security, review and operations

Architecture and evidence