How to build an AI-native software factory

Overview

Agent utilization at Uber reveals a complex landscape where the deployment of managed agents and automated migrations has not matched the anticipated adoption pace, despite significant advances in technology. Meanwhile, while engineers' productivity has seen a modest increase driven by large language models (LLMs), the associated costs have spiraled dramatically, straining the company's AI budget and leading to inefficiencies in high-volume review processes, as highlighted by similar challenges faced at Spotify.

Key points

Agent Utilization

• Each of the four limits necessitates building the factory independently of the coding tool's model.
• Uber evaluates the cost of agents based on the amount of work they complete.
• Managed agents initiate tasks without human intervention.
• Despite high agent usage, adoption at Uber has advanced slower than anticipated.
• Uber has executed over 250 automated migrations affecting 9 million lines of code.

Productivity and Costs

• Engineers' productivity increases by 10% due to LLMs, with a cost increase of $1,000 per month.
• Uber's AI expenses surged sixfold since 2024 by March, exhausting its entire annual AI budget in just four months.
• Cost is measured by various outcomes, including merged pull requests and cleanups.

Review Processes

• Review is currently the most time-consuming phase for teams, with increased code volume not expediting review processes if human approval is required.
• Spotify experienced a significant rise in pull requests after implementing AI, with a reported 441% increase in median review time among developers.