Revised software engineer role guideline follows Amazon’s shutdown of its internal AI leaderboard over tokenmaxxing
Amazon has acknowledged that the metrics it uses to track AI adoption among its software engineers are flawed, while telling staff that the same measures remain an appropriate lens for their careers and their teams' performance.
The position is set out in a memo to Amazon's software development engineers (SDEs) and managers, which the company published on Tuesday with an introduction from Terk Terkowitz, VP and Distinguished Engineer for Worldwide Amazon Stores.
The memo accompanies a revised SDE role guideline, which senior engineering leaders developed by drawing on observations of the company's most successful engineers.
Rather than defining good AI practice, which leaders said would probably look outdated within a year, the guideline focuses on impact on customers, businesses and teams.
Input metrics under scrutiny
According to the memo, Amazon has in some cases relied on input metrics such as token usage and AI tool adoption, along with throughput metrics such as code changes and reviews, because long-term outcomes can be delayed and difficult to isolate.
"We are all aware of how deeply flawed these metrics are," the memo said.
Leaders warned that treating the measures as targets could drive up costs and complexity, describing tokenmaxxing as at odds with Frugality and measuring lines of code as at odds with Invent and Simplify, two of Amazon's leadership principles.
Despite those concerns, the memo said the input metrics have correlated well with the outputs that matter, and that leaders expect this to continue for some time.
"This effect is driven both by increasing productivity and delivery on teams that have embraced AI the most (we call them frontier teams), and that teams and individuals who are not adopting AI or increasing output metrics are falling behind," the memo read.
It concluded that using these metrics with care and judgement was appropriate when engineers think about their careers and their organisation's performance. Managers were advised to focus on the 10th percentile of AI adoption, to understand why those teams are not using AI effectively, and the 90th percentile, to check whether heavy usage is producing matching gains in delivery.
The memo does not say whether the metrics are used in formal performance reviews or promotion decisions.
It also told engineers that if current AI tools do not benefit their work, this should be treated as a finding rather than a failing, and that they should document the reasons and raise them with their manager.
"An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for," the memo said.
Leaderboard scrapped
The memo follows Amazon's decision earlier this year to shut down an internal AI leaderboard after employees were found to be tokenmaxxing.
In June, the ranking known as Kirorank scored staff on their activity on Kiro, Amazon's AI developer platform. Some employees assigned autonomous AI agents to perform unnecessary tasks to climb the rankings, consuming computing capacity that Amazon had to pay for.
"Please don't use AI just for the sake of using AI", senior vice-president Dave Treadwell told staff, as quoted by The Financial Times.
Amazon said in a statement at the time that the beta dashboard had not been a formal or approved tool and had since been deprecated.
The company has set a target for more than 80% of its developers to use AI each week, and has adopted a measure it calls normalised deployments, which tracks engineers regularly using AI to write useful code rather than counting raw token totals.
The memo said Amazon would continue to evolve the SDE role as AI's effect on its business and engineering practices becomes clearer, but that customer obsession and a long-term focus on delivering results would remain what matters.
