Amazon AI Token Metrics Flawed

Amazon told its engineers this week that counting AI tokens and lines of generated code is a "deeply flawed" way to measure success.

That is a notable admission from the company that shut down an internal AI usage leaderboard in May after employees gamed it to climb the rankings.

Here is the pattern I keep seeing in enterprise AI rollouts. Leadership wants proof of adoption, so it measures what is easy to count: logins, prompts, tokens, code generated. People respond rationally and optimize the number. Costs and complexity go up, and the outcome you wanted does not move.

Amazon's own memo says these metrics can raise costs and complexity, and that long-term outcomes matter most. An AWS vice president said the company looks at feature velocity: launches and successful production changes.

The lesson reaches well beyond engineering. If your AI scorecard measures activity, you will get activity. If it measures shipped work, resolved cases, cycle time or margin, you get a business case.

The companies that win this phase will be the ones that stop asking how much AI their teams use and start asking what it changed.

What is the one outcome metric you would trust to tell you whether AI is actually working in your organization?