AI features attract a halo: leadership assumes they're working because they're exciting. Honest measurement often tells a different story, and you can't manage what you flatter.

The honest scorecard

  • Adoption depth, not just access. What fraction of users actually use it, repeatedly — not how many could.
  • Fully-loaded cost. Inference, retries, and infra per use. Many "successful" AI features are margin-negative.
  • Outcome impact. Did it move retention, conversion, or expansion — or just generate activity?
  • Counterfactual. Would the outcome have happened anyway? Be skeptical of attribution that's too convenient.

Bottom line: judge AI features on adoption, true cost, and real outcome impact — the halo hides as many money-losers as it does wins.