Contact
← Back to the blog

Blog

AI ROI Needs Process Metrics, Not Activity

· thinkai.at costgovernanceautomationanalysis

AI ROI cannot be inferred from active users, consumed tokens, or generated commits. OpenAI and AWS describe two different measurement approaches this week that lead to the same management question: what has actually improved in the business process?

Usage is only the starting point

OpenAI brings usage and spending analytics for ChatGPT Work and Codex together. Administrators can examine adoption, credits, and token consumption by group or user. According to OpenAI, these data can reveal training needs, underused integrations, and areas of concentrated cost.

The vendor also draws an important boundary: product activity alone does not measure business value. Business owners need to add what changed in the workflow and whether delivery time, quality, or profitability improved. A licence dashboard is therefore not yet evidence of ROI.

Engineering data needs a baseline

AWS applies the same idea to software delivery. A serverless architecture collects GitHub and GitLab metrics, processes them with AWS Step Functions, and displays them in Amazon Quick Sight. The design uses change detection, incremental runs, and a full refresh every 24 hours.

The architecture can expose activity. It does not automatically show whether AI accelerates development, merely increases commit volume, or introduces additional quality issues. AWS therefore recommends establishing a baseline before deploying coding tools and monitoring changes continuously afterwards.

Four layers for credible measurement

For an AI pilot, separate four layers:

  • Adoption: active users, functions used, and recurring workflows.
  • Effort: time, model cost, integration cost, and human review.
  • Process: cycle time, waiting time, rework, and handoffs.
  • Outcome: error rate, on-time delivery, quality, or financial contribution.

Define a baseline and a business owner before rollout. Do not compare only teams with and without AI; where possible, compare the same process before and after a clearly documented change. When usage data relates to individuals, its purpose, access, and retention belong in governance. Ranking individual employees by product activity is not a neutral side effect.

The decisive metric is therefore not how much AI was used. It is this: which outcome demonstrably improved, and at what total cost?

← Back to the blog