# How Olakai Measures AI ROI Olakai helps organizations answer a core business question: **Is AI creating enough value to justify the investment?** Most companies can see AI adoption growing, but far fewer can explain which investments are working, which tools are worth renewing, which agents are worth scaling, and whether AI coding tools are improving engineering output. Olakai closes that gap by measuring AI ROI across three distinct categories, each with a measurement model suited to the kind of work it represents. *These three categories are not separate products. They are delivered by the IQ modules inside Olakai's two products: **Assistive IQ** is the module of **Olakai Assistive**, while **Agent IQ** and **Coding IQ** are the two modules of **Olakai Agentic**. Which ones you see depends on which products you own — see [Core Concepts & Glossary](/docs/olakai/concepts-glossary) for the product structure.* ## Three Categories, Three Models Olakai does not force all AI into one simplistic formula. The business question is shared — "is this AI worth it?" — but the measurement approach differs because the underlying work differs. | Category | Module (Product) | What it covers | How value is measured | How cost is measured | ROI expression | |----------|-----------------|---------------|----------------------|---------------------|----------------| | **Assistive AI** | Assistive IQ (Olakai Assistive) | Employee-facing AI tools and browser-monitored AI apps | Estimated time saved per interaction, converted to dollar value | Subscription and licensing cost for the apps being used | Value created vs app/tool spend | | **Agentic AI** | Agent IQ (Olakai Agentic) | Custom AI agents, workflows, and autonomous systems | Estimated time saved per session, converted to dollar value | Execution cost from token usage and model pricing | Value Created / Execution Cost (shown as an x-multiplier) | | **Coding IQ** | Coding IQ (Olakai Agentic) | AI coding tools used by engineering teams | Measured change in the team's combined lines of code shipped per week after adoption | Fully loaded engineer cost benchmark | AI Equivalent Engineers + dollar value for the selected timeframe | --- ## Assistive AI Assistive AI covers the tools employees use directly in their daily work: chat assistants, copilots, and AI-powered SaaS such as ChatGPT, Claude, Gemini, Microsoft Copilot, and Perplexity. ### How value is estimated Olakai analyzes each AI-assisted interaction and estimates how many minutes of human work it likely saved. That time-saved estimate is then converted into a dollar value using configurable wage assumptions. **Value Created = estimated minutes saved × hourly rate / 60** The default hourly rate is $55/hour. Analysts and Admins can configure this at the account level or per user group. For Shadow AI (browser-extension-monitored apps), time-saved estimation follows a precedence chain: 1. Analytics-based estimate, whenever Olakai could analyze the interaction 2. A per-app value your admin entered, when there is no analytics estimate 3. Olakai's estimate for that app 4. Conservative fallback of 5 minutes Many monitored apps share only usage metadata, not the conversation itself, so their time saved comes from steps 2–4. Those figures are per-interaction estimates rather than measurements. ### How cost is measured Assistive cost is centered on SaaS-style software economics, not per-interaction token cost: - **Subscription cost**: monthly cost per seat × active users, prorated to the reporting period - **Licensing cost**: monthly cost per seat × total provisioned seats, regardless of active usage This distinction matters because Assistive ROI is not only about whether usage creates value — it also reveals whether the organization is paying for tools that are underused or misaligned with policy. ### What the ROI tells you Assistive ROI answers: - Are employees getting enough value from AI tools to justify what we pay for them? - Which apps save meaningful time? - Are we paying for subscriptions people actually use? - Where is there productivity upside and where is there governance risk? --- ## Agentic AI Agentic AI covers custom AI agents, workflows, and autonomous or semi-autonomous systems integrated through the SDK or API. ### How value is estimated For each agent session, Olakai evaluates the full conversation and classifies how much time it likely saved. The default classification assigns sessions to one of five buckets: 0, 3, 10, 30, or 60 minutes. That time-saved estimate is then converted into a dollar value: **Value Created = classified minutes saved × hourly rate / 60** The default hourly rate is $55/hour. Each agent can have its own rate, configured by an Analyst or an Admin. ### How cost is measured Cost basis follows how the traffic is paid for, not what category it belongs to. Two cases apply to Agentic traffic today: - **Per-token-billed traffic** — SDK-instrumented agents calling the OpenAI/Anthropic APIs directly. Execution cost is estimated per session from: - total token usage - model-specific pricing when the model is recognized (fallback token pricing when not) - a fixed server cost per request - **Subscription-billed traffic** — local developer-driven agents running inside Claude Code, Cursor agent mode, and similar tools. There is no per-session API invoice; cost is paid as a flat monthly per-seat subscription. The meaningful cost lives at the **account × traffic-type** scope, not per session, so ROI for this traffic is reported at the aggregate scope rather than per individual session. Both cases are Agentic — the defining property is the shape of the interaction (multi-step, tool-using, session-scoped), not the billing model. The same per-token vs subscription distinction can apply to Assistive traffic too; cost basis is a cross-cutting concern. ### How ROI is expressed Agentic ROI is shown as a multiplier: **ROI = Value Created / Execution Cost** For example, if an agent creates $200 of estimated value and costs $10 to run, the ROI is 20x — meaning $20 of estimated value for every $1 spent. ### Governance alongside ROI Agentic ROI in Olakai is not purely financial. Governance compliance is tracked as a companion metric — each execution is assessed for risk, and executions below the risk threshold are counted as compliant. This means the platform answers not just "did AI create value?" but also "did it create value safely?" ### What the ROI tells you Agentic ROI answers: - Which agents create meaningful labor value? - Which workflows are cost-efficient? - Which agents are expensive relative to what they produce? - Which agents should be scaled, tuned, or governed more tightly? --- ## Coding IQ Coding IQ is structurally different from the other two categories. It does not estimate time saved per interaction or per session. Instead, it measures real engineering outcomes in GitHub. ### How value is measured Coding IQ imports merged pull requests from GitHub and analyzes: - whether AI tools were used (detected through commit co-author trailers, PR markers, bot authors, AI config files, and behavioral signals) - coding time, review time, and total cycle time - lines of code shipped per week, per developer and pooled to a team total (the primary productivity metric) - adoption patterns across teams The core value signal is observed delivery improvement — are developers shipping more code per week, and faster, after adopting AI tools? ### Primary methodology: Before/After The preferred method compares each developer against themselves, measured over two fixed 60-day calendar windows: 1. Find the developer's first AI-assisted PR (`firstAiDate`). 2. The "before" window is `[firstAiDate − 60 days, firstAiDate)`. 3. Exclude a 14-day learning curve buffer after `firstAiDate` (the first AI PR itself lives inside this buffer and is discarded from both sides). 4. The "after" window is `[firstAiDate + 14 days, firstAiDate + 74 days)`. 5. Compare **lines shipped per week** and average cycle time between the two windows. A developer needs at least 3 merged PRs in each 60-day window to qualify. The before/after method becomes primary when at least 2 developers qualify. This approach is strong because it avoids comparing different developers with different skill levels — each person serves as their own baseline. Fixed calendar windows make the before and after directly comparable regardless of when the developer entered the dataset, and measuring lines-per-week instead of PRs-per-week correctly credits developers who consolidate work into fewer but bigger PRs. ### Fallback methodology: Cohort Gap When insufficient before/after history exists, Coding IQ falls back to a cohort comparison: - **Power users**: more than 70% AI-assisted PRs - **Idle users**: less than 20% AI-assisted PRs The productivity gap between these groups is used to estimate unrealized engineering capacity. ### The headline metric: AI Equivalent Engineers The measured lines-per-week improvement is converted into a business metric: **AI Equivalent Engineers = team lines-per-week gain % × developers using AI** The team gain pools the qualifying developers' output — their combined lines/week before versus after — rather than averaging their individual percentages, so higher-volume developers carry proportionally more weight. The multiplier is the count of developers using AI, not total headcount: the gain is measured on AI users and is applied only to them. For a worked example and the dollar conversion, see [the AI ROI Report guide](/docs/olakai/coding-iq-roi-report). That figure is then converted to a dollar value for **the timeframe you have selected**: **Value = AI Equivalent Engineers × (fully loaded engineer cost / 365) × days in window** It is not a quarterly figure unless you have selected a quarter-length window — at the default 7-day timeframe it is roughly 1/13 of one. The default fully loaded cost is $200,000/year, editable in the report. ### Confidence levels Coding IQ starts from the number of qualifying developers: | Starting level | Qualifying developers | Interpretation | |-------|----------------------|----------------| | **High** | 15 or more | Strong statistical basis | | **Medium** | 5 to 14 | Reasonable signal | | **Low** | 3 to 4 | Preliminary indicator | | **Very Low** | Fewer than 3 | Insufficient data — treat with caution | It then **lowers that level when those developers disagree with each other**: if the middle half of their individual uplift figures span more than 25 percentage points the rating drops one level, and more than 75 points drops it two. A large sample is not automatically a consistent one, and only a consistent one is worth planning against. See [the AI ROI Report guide](/docs/olakai/coding-iq-roi-report) for how to read the spread. ### What the ROI tells you Coding IQ answers: - Are AI coding tools increasing the amount of code developers ship per week? - Are developers shipping faster after adoption? - Which tools correlate with the strongest delivery outcomes? - How much equivalent engineering capacity is AI adding, and what is that worth? > **Important**: Coding IQ ROI is not directly comparable to Assistive or Agentic ROI. It uses a different methodology (code-volume-based, not time-saved) and different units (equivalent engineers, not dollar multiplier). All three belong under "AI ROI," but they answer the question using fundamentally different measurement approaches. --- ## Side-by-Side Comparison | Dimension | Assistive AI | Agentic AI | Coding IQ | |-----------|-------------|------------|-----------| | Primary unit | Individual interaction | Agent session / conversation | Developer PR history | | Value signal | Estimated time saved per interaction | Classified time saved per session | Measured lines-shipped-per-week improvement | | Cost basis | Subscription and licensing cost | Token + model + server execution cost | Fully loaded engineer cost benchmark | | ROI expression | Value created vs app/tool spend | x-multiplier (Value / Cost) | AI Equivalent Engineers + $ for the selected window | | Data source | Interaction records + app metadata | Event pipeline + session KPIs | GitHub PR data | | Business question | Are AI tools worth what we pay? | Are agents creating value efficiently? | Are coding tools increasing output? | --- ## What Olakai Observes vs What It Estimates Understanding where observation ends and estimation begins helps set the right expectations. ### Assistive AI - **Observed directly**: which apps are used, by whom, how often, and what subscription/license assumptions are configured - **Estimated**: how much time each interaction saved and the resulting dollar value ### Agentic AI - **Observed directly**: execution activity, token usage, model identity, and governance signals - **Estimated**: time saved per session, value created, and cost when based on pricing assumptions rather than invoice data ### Coding IQ - **Observed directly**: PR history, cycle times, review behavior, and AI adoption signals - **Estimated**: AI attribution when relying on heuristics, productivity lift magnitude, and dollar conversion --- ## Configurable Assumptions Each ROI model depends on assumptions that Analysts and Admins can customize: | Assumption | Default | Where it applies | |------------|---------|-----------------| | Hourly labor rate | $55/hour | Assistive and Agentic value calculations | | Subscription cost per seat | Configured per app | Assistive cost calculations | | License seat count | Configured per app | Assistive licensing cost | | Fully loaded engineer cost | $200,000/year | Coding IQ dollar conversion | | Shadow AI time-saved fallback | 5 minutes | Assistive Shadow AI when no better estimate is available | Keeping these assumptions accurate improves the precision of ROI figures across all three categories.