Of telemetry and why AI productivity gap is a measurement problem

Companies have spent the past few years giving employees access to AI. Yet many still cannot answer the most basic question: has it made the business more productive?

Employees say AI helps them write faster, analyse more data and complete tasks with less effort. Leaders see rising usage, growing licence costs and a steady stream of internal demos. But the gains often fail to appear in delivery times, output quality, revenue, customer satisfaction or operating costs.

This difference between apparent activity and measurable business value is the AI productivity gap.

The usual response is to improve the model, add more tools or train employees to write better prompts. Those steps may help, but they do not solve the main problem. Most firms have no clear view of how work gets done before or after AI enters the process.

They measure access. They measure usage. They may even count prompts. But they do not measure how AI changes the full path from intent to outcome.

That is where behavioural telemetry matters.

Most claims about AI productivity focus on task speed.

A worker drafts an email in five minutes rather than twenty. An analyst produces a first version of a report in an hour rather than a day. A developer creates code faster. These gains are real. Controlled trials and workplace studies have found clear gains in some tasks, with the largest gains often going to less experienced workers.

One field study involving thousands of support workers found an average productivity gain of about 14 per cent, with much larger gains among newer and lower-performing staff.

---

But task speed is not the same as business productivity.

The faster email may need more review. The report may contain claims that take longer to check. The code may increase the burden on reviewers or create faults that appear weeks later. The employee may use the saved time to produce more low-value work rather than finish a higher-value task.

This is why firms can see strong gains in a test but weak gains across the business.

AI changes the cost of producing a first draft. It does not remove the need for judgment, checks, approval, coordination or action. In many cases, it moves the main constraint from creation to review.

Before AI, the slowest part of a process may have been writing. After AI, the slowest part may be checking, choosing or getting approval. A firm that keeps measuring writing speed will report a gain while the full process remains unchanged.

Behavioural telemetry reveals this shift.

One of the worst measures of AI success is the number of active users.

High use can mean that a tool works well. It can also mean that employees need many attempts to get a useful result. A team generating ten times more AI output may be creating value, or it may be creating ten times more material for someone else to inspect.

Prompt counts, active days and generated words tell a company that people interacted with AI. They do not show that the interaction improved the result.

The same problem applies to time saved. Employees often estimate how long a task would have taken without AI. These estimates can help find broad trends, but they remain guesses. A claimed saving of five hours has little value unless the firm knows what happened to those hours and whether the final outcome improved.

The right unit of measurement is not the prompt. It is the completed work outcome.

That requires linking AI use to what happened before it, what happened after it and whether the work produced value.

The first is the adoption gap. Some employees use AI often, while others avoid it or use it only for basic tasks. A company-wide average hides this difference. Behavioural data can show which roles, tasks and teams have built AI into their real work rather than merely tried it.

The second is the skill gap. Access to the same AI does not produce the same gain for each worker. Recent research suggests that the ability to ask, test, filter and check AI output may explain much of the difference. People with strong AI interaction skills can gain far more, while weaker users may gain little or even lose time. Simple work guides can reduce this gap.

The third is the workflow gap. AI may speed up one stage but leave the rest untouched. Telemetry can show whether faster drafting leads to faster delivery or merely creates a queue at review and approval.

The fourth is the quality gap. More output can hide falling quality. Firms need to track error rates, customer response, rework, rejected work, policy breaches and later corrections alongside speed.

The fifth is the value gap. Even correct and fast work may not matter. AI may help employees create more reports, notes, summaries and messages without improving a decision or customer result. Telemetry must connect work to an outcome, not just to completion.

Behavioural telemetry can easily become staff surveillance. That would damage trust and produce bad data.

The aim should not be to rank employees by prompt count, time at a screen or volume of output. Those measures reward visible activity and punish forms of work that need thought, care and judgment.

The better approach is to study the work path.

Where does work wait? Which steps get repeated? Where do people switch between systems? Which tasks need the most review? When does AI improve the first attempt? When does it produce more edits? Which forms of AI support help people learn, and which make them dependent?

This shifts the focus from “Is this employee using AI enough?” to “Does this process work better with AI?”

It also makes the data more useful. Individual performance often depends on task type, team rules, available data, approval limits and the state of the process. A worker-level score strips away this context. Process-level analysis keeps it.

A firm should start with a small set of clear work outcomes.

For each process, it needs a baseline from before AI use. That baseline should cover four areas: time, quality, cost and outcome.

Time includes the full cycle, not just the first step. Quality includes errors, edits, approval rates and later fixes. Cost includes staff effort, system cost and added review. Outcome depends on the work: sales won, cases solved, code released, customers retained, decisions made or risks avoided.

The firm can then add behavioural signals that explain why those results changed.

These signals might include the number of hand-offs, review loops, source checks, AI-assisted steps, human overrides, abandoned drafts and time spent after AI output appears.

The goal is not to gather every possible event. More data does not ensure more insight. The goal is to collect enough evidence to explain the change in the outcome.

This lets the firm compare groups, tasks and periods. It can see where AI works, for whom it works and under which conditions it fails.

Measurement has little value if it only creates a dashboard.

The real use of behavioural telemetry is to change how work gets designed.

Suppose the data shows that AI reduces drafting time by half but doubles review time. The answer is not more prompt training. The company may need stricter source rules, set output formats, clearer review checks or limits on which work AI can draft.

• Suppose newer staff gain a lot while senior staff see little change. The firm may use AI as a learning aid for new hires rather than force the same process on everyone.
• Suppose employees keep copying facts between systems before they can use AI. The main problem is not the model. It is missing access to context.
• Suppose a team produces more work but customer response falls. The system may be cutting the cost of low-value output and causing the team to send too much of it.

These are management and work design issues. Better models alone will not fix them.

The first stage of workplace AI centred on access. Firms bought tools and encouraged staff to try them.

The second stage centred on use. Leaders tracked adoption, shared prompt tips and collected examples.

The next stage must centre on evidence.

Behavioural telemetry provides that evidence.

It turns AI adoption from a claim into something a firm can test. It shows that productivity does not come from adding AI to a task. It comes from changing the process around the task, setting the right checks and measuring the final result.

The firms that close the AI productivity gap will not be those with the most tools, prompts or generated content. They will be those that can see how work happens, learn from that evidence and redesign the work itself.