A number can be useful and still direct a decision to the wrong place. Average response time, the share of cases closed, or the proportion of people accepting an automatic recommendation reveal part of a service. On their own, they cannot show whether an answer was understandable, addressed the relevant problem, or pushed someone away from a necessary alternative. In everyday artificial intelligence, ease of counting must not be confused with the importance of what is counted. An automatic answer leaves measurable traces; the experience of the person receiving it, their doubts, and later consequences require different attention.
When a number simplifies a decision#
An organisation may see that an assistant cuts a query from eight minutes to two. That improvement may be real. Yet the average combines simple questions with situations that do not fit a form, people with strong connections with people who cannot continue, and helpful answers with confident statements lacking sufficient basis. If reducing minutes is the sole objective, the system will learn to end conversations, not necessarily to help. The metric then becomes a design instruction: it rewards what it records and leaves unprotected what does not appear on the dashboard.
An organisation may see that an assistant cuts a query from eight minutes to two.
Each number should be tied to the decision it will inform and to the behaviour it may induce. A high closure rate may mean people found a solution; it may also mean they did not know how to request review. A low rate of escalation to a person may reflect autonomy; it may also show that the human route is hidden or slow. A figure needs context, a denominator, a period, and affected groups. It should sit beside reviewed examples, complaints, corrections, and observation of complete journeys. The point is not to reject data, but to prevent a partial measurement from taking the place of judgement.
Case 1: speed in municipal support#
A city council introduced an assistant to guide housing-support applications. Its weekly dashboard celebrated that 86 percent of conversations ended without referral and that average time had fallen. At first sight, both indicators supported expansion. The service team, however, noticed repeated queries from people who had received brief answers about documents. They did not appear as incidents because each individual conversation was marked closed. Reviewing sequences showed that the assistant gave correct lists for ordinary cases, but did not explain what to do when evidence was missing or a deadline was near.
The review changed the dashboard. Alongside time and closure, it added the share of people who completed the application without asking again within thirty days, the number of applications refused for incomplete information, and time to a human answer when one was requested. A sample of conversations from different profiles was also read. Speed stopped being an automatic victory and became a condition that had to coexist with understanding and access. The assistant added clarification questions, deadline notices, and a visible referral route. The average became slightly worse; later errors fell materially.
Case 2: a school prediction turned into destiny#
A learning-support platform calculated the probability that each student would need extra help. The school watched a simple indicator: how many recommendations matched final grades. The match looked high and was used to assign limited tutoring places. The difficulty was that final grades were already shaped by support received, absences, and opportunities to recover work. Students without a stable device also appeared less often in practice records and received less favourable predictions. The number described something about a statistical relationship, but it was being used as though it expressed a fixed individual need.
Leaders suspended use of the prediction as a sole criterion. They kept the signal as one source among several, alongside teacher observation, the student's request, and a short conversation. Errors were assessed by year, connection availability, and subject, not only overall accuracy. A route to seek review was opened without requiring someone to prove a technical mistake. The change recognised an essential distinction: a model-performance metric is not a measure of the fairness of a decision. When an automatic answer affects opportunity, it is necessary to ask who benefits, who is excluded, and whether a real correction remains possible.
Case 3: satisfaction that hid a difficult exit#
A digital health service sent a survey after every automatic orientation. Satisfaction was high, and the supplier presented it as evidence of quality. Questions appeared immediately after a friendly message and before a person tried to book an appointment or follow the recommendation. Many replies judged tone rather than clinical usefulness or access to care. A patient association reported that some people with persistent symptoms followed advice to wait because they did not understand that they could consult again. That consequence lay outside both the survey and the monthly indicator.
The service redesigned measurement with patients and professionals. It separated message clarity, ease of reaching human care, understanding of warning signs, and an outcome reported several days later. Voluntary replies were not treated as a complete representation: they were compared with calls, complaints, and journey abandonment. A specific review was created for answers advising people to wait and for cases in which a person returned with greater urgency. Satisfaction remained useful data but stopped being a verdict. In a sensitive setting, a smiling face or high score cannot replace checking that help was safe and workable.
Metrics that support better answers#
Responsible measurement starts by stating the purpose in ordinary language. If a tool aims to guide an application, the desired result is not merely a short conversation; it is that a person can complete the right step with sufficient information. Complementary measures can then be chosen: verified resolution, waiting time, requested corrections, abandonment, human referral, differences between groups, and detected harm. None has value in isolation. A small set published with clear definitions reveals tensions that one figure hides. Time may improve while corrections rise; acceptance may increase because the alternative becomes worse.
Metrics also need limits on their use. They should not punish the person handling an exception or automatically decide that an answer is correct. A rigid target for closed conversations can encourage avoidance of difficult questions. A cost-reduction goal can discourage human review. Before setting incentives, a team should test impact: what will be measured, who is left out, which error would be serious, how it will be detected, and who may stop the system. Data should preserve enough traceability to review a decision without collecting unnecessary information. Proportionality matters in collection as much as interpretation.
Oversight is not looking at a dashboard once a month. It requires named responsibility, case samples, a channel for affected people, and power to change the process. Alerts should include abrupt changes, but also small repeated harms that an average dilutes. Reviews should be set before deployment, during a limited trial, and after a change of model, supplier, or purpose. When a measure worsens, the response need not be to defend the tool. It may be to narrow scope, add explanation, open an alternative, or withdraw it. The capacity to correct is a practical metric of accountability.
Public references provide useful criteria for this work. NIST's AI Risk Management Framework proposes governing, mapping, measuring, and managing risks throughout a lifecycle. The European Union's General Data Protection Regulation requires transparency, minimisation, and safeguards for certain automated decisions. UNESCO's Recommendation on the Ethics of Artificial Intelligence highlights proportionality, human oversight, and impact assessment. The European Commission has stressed human agency, diversity, and accountability for trustworthy AI. These sources do not provide a single decisive indicator, precisely because a number cannot replace examination of each context.
An automatic answer can save time and improve a service when measurement is honest. The decisive point is not allowing the most convenient number to define success on its own. People who do not finish, cases that return, corrections, alternatives, and consequences appearing after a click all need examination. Measuring is choosing what becomes visible; good governance keeps uncomfortable parts visible as well. In that way, a metric stops directing blindly and becomes an aid to answering with greater care.
Publication is part of that discipline. A service should explain, in accessible terms, what it counts, why those indicators were selected, and what action follows from a warning. Regular summaries can distinguish an experiment from a stable service and report important limitations without exposing personal data. Independent review, staff feedback, and reports from community organisations add evidence that operational systems often miss. A metric becomes more reliable when people living with its consequences can challenge its meaning, its gaps, and the response it triggers.
Official sources. NIST AI Risk Management Framework; EU General Data Protection Regulation; UNESCO Recommendation on the Ethics of AI; European Commission guidelines for trustworthy AI.
