On a delivery company's operations screen, a green arrow announces that the average time from order to doorstep has fallen by four minutes. The team is congratulated and the weekly target looks secure. Away from the screen, a courier has stopped taking one district because the planned routes no longer leave time to use a bathroom or deal with a broken bike. An older customer has stopped ordering because quick deliveries do not ring and parcels are left beyond the building entrance. Neither experience appears in the metric. The number is not lying; it has decided what counts as success and what harm may remain outside its frame.
That is the difficulty behind many algorithmic promises. A tool is said to make work more efficient, and the proof is a number that is easy to compare: response minutes, cases closed, clicks converted, risks detected, or deliveries per hour. Measurement matters, but a number is not a neutral picture of reality. It directs attention, rewards particular behavior, and tells an organization which questions can wait. Once that instruction becomes automated, it may steer thousands of decisions before anyone notices that speed is moving costs onto people with less time, knowledge, or power to object.
A metric is also a rule for distributing time#
Consider a customer-support service that wants to reduce its average closure time. At first, that sounds plainly beneficial: no one enjoys waiting. But if recognition depends only on closing quickly, easy cases are completed first while difficult ones travel from queue to queue. Resetting a password is fast; explaining a duplicate charge, an accessibility barrier, or an unfair suspension requires context. The average can improve because difficult conversations have been turned into invisible delay.
Consider a customer-support service that wants to reduce its average closure time.
The issue is not simply that an average can be gamed. An organization can accept it as evidence of good service without asking whom it serves. A speed measure distributes time. It gives immediate attention to some people while accepting that others must do the work of insisting, repeating themselves, or collecting documents. When that distribution is not examined, efficiency starts to look like transferring the burden to people with less procedural knowledge, less free time, or less ability to complain.
Algorithms intensify the effect because they connect the metric to repeated decisions. An allocation system may give better shifts to people who finish more tasks; a selection tool may prioritize applications that are easier to verify; a risk detector may concentrate checks on profiles that produce quick results. Each decision can look reasonable under the chosen objective. The pattern is what matters: people with unusual cases, older devices, irregular schedules, or different ways of communicating become friction to be reduced.
This is why the question before accuracy is not only whether a model predicts well, but which outcome has been made rewarding. A model can accurately predict which conversations will end soon and use that prediction to lower average time without improving anyone's chance of getting a solution. Treating an operational signal as the purpose of a service turns a dashboard into an ethical decision without naming it.
The story the dashboard leaves out#
A housing cooperative uses a digital form to sort repair requests. It wants to respond to urgent problems quickly, so its tool prioritizes photographs, keywords, and records from earlier reports. After a month, the dashboard looks better: first-response time has fallen and more work orders are completed each day. At a residents' meeting, another story emerges. Several older people did not know how to attach an image. A family sharing one phone submitted a report when they could. A serious damp problem did not contain the words the system treated as urgent. Their cases did not wait because they mattered less; they waited because they were less legible to the metric.
The answer is not to abandon all prioritization. It would be careless to treat a gas leak exactly like a stuck blind. The question is how to check that speed has not created a hierarchy based on how easily people can generate data. The cooperative can review a sample of slow cases, compare timelines by entry channel, ask affected residents to reconstruct their path, and inspect which reports changed priority after a human read them. It can also track abandoned requests, repeated demands for information, and repairs that worsen while waiting. Those signals do not replace average response time; they make it meaningful.
Harm also needs a precise name. “Poor experience” is too vague to guide a decision. In this setting, harm might mean spending a night without heating, losing medication to a leak, using prepaid phone credit on repeated calls, or losing confidence to report a future problem. Once it is described clearly, the cooperative can choose suitable safeguards: a phone channel, daily reading of incomplete reports, a provisional category for habitability risks, or a way to escalate a case without mastering the form.
A responsible metric should show both what improves and the limit an organization refuses to cross. It may be reasonable to promise that most reports will receive a prompt reply, but not to allow any health or housing-safety concern to go unread by a person for more than a day. That limit turns an aspiration into an operational duty. It also prevents a better headline figure from hiding a concentrated deterioration for people who already face more barriers.
Incentives change behavior before anyone gives an order#
Workers do not need an explicit instruction to follow a metric. It is enough for the figure to determine recognition, budget, or job security. A moderation team judged by the number of items reviewed is more likely to make quick decisions when evidence is ambiguous. A clinic network comparing appointments per hour may shorten the conversation in which an adverse reaction, domestic violence, or financial difficulty would otherwise emerge. A provider promising ever-faster delivery can increase unsafe journeys without ever writing “take more risk” in a policy.
The consequences usually appear far from the person who created the indicator. The dashboard designer may see lower cost per case. The worker feels the available time shrink. The person receiving the decision encounters a shorter explanation, a longer route, or a lost opportunity. Unless those views are connected, every department can honestly say it did its part while harm becomes routine.
A serious review therefore examines incentives, not only interfaces. What do workers do when the target conflicts with good care? Which cases become worse when the system succeeds by its own definition? Who can stop an automation, and with what protection? Is time reserved to investigate exceptions, or is every exception treated as a threat to productivity? These questions do not delay improvement. They stop a system from learning to win the wrong game.
It is also unsafe to assume that harm will always arrive as a complaint. Someone dependent on a service may not complain because they fear losing access, do not understand the route, or have already spent too many hours trying to solve the problem. Fewer complaints can mean satisfaction, but they can also mean exhaustion. That is why usage data should be combined with conversations, audits of closed cases, and deliberate tests with people unlike those who designed the journey.
Build a second reading into the work#
One practical approach is to pair every central metric with a counter-question. If speed is measured, measure durable resolution too: how many people return with the same issue and how many answers are later corrected. If fraud detection is measured, inspect how many reviews confirm a mistake and how long an innocent person waits to regain access. If productivity is measured, ask which work of care, explanation, or repair has stopped counting. The purpose is not an infinite dashboard. It is to prevent one number from declaring victory too early.
The second reading needs owners and consequences. A quarterly report that cannot alter priorities merely records a problem. A team can instead agree that a sustained rise in repeat contacts pauses expansion, that a pattern of channel barriers triggers an accessibility test, or that a human reviewer can change a rule after finding foreseeable harm. Governance becomes real when someone can act before the cost becomes customary.
Transparency helps, but publishing a formula is not enough. A useful explanation says what is optimized, what is not used to decide, which limits exist, and how a person can flag a damaging outcome. The housing cooperative does not need to reveal every technical weight to explain that a photo may speed classification, that it is not required, and that an urgent report without an image will be read by a person. That information changes a resident's actual ability to participate.
The final test is to reverse the perspective. Instead of immediately celebrating a reduction in minutes, ask: what would have to happen for this improvement to be unacceptable? Listen to the call handler, the parcel recipient, the screen-reader user, and the person who had to appeal a decision. Then turn their answers into test cases, limits, and repair routes. If the system cannot survive that conversation, its speed depends on somebody absorbing harm they did not choose.
The silent promise of algorithms is not merely that they will make things faster. It is that the chosen number will decide, without enough discussion, which lives fit inside success. Recovering that discussion is concrete work: review a metric, identify the harm it can hide, set a verifiable limit, and give someone authority to correct it. When a green arrow announces progress, the responsible question is not only how much the process accelerated. It is who paid for the time the screen stopped counting.
