When people discuss the risks of artificial intelligence, the conversation often starts too far away from people. Large words arrive first: future, disruption, threat, revolution. They can open a debate, but they can also stop us from seeing a particular scene. In a municipal housing office, for instance, a worker named Nuria opens a morning list of messages about rent, eviction notices and shared rooms. A new system suggests which enquiries should be answered first and drafts an initial reply. The screen appears to bring order to chaos. The risk is not that the screen is mysterious; it is what may happen if someone facing an urgent deadline falls behind a message written with less clarity.
Looking at that scene does not require us to decide that the tool is useless. It may help Nuria find important cases sooner or free time for conversations with people who need an explanation. The problem begins when an apparent improvement becomes an excuse not to ask about its consequences. Understanding a real risk means following a decision into the life it touches: what information enters, which signal is treated as important, who can correct it, what happens when it fails and who carries the work of putting things right. That way of looking is less dramatic than a prediction, but much more useful for making a decision.
Risk begins in a situation, not in a label#
Saying that a system has “bias” or that there is a “privacy risk” may be correct, yet it is not enough to guide action. Nuria needs to know something more exact. Does the system read a short enquiry as less urgent because it does not recognise a person's way of writing? Does a family sharing one phone receive messages another person can see? Does a priority recommendation cause the team to stop reading the full case? Each question names a particular relationship between a rule and a possible consequence. Without that relationship, a label becomes as comfortable as an alarm that never tells us where the smoke is coming from.
Saying that a system has “bias” or that there is a “privacy risk” may be correct, yet it is not enough to guide action.
That is why it helps to begin with a small story that can be checked. One person writes that they must leave their flat in ten days but does not use the word “eviction”. Another supplies many documents and writes in administrative language, although their problem can wait. If a programme rewards certain signals of urgency, the list may reverse those situations. We do not need to claim that it will always happen. It is enough to recognise that it can happen and that, when it does, the cost is not distributed in the abstract: someone may lose an appointment, a night's sleep or the chance to receive guidance in time.
The language of averages can hide exactly this difference. A dashboard may show that average response time has fallen. That number matters, but it does not tell us who waits longer than before or which cases are worse off. A general improvement does not erase a specific exclusion. Responsible work means looking at the distribution: responses by enquiry type, differences by language, cases corrected by staff and people who abandon the process because they do not understand what is being requested. What is not measured does not disappear; it simply remains outside the conversation that makes decisions.
Following the path of a decision#
The first practical question is which data feed a recommendation. In Nuria's office, they may include the text of a message, a date mentioned in it, a neighbourhood, attached documents or contact history. Every element has a source and a limit. A neighbourhood may help organise support routes, but it can also become a shortcut that assumes needs without listening to the person writing. A history may show that someone has sought help several times; it can also confuse persistence with severity or punish someone who did not find an answer earlier.
Next we have to ask what the system does with those data. “It calculates a priority” is not a sufficient answer. The team should be able to describe, in understandable language, which signals matter, which cases fall outside the model and what uncertainty the outcome contains. When nobody can explain why an enquiry sits in a certain position, detecting a systematic error becomes difficult. An explanation need not disclose trade secrets or become a technical manual. It needs to support a competent conversation: what we know, what we do not know and which human decision is still outstanding.
The third question is who acts. Many deployments claim to have “human oversight” because someone can press a button at the end. That is inadequate if they receive a hundred recommendations in an hour, do not know the criteria or lack time to read the context. Oversight becomes real only when it includes authority to pause the process, change a priority, record the reason and receive support for doing so. Asking Nuria to contradict a screen without giving her time or authority turns responsibility into an individual burden rather than an institutional guarantee.
We also need to follow harm through to repair. If an urgent enquiry was classified late, who notices? Can the person affected call someone who understands the case? Does the team correct only that file, or does it examine the rules that caused the error? A process limited to an apology may offer momentary relief, but it does not learn. A process that retains examples of failures, identifies patterns and changes practice protects the next person. Accountability is not about quickly locating blame; it is about having a route to recognise, limit and repair harm.
Telling inconvenience from serious harm#
Not all errors deserve the same response. A draft containing an inelegant sentence is not comparable to delayed housing advice. That difference should guide design before a tool is used at scale. A simple rule is to consider severity, reversibility and the affected person's ability to defend their position. When an outcome can alter income, health, housing, safety or access to a right, precaution should increase. A promise to save minutes does not grant permission to transfer risk to people with the least room to absorb it.
Reversibility deserves particular attention. An incorrect message may be corrected with another message. A lost place, an appointment whose deadline passes or a family that stops seeking help because it feels ignored may not be restored so easily. This is why a responsible trial does not begin with the most sensitive cases. It can begin with low-impact tasks, a limited period and a clear route back. Testing is not releasing a tool and watching from a distance; it is creating conditions to learn without turning people into experimental material.
Scale also changes the kind of harm involved. An isolated error may be resolved in a conversation between two people. A rule applied thousands of times requires records, audit and an organisational decision. It is important not to confuse repetition with normality. An incorrect answer produced automatically is not less important; it may be harder to see because nobody feels they made the complete decision. That is precisely why teams need to stop and review samples of cases rather than relying only on the indicators produced by the tool itself.
Designing controls for a difficult day#
A useful control has to work on a Monday with calls piling up. For Nuria, that could mean recommendations display a short reason and a visible uncertainty marker; that certain terms or circumstances require human review; and that an alternative queue exists for messages the system does not understand well. These are not bureaucratic ornaments. They are ways to prevent speed from closing the door on doubt when doubt matters.
The team could agree on a weekly sample of files: some answered quickly, some delayed and some changed by a person. Reviewing them together can reveal questions no automated report will ask: whether a response's tone discouraged someone, whether a translation altered a condition or whether a rule keeps moving the same kind of case to the end. Review becomes stronger when the people doing daily work take part and, where possible, when people familiar with the experience of seeking help are heard. No metric completely replaces a conversation about what happened.
The tool and its rules should also have an expiry date. Incoming messages, legal context, staffing and a neighbourhood's needs change. A decision made for six months can become permanent infrastructure through mere fatigue. Setting a review with concrete questions—what benefit was demonstrated, what harm appeared, which groups were served worse and what less intrusive alternative exists—prevents the temporary from becoming untouchable. Keeping the ability to say “this does not continue” is an essential part of collective autonomy.
The best question is not whether AI is good or bad#
At the end of a working day, Nuria does not need a perfect theory of artificial intelligence. She needs to know whether the system helps her serve people better without hiding those who need more time, whether she can challenge a recommendation and whether a person has a dignified route to request review. These questions change the debate because they move attention from fascination with a tool to the conditions for a fair relationship.
Understanding real risks does not mean living in fear or rejecting every form of automation on principle. It means refusing to call inevitable a decision that can still be designed differently. Technology can support human work when it preserves context, permits correction and distributes responsibility honestly. It can damage that work when it turns the complexity of a life into a signal nobody has time to question. The standard is not the promise on the screen, but people's ability to intervene when that promise falls short.
