On Monday, Nora opened the municipal app to request an appointment with a nurse. She had tried the previous week, when the first available option was eighteen days away. That morning, a Thursday slot appeared. She did not know that the change came neither from a newly hired professional, an expanded schedule nor a decision announced by the health centre. Overnight, an automated rule had been enabled: requests with particular answers in the form moved into a priority queue. The rule operated in the background, appeared on no organisation chart, and nobody at reception could tell Nora why she saw that opening while somebody else did not.
The problem is not that every automated classification is illegitimate. A system may help order urgent requests, prevent people from repeating information, or identify that an appointment needs more time. The problem begins when a rule changes who waits, who receives a call, or who falls out of a schedule without the institution being able to say what changed, who answered for it, and how it will be reviewed if it harms someone. A consequential decision does not stop being a decision because it runs quickly or because someone wrote it as a technical setting.
Automation does not remove responsibility#
In many organisations, a decision has a recognisable face. If opening hours change, a person or department signs the instruction. If eligibility for support changes, there is usually a policy, an effective date and a way to ask questions. When a modification lives inside a purchased tool, the chain becomes blurred. The vendor says it delivered a feature; the technical team says it enabled an option; the service department says it is still following procedure; and the person receiving the result is left with no accountable interlocutor.
In many organisations, a decision has a recognisable face. If opening hours change, a person or department signs the instruction.
That gap is not merely administrative. Nora needs to know whether her Thursday appointment reflects clinical assessment, a prioritisation rule or a data mistake. She also needs to know what to do if the form misunderstood her situation. If someone facing a language barrier, a disability or a shared phone answers differently, the rule can distribute time and attention unequally without the bias surfacing in any meeting. Technical invisibility turns a public question — who receives a scarce resource first — into a detail that appears to belong only to a screen.
Calling every system that orders cases “artificial intelligence” does not clarify matters either. Some rules are simple, while others rely on complex models. In both cases, a prior human choice exists: which information counts, which outcome is sought, when an exception applies and what cost is accepted if the system gets it wrong. An organisation must be able to name those choices without hiding behind a technical label. Complexity may require more explanation; it never warrants less responsibility.
A living rule register makes change discussable#
One basic safeguard is a living register of rules that affect rights, access, priority or treatment. It does not require publishing security secrets or personal information. It requires an understandable record: which process the rule touches, which decision it influences, what information it uses, when it began operating, which team owns it and when it is due for review. If a vendor is involved, the record should distinguish the part it controls from the part the institution retains. When a version changes, the register should preserve the earlier version and the reason for the change.
Living is the significant word. A document produced when software is bought and forgotten in a folder cannot reveal that an update has altered outcomes. Nora's rule might change because a supplier adjusts a threshold, because the council adds a question, or because the service decides to reduce appointments without notice. Each alteration deserves a trace: who proposed it, what evidence supported it, which groups might be affected and who authorised it to go live. Recording a rule does not guarantee a good decision, but it makes the decision visible, discussable and correctable before it becomes habit.
The register also changes the internal conversation. The technical team no longer receives an ambiguous request to “make it prioritise better” and can ask for a concrete definition of priority. Frontline staff can check why someone sees a particular result without inventing an explanation. Leadership can compare promised outcomes with actual outcomes. And residents have a reasonable door through which to ask questions without first having to prove programming expertise.
Institutional ownership cannot be outsourced#
A vendor may develop the software, but that should not make it the sole keeper of a public criterion. When a rule decides access to an appointment, grant, inspection or benefit, the institution using it needs a named, responsible owner. That person need not program every line. They do need to answer what purpose the rule serves, what boundaries govern its use, who can pause it and what happens when a complaint arrives.
Assigning institutional ownership prevents another common failure: nobody feels authorised to stop a system that is already causing harm. If reception staff notice that many people arrive without knowing why they could not get an appointment, they need a channel to escalate the problem. If a professional observes strange results, that observation should trigger review rather than an endless chain of emails. The process owner must have both the authority and the duty to coordinate a response with technical, legal and service teams and, where appropriate, with people affected by the rule.
Responsibility does not end when a tool performs as expected. A rule may accurately order the data it receives and still amplify an existing inequality. Someone without a stable connection may abandon the form before reaching the question that opens priority. Someone else may not understand a medical term. Looking only at technical accuracy hides this effect. Institutional ownership includes reviewing the whole experience: who can enter, who understands the decision, who can correct a record and who gives up before asking for help.
Give notice before the rule has consequences#
Advance notice does not mean overwhelming people with a twenty-page policy. It can be a short explanation where it matters: “We use answers in this form to order appointments. If your circumstances change or you think the priority does not reflect your need, you can ask for review by phone, in person or through this channel.” That message acknowledges that classification exists and offers a route before the outcome becomes irreversible.
Notice should arrive before impact, not only after a complaint. If an update will move some requests into a different queue, the organisation can tell its teams, test it with varied cases and provide accessible support. Not every change needs a public campaign, but every change that alters opportunities needs a proportionate risk assessment. Changing a button colour is not the same as changing the logic that determines whom staff call today.
Review routes matter as much as notice. A person should not have to explain their entire life to a chatbot in order to ask for somebody to look at their case. There should be a human path, clear time limits and a response that says what was reviewed. Review does not promise a different result every time. It recognises that a rule may miss relevant circumstances and that the institution retains a duty to assess its effects.
Review changes, not only the initial product#
The useful question is not “does the AI work?” but “what has this rule changed in the real distribution of time, attention and opportunity?” To answer it, a service can schedule reviews after every significant modification: compare waiting times, abandonment rates, review requests and outcomes across groups that may receive unequal treatment. Metrics do not replace the accounts of people using a service, but they can reveal that an apparent improvement consistently leaves someone behind.
Review needs the possibility of reversal. Before deploying a modification, the organisation should define who can suspend it, what signal would trigger suspension and how the earlier version will be retained. This discipline is familiar in safe systems: test, observe, record and correct. Applying it to public-service rules means accepting that people are not a silent testing environment.
Nora does not need to inspect the application's code in order to receive dignified treatment. She needs to know that her Thursday appointment follows an identifiable criterion, that someone accepts responsibility for that criterion and that she can ask for help if the result does not describe her situation. A living register, clear ownership, advance notice and meaningful review do not slow technology down. They give it the place it should have in Pacoverso: a tool serving people who can understand it, challenge it and ask for change when it stops caring.
