On Monday morning, Mara opens her housing cooperative’s app to find a repair service. The screen ranks three companies, labels one “best option,” and puts a green button beside it. She would once have clicked without thinking. Now she needs work that will affect older neighbours, a shared budget, and a date that cannot move. She does not ask whether the program is intelligent. She asks what would have to be true for this recommendation to deserve trust. That small shift changes the scene: a visual suggestion becomes a hypothesis that people can check.
Start with the decision, not the score#
A recommendation can only be assessed when the decision beside it is named. “Choose a company to repair a fault without excluding anyone who needs accessible service” is far more useful than “use the ranking.” Write that sentence before looking at the result, then add two consequences of getting it wrong: delaying an urgent repair, overspending, or leaving out a concrete need. The exercise turns attention toward the human purpose rather than the convenience of the interface. It also separates reversible choices, such as trying a book list, from choices that need more care, such as selecting a job offer or prioritising healthcare.
A recommendation can only be assessed when the decision beside it is named. “Choose a company to repair a fault without excluding anyone who needs accessible service” is far more useful than “use the ranking.”
Mara turns her purpose into four questions: can the company attend within the real deadline; does the quote explain what it includes; is there an alternative with telephone support; and who answers if the first result does not fit? None requires understanding the system’s code. All require the tool to leave room to search, compare, and correct. If the screen does not allow that, the problem is not that a person “does not know technology.” The design is asking for obedience without giving them what they need to decide.
Build a simple baseline#
To tell whether something improves, remember how the process worked before. For one week, note three modest facts: how long it takes someone to reach verifiable information, how many different options they find, and how often they must correct a result. There is no need for a dashboard full of metrics. A shared notebook may show that the app saves two minutes but forces five calls to resolve an exception, or that it shows more options while all belong to the same provider. A baseline protects against the excitement of a first impression and the cynicism of dismissing every change without observing it.
Add one qualitative observation: can the person explain why they rejected a proposal? If the answer is “because it came first,” there is no defensible decision. If it is “I compared availability, cost, and access, then chose the second because it offered an initial visit,” there is an understandable trail. Explaining does not mean revealing private information or defending every preference. It means keeping a reason another person can understand and challenge. That possibility is a sign that an algorithm is expanding people’s capacity to act rather than replacing it.
Follow the whole route of a suggestion#
Quality is not located only in the final list. Watch the route: what information the service requests, which options it discards, what it presents as urgent, and what happens when someone presses “none of these.” A course recommender may help at first and fail when it hides schedules, cost, or accessibility requirements just as a person wants to compare. A job search may match keywords while repeating contracts that conflict with someone’s caring responsibilities. Following the whole route prevents a good first screen from being celebrated while the decisive step remains opaque.
One practical test is to take one recommendation through the process with two different people. One knows the system and one does not. Ask them to find an alternative, change a preference, and locate a help channel. Note where they stop. If the second person cannot tell what to do and the first can move forward only through habit, there is a design debt. The remedy may be as concrete as a less ambiguous label, a visible filter, or a contact number. The system does not always need replacing; often the point where control is lost needs repair.
Use a contrast check#
Every two weeks, compare the automated result with a source that does not depend on the same ordering. It might be a public directory, a call to a local service, an unpersonalised search, or a list prepared by accountable professionals. The purpose is not to prove that every algorithm is wrong. It is to discover whether the same kind of alternative is repeatedly left out. Mara consults the municipal register and finds a fourth company missing from the app because it has not paid to highlight its profile. The recommendation is not useless because of that discovery, but it can no longer be presented as the whole map.
Contrast checks also help in less visible situations. A family can compare activities suggested to a child with a civic centre’s calendar; a library can compare featured titles with actual loans and user proposals; a hiring team can review a sample of applications that never reached the shortlist. When a contrast shows a difference, it is not enough to say, “that is unusual.” Ask whether it repeats, who bears its cost, and what change would reduce the problem.
Recognise positive signs without turning them into promises#
There are encouraging signs. Repeated corrections decrease; relevant alternatives appear that were previously hard to find; saved time becomes a better conversation rather than pressure to do more; and people can change a recommendation without penalty. It is also a good sign when the responsible team answers an error report with dates and actions. “We will review the rule that excluded this option and reply on Friday” is more valuable than a long technical explanation that leaves no next step.
Yet a positive sign is not a universal guarantee. A result that works for frequent users is not necessarily understandable to someone arriving for the first time. A higher average satisfaction score does not explain what happened to the person who could not complete the task. Reviews must therefore collect both patterns and exceptions. The apparently small case may reveal a barrier that always affects people using a screen reader, another language, or an unusual schedule. Listening to the exception does not abandon data; it stops an average from hiding exclusion.
Stop at concrete warnings#
There are moments when it is unwise to keep experimenting. Stop if the system demands information that does not seem necessary for the task, if there is no way to correct a record, if an important decision arrives with artificial urgency, or if nobody can say who is responsible. It is also a warning when a recommendation changes substantially without a visible explanation. Save a screenshot without sensitive data, write down the date, and describe the effect: “the accessible alternative disappeared after I changed the filter” is more useful than “the app is broken.” That record lets another person investigate without rebuilding the situation from zero.
Not all warnings have the same seriousness. A strange film order may invite more comparison. A recommendation affecting work, housing, credit, health, education, or rights needs a human channel before the result is accepted. The working rule is proportionality: the harder a decision is to undo and the greater its impact, the more evidence, time, and independent review it needs. The convenience of a quick answer does not reduce that obligation; sometimes it increases it.
Ask for help with a request someone can handle#
Asking for help does not mean writing a technical report or fighting with a screen. Prepare a six-line note: what decision was involved, when it happened, what the system showed, what effect it produced, what you checked, and what response you need. Mara writes: “On 17 July, the platform showed three repair firms and did not allow filtering for accessible service. I found a fourth option in the municipal register. I request the criteria that order the list and a way to propose a correction.” Depending on the case, that request can go to service support, a consumer association, a data-protection contact, or a public body.
It is better not to post screenshots containing names, phone numbers, or histories on social networks in order to get help. Replace identities with roles and share only what is necessary. If the matter needs professional advice, keep original documents securely and ask which information is relevant to provide. Protecting privacy does not compete with making a complaint: a well-described incident is usually more useful than a long exposure that is difficult to withdraw.
A review that ends in a small change#
Mara’s cooperative does not try to audit the entire market. Each month it reviews one real recommendation with a person who did not design the tool. They check one external alternative, look at one incident, and agree on one verifiable change. In the first month they add an accessibility filter and a notice explaining that order includes declared availability and user ratings. In the second, they create an email address for review requests. Then they compare whether questions are resolved better and whether new difficulties appear. The routine works because it has a limited scope and a return date, not because it promises perfection.
At the end of every review, write one sentence that any neighbour can use: “The list is a place to begin, but the decision needs a comparison of deadline, access, and cost; if an option is missing, we can request a review here.” That sentence does not tame complexity, but it returns an essential part of the decision to the people who live with its consequences. An algorithm genuinely helps when it makes checking, disagreeing, and correcting easier. When it prevents those three things, the next action is not silent adaptation: it is stopping, documenting, and asking for a human answer.
