On the municipal library table sits a list of books to buy and a budget that cannot cover all of them. The new recommendation tool orders requests with a score for “predicted interest.” At first sight it seems impartial: it does not know readers personally, has no literary preferences and offers a number for every title. But the number has a history. It gives more weight to earlier loans, reviews available on particular platforms and patterns of enquiry collected by an application. When the list appears on screen, books requested by reading groups that do not use that application fall to the bottom. The tool has not said that their interests matter less; it has simply turned one way of leaving a trace into the main way of being seen.
Calling that result neutral clarifies nothing. It can feel reassuring because it seems to move responsibility into a technical operation. Yet a system does not appear without choices: someone decides which data enters, what counts as success, how much error is tolerated and when a suggestion begins ordering real resources. Neutrality is a myth not because every technology hides malicious intent, but because a human decision can sit inside a rule and stop looking like a decision. Naming that presence is the beginning of a more honest conversation.
A score does not arrive by itself#
The library score might be calculated correctly according to its design. It may accurately predict which titles will receive the most clicks in the digital catalogue. The difficulty is that “more clicks” does not itself answer what a public library ought to buy. Diversity of the collection, access for people who leave no digital trace, the needs of small groups and the chance to discover authors who have not yet appeared in records may matter too. None of those priorities is mathematical by nature. They must be chosen, discussed and turned into criteria.
The library score might be calculated correctly according to its design. It may accurately predict which titles will receive the most clicks in the digital catalogue.
That distinction matters because language changes behavior. If we say that the tool is neutral, a worker may feel that questioning its order means injecting personal opinion into an objective process. If we say that it applies a configured set of priorities, another possibility opens: asking whether those priorities are suitable and whether affected people can supply information the system cannot see. The second formulation does not make the work easier, but it puts the work where it belongs: in a public decision requiring reasons and responsible people.
There is also a choice in the moment the result is used. The same score can serve as one clue among several or as a barrier that removes titles before anyone reads them. It can help prepare a meeting or replace one. It can mark a proposal as uncertain or present it with a precision nobody has tested. Auditing the calculation is not enough; we also need to observe the practice that grows around it. A recommendation gains power when other processes, calendars and targets are arranged to obey it without friction.
Data remembers some lives better than others#
Data is not a uniform portrait of reality. It is a set of traces produced by services, devices, forms and recording rules. Someone who renews books online leaves a different signal from someone who visits the desk, shares a copy at home or requests a title through a neighborhood association. If the tool treats the most visible digital signal as equivalent to interest, it turns a difference in access into a difference in value. It does not need to name anyone for inequality to appear in the result.
This observation does not require abandoning available records. It would be absurd to deny that loans and searches can offer useful information. It requires describing their limits instead. The library should be able to explain which sources feed each recommendation, who is less well represented in each source and what routes exist to correct that absence. A readable note might say that the score measures recorded activity rather than cultural need, that certain requests are reviewed outside the model, and that groups can submit reasons that enter the decision. Explanation is not a communication extra. It prevents automated output from turning its silences into truths.
Looking at data by groups also helps find problems that an average conceals. The tool might perform well for the most requested novels while consistently reducing the presence of works in minority languages, accessible material or topics requested by few people. A high average accuracy can coexist with concentrated loss. The team should therefore ask not only whether the model “works,” but for whom it works, under what conditions and what the consequence of an error is in each case. Not all mistakes weigh the same: recommending a less popular book is not the same as denying access to a necessary educational resource.
Objectivity needs a procedure, not a label#
Recognizing that automatic neutrality does not exist does not mean accepting arbitrary decisions. On the contrary, it makes a more demanding objectivity possible: one that shows criteria, tests evidence and can be corrected. For the library, a first step could be publicly agreeing what objectives its annual purchases serve and how those objectives are balanced when they conflict. The tool should then document its role in that balance and its limits. Visitors need not understand a complex model; they need understandable reasons for a decision affecting their community.
A small and diverse committee can review a sample of recommendations before each purchase. Its role would not be to approve every title by intuition, but to check whether the system persistently displaces a type of need. It should include library staff, representatives of reading groups and someone with authority to change the procedure. Meetings need a simple record: what pattern was observed, what change was agreed, who will apply it and when it will be checked again. Without authority and a date, review becomes a pleasant conversation that changes nothing.
The opportunity to challenge a result is equally important. If a group notices that its requests never appear, it needs a clear route to ask for an explanation and review without mastering technical jargon. The reply should say what information influenced the result, what data was missing and which person can reconsider the case. An organization can protect details that would let someone manipulate a system, but it should not use that protection to hide every reason. Providing a partial but useful explanation recognizes that people are not noise around a model; they are the ones living with its effects.
Changing a priority is also part of design#
The library’s tool can remain useful after such a review. It may save time by gathering dispersed signals or help identify repeated requests. The point is not to remove it to demonstrate independence, but to stop its assumptions from becoming an eternal rule. Records change, communities change, and a priority that is reasonable one year may be insufficient the next. Every configuration needs a review date and a way to tell whether it is producing results nobody intended.
A responsible test can begin with a short period. For two months, the team could compare automated recommendations with a list including in-person requests, proposals from associations and a protected share of the budget for underrepresented collections. It would then assess not only how many loans occurred, but who could reach materials previously left out and what additional work the change required. If the measure fails, it must be possible to stop or correct it without punishing the person who raised the concern. Learning from a discrepancy is more valuable than hiding it behind an accuracy figure.
The same logic applies beyond a library. A résumé filter, delivery route or health alert always prioritizes something: time, cost, frequency, safety or benefit. The democratic question is not whether such systems contain values, because they do from the moment they define what to measure. The question is who chose those values, on what evidence, who can dispute them and how harm is repaired when they fail. Answering it does not make AI less useful. It makes it visible in the only sense that matters: as a human decision that can still be reviewed.
Neutrality promises relief, as though a score frees us from justifying a difficult choice. But that relief is brief for people poorly represented by the rule. Responsible technology does not hide behind an objective appearance. It declares its priorities, keeps paths for hearing what it does not measure and accepts that a community can change direction. That is a more demanding standard than saying “the system decided,” but it is also how judgment, responsibility and autonomy remain with the people who live with the decision.
