A company introduces a system that allocates shifts, calculates targets and recommends whose contract should be renewed. During the first few weeks everything looks innovative: fewer spreadsheets, faster answers and orderly planning. Six months later nobody knows how to build the rota without the platform, managers accept its scores although they cannot explain them, and a service outage brings work to a halt. The technology did not become malicious; the organisation crossed, almost without noticing, the boundary between using a tool and depending on it.
This boundary matters especially to someone starting work. A young employee may interpret any new application as a neutral authority because it arrives with numbers, charts and words such as intelligence, optimisation or accuracy. Yet a system does not discover by itself what doing a job well means. Someone selects the data, defines the target, decides how much error is acceptable and determines who may correct an output. The useful question is not whether the company uses artificial intelligence. It is which capabilities the team retains when the system is unavailable, makes a mistake or pursues an unsuitable objective.
The International Labour Organization says generative AI is likely to transform many occupations rather than eliminate them, precisely because numerous tasks still require human input. That transformation can improve employment when workers have dialogue, training and the ability to intervene. It can also degrade work by turning professionals into operators who no longer understand the process. Innovation expands human capability; dependence removes genuine alternatives.
A simple test: what happens when the system says no?#
Imagine Sara, a recently hired technician at a maintenance company. Software assigns visits using distance, history and expected duration. Its proposal saves travel, but one day it puts an urgent repair last because the new customer has no history. Sara identifies the risk. Her coordinator can inspect the criteria, change the order and record the reason. Here the system helps: it proposes, leaves an audit trail and permits a better-informed professional decision.
Imagine Sara, a recently hired technician at a maintenance company. Software assigns visits using distance, history and expected duration.
Now change three details. The coordinator does not know the criteria, the screen provides no way to alter the route and management penalises any departure from the recommendation. Although a person formally appears in the process, effective human oversight does not exist. The supposed supervisor merely confirms. If nobody maintains an alternative way to plan during an outage, the organisation depends on a system it can neither challenge nor replace.
Five questions help analyse any tool. What decision does it prepare or make? Which relevant information does it exclude? Who can recognise an exceptional case? Who has the authority and time to change the output? How does the service continue when the tool is unavailable? A vague answer such as “a manager reviews it” is insufficient. The organisation should identify a role, channel, deadline and possible action. Oversight exists only when it can alter the outcome before harm becomes irreversible.
Innovation improves a capability; dependence replaces judgement#
A spreadsheet that detects inconsistencies can free time for conversations with customers. An assistant that summarises case files can help locate relevant history. An algorithm that forecasts workload can support decisions about extra staffing. In these examples the gain is specific, measurable and compatible with professional knowledge. People can compare the proposal with original documents and still know how to perform the essential task.
Dependence has different warning signs. First is loss of understanding: the team knows which buttons to press but cannot explain what information produces a recommendation. Second is loss of skill: after several months nobody can complete a critical operation manually, even more slowly. Third is displaced authority: contradicting the system requires far more justification than obeying it. Fourth is concentration: data, instructions and support belong to a supplier that cannot be replaced without stopping the service. Fifth is loss of organisational memory: exceptions resolved by people are not documented, so the organisation does not learn.
Not every automated process needs a permanent manual duplicate. It would be absurd to demand that every complex computation could be recreated with a pencil. Proportionality depends on impact. If a system adjusts colours in a presentation, an interruption is inconvenient. If it shapes access to employment, pay, assessment, safety or dismissal, the cost of an error is greater and the alternative must be stronger. The European Commission places certain AI uses in employment and worker management among high-risk areas and highlights requirements including traceability, information and human oversight. This suggests a practical rule: the more an output affects rights and opportunity, the less reasonable it is to accept a closed box.
The four layers of dependence#
Technical dependence is the most visible layer: servers, connectivity, licences or proprietary formats without which activity stops. It can be reduced through usable backups, data exports, continuity procedures, restoration tests and contracts that explain what happens when the service ends. Saving files is not enough if nobody can open them or reconstruct the relationships among them.
Cognitive dependence appears when a team stops practising professional judgement. It may affect a doctor who reads a recommendation first and then seeks evidence to confirm it, a recruiter who interviews only highly scored applicants or a delivery worker who follows an unsafe route because the application orders it. Consulting an aid is not the problem; allowing it to anchor interpretation is. One useful safeguard is to decide a sample of cases without seeing the automated score first and then compare the two conclusions.
Organisational dependence arises when incentives and hierarchies turn a recommendation into an order. If overriding it lowers a worker's performance rating, requires an impossible procedure or triggers a disciplinary call, the review button is decorative. Management should measure successful corrections too, protect reasoned disagreement and avoid targets that reward speed at the expense of safety or quality.
Moral dependence is the most dangerous layer: “the algorithm decided” becomes a way to dissolve responsibility. No interface can answer to an injured person, consider genuinely new circumstances or accept a duty to repair harm. There should be a named human owner of the outcome and another owner of system performance, each with clear duties. The NIST AI Risk Management Framework organises work around governing, mapping, measuring and managing risks throughout the lifecycle. It also emphasises documenting roles and distinguishing responsibilities in human-AI configurations.
A traffic-light assessment before adoption#
Green means the purpose fits into one testable sentence, such as reducing transcription mistakes without recommending discipline. Data origins are known, affected people receive understandable information, testing reflects real cases, the team retains access to original evidence and a competent person can correct the output. There is also a rehearsed incident plan and a review date. Under those conditions, technology can expand capability without capturing the process.
Amber means the supplier promises accuracy but does not explain limitations; the organisation measures savings but not errors across groups; review depends on asking someone for a favour; or only one employee can operate without the application. These signs do not always require cancellation, but they do justify limiting the pilot. The organisation can reduce the number of people affected, prohibit irreversible decisions, collect disagreements and establish exit criteria before expanding use.
Red means nobody can identify the accountable person, an explanation sufficient to review a case is unavailable, records are missing, appeals return to the same automated outcome, or an outage disables a critical function with no alternative. It is also red to use data gathered for another purpose without assessing relevance, conceal the tool from people affected by its decisions, or punish someone who reports failure. Deploying faster in this situation is not innovation. It transfers risk to workers and members of the public.
What a worker can ask without being a technical specialist#
A person does not need to code in order to ask precise questions. When a score affects a shift, bonus or application, a worker can ask what function the system performs; which personal and employment data it uses; which period it considers; what factors it cannot see; who reviews errors; how additional context can be submitted; and when a human response will arrive. It is sensible to retain relevant screenshots, communications, earlier results and concrete facts, without taking confidential material that the person is not entitled to keep.
An effective challenge describes the discrepancy instead of alleging motives: “The score counts three late deliveries, but two occurred during a recorded equipment failure. I request review by a person authorised to correct it.” If the impact is collective, speaking with worker representatives can reveal patterns that one isolated case cannot show. Laws and routes for complaints differ by country and situation, so an important decision may require trade-union, professional or legal guidance.
Workers can also ask what training the team will receive. A tutorial about buttons does not teach oversight. Training should address known limitations, examples of failure, groups or circumstances that are represented poorly, signs of drift, ways to stop the system and duties to document decisions. Someone assigned to supervise needs time, access and protection when disagreeing. Naming that person without those resources merely creates an alibi.
Design the exit before entering#
The question “How would we stop using this?” belongs before purchase. An exit plan identifies which data can be exported, in what format, how long migration would take, which knowledge must remain inside the team and which critical tasks need an alternative procedure. It assigns responsible people and is tested periodically like an emergency drill. A backup that has never been restored is a hope, not a guarantee.
For sensitive processes, an organisation can maintain an independently completed sample. The aim is not to compete with automated speed but to preserve competence and detect drift. Examples include independently reviewing a percentage of applications, comparing proposed shifts with fatigue and caring responsibilities, or assessing files before displaying their rank. Discrepancies should drive changes: revising criteria, training people, restricting use or withdrawing the tool. If incidents are collected but never change decisions, oversight is theatre.
Predefined stopping thresholds are also necessary: error rates, unresolved complaints, unequal impact, poor availability or supplier changes. Without thresholds, every failure is described as exceptional and the system becomes permanent through inertia. A genuine capacity to innovate includes the ability to leave a solution that no longer achieves its purpose.
A family conversation about the future of work#
Parents and educators can prepare a teenager without frightening them or promising that mastery of one application will guarantee employment. A useful conversation starts with an everyday task: using navigation to choose a route, a spelling tool to write or recommendations to select music. Ask what the person gains, what they stop practising, which data they give away, how they recognise an error and what they could do without the tool. Then transfer the same reasoning to a workplace.
Capabilities that resist dependence are not merely digital. They include defining a problem before opening an application, checking primary evidence, explaining a decision to another person, acknowledging uncertainty, recording exceptions, caring for the person who bears an error and learning an alternative procedure. Knowing how to use AI will be useful. Knowing when not to follow it will be an equally professional skill.
Families can try an exercise: complete one task with automated assistance and then repeat it using a primary source; record differences in time, errors and unanswered questions. The objective is not to prove human superiority but to understand how work is divided. When the tool contributes speed while the person retains context, accountability and the power to correct, there is collaboration. When a person merely lends their name to approve an incomprehensible output, there is dependence.
A checklist for keeping control#
Before calling a system innovative, a team should be able to say: we know the specific problem it solves; we have measured benefits and harms rather than speed alone; we know which data it uses and which reality it omits; affected people understand its role; an accessible human review can change the outcome; disagreements are recorded; continuity is rehearsed; internal knowledge is preserved; accountable people can be named; and conditions exist for pausing or withdrawing the system.
If several answers are missing, the organisation need not reject all technology. It can pause deployment, narrow the scope and turn those gaps into requirements. Prudence is not opposed to innovation. It prevents a commercial promise from becoming unquestionable infrastructure. The best tool is not the one that makes the most decisions, but the one that enables competent people to decide better and to answer when something goes wrong.
Further reading is available in the ILO update on generative AI and jobs, the European Commission's official explanation of the AI Act, the NIST Artificial Intelligence Risk Management Framework and its specific guidance on human-AI interaction and responsibilities.
