The question that makes official narratives about education technology uncomfortable is simple: what has genuinely improved for students after devices, platforms and licences were purchased? Counting connected classrooms or login sessions is not enough. A digital policy earns trust when it can show better opportunities to learn, fewer barriers, and decisions a school is prepared to revise when the promised results do not arrive.
Imagine a secondary school announcing that every Year 7 class will use a new platform. The presentation promises innovation, personalisation and preparation for the future. Tablets are distributed, staff receive an afternoon of training and an institutional photograph is published. Three months later, some families see more homework on screens, teachers duplicate tasks on paper and online, and pupils with unreliable connections finish work on a parent’s phone. The technology operates technically, but nobody defined the pedagogical improvement it was meant to produce.
Asking about outcomes is not opposition to technology. It protects technology’s usefulness. Schools need digital tools, but a tool is not a strategy by itself. Judgement should begin with an observable need: receiving timely feedback, practising without public embarrassment, accessing adapted materials, or collaborating with an absent classmate. A resource is then selected, limits are established and effects are examined. Reversing this order turns any purchase into a commitment that later becomes difficult to question.
From an inventory of equipment to a theory of change#
A serious evaluation distinguishes resources, activities, immediate outcomes and impact. Resources include equipment, connectivity, time and technical support. Activities are actual practices: writing drafts, receiving feedback, building a simulation or communicating with a family. Immediate outcomes may include participation, completed submissions or response times. Impact is more demanding: lasting understanding, independence, inclusion or lower dropout. Confusing these layers makes it possible to declare success because computers were distributed even though learning remained unchanged.
A serious evaluation distinguishes resources, activities, immediate outcomes and impact.
Before implementation, a school can write one testable sentence: “If we use this tool to give short feedback during the writing process, students will revise their texts more often and improve their arguments.” The sentence identifies a practice, a mechanism and an outcome. It also reveals alternatives: perhaps the problem could be addressed by reorganising tutorials or using smaller groups for one activity. The fair comparison is not technology against nothing, but the digital proposal against the best available option, with all costs included.
A baseline must be collected before launch. It might include a sample of work, feedback times, attendance, requests for help and brief interviews. Teachers do not need to become a statistics department. A few understandable indicators connected to the objective are enough. If the aim is better writing, measuring screen minutes is secondary; changes between drafts, the ability to justify revisions and the distribution of useful feedback are what matter.
Convenient metrics can tell the wrong story#
Platforms provide dashboards of logins, clicks and minutes because those data are easy to record. Activity, however, is not learning. One pupil may remain connected because the instruction is confusing; another may disconnect after downloading the material and work with deep concentration. Frequency of use does not prove quality either. It may reflect enthusiasm, an administrative requirement or an interface demanding too many steps. Every metric needs a question about meaning and a complementary form of evidence.
Averages hide distributions. If the overall result improves while outcomes deteriorate for disabled pupils, recent arrivals, rural learners or families with fewer resources, the policy cannot be called inclusive. Results should be examined across groups without publicly labelling individuals, alongside specific experiences. Equity does not mean handing everyone the same device. It means ensuring each student has sufficient access, support, formats and time to participate with dignity.
There are also costs absent from the invoice: setting up accounts, recovering passwords, adapting materials, moderating messages, handling incidents, reviewing false alerts and learning interface changes. Some fall on teachers and others on pupils and families. A pilot should record who spends this time. A solution that saves ten minutes for the school but makes forty families troubleshoot at night has not created efficiency; it has moved labour beyond the institution.
A small test before a promise becomes an obligation#
A useful pilot has limited scope, a known duration and a meaningful comparison. It might cover two teaching units over six weeks, rather than an entire year from day one. Responsible people, support and a decision date are agreed. Teachers know what is expected; students know how to report failures; families receive a short explanation of purpose, data and alternatives. Starting small allows learning without turning every error into a large-scale problem.
Success and stopping criteria are fixed before the pilot begins. For example, the proportion of students revising their work at least once should rise without widening the submission gap or adding more than an agreed amount of weekly support. Use stops if an outage prevents assessment, an important recommendation cannot be explained or accessibility barriers persist. Establishing these conditions before viewing results reduces the temptation to redefine success around a decision already made.
Comparison must respect context. It would be unreasonable to contrast a motivated volunteer group with an expert teacher against a group given the tool without support. Better options include comparing the same practice before and after, alternating similar tasks or studying several groups whose conditions are documented. Quantitative evidence should sit beside actual work, observations and conversations. A small numerical difference may matter greatly when one learner regains participation; a statistical gain may mean little if tasks simply became easier.
Questions for suppliers and public authorities#
A school leader can request written answers to five questions: what problem does the product solve; what independent evidence supports the claim; which data does it require; what happens when the contract ends; and how can a person challenge a decision? Accessibility, export of student work, support location, price changes and interoperability also matter. “It uses artificial intelligence” neither describes an educational function nor demonstrates necessity.
Evidence supplied by a vendor helps a school understand the product, but it does not replace independent evaluation. Readers should check who funded a study, which learners participated, what comparison was used, how long it lasted and who dropped out. Increased satisfaction over two weeks does not demonstrate long-term learning. When strong research is absent, caution does not always require rejection, but it does require treating the tool as experimental, limiting consequences and gathering local evidence before expansion.
Contracts should preserve a genuine exit. Schools need to recover work and necessary records in usable formats, know when copies will be deleted and avoid dependencies that prevent changing suppliers. Secondary uses incompatible with education should be prohibited, and significant changes should require notice. Governance sounds administrative until an essential feature disappears, the price rises or a family asks what information the company still holds.
Student and family voices are evidence, not decoration#
Consultation is not asking whether people “like the app”. Useful questions describe situations: could you enter without help, did you understand the task, did the feedback help you improve, did you feel watched, and which digital activity would you replace with conversation, paper or movement? Those who stopped using the tool must be included, because dashboards show only people who managed to enter. A short, diverse discussion group often reveals barriers that no technical log captures.
Families need a simple channel and answers that do not require specialist vocabulary. They should know the purpose, expected time at home, data processed and alternative when access fails. They cannot become unpaid technical support. If an activity depends upon constant adult assistance, its design needs review. Student independence is an educational outcome; moving instructions and repairs into the home may increase it for some and reduce it for others.
Student participation includes the right to disagree without a penalty. If a platform recommends a level, detects conduct or marks an answer, the student should be able to request an explanation and human review. That right also improves the system: challenges reveal patterns of error. Recording the reason and resolution helps distinguish an exceptional incident from a design that repeatedly disadvantages one group.
A dashboard a community can understand#
A quarterly report can fit on one page. It should restate the objective, present three or four indicators, explain differences between groups, summarise incidents and describe the next decision. Anonymised examples of work before and after may be included. Uncertainty should be acknowledged: a small sample, a teacher change or another intervention at the same time. Transparency does not require publishing personal data; it requires explaining what is known, what is not and how the school will investigate.
Success does not create an obligation to expand. A tool may contribute greatly to language support but little to assessment, or work in upper secondary while adding burdens in primary school. Scaling means preserving the conditions that produced the result: training, time, assistance and an alternative. If they disappear during expansion, the same programme is no longer being replicated. The reasonable decision may be to retain one specific use, renegotiate the service or close the pilot.
References for evaluation in the public interest#
UNESCO’s Global Education Monitoring Report on technology in education urges decision makers to ask whether technology is appropriate, equitable, scalable and sustainable, and to place learners and teachers before products. The European Commission’s Digital Education Action Plan frames transformation around institutional capacity, skills and high-quality ecosystems. The Joint Research Centre’s DigCompEdu framework supports examination of teaching practices, assessment, inclusion and learner agency, not merely technical command.
These references do not provide an automatic score for buying or rejecting a platform. They help communities construct public-interest questions: who benefits, under which conditions, what evidence demonstrates it, what opportunity cost is accepted, and how can the school leave if it fails? Answering requires leaders, teachers, pupils and families. A supplier may provide information, but cannot decide what counts as good education.
The uncomfortable question is a form of care#
At the next announcement, a family can ask which precise problem is being addressed and how the school will know in six months whether it has been solved. A teacher can request trial time, stopping rules and an account of added work. A pupil can demand an alternative when access fails and a person to review important decisions. Leaders can publish a short balance before renewal. These are demanding questions, but entirely compatible with a school open to change.
A mature digital policy is not afraid to discover that a tool contributes less than promised. It learns, corrects and concentrates resources where value appears. Nor does it hide good outcomes behind grand language: it shows better work, reduced barriers and time recovered for teaching. The uncomfortable question is no longer “are we innovating?” but “what can our students now do and understand that they could not before, who is still excluded, and what decision will we make with that evidence?” That is the conversation that turns technology into education.
