At the final meeting for a county's digital housing-repair pilot, the project lead opened a dashboard and pointed to a clear result: requests were being assigned faster than before. The vendor recommended a county-wide rollout. Then a tenant advocate asked how many people had been able to submit a request without borrowing a phone, navigating a long form, or disclosing a problem to a landlord first. A repair coordinator noted that staff were still transcribing requests from walk-in visitors, but that work did not appear on the dashboard. The pilot had produced a useful signal about assignment speed. It had not yet established that the service was fairer, easier, or safe to extend.
This is why a digital pilot needs a real ending rule. Its purpose is not to generate a favorable story before the budget line expires. Its purpose is to create enough accountable knowledge to choose among expansion, revision, repetition, pause, or retirement. A project can make a sensible decision to stop and still be successful, because it has prevented a weak assumption from becoming a system-wide obligation. Conversely, a pilot can look impressive and still fail its public purpose if it moves forward without explaining who benefited, who carried extra burden, and what evidence would make the organization change course.
The closing work begins by returning to the original promise. “Improve housing services” is too broad to assess. “Test whether a mobile request route shortens repair triage while preserving a practical non-digital route for tenants with unreliable access” is a claim that can be investigated. It identifies the problem, the intended population, the period of observation, the safeguard, and the boundary of the experiment. Without that specificity, teams often celebrate whatever metric improves, even if it has little connection to the reason people were asked to take part.
Measure more than the tool can count#
Product activity is not the same thing as public value. A completed form does not prove that a tenant understood their options. A shorter queue does not prove that a repair became safer. A lower cost per request does not prove that a hidden task was not shifted onto a resident, an advocate, or a receptionist. A credible pilot review separates use, outcome, and distribution. Use covers events inside the system: submissions, delivery failures, response time, and support contacts. Outcome asks whether the underlying situation changed: did urgent repairs reach the right team sooner, did repeat requests fall, did residents gain a clearer route to help? Distribution asks who received that improvement and who was excluded from the measure.
Product activity is not the same thing as public value. A completed form does not prove that a tenant understood their options.
For the housing pilot, the use data might show that online requests were assigned quickly. Outcome evidence would need to look at repair completion, urgent cases, repeat reports, and delays across comparable periods. Distribution evidence would examine whether the route worked for tenants with shared devices, limited data plans, disabilities, limited English, or a need for confidential contact. An average can improve while a smaller group waits longer or loses privacy. Naming that possibility is not hostility toward the pilot; it is the minimum discipline required before treating a partial result as a public success.
Unexpected effects deserve the same attention as planned metrics. Perhaps the form reduces duplicate reports. Perhaps its required photo upload prevents people from reporting a leak when they cannot safely take a picture. Perhaps staff save time on one step but spend it correcting addresses in another system. A weekly incident log, reviewed with frontline workers, can reveal this work long before it becomes a crisis. The goal is not to demand that a pilot be flawless. It is to ensure that surprises alter the decision rather than being removed from the final presentation.
Agree on the exit conditions before the result arrives#
A closing decision is more trustworthy when its conditions are visible before the team knows the answer. The county might decide to expand only if urgent requests are not delayed, if the offline route remains staffed, if access barriers identified in testing have a verified response, and if a named person can correct a harmful error. It might choose to revise and repeat if there is a promising but uneven benefit. It might pause if essential evidence is missing, or retire the tool if its basic design depends on a burden that cannot be made acceptable. These conditions are not a mechanical scorecard; they guard against rewriting “success” to fit the most convenient result.
Some evidence should explicitly be treated as insufficient on its own. A voluntary satisfaction survey is valuable, but it may not reach people who gave up. A fast demonstration may show that a trained participant can finish the form, not that someone under stress can find help after a failed submission. A positive vendor report may describe uptime without addressing whether the service worked for people who never entered the digital route. By naming these limits, a team gives decision-makers a more honest basis for action.
The exit rule also needs a stop signal. Who can suspend a feature if a tenant's contact details are disclosed to the wrong person? What threshold triggers a review if urgent cases are repeatedly misclassified? How quickly must the service restore a human route when the system fails? If the answer is spread across several contracts and committees, the pilot is not ready to scale. Accountability is practical: a named owner, an authority to act, evidence to review, and a deadline that preserves the chance to prevent harm.
Treat lived experience as evidence#
Dashboards cannot see all the conditions that make a service work. A repair coordinator can describe where the workflow creates double entry. A tenant can explain that receiving a notice on a shared phone creates risk. A community organization can identify language in the form that turns a simple repair request into an intimidating legal disclosure. These accounts do not replace quantitative evidence; they show what the numbers leave unmeasured.
The pilot should make room for this evidence without collecting more personal information than it needs. Short voluntary interviews, accessible feedback routes, observation by frontline staff, and discussion with representative community groups can reveal recurring barriers. The report can store anonymized patterns instead of detailed stories. If participants say they needed another person to interpret the form, the relevant conclusion is not their identity. It is that the route may compromise independent access and requires a response such as clearer language, an alternate channel, or a direct human callback.
One powerful closing question is, “What would a person do when this fails at the worst possible moment?” The answer tests whether an alternative is real. A help line available only during narrow office hours, an appeal requiring another account, or an offline option that forces people to retell every detail is not an adequate safeguard. The final record should list barriers found, changes made, and unresolved risks. That record allows the next decision to be proportionate rather than optimistic by default.
Leave a record that can be challenged#
An ending should create a concise but usable record: the original hypothesis, scope, participant routes, measures and their limits, incidents, changes made during the test, experience evidence, data-retention decisions, and the final choice. The record should separate observation from interpretation. “Assignment time decreased by twelve percent in two offices” is an observation tied to a period and method. “The digital route improves access” is an interpretation that requires much broader support. Keeping them separate makes it possible for a later team to question a conclusion without having to reconstruct the entire pilot.
The record also assigns responsibility. If a tenant later says the pilot affected their case, the county should be able to find the version in use, the available fallback, and the person able to review the issue. Responsibility cannot disappear between a software provider, a digital team, and a public service desk. Contracts may divide tasks; they do not remove the public body's duty to understand limits, respond to complaints, and stop a process that is no longer defensible.
Closing includes deleting what the test no longer needs. Pilot data can include contact details, device events, support notes, and partial applications. Retaining them indefinitely “for future learning” turns a limited experiment into open-ended collection. The team should apply the promised retention period, restrict access while review is under way, and document what aggregated evidence will remain. A pilot that asks people to trust a new route cannot treat the end of the project as the start of permanent storage.
Make the next step answer the evidence#
The county may decide that its most responsible next step is not a full rollout. It may run a second, smaller test with a shorter form, a staffed walk-in route, confidential contact choices, and a clear pause condition. It may learn that the best improvement is not another digital feature but better coordination between the repair desk and local advocates. A narrower next step can be more ambitious in the useful sense: it commits to learning from the people who bear the consequences instead of using them to validate a predetermined plan.
The final question is not, “Can we put this into production?” It is, “What decision can we explain and defend to the people who used the service and to the people who will have to correct it?” A digital pilot has done its job when it leaves a proportionate answer: expand with safeguards, revise with a specific plan, pause to understand a risk, or retire a design that should not grow. That kind of ending is not an administrative afterthought. It is the point at which a trial becomes public learning rather than a quiet route from experiment to obligation.
