CONTESTING AI DECISIONS IN COURT: PUTTING ACCOUNTABILITY TO THE TEST

 

How do you contest a decision when no one will explain how it was reached?

Consider a hypothetical case. Anokyewaa graduates with first-class honours and applies for a government-funded postgraduate scholarship. The state agency administering the scheme receives tens of thousands of applications. She is not selected. Her letter says she “did not meet the required threshold.” It gives no score and no reasons.

She later finds that several successful applicants had weaker academic records. That alone does not establish unfairness: academic results may not have been the only criterion. But she cannot find out what else counted or why her application fell short.

Anokyewaa files a request under the Right to Information Act, 2019 (Act 989), asking how her application was assessed. The agency replies that the selection was made by an automated artificial intelligence (AI) system.

She now knows that AI was involved. She still does not know what it measured, what score she received, which factors counted against her or whether an official reviewed the result. Knowing that a machine decided is not the same as knowing why. Unable to obtain a fuller explanation, she goes to court.

Her difficulty brings the article’s central question into focus: when an institution relies on AI to make a decision, can the person affected obtain enough information to challenge it? The difficulty becomes more acute where human oversight is limited or absent.

Public bodies provide a particularly demanding example because they allocate public resources and opportunities. The scholarship makes the problem concrete, but the inquiry extends to decisions about employment, licences and social benefits.

This article argues that contestability, the practical ability to challenge a decision, is a demanding test of accountability when AI is involved. A challenge tests whether records can be produced, those responsible can answer and errors can be identified and corrected. It also raises a question: if the institution cannot explain its decision, who should bear the risk of that uncertainty?

AI DOES NOT REMOVE INSTITUTIONAL RESPONSIBILITY

Anokyewaa is challenging the agency’s rejection of her application. The AI assessment matters because the agency relied on it to reach that decision.

Automation changes how an assessment is made. It does not relieve the agency of responsibility for administering the scheme. In a properly documented process, Anokyewaa should be able to identify the selection criteria, examine her assessment and obtain an explanation from those responsible. Those expectations matter whether a committee evaluates applications or uses AI to assist it.

The agency’s response does not establish that the process was defective. Records may exist but remain undisclosed. An official may have reviewed the result. Anokyewaa has simply not been given enough information to examine these possibilities. Whatever safeguards exist internally, they have not yet helped her understand her rejection.

WHAT MAKES A CHALLENGE POSSIBLE?

Anokyewaa needs to know how AI was involved, understand what happened to her application and have a practical route to seek a remedy.

Knowing that AI was used helps her frame the inquiry: what did the system assess, and how did the agency rely on its output? She does not need to master its technical architecture. She needs the criteria, the information recorded about her, the assessment against those criteria and details of any official review.

These could reveal quite different problems. Her academic results might have been entered incorrectly. The system might have applied a criterion that was not part of the stated selection rules. Alternatively, the records might show that other legitimate criteria explained her rejection. Each possibility calls for a different response.

An explanation alone cannot correct a flawed decision. She also needs a way to request review, have her concerns examined and obtain correction where an error or unfairness is established. Information helps her formulate a challenge; proceedings may help her obtain information initially withheld.

HOW THE DECISION CAN BE TESTED IN COURT

Judicial review may provide a route to question how the agency exercised its authority, applied the selection rules or treated Anokyewaa’s application. The procedures and remedies available will depend on the case. Disclosure, expert evidence and, where permitted, cross-examination may help bring the assessment into view.

Disclosure: What records explain the rejection?

Anokyewaa can seek relevant records through the applicable court procedure. These may include the selection criteria, her application data, the system’s assessment and the record of any official review. They would help her compare the stated requirements with how she was evaluated.

Commercial confidentiality may become an obstacle if the agency or vendor argues that disclosure would reveal proprietary information. The question is whether material needed to test the rejection can be disclosed with appropriate protections. She is not seeking every technical detail merely because AI was used. If the vendor holds the relevant records, the agency must also explain whether it can obtain them.

Expert evidence: What does the material establish?

An expert could help assess whether the system was suitable for the task, whether it applied the stated criteria consistently and whether the records support the agency’s explanation. A general description of the system’s capabilities may say little about Anokyewaa’s application. Useful examination requires evidence of what happened in her case.

Suitable expertise may be difficult to obtain and expensive. Access to a courtroom does not automatically provide access to the knowledge needed to test a system.

Cross-examination: Can the explanations withstand questioning?

Where witnesses give evidence and the procedure permits cross-examination, their accounts can be tested. An official may be asked what was reviewed, who approved the selection and whether an apparent error could be corrected. A vendor representative may explain the system’s operation and limitations. The agency’s officials must explain their reliance on it. Neither account necessarily answers for the other.

These tools help establish both how the system worked and whether the agency was justified in relying on its assessment. A system may calculate exactly as designed while applying an unsuitable criterion. A technically accurate score cannot, by itself, establish that the agency acted fairly or applied the selection rules properly.

LOOKING BEHIND THE DECISION

The material produced must help answer three questions: can the assessment be traced, who took responsibility and was any human review substantive?

Can the assessment be traced?

The records should connect the task assigned to the system, the data considered, the output produced and the action taken on it. For Anokyewaa, that means establishing whether her academic results were recorded correctly, how the criteria were applied and how the assessment led to rejection.

A description of how the system generally works cannot substitute for this account of her application. Missing records do not prove unfairness, but they leave a gap in the agency’s explanation.

Who took responsibility?

The vendor can explain the system’s design, represented capabilities and limitations. The agency must account for choosing and using it. Officials can explain how they interpreted and adopted its assessments.

The inquiry should identify who authorised its use, accepted its output and had authority to correct an error. An unclear internal allocation of duties should not leave Anokyewaa receiving repeated referrals to the technology instead of an answer from the institution.

Was human review substantive?

Describing the selection as automated does not establish that human judgment was absent. Did an official assess Anokyewaa’s result, or did the ranking take effect without substantive review?

A signature cannot settle that question. The official needs relevant information, sufficient time, competence and authority to question the output. The records should also show what was checked and why the assessment was accepted or changed. Human involvement has value when it enables informed scrutiny and correction; approval alone does not establish fairness.

An official may examine the assessment carefully and conclude that it is correct. The issue is whether it was evaluated before approval.

WHEN MISSING SAFEGUARDS OBSTRUCT SCRUTINY

The court’s tools depend partly on arrangements made before Anokyewaa applied. A disclosure order cannot produce a record that was never created. Questioning officials may expose missing review without reconstructing the assessment. Expert examination may be restricted if relevant technical material cannot be obtained.

Two gaps are particularly consequential: failure to retain decision records and a vendor agreement that gives the agency no effective access to documentation, performance evidence or audit. The contract does not determine the court’s powers, but it can leave the agency poorly equipped to explain a decision made in its name.

A difficulty encountered in court may therefore have begun in procurement, record-keeping or the design of the review process. Litigation can expose those choices. It cannot recreate records that were never kept or replace a review that never took place.

WHO SHOULD BEAR THE RISK?

Suppose the available records, expert assistance and protected disclosure still leave the agency unable to explain why Anokyewaa was rejected. Who should bear the risk of that uncertainty? The missing explanation now affects whether the rejection can be justified, rather than simply how much information Anokyewaa receives.

The principle I propose is that the institution should bear the risk of being unable to justify its decision. The agency chose the system and relied on its assessment. It was also better placed to secure the records and access needed to examine it. Anokyewaa controlled none of those choices. She should not be expected to prove an error hidden within a process she cannot examine.

This does not mean that every unexplained decision is automatically unlawful or that Anokyewaa should receive the scholarship. If the agency cannot explain the assessment, that should weigh against allowing the rejection to stand on that assessment alone. Where the assessment cannot be adequately explained or reconstructed, I would favour fresh, properly documented consideration of her application, subject to the court’s powers and the circumstances.

Where the inability to explain the assessment reflects a wider failure in the system or its use, reconsidering one application may not be enough. I would argue that the court should also consider, within its powers, requiring the institution to suspend use of that system for the assessment until it can demonstrate that decisions made through it can be explained and meaningfully challenged. Otherwise, other applicants remain exposed to the same failure.

That approach preserves the distinction between entitlement to the scholarship and entitlement to an accountable assessment. Anokyewaa may still be rejected after reconsideration. The institution should then be able to explain why. Opacity should not become a defence for the institution that introduced it.

REDUCING THE RISKS OF AI DECISION-MAKING

Institutions should prepare to explain and, where necessary, correct AI-assisted decisions before a dispute arises. Five measures can make scrutiny possible.

Tell affected people when AI is used. Explain its role at an appropriate stage, rather than leaving people to discover its involvement after questioning an outcome.

Keep records that make decisions traceable. Retain relevant input data, the criteria and system version used, the output produced, any human review and the reasons for the final decision. Create these records as the process unfolds. An account written later cannot replace a record kept when the decision was made.

Secure access and disclosure rights in vendor contracts. Require relevant documentation, performance evidence, independent audit access and assistance when decisions are challenged. Provide for preservation and disclosure of relevant material, with appropriate protections for personal and confidential information.

Make human review substantive and proportionate to the consequences. Give responsible officials the information, time, competence and authority to question or depart from system outputs. For decisions materially affecting people’s rights, obligations or opportunities, institutions should provide substantive review before those decisions take effect. Record what was checked and why the output was accepted or changed.

Explain decisions and provide a route to challenge them. Give the principal reasons in understandable terms and explain how people can obtain further information or seek review. A reference to an automated system or an undisclosed threshold does not explain the outcome.

These measures cannot prevent every error or dispute. They help institutions explain their decisions, correct mistakes and reduce the risk of liability arising from failures in the decision-making process. Preparing to answer a challenge is part of responsible decision-making and sound AI governance.

Institutions intending to use AI in decision-making should therefore take procurement seriously. Before adopting a system, they should establish its suitability and secure access to the records, documentation and vendor assistance needed to explain and defend decisions. These arrangements must be secured before the institution relies on the system; a dispute is too late a moment to discover that its contract leaves it unable to account for the outcome.

CONCLUSION

Anokyewaa does not yet know whether her rejection was justified. Without an explanation of how her application was assessed, she cannot test whether the criteria were appropriate or correctly applied.

An assurance of fairness means little unless the person affected can test it. That requires accessible information, an assessment that can be examined and someone who can explain the decision. If the institution cannot provide that account, it should bear the risk of uncertainty rather than require the applicant to prove a concealed error. Reconsideration may produce the same outcome, but it should produce an outcome the institution can justify. Where the inability to explain decisions reflects a wider failure, the institution should be required to suspend use of the system for those assessments until that failure is addressed.

AI can assist decision-making, but the institution remains responsible for the decisions it adopts. When Anokyewaa asks why, referring her to the system does not answer her question. Accountability is not outsourced along with the assessment.

Note: This article draws on the framework of contestability and accountability in the UNESCO / Saïd Business School course, “AI and Transparency in Justice Systems”. The hypothetical case, its application to the decisions of public bodies and the argument on who should bear the risk are the author’s own.