WHEN AI BECOMES THE DEFENDANT: THE JUDGE’S QUANDARY

 

INTRODUCTION

Artificial intelligence is moving beyond assisting human decision-makers to making decisions on its own. In the situation considered here, the system does not merely recommend, rank, score, flag or predict an outcome for a human to accept or reject. It makes the operative decision itself, without human review.

When such a decision is challenged in court, a familiar judicial problem takes an unfamiliar form. The institution that deployed the system may still be the defendant, but no human actor made or approved the decision under challenge.

This is the sense in which AI becomes the subject of litigation: not because the system acquires legal personality or literally becomes a defendant, but because the court must determine whether its decision can be understood, tested and justified sufficiently to withstand scrutiny under the rule of law.

The difficulty is practical. When a decision affects a person’s rights, obligations or interests, that person may challenge it and ask the institution responsible to explain why the decision was made. With human decisions, courts ordinarily have familiar routes to the basis of the outcome through reasons, records, evidence or the decision-maker. With autonomous AI, the institution may remain legally responsible even though no human made or approved the individual decision. The court may therefore know who must answer for the decision without yet knowing how it was reached.

This article does not address AI-generated or AI-analysed evidence, or systems that merely assist a human decision-maker. It focuses on autonomous AI systems that make the operative decision without human review.

Using a practical customs scenario, the article asks three questions: why does autonomous AI create a different judicial problem; what can the judge do to examine the decision; and what should follow if, despite reasonable efforts, the court still cannot establish why it was made?

ONE DISPUTE, TWO DECISION-MAKERS: WHY GHS150,000?

Consider an importer whose goods arrive at the port for clearance. A dispute arises over the duty payable. On the surface, it is an ordinary customs valuation case. But change the decision-maker, and the court faces a different problem.

A. If a Customs Officer Makes the Decision

Suppose a customs officer examines the importer’s declaration and supporting documents and assesses the duty at GHS150,000. The importer pays under protest to secure release of the goods and later challenges the assessment in court.

The court has familiar ways of examining how the decision was reached. The officer’s records may show what information was considered, which valuation criteria were applied and why the importer’s position was rejected. Where appropriate, the officer may also explain the decision, give evidence and be cross-examined.

The assessment may ultimately be upheld or overturned. The important point is that the decision arose from an identifiable human decision-making process that the court can examine through familiar legal means.

B. If an Autonomous AI System Makes the Decision

Now change only one fact: the decision-maker. An autonomous AI system, operating without human review or approval of the individual case, assesses the same importer at GHS150,000. The importer again pays under protest and challenges the assessment.

Customs remains the formal defendant. But no customs officer made or approved this particular assessment. An officer may explain what information was supplied to the system or why Customs chose to use it. A technical expert may explain how the system generally works. But the machine itself cannot enter the witness box to explain, or be cross-examined about, how it reached this particular decision. The judge is therefore left with the same basic question: Why GHS150,000?

The deeper difficulty is that no human exercised judgment over this particular assessment. The court may therefore have to reconstruct how the system moved from the importer’s information to the amount imposed before it can determine whether the decision was lawful or justified. The defendant has not changed. The decision-maker has.

THE JUDGE’S FIRST PROBLEM: WHO ACTUALLY MADE THE DECISION?

Once the court establishes that the GHS150,000 assessment was made by the autonomous system, an important distinction follows.

Customs may still be legally answerable because it chose to use the system and enforced the resulting assessment. But no customs officer made, reviewed or approved this particular decision. The judge therefore faces two separate questions:

The first is: Who must answer for the consequences of the decision? That may be Customs.

The second is: Whose decision is the court actually being asked to examine? Here, it is the autonomous system’s decision.

The distinction matters because identifying Customs as the defendant tells the court who must respond to the claim. It does not explain why this importer was assessed GHS150,000.

Customs may explain why it chose the system, what purpose it serves or what safeguards surround its use. Those matters may show why Customs remains responsible. But they do not explain how the system reached this particular assessment.

The judge must therefore look beyond the question of responsibility and ask whether the autonomous decision itself can be traced, explained and tested, and ultimately withstand scrutiny under the rule of law.

THE JUDGE’S NEXT QUESTION: WHY GHS150,000?

Once it is clear that the autonomous system made the assessment, the judge’s next question is straightforward: How did the system arrive at GHS150,000?

The court does not need to understand every line of code or the mathematics behind the system. Nor is it enough for Customs to say that the system is generally accurate, has been tested or is widely used. Those answers may say something about the system as a whole, but they do not explain the decision before the court.

The judge therefore needs to follow the path from the information entered into the system to the amount imposed. What information about the goods was used? Were comparable imports considered? What factors materially affected the valuation? Were any assumptions made? What do the system logs or audit records show? Most importantly, can those records explain why the system produced GHS150,000 rather than another amount? This is where three related ideas become important.

·       Transparency tells the court how the system generally works.

·       Explanation tells the court why the system reached this particular decision.

·       Contestability asks whether the importer has enough information to understand and meaningfully challenge that explanation.

The distinction matters. Customs may be able to explain that the system considers historical valuations, product descriptions, country of origin and other relevant information. That may make the system broadly transparent. But the central question remains unanswered if Customs cannot show which information mattered in this case and how it led to GHS150,000.

The issue, therefore, is not whether the judge can understand the AI system in every technical detail. It is whether the court can understand enough about this particular decision to trace how it was reached, test its basis and determine whether it can withstand scrutiny under the rule of law. And if Customs cannot provide that explanation, the next question is unavoidable: Why not?

WHAT IF CUSTOMS SAYS: “THE SYSTEM IS A BLACK BOX”?

Suppose Customs answers the judge’s question by saying: “The system is a black box.” In simple terms, a black box is a system where information goes in and a decision comes out, but the path from one to the other is not readily visible or understandable. Here, the importer’s information went in and a GHS150,000 assessment came out, but how the system moved from one to the other is unclear. That should not end the inquiry. The judge’s next question should be: What exactly makes it a black box? There are at least three possibilities.

The first is genuine technical complexity. The system may process many factors in ways that are difficult to translate into a simple human explanation. Even then, the court can still ask what can be recovered from the available inputs, outputs, logs, audit trails, validation records and other documentation.

The second is lack of specialist knowledge. The information may exist, but the judge, lawyers or parties may not have the expertise to understand it. That does not necessarily mean the decision cannot be explained. It may mean that expert assistance is needed to interpret the available records.

The third is different. Customs or the system provider may actually have the information needed to explain the decision but refuse to disclose it because of trade secrecy, security concerns or the risk that importers could game the system.

That distinction matters. “We cannot explain the decision” is not the same as “we will not disclose the information needed to explain it.” The first may be a technical or expertise problem. The second is a legal and procedural problem about access to information.

The judge should therefore not accept the label “black box” at face value. The real task is to identify what is preventing the court from understanding the decision. If the problem is technical complexity, the court must ask what can still be reconstructed. If the problem is lack of expertise, technical assistance may help.

If the problem is refusal to disclose, the issue becomes different: How far should trade secrecy, security and concerns about gaming the system be allowed to limit judicial scrutiny of the GHS150,000 assessment?

WHAT IF CUSTOMS REFUSES TO DISCLOSE HOW THE DECISION WAS MADE?

Suppose the problem is not that the system cannot be explained, but that Customs says the information needed to explain it cannot be disclosed. That changes the nature of the dispute.

The judge is no longer dealing mainly with technical complexity. The court is now dealing with a familiar legal and procedural problem: how to balance the need for judicial scrutiny against legitimate claims of confidentiality, security or commercial sensitivity.

An important distinction follows: information that cannot safely be disclosed to the public may still need to be made available to the court under appropriate safeguards.

Customs may have good reasons for refusing to make the inner workings of a valuation or risk-assessment system public. A system used to detect undervaluation, fraud or suspicious import patterns may become less effective if every relevant factor is openly disclosed. A private system provider may also have legitimate concerns about proprietary information. But those concerns do not answer the judge’s central question: Why was this importer assessed GHS150,000?

If trade secrecy is raised, the governing principle is one courts already apply everywhere: he who alleges must prove. Customs or its provider is the party making the claim here — that this information is protected, and that disclosing it would cause real harm. That is an affirmative allegation like any other, and the ordinary rule of evidence answers it without needing any special exception for AI: the party making the allegation carries the burden of proving it. The importer is not alleging anything about the trade secret; she is only asking for information relevant to her own claim that she was overcharged. She does not have to disprove a trade secret she has no access to, and she does not have to show that disclosure is safe. Customs and its provider must show, item by item, what information is genuinely proprietary and why disclosing it would cause real commercial harm. A general claim that “the system is proprietary” does not meet that burden. The question for each item is specific: is the harm plausible, is a less restrictive alternative available such as independent audit, and has similar information already been disclosed elsewhere? The existence of a trade secret does not mean that everything connected to the system must remain hidden.

If security is raised, Customs should identify the actual risk. What vulnerability would disclosure expose? Is the risk real and specific? Could the court obtain the information it needs without making sensitive details public?

The concern about gaming the system may be particularly relevant in customs administration. Customs may reasonably fear that revealing certain indicators could allow importers to manipulate their declarations or avoid scrutiny. But the court should still ask whether the particular information sought would actually create that risk.

The key point is this: secrecy may affect how information is examined, but it should not automatically determine whether the decision can be examined at all. The court may therefore need to separate what must be examined from who should be allowed to examine it. Depending on the applicable procedural law, this may involve restricted disclosure, confidentiality protections, independent expert access, in-camera review or other controlled arrangements.

The judge’s question is therefore not: “Must this AI system be completely transparent?” Rather it is: “Can I obtain enough information, under appropriate safeguards, to understand why this importer was assessed GHS150,000 and to test whether that decision can withstand scrutiny under the rule of law?”

WHAT CAN THE JUDGE DO?

By this stage, the judge knows that the GHS150,000 assessment was made autonomously, has asked why that amount was reached, and has identified what may be preventing an answer. The question now becomes practical: what can the judge do?

The answer does not require the judge to become an AI expert. Courts routinely deal with matters that require specialist knowledge—medicine, engineering, accounting, DNA evidence and complex financial transactions. AI should not be different simply because the technology is unfamiliar. The judge’s role is not to understand the technology as an engineer would, but to ensure that the decision it produced can be properly examined by the court.

The starting point is the decision itself. The court can require Customs to identify and preserve the material capable of showing how the assessment was reached: what information went into the system, what came out, what materially influenced the outcome, and what records exist of the process in between. The inquiry should remain focused on the GHS150,000 assessment rather than becoming an investigation of the entire AI system.

Where those records exist but require specialist knowledge to understand, the court can seek expert assistance. The useful expert is not one who merely tells the judge that the AI system is sophisticated or generally reliable. The expert should help answer the question that matters: can the available records explain why this importer received this assessment?

Where relevant information is sensitive, the court can consider whether access can be controlled rather than denied. Subject to the applicable procedural law, this may involve targeted disclosure, confidentiality protections, restricted expert access or in-camera examination. The objective is to protect legitimate interests without placing the decision beyond judicial scrutiny.

The importer must also be able to meaningfully challenge the decision. Producing thousands of pages of technical material does not necessarily achieve that. Nor does an explanation that only the system provider can understand. What matters is whether the information before the court enables the importer to understand the basis of the assessment well enough to challenge it. The judge can therefore bring the problem back to five practical questions:

·       Who actually made the operative decision?

·       Can I trace how the system arrived at GHS150,000?

·       If I cannot, what is preventing me—technical complexity, lack of expertise or withheld information?

·       Can expert assistance or controlled access overcome that problem?

·       Does the importer have a meaningful opportunity to challenge the decision?

These are not questions of computer science. They are questions of judicial scrutiny. If these questions can be answered, the court can proceed to determine whether the assessment can withstand scrutiny under the applicable law. But there remains the hardest case: What if, after all these steps, the court still cannot answer: “Why GHS150,000?”

WHAT IF THE JUDGE STILL CANNOT ANSWER: “WHY GHS150,000?”

Assume the court has done what it reasonably can. Customs has produced the available records. Expert assistance has been obtained where necessary. Claims of trade secrecy, security and gaming have been addressed through appropriate safeguards. The importer has been given a meaningful opportunity to challenge the assessment.

Yet the central question remains unanswered:

Why GHS150,000?

At this point, the problem is no longer mainly about technology. It becomes a legal question: can a decision that cannot be adequately explained or tested still be allowed to stand?

Customs may still be legally responsible for the assessment. It may even show that the system performs well in general. But the case before the court is not about the system’s overall performance. It is about this particular decision and this particular importer.

That does not mean that every unexplained AI decision must automatically be treated as unlawful. The consequence will depend on the governing law, the nature of the claim, the applicable burden of proof and the remedy being sought.

But one principle should matter: Where an institution chooses autonomous decision-making, its inability to adequately explain the resulting decision should ordinarily count against the institution rather than the person subjected to that decision.

The reason is simple. Customs chose the system. Customs controls, or is better placed to obtain, the information needed to explain its decisions. The importer does not.

If Customs can enforce the GHS150,000 assessment while the importer bears the burden of explaining a decision made inside a system beyond the importer’s reach, autonomy risks becoming a shield against accountability. The judge is therefore left with a final question: Who should bear the legal risk when an autonomous decision cannot be adequately explained or tested so as to withstand judicial scrutiny?

The answer may differ from one area of law to another. But the underlying rule-of-law concern remains the same: a person should not be bound by a consequential autonomous decision merely because the institution that chose the system cannot adequately explain how that decision was reached.

CONCLUSION

The customs example brings the issue into focus: autonomy cannot place a decision beyond judicial scrutiny. If a customs officer assesses GHS150,000, the court has familiar ways of examining how that decision was reached. If an autonomous AI system makes the same assessment, Customs may still be legally answerable, but the court must find another route to the basis of the decision.

That route may lie through system records, expert assistance, targeted disclosure, confidential review or other procedural measures capable of showing how the assessment was reached. Where those measures work, the fact that AI made the decision should not, by itself, determine the outcome. The harder case is where they do not.

If, after reasonable efforts, neither the court nor the person affected can adequately understand why the decision was made or test its basis, the issue is no longer whether the technology is sophisticated or generally reliable. The issue is whether that decision should remain enforceable despite being beyond meaningful judicial scrutiny.

This is the judge's quandary: who should bear the legal risk when an autonomous decision that has caused, or is alleged to have caused, harm cannot be adequately explained or tested? That risk should not fall on the person challenging the decision. The provider or deployer is ordinarily better placed to access the records, technical information and expertise needed to explain how the decision was reached.

Even where the technology itself resists perfect explainability and no one can fully reconstruct how it moved from the information received to the decision reached, that limitation does not remove the deployer's responsibility. The decision to use such a system in a high-stakes public function is an institutional choice, not one made by the person subjected to its outcome.

Where adequate explanation remains impossible, the legal risk of the quandary ought to fall on the provider or deployer that introduced and relied upon the system, not on the person affected by its decision. The precise legal consequence may depend on the applicable law, but the principle is important: opacity should not become an evidential advantage.

The technology may be novel, but the principles of justice are not. The court need not master the algorithms or mathematics behind the system. It must focus on the decision-making process by asking what information entered the system, what emerged as the outcome, and what materially influenced that particular result. The task is not to understand the AI; it is to understand the decision. That is a task courts have always performed.

Autonomy cannot therefore become immunity from accountability, nor should it place a consequential decision beyond the reach of the rule of law.