The Hallucination Problem: Fake Citations, Real Consequences

In December 2024, the Bengaluru bench of the Income Tax Appellate Tribunal did something tribunals almost never do: it recalled its own order within a week of passing it. The matter — Buckeye Trust — involved a trust taxation dispute reported at ₹669 crore. The order had relied on four judgments, three attributed to the Supreme Court and one to the Madras High Court. None of them existed. They had been generated by an AI tool and had passed, unchecked, from a representative's research into the record and then into the tribunal's own reasoning.

That episode is the cleanest illustration of what the profession now calls the hallucination problem, and it is worth being precise about what is actually going wrong, because the remedy follows from the mechanism.

A general-purpose generative model does not look anything up. It produces the most statistically plausible continuation of the text it is given. A legal citation is an extremely regular object — party names, a year, a reporter, a paragraph number — which means the model can produce something with the exact shape of authority while having none of its substance. The output is not a lie in the ordinary sense; the system has no concept of the difference between a judgment that exists and one that merely could. Law is close to the worst-case domain for this failure, because a fabricated citation looks precisely as checkable as a real one, and because for decades the profession's habit has been to trust a well-formed citation on sight. That habit is now a liability.

The Indian record since then has grown uncomfortably fast. In September 2025, in the Greenopolis Welfare Association matter before the Delhi High Court, a filing was found to contain not only non-existent precedents but invented paragraphs attributed to a real, landmark judgment; counsel was admonished and the petition was withdrawn. In October 2025, the Bombay High Court quashed a tax order involving a demand of ₹27.91 crore after finding it rested on precedents that did not exist. Earlier flags had already gone up in the Punjab and Haryana High Court in 2023 and the Manipur High Court in 2024, where reliance on chatbot output in proceedings drew judicial comment.

The case that forced the issue to a head began in a Vijayawada trial court. In August 2025, the court dismissed a party's objections to a commissioner's report by relying on four Supreme Court judgments that do not exist. On challenge, the Andhra Pradesh High Court accepted that the citations were AI-generated — and then affirmed the order anyway, recording only a word of caution, reasoning that fake citations do not vitiate an order if the legal principle applied is otherwise sound. The Supreme Court disagreed in the strongest available terms. Taking the matter up in early 2026, it held that a decision tainted by hallucinated material is "no decision in the eyes of the law", characterised the citation of unverified AI-generated case law by counsel as misconduct, appointed an amicus curiae, and sought responses from the Attorney General, the Solicitor General and the Bar Council of India. Weeks earlier, a bench of Justices Nagarathna and Bhuyan had put the underlying principle beyond argument: cross-verifying citations is a fundamental duty of every advocate.

The institutional response has been unusually coherent. The Kerala High Court's July 2025 policy bars the district judiciary from using AI to arrive at findings or draft orders. The Supreme Court's November 2025 White Paper on AI and the judiciary names fabricated case law as a primary risk and mandates independent verification of AI output. The draft AI regulations released in June 2026 would keep AI out of adjudication and require disclosure of its use. None of this is anti-technology; the same Court runs SUVAS, SUPACE and TERES. The message is narrower: use the machine, verify the machine, answer for the machine.

For an advocate, the exposure runs on three tracks, and it is worth seeing all three at once. The first is costs: Indian courts have grown steadily more willing to impose exemplary costs on parties who burden the record with fabricated material, and an invented citation sits squarely in that territory. The second is the fate of the proceeding itself: orders recalled, petitions withdrawn under the shadow of contempt, assessments quashed — each one a real injury to a real client who did nothing wrong. The third, now the sharpest, is professional discipline. Misconduct is the statutory trigger under the Advocates Act, 1961, and the disciplinary machinery of the State Bar Councils and the Bar Council of India runs on exactly that word — the word the Supreme Court has now attached to unverified AI citation. This is not a hypothetical. The global pattern is the same: the well-known American example, Mata v. Avianca, ended in sanctions as far back as June 2023, and courts abroad have been fining lawyers for it ever since.

The discipline that keeps you safe is short enough to pin above a desk.

One: never cite what you have not opened. Not the summary — the judgment. If you cannot produce the text of an authority, it does not go in your filing.

Two: verify in an authoritative repository. The e-SCR portal, the court's own website, an established reporter. A citation's existence is a fact, and facts are checked at the source.

Three: prefer tools that retrieve over tools that imagine. This is the structural fix. A retrieval-grounded legal platform answers from a maintained database of real judgments and links you to the full text of everything it relies on; a general-purpose chatbot writes from statistical memory and cannot tell you where its answer came from because, in the relevant sense, it came from nowhere. If a tool cannot show you the authority, it has no business anywhere near your pleadings.

Four: treat AI output as a junior's overnight draft — genuinely useful, and unsigned. The signature stage, where an advocate reads what is about to be filed, is precisely the stage every one of the cases above skipped.

The hallucination cases are usually told as stories about technology. They are better understood as stories about verification — the oldest duty in the profession meeting the newest way to breach it. The advocates who thrive with these tools will not be the ones who avoided them, but the ones who never let a machine's confidence substitute for their own diligence.

---

— Every answer in LawWorld's LawChat and Case Laws is drawn from LawWorld's own database of judgments and legislation, with the full text one click away. Verification is designed into the product, not left to chance.