2026-11-04 · Fonte:Ars Technica

The attorney's admission — "I didn't know that AI could hallucinate facts" — came after discipline over ChatGPT-cited testimony from witnesses who never existed.
A New Mexico defense lawyer has been punished after filings prepared with ChatGPT's help cited testimony from witnesses who never existed. Pressed on the fabrication, the attorney offered a blunt explanation: "I didn't know that AI could hallucinate facts." The case stands out among AI-related disciplinary matters because the invented material was not a miscited precedent but purported statements from made-up people — fabricated evidence rather than a garbled reference.
The distinction matters. When generative AI goes wrong in court, it most often produces fabricated case citations — plausible-sounding rulings that collapse the moment anyone checks the docket. Testimony attributed to witnesses who do not exist goes a step further, inserting phantom evidence directly into a legal record. That a disciplinary punishment followed underscores how regulators and courts are treating unverified AI output not as a technical glitch but as a professional failure attributable to the human who signs the filing.
The lawyer's stated rationale — not knowing that AI could hallucinate facts — is itself the story. Hallucination, the tendency of large language models to generate fluent, confident text that is simply false, is a documented characteristic of systems like ChatGPT, not a rare anomaly. Models predict plausible language; they do not verify that names, events, or quotations correspond to reality. A practitioner deploying such a tool in court filings without any verification layer was, in effect, outsourcing factual accuracy to a system engineered for plausibility rather than truth.
For the AI industry, the episode lands in the middle of the reliability debate that shapes enterprise adoption. Each new courtroom sanction tied to fabricated AI output raises the perceived cost of deploying generative tools in high-stakes professional workflows, reinforcing demand for retrieval grounding, citation checking, and human review. The New Mexico case is a reminder that as ChatGPT and its peers spread into regulated professions, accountability for hallucinated facts rests — legally and professionally — with the user, a dynamic vendors and buyers alike are still learning to price.
Tags: ChatGPT · OpenAI · AI Hallucination · Generative AI · Legal Profession
萬安算交所 AIXX · AIXX · Exchange verificado · Ir ao mercado