In May 2023, a federal judge in New York fined a lawyer five thousand dollars and ordered him to submit a sworn affidavit verifying every case he had cited in the past five years.
The reason was almost surreal: the lawyer had filed a brief containing citations to six legal cases that did not exist.
The case names looked plausible. The citations were formatted correctly. The legal reasoning attached to them was coherent and apparently learned. They had been generated, entirely and convincingly, by ChatGPT.
The lawyer — Steven Schwartz, of the firm Levidow, Levidow & Oberman — had asked the AI to assist him in finding precedents for a personal injury case against an airline. The AI produced six cases. He included them in his brief. When opposing counsel could not locate any of them, the judge asked for confirmation. Schwartz submitted additional ChatGPT-generated summaries, still apparently unaware the cases were fictional. The deception — though 'deception' is the wrong word for what was, at its core, a catastrophic failure of professional judgment — came fully to light only when the judge demanded the actual court documents.
Schwartz told the court he had not known the AI was capable of fabricating citations. He had assumed, in the way one assumes that a reference book contains accurate information, that the confident and authoritative-sounding output reflected real legal reality.
He had trusted. He had not checked.
This case became, inevitably, a cautionary tale about AI hallucination — the tendency of large language models to generate false information with the same fluency as true information. Every piece of coverage drew the same lesson: be careful, because AI makes things up.
I want to tell you something I think matters more.
This lesson, while accurate, is the least important one available. The case is not primarily a story about a technology that sometimes produces incorrect outputs. It is a story about a professional whose relationship to authoritative-sounding information had changed so completely that the concept of verification had, in a practical sense, ceased to operate.
He had trusted. He had not checked.
What had erased the checking?
It helps to trace the evolution of how we relate to digital information. There have been three stages, and each was experienced, at the time, as pure progress.
Stage one: search. In the early years of the web, search engines told you where to look. They returned a list of sources, some reliable and some not, and you — the user, the person with the question — were responsible for what came next. You navigated to the sources. You read them. You evaluated their credibility. The tool was genuinely in service of your cognition: it reduced the time required to locate relevant material, but the cognitive work of assessing, synthesizing, and concluding was entirely yours.
This stage was imperfect. But the structure of the interaction at least created the occasion for evaluation. The user received sources, not conclusions.
Stage two: recommendation. Algorithms began not just directing you toward information but selecting, shaping, and curating what you would see. Your search results were increasingly personalized. Your social feed was arranged by calculated impact, not chronology. You were no longer searching a neutral catalog. You were receiving a curated presentation, assembled by a system that had learned a great deal about what kept your attention.
The cognitive work of deciding what was worth reading had been quietly removed.
Stage three: AI synthesis. This is categorically different. Search gives you sources to evaluate. Recommendation gives you curated sources to evaluate. AI gives you synthesized conclusions and implicitly invites you not to evaluate them at all — because the synthesis has already been done, the conclusion has already been drawn, and the answer arrives with a fluency and completeness that makes further inquiry feel redundant.
When you type a question into a search engine, you receive a list. The list is a prompt: there is more to find; go find it. When you type the same question into an AI, you receive an answer. The answer is the end of the process, or feels like one. It is organized, confident, complete-seeming, authoritative in tone. It does not say here are some sources that may be relevant. It says here is what is true.
And the part of your mind that should have asked is it? has, by this point, been trained over decades not to ask.
The lawyer did not fail because he was lazy. He failed because his System 2 — the slow, skeptical, evaluative mode of thinking — had been trained, over years of working with reliable reference materials, to stand down when the tool was consulted. Reference books are reliable. Encyclopedias are reliable. Peer-reviewed journals, within their limitations, are reliable. If the AI produced outputs with the confident, organized form of reliable reference material, and if it was, most of the time, producing accurate outputs, the habit of acceptance without verification was not unreasonable.
The problem is that 'most of the time accurate' is not adequate for filing legal documents. And the problem is that the habit of acceptance does not stay confined to contexts where it is safe. It generalizes. The person who accepts AI outputs without checking them when writing emails also, eventually, accepts them without checking them when writing briefs.
The muscle of independent evaluation does not maintain itself through selective use.
Steven Schwartz lost five thousand dollars and the trust of his firm. That cost is visible. It made the news. It became a teaching anecdote in legal education programs.
The cost the rest of us are paying — every time we accept an output without checking it, in low-stakes contexts where it almost certainly didn't matter — is invisible, slow, and adds up.
The next time the cost matters, the muscle will not be there.