Did Your AI Actually Read That Case? How to Check

On Nov. 6, 2026, Midpage, which makes legal research tools, said it ran the same AI model on 100 legal research questions twice. Using ordinary web search, the AI had opened only 33% of the cases its answers relied on, compared with 82% using Midpage's own case law tools. That's the company's own test, graded by an AI model, not an independent study. The lesson works with any tool: open and read every case an AI answer cites before you rely on it, and keep a short record that you did.

Where does this come from?#

This guide is the written companion to Episode 5 of our podcast, "Did Your AI Actually Read That?". The episode covers legal AI news from Nov. 6 to 10, 2026. Every fact here comes from the sources listed in the episode's show notes. The episode also covers document review that keeps up with new files, how to read a vendor's security report, and a new AI model release. This guide sticks to the research test, because it applies to every firm that uses AI to find case law. The advice sections are our practical suggestions for solo and small firms, not rules and not legal advice.

If you'd rather listen, the episode page has the audio and the full transcript. Our news page for Nov. 9 has the short version.

What did the test find?#

A quick definition first. An AI model is the engine that reads and writes text inside an AI tool. Many different research products can run on the same model.

Midpage says it gave one model, Anthropic's Opus 5.5, the same 100 legal research questions twice:

• Once with ordinary web search. The AI had opened only 33% of the cases its answers relied on.
• Once with Midpage's case law tools. The AI had opened 82% of the cases its answers relied on.

Put plainly, in the web search version, about two out of three cases the answers leaned on were never opened by the AI.

Midpage also says an AI model acted as the judge and preferred the Midpage answer on 89 of the 100 questions.

Should you trust a company's test of its own product?#

Treat it as a useful warning, not proof that one product beats another. Two things limit it:

• Midpage ran it. It's the company's own test of its own tools, not an independent study.
• An AI graded it. The "judge" was an AI model, not a panel of lawyers.

What the test does show is how an AI answer can sound complete while resting on cases nobody, human or machine, actually read. That risk doesn't depend on which tool you use.

What should your firm do about it?#

Make one habit non-negotiable: open and read every case an AI answer cites before you rely on it. Not the AI's summary of the case. The case.

As one of the hosts put it on the episode: "The tool finds. You read. You decide."

Here's a simple way to build that into a small firm's week:

1. Pull every citation. Before an AI-assisted memo, brief or letter goes anywhere, list every case it cites.
2. Open each one in a source you trust. Use the court's own site or the research service your firm already pays for.
3. Check three things. Does the case exist? Does it say what the answer claims? Is it still good law (not overturned or limited by a later decision) where you practice?
4. Note that you did it. A one-line entry in the matter file, with who checked and when, is enough for most firms.
5. Ask whoever drafted it. If a staff member or contract lawyer used AI, ask them to confirm they read every cited case, too.

Does this mean you should switch research tools?#

Not on the strength of one company's test. If you're comparing research tools anyway, ask each vendor the same questions:

• Does the tool open and read the full text of the cases it cites, or work from snippets and summaries?
• Does every citation link to the full case, so you can check it quickly?
• Which AI model sits underneath, and does it change without notice?

Write down the answers. Our legal research category lists research tools in our directory. Whatever you choose, the habit above stays the same.

What about the ethics rules?#

AI tools can sound confident and still be wrong. When an AI confidently makes something up, people say it hallucinates. A citation to a case that doesn't exist, or doesn't say what the answer claims, is the classic example.

Two common ethics duties come up most here. Check how your state's rules handle each one:

• Competence: know roughly what the tool does, and check its work. Reading every cited case is the most direct check there is.
• Candor to the court: what you file has to be accurate. Your name goes on the filing, not the tool's.

Check your state's ethics rules and any bar guidance on AI before you roll this out. This isn't legal advice. Your state's rules control.

Frequently asked questions#

Does this only apply to free AI chat tools?#

No. The test compared ordinary web search with a research company's own case law tools, and neither version opened every case. Check every citation, whatever the tool costs.

Is a summary of the case good enough?#

No. A summary is the AI's description of the case. Read the case itself before you rely on it.

Is this legal advice?#

No. This guide summarizes the sources behind Episode 5 of our podcast and offers general practice suggestions. Ethics rules and bar guidance on AI vary by state, so check yours.

Next step#

Listen to Episode 5, "Did Your AI Actually Read That?" and read the full transcript. Then add the five steps above to your firm's checklist for any AI-assisted filing. For more episodes, see the podcast page, or get the written version in our newsletter.

This guide is general information, not legal advice. Facts come from the sources listed in the show notes for Episode 5 of our podcast; test figures are Midpage's own. Some listings on AI Tools For Law may be sponsored or include affiliate or referral links, and paid placements are always labeled "Sponsored"; see our disclosure. None of the links in this guide are paid placements, and nothing in the episode is sponsored.