Why your firm's AI search returns confidently wrong answers

A partner asks the firm's AI assistant for engagement letters from the last two years with a particular fee arrangement. It returns six documents, written fluently, formatted well, and three of them are from 2019. Nothing is broken in an obvious way, which is what makes this failure dangerous. The system did exactly what it was built to do. It was built to find things that sound similar, and it has no idea that a date was a requirement rather than a preference.

Meaning-based search does not understand the word only

Most AI search over firm documents works by converting text into a mathematical representation of meaning and finding the closest matches. This is genuinely powerful. It finds a document about fee disputes when you searched for billing disagreements, which keyword search would miss entirely.

But similarity is a spectrum, not a rule. When you say only closed matters, or excluding this client, or after the amendment, the system does not treat those as constraints. It treats them as flavor. A 2019 document that closely matches everything else you described will outrank a 2026 document that matches slightly less well, because the date was never a gate.

The fix is an order of operations

Good systems do not choose between filters and meaning. They sequence them. First the request is read and the hard requirements are pulled out as structured values: this date range, this practice area, this status, this jurisdiction. Then those requirements eliminate everything that does not qualify. Only then does meaning-based ranking run, across the set that already satisfies the rules.

The order is the whole idea. Filtering first means a non-qualifying document cannot appear no matter how similar it reads. Ranking second means the surviving documents come back in a sensible order. Many firm deployments run these in the opposite order, or skip the filter entirely, which produces exactly the failure above.

Which means your metadata is the real project

Filtering only works if the attributes exist and are trustworthy. A date range filter needs reliable dates. A practice-area filter needs practice areas recorded consistently. A status filter needs someone to have marked matters closed when they closed.

This is the part firms do not expect and where most of the effort actually goes. The AI work is comparatively quick. The document housekeeping is the engagement. It is also the part with value beyond AI, since a firm that can reliably filter its own documents has solved a problem that predates any of this.

A second pass for the top results

There is an optional refinement worth knowing about. After filtering and ranking, a slower and more careful model can re-order just the top handful of candidates. It is too expensive to run across everything, but across ten documents it is cheap, and it meaningfully improves which one lands first.

The practical effect is that the best answer appears at the top rather than fourth, which matters because the person asking usually reads one or two results and stops.

How to test whether yours works

Do not test with vague questions. Test with questions carrying hard constraints where you already know the right answer:

• Ask for documents from a specific date range, then check every result's date.
• Ask for one client's matters only, and see whether anything from another client appears.
• Ask for something you know does not exist, and see whether the system says so or invents a plausible near-match.
• Ask the same question two different ways and compare. Wildly different results mean nothing is being filtered.

The failure mode worth taking seriously

The third test above is the important one. A retrieval system with no filtering rarely returns nothing, because something is always the closest match. So when the answer does not exist, you get the nearest thing dressed up as the answer, and a busy person reading quickly has no signal that anything went wrong.

That is the real exposure in a professional setting. Not a system that fails visibly, but one that succeeds in form while failing in substance, on a question where being approximately right is the same as being wrong.

How Aday Interactive, Inc. builds retrieval you can rely on

In our Custom AI engagements, the retrieval layer gets designed before anything conversational is built. We identify which attributes are genuine constraints in your practice, confirm the metadata supports filtering on them, and enforce them before any meaning-based ranking runs.

We also build the test set: real questions with known correct answers, run against the system regularly. Retrieval quality drifts as documents accumulate, and a firm without a test set finds out from a partner rather than from a report.