No similarity threshold can reach 100%. I used to treat that as a slogan. Then I measured it on a real supplier database for a global CPG manufacturer: the cosine scores of correct hits and wrong hits overlap across the range 0.20 to 0.62. There is no line you can draw through that. The only way to get precision was to stop asking the vector to answer a question it cannot answer.
This is the story of how our supplier discovery pipeline went from a plausible RAG design to a measured one, and what the measurement changed.
The setup
We build a sourcing platform. One of its agents takes a project charter (“we need corrugated packaging for three plants in Eastern Europe”) and proposes suppliers. The internal half of that search runs against a read-only semantic layer in Postgres 17 with pgvector: about 45K global supplier profiles, each with a 1024-dimension embedding from Amazon Titan Embed v2, an HNSW cosine index, and the client’s own spend history alongside.
The app embeds the charter with the same Titan model, so query vectors and stored vectors are directly comparable. The retrieval code runs three searches concurrently with asyncio.gather: the client’s own suppliers, a structured category lookup over the reference pool, and the pure vector pass. Results merge in that order, with fuzzy name dedup at 0.80 similarity and small boosts (CLIENT_BOOST = 0.10, REF_BOOST = 0.05) for suppliers the client already uses.
A full-text fallback (ts_rank over websearch_to_tsquery) fills in rows the vector pass missed, because at one point about 32% of profiles had no completed embedding. The fallback deliberately ignores the similarity threshold. Cosine and ts_rank are different scales, and the comment in the code says why: a strict threshold would otherwise “silently drop them from both paths.”
On paper this is a reasonable hybrid search. It is also what most RAG tutorials would have you build.
What the numbers said
In October I sat down with the live database, read-only, and ran eight realistic sourcing queries for the client. Each one has a known target category, so precision is measurable: what share of the returned suppliers actually supply that category, either by their profile or because the client has paid them for it.
| Query | Pure vector, P@25 | The app’s real pipeline, top 100 |
|---|---|---|
| Integrated facility management | 0.96 | 0.46 |
| Corrugated boxes | 0.72 | 0.67 |
| Consumer market research | 0.92 | 0.89 |
| Temporary staffing | 0.68 | 0.83 |
| Full-truckload road freight | 0.56 | 0.72 |
| Laptops and IT hardware | 0.60 | 0.55 |
| Office cleaning | 0.96 | 0.75 |
| Hotels for business travel | 0.40 | 0.36 |
| Mean | 0.72 | 0.65 |
The app’s pipeline (top 100, threshold 0.2) averaged 0.65. Pure vector search over the global pool averaged 0.72. The worst query, hotels, was 0.36 either way. A banking services firm came back for packaging because of a name. A freight company came back for trucking because its profile said hotels and lodging.
Then I looked at the similarity distributions. Right answers and wrong answers both live between 0.20 and 0.62. Raising the threshold throws away good suppliers before it throws away bad ones. Lowering it adds noise. The threshold is a dead lever.
That is the moment the design rule became obvious. In the words I wrote in the report: precision has to come from structured filters; vectors may only rank.
What the data knew that the model did not
Measuring precision forced me to look at the structured data properly, and that turned up more than the vector problem.
Two taxonomies, two matches. The reference pool and the client’s spend lines use one taxonomy: 16 top-level categories, 182 subcategories. The client’s preferred-supplier table uses a different one, about 50 pairs. Only 2 of those pairs exist in the first taxonomy. No crosswalk can be derived from the data; it needs a curated mapping from the business.
A supplier’s profile category is wrong three times out of four. A supplier’s global profile has one subcategory. It matches the subcategory the client actually spends the most on with that supplier only 25% of the time. The client buys from most suppliers across two or more subcategories. A profile is a summary; the spend table is the truth.
Thirteen percent of the client’s suppliers could never be returned. Every query joined a name-alias table with an INNER JOIN. 13% of the client’s spend suppliers had no row in it. They were silently excluded from every result, forever, with no error.
“Preferred” meant “has a row.” The UI showed a preferred badge for any supplier with a row in the preferred-supplier table. For this client that flagged all but two of roughly 7,200 spend suppliers. The number of truly preferred suppliers is a few hundred.
The category filter did nothing. The intake agent produced its own canonical list of categories. They were not real top-level categories in the taxonomy, so the filter matched nothing and silently fell through.
None of these are embedding problems. All of them cap precision harder than the embedding does.
The failure you cannot see
There was one more finding, and it is the one I keep coming back to. The tenant-to-schema mapper computed a schema name with one suffix. The ETL had loaded the data under a different suffix. Every query against the client’s own data failed.
Nobody noticed, because the database layer wrapped every call in a tenacity retry (three attempts, exponential backoff) with a fallback that returns an empty list or a {"_db_error": True} sentinel. Empty results looked like “no matches.” The UAT environment masked it further with an override that pointed the client at a demo tenant’s data. Users were seeing somebody else’s suppliers and nobody could tell.
Retry-and-fallback is a good pattern for transient failures. It is a terrible pattern for configuration errors, because it converts a loud failure into a quiet wrong answer. Make empty loud.
The redesign
The plan that came out of the measurement is short.
-
Classify first, with a live enum. The LLM classifies the charter into one of the 182 subcategories. The allowed values are read from the taxonomy table at runtime, never hard-coded, so the model cannot invent a category that does not exist. The result is stored on the project and shown as an editable chip. The user confirms it.
-
Pool A: the client’s own suppliers. Everything the client has actually paid for in that subcategory. Precise by construction, because membership comes from the spend table, not from a score. Today this is a sequential scan of about 0.8 seconds per category on a table with no primary key and no indexes. With an index it is milliseconds.
-
Pool B: the wider network. Suppliers whose profile subcategory matches, filtered by country, excluding Pool A, and only then ranked by cosine similarity to the charter embedding. The vector orders the set. It never decides who is in it.
-
Web discovery stays out of the database matches. It cannot be made precise, so it goes in a separate tab, clearly labelled as unverified, or gets switched off for this client.
The embedding did not get worse in this design. It got a smaller job, and it does that job well.
What I’d tell you to do
- Measure precision on real queries before you tune anything. A P@25 table took one afternoon and changed the architecture.
- Plot the similarity distribution of right and wrong hits. If they overlap, stop tuning the threshold.
- Let structured data decide membership and let the vector rank inside it. Joins, categories and spend history are your precision; embeddings are your ordering.
- Read the allowed values for any LLM classification from the source of truth at runtime. A hard-coded list drifts and the filter dies silently.
- Audit every retry-with-empty-fallback. Ask what a configuration error looks like through it. If the answer is “an empty page,” add a loud path.