Don’t Classify—Hallucinate (Then Match): A Cheaper Way to Hit Your Real Taxonomy

Picture the last time you tried to “force” an LLM to choose from a strict list of categories.

You hand it a giant menu: hundreds of brand-safe paths like:

• Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables
• Décor & Pillows / Decorative Pillows & Blankets / Throw Pillows

Then the model comes back with either malformed output (the JSON is wrong), a near miss (spacing/casing differences), or something that your downstream code can’t trust.

Structured outputs help, but they come with a catch: the moment the allowed values grow large and nested, the schema you send to the model grows too. Eventually you hit practical limits—schema size, enum limits, and context bloat—long before you hit the “learning” ceiling.

So here’s the pivot behind Don’t classify. Hallucinate!: ask a small model to invent plausible category strings, then “snap” those strings back onto your real taxonomy using embeddings.

Why does that work? Because you’re separating the problem into two simpler steps:

1. Creative hypothesis generation (cheap, unconstrained)
2. Deterministic mapping to your real vocabulary (embedding similarity)

No huge schema. No shipping the whole taxonomy on every request. Still, end up with real, legal category paths.

The boring way: structured outputs with a giant schema

Structured outputs means the model is told to output something like a JSON object that matches a schema. In Python, a common approach is using Pydantic, a library that defines data shapes with types.

When a taxonomy has hundreds of legal values, people often encode them as a giant Literal[...] list.

A Literal in Python typing means “this variable must be exactly one of these strings.” Pydantic can then parse the model’s response into a typed object, rejecting values outside the allowed set.

That works, until the taxonomy gets large. The schema becomes big, nested, and expensive to send. Worse, it encourages a weird failure mode: the model spends more effort producing syntactic correctness than semantic usefulness.

Sometimes it even “refuses” by accident because the request is too constrained.

The pattern: invent candidates, then map them back

The replacement pattern is surprisingly human:

• Instead of asking the LLM to pick from your list, ask it to make plausible guesses.
• Those guesses can be wrong in your taxonomy, but that’s okay.
• Then you map each guess to the closest real taxonomy entry using embeddings.

Step 1: hallucinate plausible classifications

For example, given the query:

brown coffee table

Ask a small model to output a list of category paths it thinks would fit. It might produce something like:

• Furniture / Living Room / Tables / Coffee
• Furniture / Home & Kitchen / Coffee Tables
• Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables (sometimes it lands exactly)

The point isn’t correctness at this stage. The point is to generate candidate strings that are semantically close.

A small model is fine here because this task is basically “guess what these words are probably related to.” It doesn’t need to perfectly obey your taxonomy rules.

Step 2: embed your real taxonomy once

Now build an in-memory searchable index of your real legal categories.

An embedding is a vector (an array of numbers) that represents meaning. Similar meanings end up with vectors that are close together in vector space.

A vector index is the data structure that lets you quickly find the closest vector(s) to a query vector.

Here’s the key move:

• Precompute embeddings for every real category path.
• Store them once (in memory, or in a lightweight vector store).

If your taxonomy has 10,000 categories, you pay the embedding cost upfront. After that, matching is fast.

In the source inspiration, the “in-memory set of embeddings” used a MiniLM-style sentence embedding model. The exact model choice isn’t sacred. What matters is that:

• Your category strings embed consistently
• Your hallucinated candidate strings embed into the same space

Step 3: embed the hallucinated candidate, then nearest-neighbor match

For each fake candidate classification string:

1. Embed the candidate string into a vector.
2. Compare it to all real category vectors (or the top N candidates from an index).
3. Select the most similar real category path.

Similarity is usually computed with cosine similarity, which is the cosine of the angle between two vectors.

A dot product can be used too, but cosine similarity is normalized for vector length, so it’s typically the safer choice.

Once mapping is done, your final output is guaranteed to be a member of your real taxonomy.

A concrete implementation sketch (Python)

Below is a minimal, beginner-friendly sketch that focuses on the idea.

1) Precompute embeddings for the real taxonomy

import numpy as np

# Pretend this returns a vector embedding for a string.
# In practice, this comes from a sentence embedding model.
def embed_text(text: str) -> np.ndarray:
 # e.g. MiniLM or another embedding model
 raise NotImplementedError

real_categories = [
 "Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables",
 "Furniture / Living Room Furniture / Coffee Tables & End Tables / End & Side Tables",
 "Décor & Pillows / Decorative Pillows & Blankets / Throw Pillows",
 #... thousands more
]

real_vectors = np.stack([embed_text(c) for c in real_categories])

# Optional: normalize for cosine similarity
real_vectors = real_vectors / np.linalg.norm(real_vectors, axis=1, keepdims=True)

2) Ask a small model for hallucinated candidates

# Pseudocode: call an LLM that returns a list[str]
# candidates = llm_hallucinate_categories("brown coffee table")

The model might return strings that don’t exist in real_categories. That’s expected.

3) Map each candidate to the closest real category

def cosine_top1(candidate: str) -> str:
 v = embed_text(candidate)
 v = v / np.linalg.norm(v)

 # Cosine similarity via dot product because vectors are normalized
 sims = real_vectors @ v
 best_idx = int(np.argmax(sims))
 return real_categories[best_idx]

# candidates = [...]
# mapped = [cosine_top1(c) for c in candidates]

At the end, mapped is made of real legal taxonomy strings only.

Why this is cheaper in practice

The costs you’re avoiding are mostly “schema shipping” and “schema struggle.”

• Schema shipping: sending thousands of allowed enum values every time bloats prompts.
• Schema struggle: hard constraints can reduce the semantic value of the model’s output.
• Retry loops: structured output systems sometimes force retries when parsing fails.

In this pattern, you still use structured parsing—but the schema is tiny: a list of strings (candidate guesses). The expensive part (large taxonomy) happens once as embeddings.

This also makes throughput friendlier: the model generates a handful of candidates, and matching runs in-process.

The one tricky part: embeddings can still make the wrong snap

Embeddings aren’t magic. They can get fooled by vague text.

If the query is “table” with no modifiers, multiple categories are plausible. If the candidate generation step is too weak, hallucinated candidates might skew toward the wrong neighborhood.

Two practical safeguards usually keep things sane:

1. Generate multiple candidates, not one. Diversity gives embeddings more chances to land near the correct taxonomy region.
2. Use a similarity threshold or top-k. Even when you pick the top-1, keeping the top-3 with scores can prevent obvious mistakes from silently passing.

These safeguards don’t reintroduce the massive schema. They just make the deterministic mapping less brittle.

The mental model to keep

This pattern is easy to remember because it mirrors how humans classify in the real world.

You rarely open a dictionary and hunt for the one legal phrase. You form a guess (“this seems like a coffee table”), then you align it with the store’s category system.

In code:

• The LLM does guessing.
• Embeddings do alignment.

That split is what keeps both quality and cost under control.

Conclusion

“Don’t classify. Hallucinate!” isn’t about giving up on correctness. It’s about moving correctness to where it’s cheapest and most reliable.

Generate plausible category candidates with a small, unconstrained model, then map them to your real taxonomy using embeddings and nearest-neighbor similarity. You avoid giant schemas, reduce prompt bloat, and still guarantee that the final category paths come from your legal vocabulary.

And once that mental split clicks, it starts feeling weird that we ever tried to force the model to memorize every option in the first place.