TL;DR
The conference-talk pipeline of embed everything and take the nearest neighbors does not work on its own for technical documents. Keyword search catches what embeddings miss, a reranker sorts the union, and neighbor pages give the model the context it needs.
The problem
Sit through enough conference talks on retrieval for LLM prompts and you will see the same pipeline: chunk the documents, embed the chunks, embed the question, take the nearest neighbors, done. Semantic search as the whole answer. It demos well on prose, blog posts, and support tickets. My experience with technical documents is that it is inadequate on its own, and the gap shows up exactly on the questions a domain expert would ask.
I have built many chatbots with construction documents as their knowledge base. The one that taught me this lesson sits on top of electrical construction specifications. The questions professionals ask it are dense with trade vocabulary: Schedule 80 PVC, sherardized steel, Type FS/FD, EMT, set screws. The specs are just as dense.
My first attempts (a few years ago) were pure semantic search. Embed the question, find the nearest page vectors, hand them to the model. Conversational prompts worked. The jargon-heavy questions did not. An embedding squeezes a phrase into a vector, and the trade-specific meaning is what gets lost.
“Set screws” is the example I still use. “Set” is one of the most common words in English, with dozens of meanings. The embedding may read the query as generic fastening, or miss the hardware sense entirely. When only vector search is used, pages that actually contain “set screws” may land below pages about anchors and clamps, or not appear at all.
Keyword search has the opposite failure. It nails EMT and misses the page that says Electrical Metallic Tubing.
I wrote up the first fix in Keyword and Semantic Search with Contextual Reranking for Enhanced LLM Queries in December 2024. That version extracted keywords with an LLM, ran a SQLite regex search and a Pinecone vector query in sequence, merged them, and reranked with Voyage. The method still stands. Almost every piece of the implementation has been updated, thus this post is about the current shape of the pipeline.
The method
- Extract 3-5 search terms from the query with an LLM.
- Run the keyword leg and the vector leg at the same time.
- Text-search the extracted terms in SQLite with regex.
- Embed the query and score it against every page vector.
- Merge the text hits with the vector hits, drop exact duplicates, and rerank the set with Voyage.
- Expand each top-k hit with its previous and next page from the same document.
- Tag each page as the actual hit or a neighbor, so the LLM cites the right one, and send the bundle.
Steps 6 and 7 are new. Steps 1 through 5 are the old post rebuilt.
1. Keyword extraction
The original prompt returned one Boolean string like ground OR grounding OR earthing. The current one returns a JSON array. That lets me clean each term, join them myself, and grade the terms separately later.
<extraction_rules>
1. PRIORITIZE electrical-specific nouns and phrases (equipment type, environment, conductor/insulation family, NEC article, spec section numbers) exactly as they appear in the query or context.
2. ALWAYS carry through at least one verbatim term from the user query.
3. Add at most one complementary synonym or code reference per idea when it sharpens the electrical scope (e.g., wet location → damp location; raceway → conduit). Skip generic fillers like "types" or "requirements".
4. Keep the list to three-to-five unique items ordered from most specific to most general.
5. Each list item should be a single string (multiword phrases are allowed), ready to combine with OR logic later.
</extraction_rules>
<output_format>
{
"important_terms": ["below-grade insulation", "XPS", "polyiso"],
"explanation": "..."
}
</output_format>For “What are the requirements for underground junction/pull boxes?” it produces underground junction box OR pull box OR handhole OR NEC Article 314 OR quazite. The verbatim term is the safety net. The synonyms and code references are the reach.
The extractor is google/gemini-3-flash-preview through OpenRouter in JSON mode. The terms are joined with OR to become the text-search phrase.
2. Run the legs in parallel
Vector search and keyword extraction are both network-bound and independent. Text search depends on the keywords. So the vector leg and the extractor start together, and the text leg fires the moment the extractor returns while the embedding call may still be in flight.
def _run_parallel_retrieval(query, top_k):
with ThreadPoolExecutor(max_workers=3) as executor:
vector_future = executor.submit(search_documents_vector, query, top_k)
kw_future = executor.submit(
extract_important_term_openrouter, query, key_phrase_system_prompt
)
parsed = parse_keyword_response(kw_future.result())
important_term = parsed["important_term"]
text_results = executor.submit(search_phrase, important_term, db_files).result()
vector_results = vector_future.result()
...Wall time is roughly the slowest single leg instead of the sum of three.
3. Text search
The Boolean parser survived from the old post nearly intact. It splits on AND, OR, NOT, wraps each term in a word-boundary regex that tolerates a trailing s, and builds a WHERE clause against the page text. It now returns the row id alongside the text so the id can reach the neighbor-page step.
def create_word_pattern(word):
if word.endswith('s'):
return f"\\b{re.escape(word[:-1])}s?\\b"
return f"\\b{re.escape(word)}s?\\b"
def search_phrase(phrase, db_files):
query, params = parse_search_input(phrase)
for db_file in db_files:
conn = sqlite3.connect(db_file)
conn.create_function(
"REGEXP", 2,
lambda expr, item: re.search(expr, item, re.IGNORECASE) is not None if item is not None else False,
)
rows = conn.execute(query, params).fetchall() # [(id, text), ...]
...I tried an FTS5/BM25 replacement and it made little measurable difference. The lexical leg has one job, getting the page with the exact trade term into the candidate pool. Ranking is the reranker’s job. I kept the regex.
4. Semantic vector search
My earlier methods used Pinecone. The vectors now live in the same SQLite database as the text: each spec page is a row with id, text, and a vector BLOB of 3072 float32 values from text-embedding-3-large. On first use I load every vector into a row-normalized numpy matrix and cache it for the life of the process.
def _load_vectors_matrix(db_path):
...
for doc_id, text_content, vector_blob in cursor:
vec = np.frombuffer(vector_blob, dtype=np.float32)
norm = float(np.linalg.norm(vec))
if norm == 0.0:
continue
ids.append(doc_id)
texts.append(text_content)
vectors.append((vec / norm).astype(np.float32, copy=False))
matrix = np.stack(vectors)
...
def semantic_vector_search(query, db_path=None, top_k=20, ...):
query_embedding = _embed_query_openrouter(query, api_key, EMBEDDING_MODEL)
q_normalized = query_embedding / np.linalg.norm(query_embedding)
ids, texts, matrix = _load_vectors_matrix(db_path)
scores = matrix @ q_normalized # cosine for every page in one gemv
top_idx = _top_k_indices(scores, top_k) # argpartition, then sort the slice
...A single matrix-vector multiply gives cosine similarity for every page. For a few thousand pages this is faster than a round-trip to a hosted index, and the vectors and text can never drift apart. If your corpus fits in memory, there is no reason to pay for Pinecone or fight with its free tier. The vector leg is the easy half of this problem, and it was never the half that failed.
5. Merge, dedupe, rerank
Vector hits come first because they are already scored. Text hits are capped at 180 rows and appended. Anything whose text exactly matches a page already in the list is dropped so the reranker does not spend a slot on the same page twice.
def _assemble_rerank_candidates(vector_results, text_results, text_cap=180):
text_pairs = [(row_id, content)
for db_results in text_results.values()
for row_id, content in db_results][:text_cap]
seen_texts, pairs = set(), []
for doc_id, text in list(vector_results) + text_pairs:
if not text or text in seen_texts:
continue
seen_texts.add(text)
pairs.append((doc_id, text))
return pairsVoyage’s rerank-1 takes up to 100 documents per call, so larger candidate lists are chunked and reranked concurrently, then sorted by relevance score.
def rerank_results(flattened_results, test_query, voyage_api_key):
vo = voyageai.Client(api_key=voyage_api_key)
chunks = [flattened_results[i:i + 100] for i in range(0, len(flattened_results), 100)]
all_reranked = []
with ThreadPoolExecutor(max_workers=min(len(chunks), 4)) as executor:
futures = [executor.submit(_rerank_chunk, vo, test_query, c) for c in chunks]
for fut in futures:
all_reranked.extend(fut.result())
return sorted(all_reranked, key=lambda x: x.relevance_score, reverse=True)The reranker is what makes the union safe. Without it, adding 180 regex hits to 20 vector hits would bury the good pages in noise. With it, the regex leg only has to get the right page into the candidate pool; the reranker decides where it lands. The reranker only sees text, so after it returns I map each document back to its id through the deduped text -> doc_id table.
6. Neighbor-page expansion
This was the change that moved the needle most. Retrieval returns page-sized slices. When the matched phrase sits at the top or bottom of a page, the model sees the hit line and misses the paragraph before or after: the definition, the exception, the “see Drawings”, the table that continues overleaf. Adjacent pages of a spec almost always continue the same CSI subsection.
So for each of the top-k reranked hits, I also pull the previous and next page from the same document. Page ids look like 26-05-33-CC-RACEWAYS-AND-BOXES_Rev_0_page_06, so the prefix and page number split on the last _page_. Expansion happens after rerank so neighbors cannot steal rerank slots, and everything is deduped by id so adjacent hits on pages 20 and 21 yield 19, 20, 21, 22.
for rank, result in enumerate(top_hits, 1):
prefix, page_num = _parse_page_id(result.doc_id)
pages = _load_doc_pages(prefix) # {int_page: (doc_id, text)}, cached
_add(pages[page_num], rank=rank, page_role="primary_hit", ...)
if (page_num - 1) in pages and pages[page_num - 1][0] not in primary_doc_ids:
_add(pages[page_num - 1], rank=rank, page_role="previous_context", ...)
if (page_num + 1) in pages and pages[page_num + 1][0] not in primary_doc_ids:
_add(pages[page_num + 1], rank=rank, page_role="next_context", ...)Context roughly doubles or triples. That is the cost. The gain shows up most where the answer is a table or a numbered list that straddles a page break: phase color codes, conduit schedules, backfill depths.
7. Tag each page as the hit or a neighbor
After step 6 the bundle holds two kinds of pages: the ones the reranker chose, and the neighbors added only for continuity. To the model they all look like plain text. So every page goes in with a header that says which kind it is and how to cite it.
--- Retrieved context page ---
Retrieval rank: 1
Page role: primary_hit
Citation guidance: Cite this page when it supports the answer; it is the actual top-K retrieval hit.
Document page token: 26-05-33-CC-RACEWAYS-AND-BOXES_Rev_0_page_06
Parsed page number: 6
Relation to hit: reranked top-k page
Rerank score: 0.8731
<page text>Neighbors get Page role: previous_context or next_context and the guidance “Context only. Cite this page only if the exact answer support is on this neighbor page rather than the primary hit.” Without that hint the model would cite a neighbor page for a claim already supported on the hit, one page off from where a reader would look.
The whole bundle is wrapped as <context>...</context>\n<query>...</query> and sent with the system prompt to x-ai/grok-4.3, also through OpenRouter.
Where it stands
For “grounding requirements” the extractor gives ground OR grounding OR earthing. For “set screws” the keyword leg is still what saves the retrieval, nearly two years on. Semantic search alone was never going to find that page. What changed is everything around the two legs: they run at the same time, the vectors live next to the text, duplicates are removed before the reranker sees them, and every hit arrives with its neighbors and a label. Keywords plus vectors, reranked, expanded, labeled, then the LLM.