I spent an afternoon turning chatbot improvement into a loop I can rerun. Generate question-answer pairs the same way production does, score them with a different model, then change prompts or retrieval and run again. A JSON file and a Markdown report are enough.
This is for retrieval-augmented chatbots, the kind that answers from a corpus. Mine was construction specifications.
The loop
- Run real questions through the same hybrid retrieval and system prompt as production. Save each pair: prompt, response, model, keywords, search terms.
- Judge each pair with an independent LLM. Two scores: response quality, and keyword quality for the terms that drive text search.
- Read the report. Fix the prompt, the boolean search, or the context formatting.
- Rerun the same question set. Keep what moved. Revert what did not.
I do not treat the judge as a grade. The numbers are a ranking. I read the low items and the explanations.
Hybrid retrieval
Answer quality is bounded by what you retrieve. I use three pieces:
- Vector search for semantic recall.
- Text search with LLM-extracted terms joined as
term1 OR term2 OR term3. - A reranker on the merged list. If the reranker is down, skip it and keep the merge.
Production and QA share this stack so the scores reflect real behavior.
Retrieval notes (practical behavior)
- Boolean grammar: text search supports
AND,OR, andNOTwith case-insensitive word boundaries. Basic plural handling applies (e.g.,conduitmatchesconduits). - Keywords vs search terms: a small list of focused keywords is joined with
ORinto a boolean string for the text search. - Rerank fallback: if a reranker isn’t available, return results without reranking (still usable, just less precise ordering).
- Context length: long candidates are truncated and the final set is limited to keep prompting efficient.
Two judges
Response quality (0.0-1.0): completeness, facts from the supplied context, structure, citations. Penalize unsupported claims.
Keyword quality (0.0-1.0): relevance, coverage, not too generic, not redundant.
Keep the judge model and rubric fixed when you compare runs. Use a different model than the generator. Temperature low. JSON only.
Evaluation Prompt Samples
Use these as a starting point. Keep temperature low and require strict JSON to reduce drift.
Response quality (single score)
System:
You are an impartial evaluator. Score how well the assistant’s response answers the user’s prompt, relying only on the provided context when present. Use a 0.0–1.0 scale with decimals. Penalize unsupported claims, factual errors, missing key steps, weak structure, and citation issues (when sources are included). Reward completeness, correctness, clarity, and good use of supplied context. Do not add new facts. Return ONLY a JSON object.
Rubric (guide):
1.0 = comprehensive, correct, well-structured, grounded in context (if provided)
0.8–0.9 = strong, minor gaps/omissions
0.6–0.7 = adequate but misses important points or has clarity issues
0.4–0.5 = weak coverage or notable errors
0.0–0.3 = largely incorrect/unsupported/off-topic
Output JSON schema:
{ "score": number, "explanation": string }
User:
<prompt>
{{USER_PROMPT}}
</prompt>
<context>
{{OPTIONAL_CONTEXT_OR_EMPTY}}
</context>
<response>
{{ASSISTANT_RESPONSE}}
</response>Keyword quality (retrieval terms)
System:
You are evaluating a list of keywords proposed for retrieving documents relevant to the user’s prompt. Score 0.0–1.0 for relevance, coverage of core concepts, specificity (not too generic), and non-redundancy. Penalize missing key terms or overly broad terms. Return ONLY a JSON object.
Rubric (guide):
1.0 = highly relevant, comprehensive, specific, minimal redundancy
0.8–0.9 = mostly relevant with minor gaps or minor redundancy
0.6–0.7 = generally relevant but misses important concepts or is too generic
<=0.5 = poor coverage or many irrelevant items
Output JSON schema:
{ "score": number, "explanation": string }
User:
<prompt>
{{USER_PROMPT}}
</prompt>
<keywords>
{{["term 1", "term 2", "term 3"]}}
</keywords>Pairwise A/B (optional)
System:
You are an impartial judge. Compare Response A vs Response B for the same prompt (and optional context). Choose the better answer based on correctness, completeness, clarity, and grounding in the provided context. Do not invent facts. Return ONLY a JSON object.
Output JSON schema:
{ "winner": "A" | "B", "margin": number, "rationale": string }
Notes:
- margin in [0.0, 1.0] where higher = stronger win
- keep rationale concise
User:
<prompt>
{{USER_PROMPT}}
</prompt>
<context>
{{OPTIONAL_CONTEXT_OR_EMPTY}}
</context>
<response_A>
{{ASSISTANT_RESPONSE_A}}
</response_A>
<response_B>
{{ASSISTANT_RESPONSE_B}}
</response_B>Compare models the same way
Generation and judging are separate, so I can swap the generator, keep the judge, and compare. Same questions, same retrieval, different model. Look at averages and at which items flipped. Do not treat the judge as a finished grade.
Failure types I actually see
- Retrieval miss (nothing relevant surfaced)
- Weak sourcing (claims not supported by provided sources)
- Unsupported claim (hallucinated requirement)
- Formatting drift (missing sections or citations)
- Keyword gap (too generic, or misses core terms)
Reproduce it
- Ten to twenty-five representative questions.
- A corpus indexed for vector and text search.
- Generate pairs:
{prompt, response, model, timestamp, keywords, search_terms, explanation}. - Score response quality and keyword quality. Write an evaluation JSON plus a Markdown report.
- Change one thing. Rerun the same questions.
keywords is the extracted list. search_terms is the boolean string, e.g. term1 OR term2 OR term3.
Version each run: model names, prompt version, a commit hash. Incremental writes so a 429 does not wipe the file.
Simplified Code Sketches (illustrative)
Generation
def generate_pairs(questions, system_prompt, retrieve, llm):
items = []
for q in questions:
terms = llm.extract_terms(q) # ["Section 26 05 33", "EMT", ...]
search_terms = " OR ".join(terms)
context = retrieve(q, search_terms) # hybrid search returns concatenated snippets
prompt = f"<context>{context}</context>\n<query>{q}</query>"
resp = llm.generate(system_prompt, prompt)
items.append({
"prompt": q,
"response": resp,
"model": llm.name,
"keywords": terms,
"search_terms": search_terms,
"explanation": "Term extraction rationale here"
})
return items # write to a JSON fileEvaluation
def judge_pairs(pairs, judge):
results, r_scores, k_scores = [], [], []
for p in pairs:
rq = judge.score_response(p["prompt"], p["response"]) # {score, explanation}
kq = judge.score_keywords(p["prompt"], p.get("keywords", []))
results.append({
"prompt": p["prompt"],
"score": rq["score"],
"explanation": rq["explanation"],
"keywords": p.get("keywords", []),
"search_terms": p.get("search_terms", ""),
"keywords_evaluation": kq,
})
r_scores.append(rq["score"])
k_scores.append(kq["score"])
summary = compute_stats(r_scores, k_scores)
return {"results": results, "summary": summary} # write to an evaluation JSONModel comparison (generation side only)
def compare_models(questions, system_prompt, retrieve, models, judge):
leaderboard = []
for m in models:
pairs = generate_pairs(questions, system_prompt, retrieve, m)
evald = judge_pairs(pairs, judge)
leaderboard.append({"model": m.name, "avg": evald["summary"]["response_quality"]["average_score"]})
return sorted(leaderboard, key=lambda x: x["avg"], reverse=True)Stats helper
def compute_stats(resp_scores, kw_scores):
def stats(xs):
xs = sorted(xs)
n = len(xs)
mid = n // 2
return {
"average_score": sum(xs) / n if n else 0.0,
"median": (xs[mid] if n % 2 else (xs[mid - 1] + xs[mid]) / 2) if n else 0.0,
"p25": xs[max(0, (n * 25) // 100 - 1)] if n else 0.0,
"p75": xs[max(0, (n * 75) // 100 - 1)] if n else 0.0,
"min_score": xs[0] if n else 0.0,
"max_score": xs[-1] if n else 0.0,
"high_scores_count": sum(1 for s in xs if s >= 0.8),
"medium_scores_count": sum(1 for s in xs if 0.6 <= s < 0.8),
"low_scores_count": sum(1 for s in xs if s < 0.6),
}
return {
"response_quality": stats(resp_scores),
"keyword_extraction_quality": stats(kw_scores)
}Iterate from the report
When the same failure repeats, I change one thing and rerun. Sometimes I feed the report and the current system prompt to another model and ask for surgical prompt edits. Then I rerun. I do not ship a prompt change because an agent said it would help. I ship it if the next report looks better on the same questions.
Start small. Expand later. Let the low items tell you what to fix.