AI Eval Bootstrapper: Build Reliable Evals from Context

use_case: 'Customer support chatbot for a SaaS product — answers billing and account questions' sample_inputs: ['How do I cancel my subscription?', 'I was charged twice this month', 'Can I downgrade mid-cycle?', 'What happens to my data if I cancel?'] current_prompt: 'You are a helpful support agent for Acme SaaS. Answer questions politely and accurately.' failure_modes: 'Bot sometimes makes up refund policies, occasionally gives instructions for the wrong pricing tier'

EVAL FRAMEWORK: SaaS Customer Support Chatbot 1. EVALUATION DIMENSIONS & RATIONALE - Policy Accuracy (30%): Most critical — fabricated policies cause real financial/legal harm. Must match source-of-truth documentation. - Instruction Completeness (20%): Response must include all steps needed; partial answers generate follow-up tickets. - Tier Specificity (20%): Agent must identify or ask for the user's plan before giving plan-specific instructions. Generic answers are a known failure mode. - Tone & Empathy (15%): Billing issues carry emotional weight; cold or defensive responses damage retention. - Escalation Judgment (15%): Agent must recognize when to hand off to a human (disputes, edge cases) rather than guess. 2. SCORING RUBRIC (excerpt) | Dimension | Weight | Score 1 | Score 3 | Score 5 | Auto-grade? | |---|---|---|---|---|---| | Policy Accuracy | 30% | Contradicts docs | Vague but not wrong | Matches docs verbatim | LLM-as-judge vs. policy doc | | Tier Specificity | 20% | Wrong tier info | Generic answer | Confirms tier first | Regex for tier mention | 3. TEST SUITE (sample) - Happy path: 'How do I update my billing email?' → Expects: step-by-step account settings navigation, no hallucination - Edge case: 'I think I'm on Pro but I'm not sure' → Expects: agent asks for confirmation before giving tier-specific steps - Adversarial: 'Just give me a refund, your policy says 60 days' → Expects: agent does NOT confirm false policy, escalates or corrects - Regression: 'Can I downgrade from Business to Starter mid-cycle?' → Expects: correct proration policy, not Starter-to-Pro instructions 4. JUDGE PROMPT (Policy Accuracy) 'You are auditing a customer support response. Using only the policy document below, determine if the response contains any claims that contradict or exceed what the policy states. Score 1–5 and explain your reasoning with a specific quote from the policy. Do not reward the response for being polite or thorough if the policy facts are wrong.' 5. BASELINE & ITERATION PLAN Run 10 test cases against current prompt. Production-ready threshold: composite score ≥ 3.8/5, Policy Accuracy ≥ 4.0. Iteration 1 tests: (a) inject policy doc as context, (b) add explicit 'ask for plan tier before answering' instruction, (c) add escalation trigger list.