Building a Prompt Eval Harness in an Afternoon
Why prompts regress
A tweak that fixes one case breaks three others — silently, because nobody re-runs the old cases. An eval harness is a test suite for behavior: fixed inputs, recorded outputs, scored diffs. Without one, prompt engineering is superstition.
The minimal harness
Collect 20–30 real inputs with expected outcomes (exact match for extraction, rubric-graded for generation). Score deterministically where possible (regex, JSON validity, keyword presence) and use LLM-as-judge only for nuance, with the judge prompt versioned alongside. Run on every prompt change; store scores over time.
CI integration
Evals run in CI like unit tests: block merges that drop the pass rate. Keep the golden set in the repo next to the prompts. Refresh quarterly with production failures — each incident becomes a permanent regression case.
What not to eval
Don't chase single-number quality scores for open-ended generation; track task-completion proxies instead (resolution rate, escalation rate, user thumbs). Metrics must connect to outcomes or they optimize theater.
Related Articles
AI Guardrails for Production — PII, Jailbreaks, and Human Review
Layered defenses for LLM apps: input filters, output checks, and escalation paths that actually work.
Choosing a Vector Database — pgvector vs Pinecone vs Qdrant
Scale thresholds, filtering needs, and ops burden: picking vector storage without over-engineering.
Document Extraction Pipelines — From PDFs to Structured Data
OCR, layout-aware chunking, and validation loops for invoices, resumes, and lab reports.
Written by AI Engineer
Specialized engineering teams at Omnetra focus on writing high-performance code, ensuring API security, and optimizing layouts for client success.