AnswerAI EngineeringPlaceholder
How do you evaluate an LLM feature before launch?
Short answer
Build an evaluation set of at least 50 real inputs with expected outputs, grade responses with a mix of exact checks, model-based graders and human review, and set a pass threshold before launch. Run the same set on every prompt or model change so regressions are caught before users see them.
Last verified24 Sep 2026
The detail
Pick examples that reflect real usage, including the awkward cases. Add production failures back into the set over time.
Agree the pass threshold with the product owner before building, so the launch decision is a number rather than a debate.