Evaluation often arrives at the end of a generative AI project. Teams choose a model, build the feature, and prepare to launch before they decide what a good result actually looks like. The order is often backward.
Airbnb's evaluation-driven development approach treats evaluation as a development discipline, not a final check. It does not try to predict every failure in advance. Instead, the team finds failures as they appear, turns them into criteria, and keeps testing them from then on.
The process begins modestly. Run one hundred inputs through a prototype and have people read every output. Group the failures by type, then select fifty to one hundred examples for a golden set. Even when an AI judge is introduced, it is not trusted immediately. Its scores are compared with human scores, and disagreements are used to refine the criteria.
What stood out to me was not the number of examples but the organizational rules. One person is named as the final decision-maker when opinions diverge. Automation stops when experts cannot agree. After launch, product managers continue reading outputs every week and add new failures to the evaluation set.
When evaluation belongs only to the engineering team, what remains is often what is easiest to measure. Format errors and response times are easy to count. Why a user feels frustrated, or what makes an answer good, cannot be decided by technology alone. Evaluation is ultimately the work of making the product's standards explicit.
Years ago, when we built an internal chatbot, we made it refuse to answer below a confidence threshold. At the time, it felt like a compromise that sacrificed answer rate. Looking back, it was our simplest evaluation system: we had defined the moment when the system should step aside.
The golden-set file itself is not much of a moat. It can be copied. The more important asset is the agreement formed while creating it: what the organization calls a good result, who decides when people disagree, and how a new failure becomes part of the next standard.
Models can be bought. What our organization chooses to call good is something we have to decide for ourselves.