I spent two years treating automated LLM judges as the final word on model selection for ministry-facing features. The assumption was simple: if the judge ranked outputs consistently across blind tests, the ranking would hold once real users encountered the same material. That shortcut produced a model choice that looked strong on every internal metric but collapsed the moment a children’s ministry coordinator tried to adapt the generated lesson for a seven-minute volunteer slot.
The cost showed up in completion rates. Volunteers abandoned the flow at twice the previous rate, and the drop traced directly to phrasing the judge had scored highest for clarity and warmth. When I finally ran the same outputs past three actual coordinators and asked them to pick the version they would hand a volunteer tomorrow morning, the ranking inverted on 64 of the 100 items. The model we had shipped was the one they consistently ranked last.
This is the foundational misread that causes product teams to lock in the wrong model for pastoral tasks. Automated judges optimize against preference data that does not reflect the constraints of Sunday morning, or the volunteer who has never opened the tool before. Teresa Torres’s continuous discovery framework insists that product decisions stay tethered to direct user contact rather than proxy signals. The same discipline applies to evaluation: the only reliable signal is repeated, lightweight human preference collection on the exact workflow the model will serve.
The 64-generation flip that changed the ranking
The inversion happened on a narrow slice of curriculum generation for Sermons4Kids. We had prompted three models to produce a full small-group lesson from a given Bible passage, then fed the outputs to an LLM judge with a rubric focused on theological accuracy, age appropriateness, and engagement. Model B won by a clear margin.
Three coordinators then scored the same set without seeing the model labels. They were told only to choose the version they would actually print and give to a volunteer that week. Model B dropped to last place on 64 items. The reasons clustered around sentence length, the number of steps that required additional explanation, and the absence of any margin notes a volunteer could use when a child asked an unexpected question.
The judge had rewarded fluent, self-contained prose. The coordinators rewarded material that left room for the volunteer to insert their own voice. That single mismatch flipped the selection.
How ministry taste differs from generic preference data
Generic preference datasets reward completeness and polish because those qualities score well in broad user studies. Ministry contexts reward something closer to scaffolding. A children’s ministry volunteer needs the output to function as a script they can deviate from without losing the thread, not a finished product they must deliver verbatim.
The difference appears most clearly on edge cases. When a passage involves difficult imagery or a concept that does not map neatly to a seven-year-old’s experience, the generic judge prefers the model that offers the most elaborate explanation. Coordinators prefer the model that flags the difficulty and suggests one concrete question the volunteer can ask instead. The two priorities pull in opposite directions once the output leaves the product team’s laptop.
Torres’s continuous discovery approach treats this gap as expected rather than surprising. It requires the team to keep bringing fresh user contact into the loop instead of declaring the evaluation problem solved after the first benchmark run.
Building the Sunday morning checkpoint into the eval loop
The practical fix is to treat human preference collection as a recurring checkpoint rather than a one-time validation. After every automated ranking round, pull the top two model outputs and run them through a small set of real users who perform the actual task. The checkpoint does not need to be large; five to seven scorers who match the target persona are enough to surface inversions before they reach production.
The checkpoint must also match the time pressure of the real workflow. For curriculum tools, that means giving scorers the same seven-to-ten-minute window a volunteer would have. Outputs that require more time to evaluate are already disqualified, regardless of what the automated judge said.
Over successive checkpoints, the team builds a running record of which model characteristics survive human review. That record becomes the lightweight preference layer that sits on top of every future automated benchmark. When the layer and the benchmark disagree, the layer wins until the models are retested with fresh users.
Your Turn: Apply This Today
- Choose the single workflow your team runs most often, curriculum generation, sermon illustration drafting, or volunteer onboarding copy, and write down the exact user persona who will touch the output on Sunday morning.
- Generate twenty paired outputs from your current top two models using real inputs from the last month of that workflow.
- Recruit three users who match the persona and give them a shared scoring sheet that asks only which version they would use tomorrow morning and why, in one sentence.
- Run the scoring session in under thirty minutes total so the time cost matches the real constraint the users face.
- Record every inversion where the automated ranking and the human preference disagree, then store the twenty pairs with their scores in a shared folder the whole team can reference.
- Repeat the same twenty-item set with a new cohort of three users next month and compare the two runs before making any model swap.
“The 45-Minute Benchmark That Exposed My LLM Judge Habit” and “The Bias That Quietly Killed Three Ministry Pilots” both trace the same pattern of over-reliance on proxy signals.
I consult with product leaders shipping AI features into ministry and nonprofit contexts on continuous discovery loops, human preference layers for model evaluation, and workflow-specific checkpoints. Let’s talk.