The best benchmarks for ministry AI still require a human gate, even when the model scores 92 on every automated test.
Teams treat higher automated scores as proof that the system understands the real constraints of a children’s ministry volunteer printing materials on Saturday night. In practice those scores only confirm the model stayed inside the distribution of its training data. They reveal nothing about whether the output survives contact with a person who has seven minutes and a printer low on toner.
This is the foundational misread that causes product teams to ship workflows that look efficient in staging and collapse on Sunday. Teresa Torres’s continuous discovery framework exposes the gap: discovery happens through repeated, lightweight contact with users performing their actual jobs, not through static test sets that grow stale the moment the first volunteer deviates from the expected path.
Why 45-Minute Human Benchmarks Beat the Dashboard
Torres emphasizes that discovery is continuous precisely because user context shifts faster than any fixed dataset can track. A dashboard of LLM-as-judge scores creates the opposite condition. It freezes the evaluation criteria at the moment the prompt was written, then rewards the model for staying inside that frame.
A 45-minute human benchmark works differently. One person pulls a real, frozen input set from last month’s curriculum export, runs it through the new model, and scores the output against the single observable behavior that matters: whether the volunteer can complete the activity without opening a second tab or asking for help. The score is binary and tied to time-on-task, not to semantic similarity.
Teams that adopt this pattern quickly notice the dashboard and the human score diverge. The model that aces BLEU, ROUGE, and custom LLM judges still produces instructions that assume the volunteer has already read the full lesson plan. The 45-minute check catches the failure before it reaches the print queue.
The Ministry Failure Mode LLM Judges Miss
LLM judges optimize for coherence and doctrinal alignment because those dimensions are easy to express in a grading rubric. They miss the failure that actually costs ministries users: an output that looks complete but leaves the volunteer without the next physical action.
Consider a children’s ministry workflow that generates small-group discussion questions. An automated judge will rate the questions as age-appropriate and on-topic. The same judge will not notice that the questions require the leader to locate three separate craft supplies that are not listed in the materials section. The volunteer hits that gap at 6:43 p.m. on Saturday, and the activity is abandoned.
Continuous discovery treats this as a signal to update the frozen input set, not as noise to be averaged out. Each new human-scored run adds the exact edge case the previous version missed. Over six weeks, the benchmark grows into a living record of what actually breaks in the field rather than a polished but brittle leaderboard.
Running Your First Frozen-Input Check This Week
The first check does not require new tooling. Export the last three weeks of curriculum PDFs that were actually printed and used. Strip any date-specific or church-specific identifiers so the set can be reused. Load those files into whatever generation system you are evaluating and treat the output as a single frozen artifact.
Run the artifact through one human reviewer who has performed the volunteer role before. Give that person a timer and the exact instructions a real volunteer receives. Record only whether the task finished inside the time budget and whether any required external resource was missing. Do not collect qualitative feedback on this first pass; the binary outcome is the only data that survives contact with next week’s schedule.
Repeat the identical frozen set after each model or prompt change. The moment the human score drops, the change is rejected regardless of what the automated judges report. This single loop implements Torres’s core practice: the product team stays in regular, low-cost contact with the job the user actually performs.
Your Turn: Apply This Today
- Export the last three curriculum PDFs that were actually printed and used in a children’s ministry setting this month and strip identifying details to create your first frozen input set.
- Recruit one former volunteer who still remembers the seven-minute constraint and schedule a single 45-minute session this week to run the frozen set through your current generation workflow.
- Score the output using only two criteria: task completed inside the time limit, and no missing physical materials required.
- Store the scored output and the exact model version or prompt hash so the same input can be re-run after any change.
- Reject the next model or prompt update that drops the human score below the previous run, even if automated metrics improve.
- Repeat the identical frozen-input check once every two weeks for the next six weeks and log the human completion rate alongside dashboard numbers.
Two earlier posts trace the same pattern from different angles: “The Bias I Let Shape Every Eval Until a Human Scored It” and “The 45-Minute Benchmark That Exposed My LLM Judge Habit.”
I consult with product leaders building AI features for ministry and nonprofit tools on human-scored benchmarks, continuous discovery loops, and frozen-input checks for Sunday workflows. Let’s talk.