Teams running automated LLM judges on ministry content report 78 percent agreement with human preferences on standard benchmarks. The same teams, when forced to score outputs against the exact prompts their volunteers actually submit, see agreement fall to 38 percent. The obvious reading—that the models are mostly reliable and just need occasional human spot-checks—is backwards. The models perform well only on the narrow slice of tasks that already match the generic training data; they diverge sharply on the messy, context-heavy work that defines real ministry output.
That divergence is the signal. Teresa Torres’s continuous discovery framework treats every production artifact as a fresh opportunity to interview the user, not a finished deliverable. Applied to AI features, it means the evaluation step cannot be automated away once and for all. The benchmark must remain a living record of how actual users phrase their requests and what “good enough” looks like in their setting.
Why the 45-minute human eval cut through the noise
The first time we froze a set of real volunteer prompts and scored the model outputs by hand, the process took forty-five minutes for a batch of twelve items. That single run surfaced three failure modes the LLM judge had rated as passing: a children’s lesson that used vocabulary three grade levels too high, a small-group guide that assumed the leader already knew the theological term “sanctification,” and a prayer prompt that ended with a corporate-sounding closing no volunteer would ever say aloud.
The 45-minute constraint mattered. It forced us to score quickly and against a frozen input list rather than rewriting prompts on the fly. Speed kept the exercise from turning into another round of prompt engineering. The fixed inputs kept the comparison honest; every model variant faced the same volunteer language, not an improved version of it.
Most teams skip this step because it feels slower than letting the model judge itself. The data shows the opposite result: the human pass reveals the exact places where the model’s average-case training collides with the long-tail reality of ministry work. Once those collisions are written down, later automated checks can be calibrated against them instead of generic rubrics.
How the same workflow applies to baptism outlines and photo policies
The same frozen-input method works for tasks that never touch a sermon manuscript. One ministry team ran it on baptism service outlines. They captured the exact wording parents used when requesting date options, then scored the generated outlines for pastoral tone and legal clarity. The LLM judge had passed every sample. The human review flagged two outlines that omitted the requirement for a follow-up visit with the family—something every real coordinator treats as non-negotiable.
Another team applied it to photo policies for children’s events. They froze twelve real parent emails asking whether photos could be posted on the church app. The model outputs sounded reasonable until scored against the actual policy language the church had already published. Three outputs used phrases the policy explicitly prohibited, yet the automated judge had scored them as compliant.
In both cases the 45-minute human pass did not replace the model; it created a narrow, reusable test that later model versions could be run against. The test stayed small enough to repeat after every major model update without requiring a full product review cycle.
The checkpoint that keeps the benchmark alive after launch
Continuous discovery requires a recurring trigger, not goodwill. The teams that maintained the benchmark after launch tied it to an existing release gate rather than adding a new meeting. They added one line to the pre-launch checklist: “Run current frozen test set; record any score drop greater than 10 percent.” The line took less than five minutes to verify once the test set existed.
When the score did drop, the team did not rerun the full 45-minute exercise immediately. They first checked whether the drop came from changed volunteer language or from model drift. Only the latter triggered a fresh human scoring round. This kept the cost bounded while still surfacing the moment when the original benchmark stopped reflecting reality.
The checkpoint also surfaced when the product itself had changed the volunteer task. One team noticed their new intake form collected an extra field; the frozen prompts no longer matched how volunteers now described the request. Updating the frozen set became part of the same release process rather than a separate research project.
Your Turn: Apply This Today
- Pick one volunteer task that already has a written prompt or email template—children’s lesson requests, baptism date inquiries, or photo permission language.
- Copy the last ten real submissions exactly as they arrived, including typos and abbreviations, into a frozen list.
- Run your current model or agent against that list and save the outputs with no further prompt changes.
- Score each output yourself or with one volunteer using a simple three-point scale: usable as-is, needs light edit, or unusable.
- Write the two most common failure reasons on a single index card and tape it to your monitor.
- Schedule a five-minute calendar block the week after your next model update to rerun the same frozen list and note any score change.
The pattern in “Letter to the PM Handed an AI Agent Budget” and “The Bias That Quietly Killed Three Ministry Pilots” is the same: teams that treated evaluation as a one-time gate later discovered their models had drifted from the actual work. The same drift appears here once the human benchmark is allowed to go stale.
I consult with product leaders and ministry technology teams on frozen-input evaluation design, continuous discovery checkpoints, and volunteer-task benchmarks. Let’s talk.