I sat with three kids ministry coordinators last month and watched them tear through fifty AI-generated lesson outlines. One page in, the first coordinator stopped. “This kid has never met an actual seven-year-old,” she said, and slid it aside. By the time they finished, thirty-four of the fifty were in the same pile.
The model had scored 87 percent on the automated check. Same outputs. Same frozen generations. The difference was that the machine only cared whether the words stayed inside the prompt. The humans cared whether the words would survive a real Wednesday night when the volunteer shows up late, half the kids are wired from practice, and the craft supplies are already missing three pieces.
That gap is not a scoring error. It is what happens when the judge has never stood in the room.
Teresa Torres’s continuous discovery framework makes the misread visible. Torres insists that discovery is not a phase but an ongoing loop of putting real artifacts in front of real users and watching what happens. When teams replace that loop with an automated judge, they stop updating the criteria that actually matter. The benchmark becomes a mirror of the team’s initial assumptions rather than a test of whether those assumptions still hold.
The same error shows up in agentic ministry tools that promise to route prayer requests, volunteer scheduling, or sermon prep. The model passes the automated rubric because the rubric was written before anyone had seen the model interact with a grieving parent at 11 p.m. on a Tuesday. Continuous discovery requires the team to keep running the same fixed inputs against the same fixed human scorers so that any drift in quality becomes impossible to ignore.
Where frozen inputs catch what rubrics miss
Human scoring on frozen inputs surfaces routing failures early. The 50-generation set includes a request for “a simple object lesson using only items found in a typical church kitchen.” Automated judges mark the output correct when it lists three common kitchen items and offers a spiritual application. Human scorers reject it when the suggested object lesson requires a microwave that the actual volunteer does not have access to after 8 p.m.
The difference is not subjective taste. It is the difference between a response that satisfies a written rubric and a response that survives contact with the physical constraints of the room where the lesson will actually be taught. Running the frozen set every two weeks forces the routing layer to surface these mismatches before the agent is exposed to live traffic.
Ministry pilots that skipped this step lost security reviews. One team built an agent that could draft follow-up messages for first-time guests. The automated judge scored the drafts highly for warmth and clarity. When the security team ran the same fifty inputs through the agent, it produced a message that referenced a specific family situation the guest had mentioned only in a private intake form. The pilot was paused for six weeks while the routing policy was rewritten. The failure had been latent in every automated pass; only the human gate on frozen data made it visible in time.
The cost of skipping the gate is not theoretical. Each lost security review consumes engineering cycles that could have been spent on the next routing improvement. The frozen human-scored benchmark acts as an early warning system that is cheaper than a formal review board.
How to run the 50-generation loop without slowing release cadence
The team selects one workflow and freezes the input set once. New model versions or prompt changes are scored against that fixed set by the same three human reviewers on a two-week cadence. Review sessions last forty-five minutes because the inputs never change. The output is a simple pass-fail plus one-sentence reason for each failure.
The cadence stays fast because the artifact under review is small and stable. The signal stays sharp because the humans are always scoring the same cases. Drift shows up as a rising failure count rather than as a vague sense that quality has slipped.
Your Turn: Apply This Today
- Pick one live ministry workflow that already has an agent in production or in late-stage pilot, such as volunteer lesson assignment or guest follow-up drafting.
- Collect the last fifty real inputs from that workflow, strip any PII, and freeze the list in a shared document with version control.
- Identify three human scorers who have direct experience with that workflow and confirm they can commit to forty-five minutes every other Friday for the next eight weeks.
- Run the frozen set through the current agent, present the outputs blind to the scorers, and record pass-fail plus one-sentence reason for each failure.
- Schedule the next scoring round for two weeks from today and commit to treating any rise in failure rate as a blocking signal for the next release candidate.
- After the third round, publish the cumulative failure trend inside your team and decide whether the routing policy needs revision before any further model changes ship.
The Benchmark Number That Still Needed a Human Gate and The Bias I Let Shape Every Eval Until a Human Scored It both trace the same pattern across different agent surfaces.
I consult with product leaders shipping agentic tools for ministry contexts on continuous discovery loops and human-scored frozen benchmarks. Let’s talk.