Blog

I spent two years treating automated LLM judges as the final word on model selection for ministry-facing features. The assumption was simple: if the judge ranked outputs consistently across blind tests, the ranking would hold once real users encountered the same material. That shortcut produced a model choice that looked strong on every internal metric but collapsed the moment a children’s ministry coordinator tried to adapt the generated lesson for a seven-minute volunteer slot.

The cost showed up in completion rates. Volunteers abandoned the flow at twice the previous rate, and the drop traced directly to phrasing the judge had scored highest for clarity and warmth. When I finally ran the same outputs past three actual coordinators and asked them to pick the version they would hand a volunteer tomorrow morning, the ranking inverted on 64 of the 100 items. The model we had shipped was the one they consistently ranked last.

This is the foundational misread that causes product teams to lock in the wrong model for pastoral tasks. Automated judges optimize against preference data that does not reflect the constraints of Sunday morning, or the volunteer who has never opened the tool before. Teresa Torres’s continuous discovery framework insists that product decisions stay tethered to direct user contact rather than proxy signals. The same discipline applies to evaluation: the only reliable signal is repeated, lightweight human preference collection on the exact workflow the model will serve.

The 64-generation flip that changed the ranking

The inversion happened on a narrow slice of curriculum generation for Sermons4Kids. We had prompted three models to produce a full small-group lesson from a given Bible passage, then fed the outputs to an LLM judge with a rubric focused on theological accuracy, age appropriateness, and engagement. Model B won by a clear margin.

Three coordinators then scored the same set without seeing the model labels. They were told only to choose the version they would actually print and give to a volunteer that week. Model B dropped to last place on 64 items. The reasons clustered around sentence length, the number of steps that required additional explanation, and the absence of any margin notes a volunteer could use when a child asked an unexpected question.

The judge had rewarded fluent, self-contained prose. The coordinators rewarded material that left room for the volunteer to insert their own voice. That single mismatch flipped the selection.

How ministry taste differs from generic preference data

Generic preference datasets reward completeness and polish because those qualities score well in broad user studies. Ministry contexts reward something closer to scaffolding. A children’s ministry volunteer needs the output to function as a script they can deviate from without losing the thread, not a finished product they must deliver verbatim.

The difference appears most clearly on edge cases. When a passage involves difficult imagery or a concept that does not map neatly to a seven-year-old’s experience, the generic judge prefers the model that offers the most elaborate explanation. Coordinators prefer the model that flags the difficulty and suggests one concrete question the volunteer can ask instead. The two priorities pull in opposite directions once the output leaves the product team’s laptop.

Torres’s continuous discovery approach treats this gap as expected rather than surprising. It requires the team to keep bringing fresh user contact into the loop instead of declaring the evaluation problem solved after the first benchmark run.

Building the Sunday morning checkpoint into the eval loop

The practical fix is to treat human preference collection as a recurring checkpoint rather than a one-time validation. After every automated ranking round, pull the top two model outputs and run them through a small set of real users who perform the actual task. The checkpoint does not need to be large; five to seven scorers who match the target persona are enough to surface inversions before they reach production.

The checkpoint must also match the time pressure of the real workflow. For curriculum tools, that means giving scorers the same seven-to-ten-minute window a volunteer would have. Outputs that require more time to evaluate are already disqualified, regardless of what the automated judge said.

Over successive checkpoints, the team builds a running record of which model characteristics survive human review. That record becomes the lightweight preference layer that sits on top of every future automated benchmark. When the layer and the benchmark disagree, the layer wins until the models are retested with fresh users.

Your Turn: Apply This Today

• Choose the single workflow your team runs most often, curriculum generation, sermon illustration drafting, or volunteer onboarding copy, and write down the exact user persona who will touch the output on Sunday morning.
• Generate twenty paired outputs from your current top two models using real inputs from the last month of that workflow.
• Recruit three users who match the persona and give them a shared scoring sheet that asks only which version they would use tomorrow morning and why, in one sentence.
• Run the scoring session in under thirty minutes total so the time cost matches the real constraint the users face.
• Record every inversion where the automated ranking and the human preference disagree, then store the twenty pairs with their scores in a shared folder the whole team can reference.
• Repeat the same twenty-item set with a new cohort of three users next month and compare the two runs before making any model swap.

“The 45-Minute Benchmark That Exposed My LLM Judge Habit” and “The Bias That Quietly Killed Three Ministry Pilots” both trace the same pattern of over-reliance on proxy signals.

I consult with product leaders shipping AI features into ministry and nonprofit contexts on continuous discovery loops, human preference layers for model evaluation, and workflow-specific checkpoints. Let’s talk.

The “autonomous first, review later” guidance that ships with most agent frameworks assumes builders can bolt human oversight onto an already-running loop without rewriting the loop itself. Shreyas Doshi’s account of status-quo bias shows why that assumption collapses in practice: once the default path encodes the status-quo routing, every later iteration protects that path instead of exposing its failure points. Ministry teams that adopt this pattern discover the breakage only after a volunteer or pastor receives an answer that should never have left the system.

The same pattern shows up in every pilot that treats pastoral review as a nice-to-have guardrail. The guardrail arrives too late, after the agent has already committed to an answer, a schedule, or a referral. The result is not a slow rollout. It is eroded trust that no amount of later tuning repairs.

Doshi describes how teams repeatedly optimize the metric that is easiest to instrument while the metric that reveals real breakage stays unmeasured. In agent work, the easy metric is task completion inside the model. The hard metric is whether the human who must act on the output can do so without introducing new risk or labor. When that second metric stays invisible, the bias locks in.

Voice agent creation that skips the pastoral review step

Most voice agents for ministry start with a prompt tuned for speed and tone. The team ships a demo that sounds natural on the first try, and pastoral review gets scheduled for week three or four, once “the core loop is stable.”

By that point the agent has already learned which questions it can answer without escalation. Questions that should surface to a human, like a custody concern mentioned inside a children’s ministry query, receive plausible but unverified replies. The review session becomes an exercise in adding exceptions rather than redesigning the default path. The status-quo bias has done its work: the original routing is treated as given, and every fix adds complexity on top of it.

Teams that instead require a pastoral sign-off before the first external call see the opposite pattern. The reviewer flags the exact class of query that must never receive an autonomous answer, and the agent is built around an explicit handoff rule rather than an after-the-fact correction. Completion rates inside the model drop, but the rate at which real ministry leaders trust the output rises.

Where the default routing erases the security log

Agent platforms default to storing conversation transcripts for model improvement. When the same platform is pointed at ministry data, the transcript often contains names, prayer requests, or attendance patterns that fall under church child-protection policies. The default retention setting keeps those records for ninety days or more, and the routing layer never surfaces that choice to anyone on the ministry side.

The bias shows up in the decision to treat retention as an infrastructure detail rather than a product decision. Because the agent still completes its assigned task, the missing log entry never registers as a defect. Only after an audit or a parent inquiry does the team realize the system has no record of what was said or to whom it was escalated. Rebuilding the log after the fact requires changes to the core data model, not a configuration toggle.

One team I worked with moved the retention decision into the first configuration screen an admin sees. The agent cannot start a session until the ministry chooses either “no persistent transcript” or “transcript stored under role-based access only.” The choice forces the status-quo setting into the open before any conversation occurs.

The one checkpoint that forces the bias into the open

The checkpoint is a fifteen-minute review of the exact moment the agent decides it can answer without a human. In practice, this means printing the decision tree or prompt branch that determines escalation and walking through it with the person who would actually receive the handoff. Every branch that ends in “model responds” must be justified against the specific risk it carries in a ministry context.

Teams that run this checkpoint before any public pilot surface assumptions that would otherwise remain invisible: that volunteer schedules are public information, that a prayer request can be summarized without consent language, that a children’s curriculum question never touches custody issues. Once those assumptions are listed, the routing logic must change or the pilot must be scoped down until they are resolved.

The checkpoint does not slow shipping. It prevents the later rewrite that occurs once trust is already damaged, and it gives the ministry leader a concrete artifact they can show their board or insurance carrier.

Your Turn: Apply This Today

• Open your current agent pilot’s escalation prompt and list every condition that currently leads to an autonomous answer; mark any condition that touches children, counseling, or personal data.
• Export the last twenty live sessions and highlight every response that a pastor would need to see before it reached a volunteer or attendee.
• Schedule a fifteen-minute call with the actual person who would receive the handoff and walk through the marked items one by one.
• Change the default retention setting in the agent configuration to “no transcript” and require an explicit admin override before any session is stored.
• Write the single sentence that will appear to the user when the agent refuses to answer, and test whether that sentence matches the tone your ministry partners expect.
• Re-run the same fifteen-minute bias audit after the next code change and record which assumptions have reappeared.

The same status-quo bias appears in the pilots described in “The Bias That Quietly Killed Three Ministry Pilots” and in the budget conversations captured by “Letter to the PM Handed an AI Agent Budget.”

I consult with product leaders building AI agents for ministry contexts on surfacing hidden handoffs and running bias audits. Let’s talk.

The volunteer leaned across the check-in table, phone wedged between shoulder and ear, two kids tugging at her sleeve. She tapped the new kiosk screen and spoke low: “Confirm the peanut allergy for the Johnson girl.” The speaker hummed for half a second, then answered in the exact warm cadence Pastor Mike used when he closed services three years ago. The volunteer froze, eyes wide, and glanced around as if someone had stepped into the room.

No one had. The system had simply pulled the pastor’s recorded greeting from an old Easter service and routed it through the new voice agent. The child’s record updated. The line moved forward. The volunteer kept working, but the moment stayed on her face.

That single exchange exposed the real problem. Voice agents can now match pastoral tone with almost no friction. The moment they do, they borrow authority that was never granted to a piece of software. John Wesley’s three simple rules give us a way to test whether that borrowing is responsible.

The rule of do no harm applied to cloned voices

Wesley’s first rule begins with refusal: do no harm. In product terms, this means the voice itself must not carry emotional weight it cannot answer for. When the kiosk spoke in Pastor Mike’s cadence, it triggered recognition before it delivered a single fact. Recognition is not neutral. It pulls on relationships built in sanctuaries and hospital rooms.

The harm shows up in small places first. A parent hears the familiar voice and assumes pastoral review already happened. A volunteer treats the answer as settled because the tone matches someone she trusts. Neither party asked for that shortcut. The product supplied it anyway.

Teams building these systems usually focus on transcription accuracy and latency. They rarely test what happens when the voice carries three years of accumulated trust. Without an explicit refusal layer, the agent keeps borrowing that trust every time it answers.

The rule of do good applied to recorded context

The second rule pushes toward active good: record the surrounding context, not just the answer. In the check-in scene, the kiosk had access to the allergy note but nothing about the family’s last conversation with staff. It had the pastor’s voice but none of the pauses or follow-up questions he would have asked. Good requires that missing layer.

Product teams can meet this rule by logging who last touched the note and when. The voice becomes a pointer, not the authority. The volunteer still decides what to do with the information. This also changes the data contract: families need to know their child’s record will be read aloud by a system that identifies its sources. The good here is transparency, not speed. Wesley would recognize the difference.

The rule of stay in love with God applied to presence versus simulation

Wesley’s third rule points to ongoing relationship with God. For product work, this becomes a test of presence. Simulated pastoral voice can reduce friction, but it cannot substitute for actual presence when the moment requires it. The kiosk answered the allergy question cleanly, yet it could not notice the volunteer’s hesitation or ask why the question felt urgent that day.

The distinction matters most at the edges. A medical update that also carries grief, fear, or family conflict needs a human who can stay in the room. The agent should surface those flags immediately and hand off rather than try to carry the tone across that gap.

Staying in love with God in this setting means refusing to let simulation stand in for the relationship it imitates. The product decision is simple in principle: any voice that sounds pastoral must carry an immediate, visible path back to a real person who can be reached that same hour.

Your Turn: Apply This Today

• Before your next event, add a two-rule override prompt to the voice agent configuration: first, require the agent to name the last human who touched the record; second, require an immediate handoff option if the query involves medical, safety, or family conflict flags.
• Export the last thirty kiosk answers from your check-in system and label which ones carried cloned pastoral tone. Replace any that did with a neutral system voice plus the required human-source line.
• Write the two override rules on a single index card and tape it to the kiosk hardware so every volunteer sees them before the first family arrives.
• Run a ten-minute test with two volunteers and one simulated allergy question. Confirm both overrides trigger without extra clicks.
• Update the family consent language on your registration form to state that recorded pastoral voices will never be used without an identified human source attached.
• Schedule a fifteen-minute review with whoever owns the voice model contract and confirm the two overrides are now non-removable defaults.

The same authority questions surface in ministry pilots that touch pricing data or agent budgets. The posts “The Question Every PM Asks When an Agent Hits Pricing Data” and “Letter to the PM Handed an AI Agent Budget” trace how those pressures play out once the system is already running.

I consult with product leaders and ministry technology teams on voice agent accountability layers, human-override design, and source transparency requirements. Let’s talk.