Blog

I spent eighteen months assuming that any AI we added to our content tools would reduce the load on our volunteer coordinators. The faster we generated lessons, the more time they would have for actual relationships with kids. That assumption cost us three coordinators in one quarter and left the rest of the team questioning whether the product still served the people who showed up every Sunday.

The real fracture was not about speed or even about quality. It was about who decided when the output was good enough to hand to a volunteer working on a Tuesday night with seven minutes and a printer. Once that decision moved to an external model and a dashboard no one on the ground could touch, the people closest to the work stopped trusting the system they were supposed to maintain.

This is the same pattern Jensen Huang has warned about with sovereign AI. When an organization gives up control over the model, the data, and the evaluation criteria, it ends up optimizing for someone else’s metrics. In ministry products the someone else is usually an LLM vendor whose success is measured in tokens, not in whether a volunteer finishes the lesson without rewriting it at 10 p.m.

The split showed up first in the curriculum workflow

The first visible break happened in the weekly curriculum handoff. We had built a generation step that turned a short prompt into a full lesson outline, complete with discussion questions and craft ideas. On paper it looked like a win for the 7-minute volunteer. In practice the outlines arrived with the wrong tone for our age group or referenced objects that did not exist in most church supply closets.

Coordinators began spending more time fixing the generated material than they had spent writing it from scratch. One of them printed a side-by-side comparison for the team: the original handwritten lesson took twenty-two minutes to produce and needed almost no changes; the AI version took four minutes to generate and then forty minutes to repair. The time savings had moved from the volunteer to the coordinator, exactly the opposite of the intended effect.

The team divided along a line no one had predicted. The product managers and engineers saw throughput numbers rising and assumed morale was improving. The coordinators saw their evenings disappear into revision cycles and stopped suggesting improvements. The product kept shipping, but the people who actually delivered it on Sunday had checked out.

Control over the model mattered more than speed

We tried the usual fixes first: better prompts, retrieval-augmented generation, even a small fine-tune on past approved lessons. None of them addressed the core issue. The model still sat outside the team’s direct influence. When it produced something off, the only options were to accept it or to open a ticket that might be resolved in the next sprint.

Huang’s point about sovereign control applies here in a practical way. A ministry product team needs the ability to freeze a model version, to define its own scoring rubric, and to override outputs without waiting for an upstream change. Without that, every improvement in generation speed simply increases the volume of work that must be manually corrected by the same small group of coordinators.

We eventually moved a smaller, locally hosted model behind a simple internal interface. Generation slowed by a factor of three, yet the coordinators stopped rewriting lessons. The difference was that they could now adjust the temperature, swap the reference set, and reject an output before it reached the print queue. The product became slower on the server and faster in the real workflow.

Sovereignty starts with who owns the eval loop

The decisive change came when we handed the evaluation criteria back to the people who would use the output. Instead of a shared dashboard with vendor-defined scores, each coordinator kept a personal frozen set of ten lessons they had already approved by hand. Any new generation had to be compared against that set before it moved forward. If it failed the comparison, the coordinator could veto it without further discussion.

This small shift restored ownership. The eval loop was no longer an abstract quality metric; it was a concrete personal benchmark that traveled with the person doing the work. Morale improved not because the model got smarter, but because the decision rights returned to the same individuals who carried the Sunday responsibility.

The pattern repeats across other ministry tools. When an AI feature is introduced without a corresponding transfer of control over its outputs, the people closest to the mission absorb the hidden cost. The line that split our team was never about technology preference. It was about whether the coordinators still recognized the product as something they could shape.

Your Turn: Apply This Today

• Choose one coordinator on your current project and give them explicit veto authority over any AI-generated content before it reaches volunteers; document the veto rule in the next sprint planning note.
• Have that same coordinator assemble a frozen personal benchmark of eight to ten past lessons they consider acceptable and store it as a simple shared folder they control.
• Replace the next automated quality score with a two-question checklist the coordinator answers against their benchmark before any output moves forward.
• Schedule a thirty-minute weekly review where the coordinator can adjust the reference set or reject a model version without needing engineering approval.
• Measure coordinator time spent on revision rather than generation for the next four weeks and share the delta with the product team in raw minutes, not percentages.
• If the revision time does not drop by at least half, pause the AI step for that workflow until the coordinator can describe what control they still lack.

The same ownership question appears in The Benchmark Number That Still Needed a Human Gate and The Bias I Let Shape Every Eval Until a Human Scored It.

I consult with ministry product leaders on AI tool ownership, volunteer workflow metrics, and eval loops that stay in human hands. Let’s talk.

You are the product manager who just got told to lead a three-person pod that will own every workflow from volunteer onboarding to curriculum delivery. The mandate arrived with an org chart that looks clean on paper and a budget line that says “AI-native.” No one mentioned the actual people who will use what you ship.

You know the risk. Three generalists can move fast when every decision loops through one operator who still touches the real work. But the moment that operator starts letting the model close the loops, the whole setup quietly stops serving the people it was built for.

Teresa Torres built her continuous discovery habit on the simple claim that teams only ship useful products when they keep running lightweight interviews with real users every week. The framework forces the product person to stay in the room where the work actually happens instead of letting assumptions harden inside a document or a prompt history. When the pod is small, that single discipline becomes the only thing that keeps the model from becoming the default decision maker.

The pod collapses when discovery stays inside the chat window

Teresa Torres always started with the assumption that the team’s current understanding is incomplete. In a three-person generalist pod the temptation is to treat the model as the stand-in for that missing understanding. A quick prompt produces a volunteer flow, the flow looks reasonable, and the sprint begins. The problem is that the model has never watched a children’s ministry volunteer try to print a lesson on a church copier that only works half the time.

The collapse happens quietly. The pod ships the feature. Usage numbers look flat. No one can point to a single conversation with an actual volunteer that would explain why. Continuous discovery, in Torres’s terms, requires the product owner to schedule and run the interview themselves, not delegate the synthesis to a model. When the pod skips that step, every later decision rests on an untested mental model of ministry work.

One person must still watch the child-protection log

Ministry tools carry constraints that most enterprise software never faces. A lesson plan that accidentally surfaces a child’s photo to the wrong volunteer is not a usability issue; it is a safety failure. The model can generate clean copy and attractive layouts, but it cannot carry the liability or the institutional memory of past incidents.

In practice this means the single operator in the pod has to keep a standing review of every change that touches identity data or permissions. Torres’s continuous discovery habit helps here because the weekly interview often surfaces the edge case the model missed. The volunteer mentions that their phone number changed and the old one still routes to the previous church administrator. That single sentence forces a design change no amount of prompt refinement would have produced.

The operator who skips this review is not being efficient. They are creating the conditions for the first real incident that will later require the entire pod to stop and rewrite the permission model under pressure.

Authenticity comes from the friction the model cannot remove

Torres’s framework treats ongoing contact with users as the source of product judgment, not as a research phase that ends. In a generalist pod the operator who keeps running live sessions with volunteers develops a feel for what actually lands. The language that feels natural, the step that always gets skipped, the moment when a volunteer reaches for a printed page instead of the screen.

That friction is the signal. A model can remove every visible obstacle and still produce something that feels off to the person who has to use it on a Sunday morning. The operator who protects time for direct observation keeps the pod’s output tethered to the actual texture of ministry work. When that loop breaks, the product begins to optimize for the model’s idea of cleanliness rather than the volunteer’s idea of getting the job done.

Your Turn: Apply This Today

• Book one 20-minute call this week with a children’s ministry volunteer who has used your current tool at least twice; do not use the model to draft the questions.
• Run the session using the exact screen or printout the volunteer would have in front of them and take notes by hand.
• Within 24 hours write the single most surprising sentence from that call and share it with the other two pod members before the next planning meeting.
• Identify one workflow step the volunteer skipped or worked around and add it to the backlog as a discovery item rather than a build item.
• Schedule the next identical call for the same time next week before you close the calendar invite for this one.
• Refuse any model-generated summary of the session; keep the raw notes as the only record the pod uses for the next two sprints.

The same pressure to collapse discovery into the model appears in Letter to the PM Who Inherited the Agent Pilot and The Benchmark Number That Still Needed a Human Gate. Both posts trace what happens when the operator stops running the live loop.

I consult with product leaders running small AI pods on continuous discovery and human oversight in ministry tools. Let’s talk.

Token caching in ministry agents produces the 10x cost reduction that product teams celebrate until a background check rule or baptism record policy updates and the cached context continues to serve the old decision.

The mismatch appears harmless in logs because the agent still returns an answer. It surfaces only when the live system would have blocked or rerouted the request. Most teams discover the gap after the first volunteer completes a form that the cached prompt should have rejected.

This pattern repeats because teams treat reuse as a pure engineering win rather than a policy surface that must stay synchronized with changing requirements.

The 10x saving hid the policy drift

Ministry agents built for children’s ministry often cache the first several turns of a volunteer intake flow. The template holds the current background check criteria and the rule for when a record routes to a pastor for review. Teams that built the cache assumed the criteria would remain static for the quarter. In practice, denominational policies and state requirements shift on shorter cycles. The 10x token reduction therefore trades one-time savings for ongoing exposure.

Charlie Munger’s latticework of mental models treats this as an inversion problem. Instead of asking how to cache more aggressively, the latticework asks what must remain uncached for the decision to stay valid. The model that surfaces is second-order consequences: the first-order win on cost produces a second-order loss on correctness that compounds with every new volunteer record.

Power users noticed the mismatch first on Sunday

Children’s ministry coordinators who run the same intake form every weekend hit the stale cache first. One coordinator saw the agent approve a volunteer whose recent record should have routed to a pastor for review. The cached context still referenced the older threshold. The coordinator caught it because she compared the agent’s answer against the printed policy sheet she still kept on her desk.

The same gap appeared in agents handling baptism record lookups. Cached summaries continued to list a candidate as eligible after the congregation updated its membership requirements. The mismatch only became visible when the family arrived for the scheduled meeting and the printed record differed from the agent’s response.

The latticework that catches cached errors

Munger’s latticework requires multiple models to be applied before a change is considered complete. The relevant models here are redundancy, feedback loops, and inversion. Redundancy demands an explicit policy check outside the cached block. Feedback loops require the agent to log every cached decision against the live rule so drift becomes measurable. Inversion asks what the agent must not do even when the cache is valid.

Teams that apply only the cost model miss the other two. They ship the cached template because the token count falls and the latency improves. They do not run the live policy check that would catch drift before a volunteer or family feels the consequence. Splitting the prompt so the policy portion becomes a small, always-fresh query that runs before the cached material is used preserves most of the saving while keeping the decision current. The 10x saving remains on the non-policy turns while the decision surface stays current.

Your Turn: Apply This Today

• Open the prompt template for your current ministry agent and locate every cached section that references background checks or membership records.
• Insert a single live API call that fetches the current policy version and compares it to the cached values before any further tokens are generated.
• Add a one-line log entry that records the policy version used for each cached decision so drift becomes visible in weekly reviews.
• Reduce the cached block to only the non-policy turns and keep the policy portion outside the cache for the next agent update.
• Run the revised template against the last ten real volunteer records and confirm every output now matches the live rule.
• Schedule the same explicit check in the deployment checklist before any future cached prompt ships.

The Routing Step Ministry Teams Still Can’t Automate and Letter to the PM Who Inherited the Agent Pilot both examine where small automation choices create larger downstream mismatches in ministry systems.

I consult with product leaders shipping ministry agents on token caching risks, policy synchronization, and mental model application to prompts. Let’s talk.