Blog

You’re the product manager who just inherited an AI mandate inside a denomination where half the staff still prints bulletins on Thursday afternoons. The mandate arrived with a deck that promised “scalable pastoral care” and a timeline that assumed the hard part was the model. No one mentioned the volunteers who answer the phones, or the fact that trust here moves through names and clear trust signals, not notifications.

You made the first change quickly. You swapped the bot greeting for a real person’s name because the early logs showed people typing “who is this” before they typed anything else. That move felt small. It also exposed the larger pattern no one had named yet.

John Wesley’s three rules give the frame. Do no harm. Do good. Stay in love with God. In product terms the first rule means never letting an interface pretend to be a relationship it cannot sustain. The second means designing every surface so it actually routes people toward living people. The third means keeping the goal of the work pointed at real formation instead of simulated engagement.

The greeting change we made after the first week of complaints

The first version opened with a clean line: “Hi, I’m here to help with your ministry questions.” Within days the support inbox filled with messages that ranged from polite confusion to outright anger. People wanted to know who was on the other side and whether their question would reach anyone who actually knew their church.

We replaced the line with a name and a short role: “Sarah, children’s ministry coordinator, responds here.” The complaints dropped. Reply volume stayed flat for a couple of days, then climbed. The difference was not the model. It was the removal of the pretense.

Wesley’s first rule showed up immediately. Pretending the system was a person did harm because it trained users to expect a relationship that would never appear. Naming Sarah removed the deception before it could compound.

Why the photo and direct number mattered more than the model behind it

The next iteration added a small headshot next to Sarah’s name and a direct extension that routed to her desk line during posted hours. Session length did not change. First-reply rate did. People who saw the photo and the number were noticeably more likely to send a second message instead of closing the tab.

The photo worked because it signaled that a real human had agreed to be visible. The number worked because it gave an escape hatch. Both moves followed Wesley’s second rule. They turned the interface into a handoff instead of a trap.

We tested the same pattern on a different surface, a volunteer scheduling agent. When we hid the coordinator’s name and photo behind a generic “Support” label, completion rates fell. When the name and photo returned, completion rose again. The model stayed identical. The trust signal changed the outcome.

The metric we now track instead of session length

Session length rewarded long conversations with the agent. We stopped measuring it. The number we watch is the share of conversations that end with an explicit handoff to a named person within two messages. That metric now sits on the same dashboard as weekly active volunteers.

Wesley’s third rule keeps the target honest. The work is not to keep users inside the product. It is to keep them connected to the actual body they joined. When the handoff rate rises, we know the interface is doing its job instead of substituting for one.

The same shift appeared when we applied the pattern to a prayer request form. Adding the name and photo of the intercessor who would receive the request meaningfully increased the number of follow-up emails from church members. People treated the request as a relationship because the surface told them it was one.

Your Turn: Apply This Today

• Pick the single ministry surface that currently opens with a generic bot line and replace it with one named person’s name plus role this week.
• Add that person’s actual headshot from the staff directory, sized no larger than the name itself.
• Include their direct extension or Slack handle with posted availability hours.
• Run the change for seven days and record the reply rate on day one and day seven.
• Compare it against the prior seven-day average for the same surface.
• If reply rate rises, leave the change live and apply the same pattern to the next surface on your roadmap.

Two recent posts that trace the same thread are Letter to the PM Who Just Inherited the Ministry Agent Schedule and The Context Layer Ministry Teams Skipped When Agents Arrived.

I consult with ministry product leaders on human-first AI interfaces, trust signals in faith-facing products, and metrics that track handoffs rather than dwell time. Let’s talk.

A ministry AI pilot reported a dramatic drop in per-query compute after the first routing layer went live. The number looked decisive on the dashboard because it measured only the traffic the agent handled end to end.

What the dashboard never captured was the volume of queries that left the agent and landed on a human volunteer or staff pastor. Those escalations carried full context, full token history, and an entirely different cost profile. When the team finally added the handoff events to the same ledger, the original headline savings shrank sharply and then disappeared within a couple of weeks.

The obvious reading — that routing alone had solved the escalation cost problem — was exactly backwards. Routing had simply moved the expensive work off the visible chart.

This is the foundational misread that causes product teams to ship agents they later have to patch with ever-larger evaluation layers. They optimize for the segment the model can finish alone and treat every handoff as an exception instead of the actual operating condition.

Charlie Munger’s latticework insists that any single model is only as reliable as the second and third models you run against it. In this case the second model had to be the ministry handoff itself. Without that second lens the first number stayed seductive and wrong.

What the initial benchmark hid about escalation volume

The early benchmark ran on a curated slice of children’s ministry questions: simple story summaries, basic memory-verse lookups, and scheduling reminders. Those queries resolved inside the agent. Escalation rate sat in the low single digits.

When the same routing logic met the live queue, escalation climbed into the range of about a third of all traffic within the first month. Most of the new volume came from edge cases the training set had never seen: a volunteer asking how to adapt a lesson for a child with sensory processing needs, a pastor needing last-minute substitute material after a family emergency, and follow-up threads that referenced prior conversations the agent could not retrieve.

Each escalation carried the full preceding context window. The compute cost after the handoff was often several times the cost of the agent leg alone. The original headline savings had been calculated only on the easy majority of traffic.

The routing change that surfaced the real weekly burn

We added a mandatory second hop for any query that referenced a specific person, date, or prior interaction. The router now logged the token count at the moment of handoff and again at the moment the human closed the thread.

The first full week of data showed thousands of escalated threads, each carrying a modest but real per-thread cost once the human leg was counted. The weekly total moved from a projected figure to several times that once the human leg was included. The routing change had not increased cost; it had simply made the existing cost visible.

Teams that keep the handoff invisible continue to celebrate savings that exist only inside the agent’s boundary. The latticework requires the second measurement before any claim is trusted.

The filter we now apply before any new agent ships

No new agent reaches production until we run last week’s production threads through the proposed router and record the cost after the first human touch. If the post-escalation number exceeds the current baseline by any meaningful margin, the agent is rejected or narrowed.

The filter also requires an explicit escalation taxonomy: which categories of query are allowed to stay inside the agent, which must always surface, and which trigger a human review after a few turns. Without that taxonomy the model drifts toward the easy subset and the hidden cost reappears.

The discipline is tedious, but it prevents the cycle of shipping an optimistic benchmark and then spending the next quarter building guardrails that should have been required at the start.

Your Turn: Apply This Today

• Export every ministry thread from the last seven days that touched a human volunteer or pastor.
• Replay each thread through your current router and stop the simulation at the first handoff.
• Log the cumulative token count from the moment the human receives the context onward.
• Calculate the average cost per escalated thread and compare it to your agent-only average.
• Tag the three highest-volume escalation categories and write one-sentence rules for how each should be routed next week.
• Schedule a 30-minute review with the person who actually answers those threads to confirm your tags match reality.

The Token Cost Drop That Only Appeared After We Added the Knowledge Graph showed how surface-level metrics shift once hidden structure is added. The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer walked through the same pattern on a different product surface.

I consult with ministry product teams on routing logic, escalation measurement, and the filters required before any agent reaches production. Let’s talk.

Several widely-circulated AI infrastructure playbooks push a “frontier model default,” the assumption that every production query benefits from the largest available model. That assumption collapses once you measure actual model routing costs against the queries coming from volunteer coordinators at a mid-size ministry organization. Those playbooks treat capability as a one-dimensional scale. They ignore that most ministry interactions are narrow classification or retrieval tasks that smaller open models handle at a fraction of the cost.

This single-default habit produces budgets that grow faster than usage. Teams ship an agent, watch the first invoices, then discover that most of the load never needed the frontier tier at all.

Charlie Munger’s latticework of mental models insists you keep multiple tools sharp and switch between them deliberately. The same principle applies to model routing: you hold several models in inventory, each suited to a distinct slice of work, and you define the switch rules in advance.

The single-model habit that hid real costs

Teams default to the strongest model because the prompt works on the first try. In practice that choice masks the real distribution of work. A children’s ministry volunteer asking for a printable activity sheet triggers the same frontier-tier call as a pastor drafting a full sermon series. Both return usable text, so the cost difference never surfaces in testing.

The hidden expense compounds across volume. In the curriculum-support logs we reviewed for a volunteer-curriculum team we worked with, simple requests outnumbered theological research queries by a wide margin. Running every request through the same model turns a predictable pattern into an unpredictable monthly bill.

Open-model benchmarks run regularly against actual query logs expose the mismatch quickly. A smaller, fine-tuned open model matched frontier accuracy on the large majority of volunteer requests while cutting per-token cost substantially. The remaining harder cases still route upward, but the overall spend drops right away.

How routing exposed the coordination failures

Once multiple models sit in the stack, the routing layer must decide which one owns each incoming request. Without explicit rules, the decision falls to whoever last touched the prompt. One week the agent tries to answer everything locally; the next week someone flips a flag and every request escalates. The inconsistency shows up in both cost spikes and user frustration when answers suddenly lose context.

Munger’s latticework requires the decision criteria to be written down and tested, not left to individual judgment. Ministry teams that adopted a simple decision tree — query length, presence of curriculum keywords, presence of personal data — noticeably cut coordination meetings. The tree lives in the same repo as the agent code so changes require a pull request rather than a Slack thread.

The same tree also surfaces coordination failures between agents. A routing decision that sends a volunteer request to the small model sometimes needs to hand off to a knowledge-graph lookup agent next. Without the handoff rule documented, the small model simply hallucinates the missing data instead of calling the graph.

Building the handoff rule that stuck

The durable rule set three conditions for escalation. First, the query must contain a named person or family identifier. Second, it must reference content older than the current quarter. Third, the small model’s first-pass answer must score below a fixed similarity threshold against the local knowledge base. Any one condition triggers the frontier model; all three must fail to stay on the cheaper tier.

Teams tested the rule for several consecutive weeks on live traffic from a volunteer-facing curriculum platform we advised. The first week the rule was too strict and escalated far more queries than expected. By the third week the threshold settled at a point that escalated a much smaller share of traffic while keeping user-reported accuracy comfortably high. The rule survived because it was measured against real queries rather than synthetic test sets.

The handoff also preserved audit trails. Every escalation logged the three conditions that fired, so product managers could review edge cases without re-running the entire conversation history. That audit layer turned a cost-saving tactic into a repeatable evaluation practice.

Your Turn: Apply This Today

• Export a sample of real queries from your ministry agent logs this week and tag each one with expected model tier.
• Run the same queries against a couple of open-weight models using identical prompts and record accuracy plus cost per query.
• Write a three-condition decision tree in a single markdown file and commit it to the agent repo before Friday.
• Deploy the tree behind a feature flag and route a small slice of live traffic through it for a week.
• Compare the flagged cohort’s spend and accuracy against the control cohort in a shared dashboard.
• Adjust one condition in the tree based on the week’s results and schedule the next test run for the following Monday.

The Token Cost Drop That Only Appeared After We Added the Knowledge Graph and Graph Engineering: Stop Waiting on Agents That Don’t Need Each Other both show how routing decisions only stabilize once the surrounding data layer is explicit.

I consult with ministry product leaders on task routing for agents, open model evaluation, and compute cost control in faith-tech deployments. Let’s talk.