The Compute Number That Only Appeared After We Routed Ministry Queries

A ministry AI pilot reported a dramatic drop in per-query compute after the first routing layer went live. The number looked decisive on the dashboard because it measured only the traffic the agent handled end to end.

What the dashboard never captured was the volume of queries that left the agent and landed on a human volunteer or staff pastor. Those escalations carried full context, full token history, and an entirely different cost profile. When the team finally added the handoff events to the same ledger, the original headline savings shrank sharply and then disappeared within a couple of weeks.

The obvious reading — that routing alone had solved the escalation cost problem — was exactly backwards. Routing had simply moved the expensive work off the visible chart.

This is the foundational misread that causes product teams to ship agents they later have to patch with ever-larger evaluation layers. They optimize for the segment the model can finish alone and treat every handoff as an exception instead of the actual operating condition.

Charlie Munger’s latticework insists that any single model is only as reliable as the second and third models you run against it. In this case the second model had to be the ministry handoff itself. Without that second lens the first number stayed seductive and wrong.

What the initial benchmark hid about escalation volume

The early benchmark ran on a curated slice of children’s ministry questions: simple story summaries, basic memory-verse lookups, and scheduling reminders. Those queries resolved inside the agent. Escalation rate sat in the low single digits.

When the same routing logic met the live queue, escalation climbed into the range of about a third of all traffic within the first month. Most of the new volume came from edge cases the training set had never seen: a volunteer asking how to adapt a lesson for a child with sensory processing needs, a pastor needing last-minute substitute material after a family emergency, and follow-up threads that referenced prior conversations the agent could not retrieve.

Each escalation carried the full preceding context window. The compute cost after the handoff was often several times the cost of the agent leg alone. The original headline savings had been calculated only on the easy majority of traffic.

The routing change that surfaced the real weekly burn

We added a mandatory second hop for any query that referenced a specific person, date, or prior interaction. The router now logged the token count at the moment of handoff and again at the moment the human closed the thread.

The first full week of data showed thousands of escalated threads, each carrying a modest but real per-thread cost once the human leg was counted. The weekly total moved from a projected figure to several times that once the human leg was included. The routing change had not increased cost; it had simply made the existing cost visible.

Teams that keep the handoff invisible continue to celebrate savings that exist only inside the agent’s boundary. The latticework requires the second measurement before any claim is trusted.

The filter we now apply before any new agent ships

No new agent reaches production until we run last week’s production threads through the proposed router and record the cost after the first human touch. If the post-escalation number exceeds the current baseline by any meaningful margin, the agent is rejected or narrowed.

The filter also requires an explicit escalation taxonomy: which categories of query are allowed to stay inside the agent, which must always surface, and which trigger a human review after a few turns. Without that taxonomy the model drifts toward the easy subset and the hidden cost reappears.

The discipline is tedious, but it prevents the cycle of shipping an optimistic benchmark and then spending the next quarter building guardrails that should have been required at the start.

Your Turn: Apply This Today

• Export every ministry thread from the last seven days that touched a human volunteer or pastor.
• Replay each thread through your current router and stop the simulation at the first handoff.
• Log the cumulative token count from the moment the human receives the context onward.
• Calculate the average cost per escalated thread and compare it to your agent-only average.
• Tag the three highest-volume escalation categories and write one-sentence rules for how each should be routed next week.
• Schedule a 30-minute review with the person who actually answers those threads to confirm your tags match reality.

The Token Cost Drop That Only Appeared After We Added the Knowledge Graph showed how surface-level metrics shift once hidden structure is added. The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer walked through the same pattern on a different product surface.

I consult with ministry product teams on routing logic, escalation measurement, and the filters required before any agent reaches production. Let’s talk.