The Week I Kept Routing Ministry Queries to the Wrong Model

I routed every incoming ministry query to the lowest-cost model for months because the benchmark numbers looked clean. The decision felt responsible at the time. We were shipping fast and the variance on simple prompts stayed low enough that nobody raised an alarm.

Bad ministry model routing is what nearly cost us a volunteer’s trust. One Wednesday the system handed a children’s ministry volunteer a rewritten lesson that dropped the key application point and added a sentence about adult small groups instead. The volunteer printed it, taught it, and only noticed the mismatch when a parent asked why the craft didn’t match the story. That single handoff cost two hours of follow-up calls, an apology to the family, and the volunteer’s willingness to try the tool again.

The pattern was clear once I stopped defending the choice. One-time model selection hid the exact places where ministry work moves between people with different constraints and different levels of context. The cost showed up later, in coordination failures, not in the token ledger.

John Wesley’s three rules give a better lens than any benchmark table. Do no harm. Do good. Stay in love with the work. Applied to model routing, the first rule forces you to measure whether a response damages the next human step. The second pushes you to track whether the output actually advances the volunteer or pastor who receives it. The third keeps you from outsourcing the judgment to a pricing page.

The model that looked cheap on benchmarks failed the Wednesday night handoff because it optimized for the wrong unit. Token cost per thousand queries hid the fact that the output still needed a children’s ministry lead to rewrite the closing paragraph before it could reach a volunteer. That rewrite step was invisible in the lab. In the field it became the place where trust broke.

We had chosen the model on a set of generic instruction-following tests. None of those tests included the handoff between a paid staff person writing curriculum and a volunteer who only has seven minutes between arriving at church and starting the lesson. The model passed the test and failed the actual work.

Routing rules emerged only after we logged every Slack thread outcome. For six weeks we captured what happened after each AI suggestion left the system. We noted whether the next person accepted it, edited it, or started over. The logs showed that the cheap model triggered a full rewrite far more often in children’s ministry threads than in adult sermon threads. The difference was not in the model’s intelligence. It was in the distance between the generated text and the next human’s constraints.

Once the data existed, the routing decision became simple. Children’s ministry queries went to the model that preserved application points without extra explanation. Adult sermon queries could stay on the cheaper model because the pastor already treated the output as raw material. The rules were not elegant, but they matched the actual movement of work.

Open models forced us to own the evaluation loop instead of outsourcing it. With closed APIs the only visible signal was price and latency. With open weights we could run the same prompt across three models every week and keep the outputs side by side. The habit revealed that one model consistently preserved the volunteer-friendly structure on curriculum tasks while another drifted toward generic language. That difference only appeared because we kept the comparison running instead of declaring a winner once.

The loop also surfaced when a model began to degrade. A small change in the fine-tune data would shift the tone on lesson endings. We caught it in the weekly sample instead of waiting for a volunteer complaint. Owning the loop turned model choice from a procurement decision into an ongoing product measurement.

The model that looked cheap on benchmarks failed the Wednesday night handoff

The failure was not dramatic in the logs. A single paragraph changed, a craft instruction disappeared, and the volunteer adapted on the fly. The cost appeared in the follow-up conversation and in the decision not to try the tool again.

Ministry handoffs carry asymmetric risk. A bad suggestion to a pastor can be rewritten in private. A bad suggestion to a volunteer reaches families before anyone sees it. The benchmark never measured that exposure.

Task-level measurement starts with the handoff, not the prompt. We now track whether the next human accepts the output unchanged, edits it lightly, or discards it. That single metric changed which model we route to for different ministry surfaces.

Ministry model routing rules emerged only after we logged every Slack thread outcome

Before the logs we guessed at routing logic. After six weeks the data showed clear patterns by ministry role and time pressure. Children’s ministry threads needed structure that survived printing. Adult teaching threads needed raw ideas that a pastor could expand.

The rules stayed narrow. Three categories, two models, and a simple tag on the incoming query. Nothing more complex survived contact with real threads.

Logging forced the team to treat every outcome as data rather than anecdote. The habit replaced the earlier assumption that one model would eventually be good enough for everything.

Open models forced us to own the evaluation loop instead of outsourcing it

Closed APIs hide the comparison. You receive one output and move on. Open weights let you keep the last three weeks of real ministry prompts and rerun them whenever a new checkpoint appears.

The weekly run takes about thirty minutes and surfaces drift before users notice. It also keeps the product team in direct contact with the actual tasks instead of relying on vendor benchmarks.

Owning the loop changed the conversation from “which model is best” to “which model is best for this exact handoff this week.” That shift matches the work ministry teams actually do.

Your Turn: Apply This Today

• Choose three real ministry tasks that currently route through Slack or email this week.
• Run each task against both your current closed model and one open-weight alternative, keeping the outputs side by side.
• Log what the next human actually does with the output: accept, edit, or discard.
• Tag each outcome by ministry role and time pressure so patterns appear after two weeks.
• Write a one-sentence routing rule based on the data and apply it to the next ten incoming queries.
• Repeat the three-task comparison next Monday and note any change in acceptance rate.

See also Graph Engineering: Stop Waiting on Agents That Don’t Need Each Other and The Token Cost Drop That Only Appeared After We Added the Knowledge Graph for related evaluation practices.

I consult with ministry product leaders on task-level model evaluation and routing rules for handoffs. Let’s talk.