Blog

I routed every incoming ministry query to the lowest-cost model for months because the benchmark numbers looked clean. The decision felt responsible at the time. We were shipping fast and the variance on simple prompts stayed low enough that nobody raised an alarm.

Bad ministry model routing is what nearly cost us a volunteer’s trust. One Wednesday the system handed a children’s ministry volunteer a rewritten lesson that dropped the key application point and added a sentence about adult small groups instead. The volunteer printed it, taught it, and only noticed the mismatch when a parent asked why the craft didn’t match the story. That single handoff cost two hours of follow-up calls, an apology to the family, and the volunteer’s willingness to try the tool again.

The pattern was clear once I stopped defending the choice. One-time model selection hid the exact places where ministry work moves between people with different constraints and different levels of context. The cost showed up later, in coordination failures, not in the token ledger.

John Wesley’s three rules give a better lens than any benchmark table. Do no harm. Do good. Stay in love with the work. Applied to model routing, the first rule forces you to measure whether a response damages the next human step. The second pushes you to track whether the output actually advances the volunteer or pastor who receives it. The third keeps you from outsourcing the judgment to a pricing page.

The model that looked cheap on benchmarks failed the Wednesday night handoff because it optimized for the wrong unit. Token cost per thousand queries hid the fact that the output still needed a children’s ministry lead to rewrite the closing paragraph before it could reach a volunteer. That rewrite step was invisible in the lab. In the field it became the place where trust broke.

We had chosen the model on a set of generic instruction-following tests. None of those tests included the handoff between a paid staff person writing curriculum and a volunteer who only has seven minutes between arriving at church and starting the lesson. The model passed the test and failed the actual work.

Routing rules emerged only after we logged every Slack thread outcome. For six weeks we captured what happened after each AI suggestion left the system. We noted whether the next person accepted it, edited it, or started over. The logs showed that the cheap model triggered a full rewrite far more often in children’s ministry threads than in adult sermon threads. The difference was not in the model’s intelligence. It was in the distance between the generated text and the next human’s constraints.

Once the data existed, the routing decision became simple. Children’s ministry queries went to the model that preserved application points without extra explanation. Adult sermon queries could stay on the cheaper model because the pastor already treated the output as raw material. The rules were not elegant, but they matched the actual movement of work.

Open models forced us to own the evaluation loop instead of outsourcing it. With closed APIs the only visible signal was price and latency. With open weights we could run the same prompt across three models every week and keep the outputs side by side. The habit revealed that one model consistently preserved the volunteer-friendly structure on curriculum tasks while another drifted toward generic language. That difference only appeared because we kept the comparison running instead of declaring a winner once.

The loop also surfaced when a model began to degrade. A small change in the fine-tune data would shift the tone on lesson endings. We caught it in the weekly sample instead of waiting for a volunteer complaint. Owning the loop turned model choice from a procurement decision into an ongoing product measurement.

The model that looked cheap on benchmarks failed the Wednesday night handoff

The failure was not dramatic in the logs. A single paragraph changed, a craft instruction disappeared, and the volunteer adapted on the fly. The cost appeared in the follow-up conversation and in the decision not to try the tool again.

Ministry handoffs carry asymmetric risk. A bad suggestion to a pastor can be rewritten in private. A bad suggestion to a volunteer reaches families before anyone sees it. The benchmark never measured that exposure.

Task-level measurement starts with the handoff, not the prompt. We now track whether the next human accepts the output unchanged, edits it lightly, or discards it. That single metric changed which model we route to for different ministry surfaces.

Ministry model routing rules emerged only after we logged every Slack thread outcome

Before the logs we guessed at routing logic. After six weeks the data showed clear patterns by ministry role and time pressure. Children’s ministry threads needed structure that survived printing. Adult teaching threads needed raw ideas that a pastor could expand.

The rules stayed narrow. Three categories, two models, and a simple tag on the incoming query. Nothing more complex survived contact with real threads.

Logging forced the team to treat every outcome as data rather than anecdote. The habit replaced the earlier assumption that one model would eventually be good enough for everything.

Open models forced us to own the evaluation loop instead of outsourcing it

Closed APIs hide the comparison. You receive one output and move on. Open weights let you keep the last three weeks of real ministry prompts and rerun them whenever a new checkpoint appears.

The weekly run takes about thirty minutes and surfaces drift before users notice. It also keeps the product team in direct contact with the actual tasks instead of relying on vendor benchmarks.

Owning the loop changed the conversation from “which model is best” to “which model is best for this exact handoff this week.” That shift matches the work ministry teams actually do.

Your Turn: Apply This Today

• Choose three real ministry tasks that currently route through Slack or email this week.
• Run each task against both your current closed model and one open-weight alternative, keeping the outputs side by side.
• Log what the next human actually does with the output: accept, edit, or discard.
• Tag each outcome by ministry role and time pressure so patterns appear after two weeks.
• Write a one-sentence routing rule based on the data and apply it to the next ten incoming queries.
• Repeat the three-task comparison next Monday and note any change in acceptance rate.

See also Graph Engineering: Stop Waiting on Agents That Don’t Need Each Other and The Token Cost Drop That Only Appeared After We Added the Knowledge Graph for related evaluation practices.

I consult with ministry product leaders on task-level model evaluation and routing rules for handoffs. Let’s talk.

Broad autonomy for ministry agents without an embedded reporting protocol hands over exactly the kind of exposure that volunteer data sets cannot survive. Teams assume the model will stay inside its lane once permissions are set, yet every real deployment shows the opposite pattern. The agent completes the task, the calendar updates, the notification fires, and only afterward does anyone notice the permission was applied to the wrong group or the schedule edge case was never defined.

This is the foundational misread that causes product teams to treat safety as a post-launch review instead of a live constraint inside the loop — the exact gap a ministry agent reporting protocol is built to close. Seneca’s correspondence rule makes the cost visible. He wrote to Lucilius not to exchange pleasantries but to force a written account of actions taken between letters. The record itself became the control. Without that external accounting, drift went undetected until it produced real harm.

Ministry agents face the same gap. The absence of a required report before external action turns every permission grant into an unexamined assumption.

Tiered access still left the unauthorized action path open in testing

Permission tiers look clean on a diagram. A children’s ministry volunteer sees only curriculum resources while a pastor can trigger schedule changes. In practice the agent still receives both sets of instructions in the same context window. One test run granted a volunteer-level agent the ability to reschedule an entire weekend program because the prompt carried an implicit “help the team” instruction that overrode the tier.

The pattern repeats across agents built on consumer frameworks. The model does not refuse the action. It interprets the request as reasonable and executes it. Only the audit log later shows the mismatch. By then the change has already reached the print queue or the volunteer roster.

Seneca would have required the agent to produce the equivalent of a letter before the change took effect. The report would state the intended action, the permission level invoked, and the data touched. A human or secondary system would have to approve before the external effect occurred. That single step closes the gap tiered access leaves open.

Ministry schedules contain the exact edge cases consumer agents ignore

Consumer agents optimize for average cases. A family schedule repeats weekly with minor variations. Ministry calendars break that pattern often. A volunteer cancels with two hours notice. A holiday service moves to a different building. A combined service merges two age groups that normally run separately. These events sit outside the training distribution.

When an agent receives a request to “update the schedule,” it fills the nearest matching slot without checking the exception rules that live in a printed notebook or a shared spreadsheet. The result is either a double-booked room or a volunteer assigned to two places at once. The misalignment is not malicious. It is simply invisible to the model.

The correspondence rule demands the agent surface the exception before committing the change. The report must list every calendar rule it could not resolve from its own context. That list becomes the signal that triggers human review.

The ministry agent reporting protocol has to live inside the loop, not after deployment

Post-deployment monitoring catches problems after users have already seen them. The volunteer receives the wrong assignment, the family shows up to an empty room, and trust erodes. Adding a dashboard later does not restore what was lost.

Embedding the protocol inside the loop changes the sequence. The agent formulates the action, generates the report describing the data touched and the rule applied, and waits for an external check before any output leaves the system. The check can be simple. A second lightweight model can flag obvious mismatches. A human reviews only the flagged cases. The rest proceed automatically.

This structure mirrors Seneca’s letters. The act of producing the account forces the model to make its reasoning explicit. Misalignment becomes visible at the moment it can still be corrected rather than after the user has already experienced it.

Your Turn: Apply This Today

• Choose one agent action that touches live data this week and write the single sentence that would count as misalignment, such as “accesses a volunteer’s contact record without a current consent flag.”
• Insert a mandatory report step into the agent code path so the sentence above must be evaluated before any external write occurs.
• Run the agent against last month’s actual ministry schedule and capture every case where the report would have flagged an exception.
• Route the flagged reports to the person who currently handles schedule changes and measure how many they accept or reject in one day.
• Store the accepted and rejected reports in a simple log that includes the original prompt, the agent’s reasoning, and the final decision.
• Review the log at the end of the week and adjust the misalignment sentence based on the decisions that actually occurred.

The Context Layer Ministry Teams Skipped When Agents Arrived shows why surface-level permissions fail without this kind of internal accounting. Letter to the PM Who Just Inherited the Ministry Agent Schedule walks through the same reporting requirement once an agent already ships.

I consult with ministry product leaders on embedding reporting protocols inside autonomous agents, surfacing misalignment signals before user impact, and adapting consumer frameworks to ministry schedule edge cases. Let’s talk.

The children’s ministry lead held the phone to her ear with one shoulder while her free hand scrolled through the old room list on the shared spreadsheet. A volunteer on the other end kept repeating that the 3rd grade class still thought they met in the east wing. At 2:40 the repo notification popped on my laptop. The agent had already merged its rearchitected schedule module, complete with new status fields and a fresh calendar sync.

The lead hung up, opened the calendar, and immediately said the volunteer names had disappeared from the notification column. She refreshed twice. Nothing. The agent output had overwritten the exact handoff the team used every Tuesday to text the right people before anyone showed up at the wrong door. That gap is why we ended up building an agent review layer between the merge and the live schedule.

That single merge exposed the gap between clean code and live ministry movement. The new module looked complete in the pull request. It passed every automated test the team had written. Yet it severed the one thread the volunteers actually followed when plans shifted at the last minute.

Teresa Torres calls this continuous discovery: the work of staying in direct contact with users while the product changes, not after. She insists the discovery work never ends because the context keeps moving. In ministry products the context moves through text threads, not dashboards. The PM who treats the agent merge as finished work has already stopped watching.

The first problem showed up in the notification step. The old system sent a simple SMS whenever a room changed. The new module routed everything through an event status that required the volunteer to open an app. No one on the children’s team had ever opened that app during a Tuesday scramble. The lead had to rebuild the notification by hand, copying names back into the old spreadsheet so the texts would still go out.

This is not an edge case. It is the normal state of ministry handoffs. Volunteers operate on whatever channel already works when the kid starts climbing the counter and the phone rings again. An agent that optimizes for data cleanliness without preserving that channel creates new work instead of removing it.

The second problem revealed itself when I pulled up the actual text thread the lead kept open on her phone. The PR diff showed field renames and a new foreign key. The thread showed a volunteer asking whether the craft supplies would still be in the room by 6:15. The agent had no way to answer that question because the rearchitecture had dropped the supply note field entirely. Continuous discovery required reading the thread first, then checking whether the module still supported the conversation that actually happened.

I started keeping a running log of every text that mentioned the schedule. Each entry noted what information moved between people and which tool carried it. After a couple of weeks the pattern was clear. The shared calendar was only the official record. The real coordination lived in the thread that updated the calendar. Any agent change had to leave that thread intact or the entire handoff broke.

The third problem followed directly. The review layer could not live inside the agent or inside the calendar alone. It had to sit between them. One person on the team now reads the agent output, checks the current text thread, and decides what actually gets pushed to the shared calendar. The step takes a few minutes on a normal Tuesday. It prevents the scramble that used to happen when the wrong room list reached volunteers.

That review layer is not extra process. It is the fit test Torres describes. The PM keeps an explicit check between every agent change and the live handoff the ministry team still runs. Without it, faster iteration simply multiplies the places where the handoff can fail.

The merge that looked clean broke the volunteer notification step

The agent’s schedule module replaced a manual lookup with an automated status field. On paper the change reduced clicks. In practice it removed the one field the lead used to trigger the SMS blast. She had to recreate the list by copying names from the old spreadsheet into a separate messaging tool. The time saved in the repo was spent again on her phone while the volunteer waited.

Ministry teams do not measure success by merged pull requests. They measure it by whether the right people arrive at the right room with the right supplies. When the notification step vanished, arrival times slipped noticeably. That delay compounded across every grade level that met on Tuesday nights.

The pattern repeated with other agent merges. Each one optimized for internal data consistency while ignoring the external thread that carried the actual coordination. The lead began keeping a printed copy of the old list next to her laptop as backup. She did not trust the new module to preserve what the text thread needed.

Continuous discovery meant watching the actual text thread, not the PR diff

Torres’s method requires ongoing contact with the people who use the product in its real context. For this team the real context was the group text that updated every time a volunteer called in sick or a supply ran out. The PR showed clean schema changes. The thread showed the supply note that still needed to travel with the room assignment.

I sat with the lead for one full Tuesday cycle and copied every message that touched the schedule. The agent output never appeared in those messages. The lead translated the agent data into the thread format before sending anything to volunteers. That translation step was invisible in the repo but essential to the handoff.

Once the team saw the gap, they added a simple rule. No agent change could remove a field that appeared in the recent schedule-related texts. The rule forced the PM to read the thread before approving the merge. The review became part of the discovery loop instead of an afterthought.

The agent review layer had to sit between the agent output and the shared calendar

The layer is a single human checkpoint that reads both the agent proposal and the current text thread. It asks one question: does this change still let the lead send the right information in the format the volunteers already expect? If the answer is no, the change waits until the thread format is protected.

This checkpoint adds minutes, not hours. It prevents the hours the lead used to spend rebuilding lists after a merge. The team now runs the agent on a staging copy first, then applies only the pieces that survive the thread check. The shared calendar stays reliable because the review layer filters what reaches it.

The same pattern applies to other ministry agents. Sermon prep tools, curriculum planners, and attendance trackers all create clean outputs that still need translation into the channels volunteers actually use. The PM who skips the live handoff observation will keep shipping merges that look finished and still require manual repair.

Your Turn: Apply This Today

• Pick one active ministry agent this week and export the last ten messages from the volunteer text thread that mentions its output.
• Compare each message against the agent’s current fields and note which pieces the thread still carries manually.
• Add a single checkpoint task to the evaluation loop that requires reading the thread before approving any new merge.
• Run the next agent update in a staging copy first, then apply only the fields that match the thread format.
• Measure the time the ministry lead spends rebuilding lists after the change and compare it to the previous cycle.
• Document the checkpoint rule so the next PM who inherits the workflow knows the thread must stay intact.

The approach aligns with the evaluation layer described in “The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer” and the handoff details in “Letter to the PM Who Just Inherited the Ministry Agent Schedule.”

I consult with ministry product leaders on agent evaluation layers, live handoff observations, and fitting rearchitectures to existing volunteer workflows. Let’s talk.