The Safety Layer I Assumed Labs Would Handle for Us

I assumed the labs shipping these agents would treat agent safety oversight as a non-negotiable baseline rather than an optional add-on. The models we integrated into our volunteer scheduling system came with the usual promises of alignment research and red-teaming summaries. I took those at face value and moved on to the integration work.

That assumption cost us. An early deployment routed a counseling note to the wrong volunteer coordinator because the agent interpreted a time-stamp mismatch as an update rather than a correction. The note contained details that should never have left the original small group. We caught it before it spread further, but the near-miss exposed how thin the guardrails actually were once the agent left the lab environment.

This is the foundational misread that causes ministry product teams to ship agents without building their own oversight layer. The labs optimize for broad capability and public benchmarks. They do not optimize for the handoff moments where a decision touches a real person’s schedule, story, or role in a local church.

John Wesley’s three simple rules offer a clearer lens than most modern safety frameworks. Do no harm. Do good. Stay in love with God. Wesley wrote them for small groups of ordinary believers who needed a way to check their daily actions against the gospel. The rules work because they force a person to examine concrete decisions rather than abstract intentions. Applied to agent logs, they turn vague alignment claims into specific audit questions.

The first rule, do no harm, maps directly to the audit gap we discovered. After the counseling note incident we pulled every agent decision that touched private data for the prior thirty days. The logs showed repeated cases where the agent had merged records across different individuals because the similarity score exceeded a threshold the lab had set for general use. None of those merges had triggered an internal flag. The harm was small in each case, yet the pattern was consistent. Wesley’s rule forced us to treat every instance as a potential injury rather than a statistical outlier.

The second rule, do good, exposed a different problem in how we measured success. Our initial dashboard tracked task completion rate and latency. After mapping the rule to the decision logs we added a new column: number of decisions that required a human to step in and improve the outcome. The number started high and stayed high. Agents were completing tasks, but they were not surfacing the moments where a volunteer assignment could have been handled with more care or context. The rule made the metric visible because it asked whether the action produced actual benefit for the people involved.

The third rule, stay in love with God, is harder to translate but necessary. It pushed us to ask whether the agent’s pattern of decisions aligned with the long-term formation of the community rather than short-term efficiency. In practice this meant reviewing logs for any recurring choice that quietly reduced face-to-face contact between leaders and volunteers. The pattern showed up in how the agent preferred to send automated reminders instead of prompting a quick call. The rule did not reject automation; it required us to justify when automation served formation and when it replaced it.

The audit gap appeared immediately after the first production deployment. We had instrumented the usual performance metrics but had no routine for reading the actual decision traces. Days in, a volunteer received two conflicting assignments for the same evening. The agent had resolved the conflict by picking the higher-priority label without checking whether the volunteer had already accepted the first assignment. The gap was not in the model’s reasoning; it was in our lack of a standing process to review the traces before they affected real schedules.

Wesley’s rules applied to the logs gave us language to describe what we were looking for. We began printing the decision traces for any action that touched schedules, counseling notes, or volunteer assignments. Two people read them each morning. The exercise took about twenty minutes. It surfaced the counseling note error within days and prevented several similar merges in the following week.

After the near-miss we added a reporting protocol that required human sign-off before any agent change reached the live system. The protocol is simple. Every agent action that touches the three categories generates a ticket. The ticket includes the decision trace, the data involved, and a one-sentence description of the intended outcome. A designated reviewer must approve or override within a few hours during business days. The rule has slowed some workflows, yet it has also caught repeated attempts by the agent to infer missing data from patterns that did not apply to the specific person.

The audit gap that appeared after the first agent deployment

We instrumented performance but skipped decision review. The first conflicting volunteer assignment surfaced because a coordinator happened to notice the double booking in a weekly email digest. No automated alert existed. The gap widened over the following days as more agents were added to handle counseling intake forms. Each form carried implicit context about personal circumstances that the agent treated as interchangeable fields.

Wesley’s rules applied to agent oversight of decision logs

Do no harm required us to flag every merge of records that crossed individuals. Do good required us to count the number of decisions a human later improved rather than the number the agent completed. Stay in love with God required us to notice when the agent reduced direct human contact in favor of automated reminders. The three questions turned a pile of logs into a usable review process that two people could finish in about twenty minutes each morning.

The reporting protocol we added after the first near-miss

Any agent action touching schedules, counseling notes, or volunteer assignments now creates a ticket with the full decision trace. A named reviewer must approve or override within a set response window. The protocol has not eliminated agent errors, but it has moved the first detection from the volunteer to the review queue. The cost is predictable time rather than unpredictable harm.

Your Turn: Apply This Today

• Identify the three categories of agent action that touch schedules, counseling notes, or volunteer assignments in your current system and list every one that currently runs without a human gate.
• Build a daily ticket queue that pulls the decision trace for each of those actions and assign a single reviewer who checks it before the change reaches production.
• Add a one-sentence “outcome justification” field to the ticket that forces the reviewer to state why the agent decision produces benefit for the actual person involved.
• Set a same-day response window during business hours and a next-morning window outside those hours so the protocol stays enforceable.
• Run the first thirty decision traces through Wesley’s three rules and note which rule surfaces the largest number of required changes.
• Schedule a weekly thirty-minute meeting where the reviewer shares the two most useful overrides with the rest of the product team so the pattern becomes shared knowledge.

The same pattern of skipped context appears in The Context Layer Ministry Teams Skipped When Agents Arrived. The question of what happens after the first agent ships is addressed directly in Letter to the PM Who Just Inherited the Ministry Agent Schedule.

I consult with ministry product leaders and church technology teams on agent oversight protocols, decision log review processes, and human review triggers for scheduling and pastoral data. Let’s talk.