Blog

I watched the model hand back a clean volunteer roster one Tuesday morning. Every shift covered, no obvious clashes on the calendar. Then the first reply hit my phone—an elder who’d been double-booked into both the food-bank run and his own kid’s surgery.

The schedule looked balanced because the model only scored what it could see. It never felt the weight of someone having to choose between two people who both needed them.

We had treated the output like a finished decision instead of raw material that still needed an outside rule to catch what the model couldn’t afford to break.

Why pure model scoring failed on real schedules

Model scoring systems rate candidate schedules on metrics such as total coverage, average travel time, and stated preference match. Those numbers look decisive until a children’s ministry volunteer who agreed to two consecutive weeks is assigned a third week that conflicts with a prior commitment the model never saw. The score remains high because the conflict lives outside the prompt context.

Observed cases show the model consistently preferring the schedule that balances load across names rather than protecting the single volunteer whose absence would collapse an entire age-group rotation. The numeric optimum therefore produces the outcome that requires the most manual rescue calls the following week.

The deeper issue is that model scoring treats every constraint as soft. A hard rule such as “no volunteer may receive two consecutive weeks in the same role without explicit confirmation” never enters the evaluation function unless it is injected as code rather than left to the model’s judgment.

The simple rule check that caught the three-way fork

One team added a deterministic pre-filter that rejected any schedule containing a volunteer assigned to three distinct locations within a fourteen-day window. The filter ran after the model produced its top three candidates and before any human opened the file. In the first month the filter discarded seventeen schedules that the model had ranked highest.

The three-way fork surfaced when the same volunteer name appeared in the output for a Wednesday rehearsal, a Saturday outreach, and a Sunday service that required travel across town. The model had scored the combination acceptable because each individual assignment matched stated availability. The rule check flagged the cumulative load that no single prompt had surfaced.

Adding the filter required fewer than twenty lines of code and ran in under a second. It removed the need for coordinators to discover the conflict only after the volunteer had already been contacted and had to decline.

Where human review still has to sit in the loop

Human review belongs after the rule layer has already eliminated the mechanically invalid options. The reviewer then faces a short list that contains only candidates that satisfy every hard constraint. At that point the reviewer’s task narrows to weighing relational factors the rule set cannot encode, such as whether two volunteers who share a carpool should be kept on the same rotation.

Placing the human earlier, before rules run, wastes attention on schedules that will be rejected anyway. Placing the human later, after the schedule has already been published, forces recovery work that could have been avoided. The Solomon pattern requires the costly test to occur before the decision is finalized, not after.

Teams that moved the human into the final gate reported fewer last-minute replacements and higher volunteer retention on the same set of assignments. The model still generated the initial candidates; the added layers simply prevented the model from declaring victory on its own output.

Your Turn: Apply This Today

• Pick the single agent output your team reviews most often and write one deterministic rule that must pass before any human sees the file.
• Run last week’s rejected outputs through that rule and count how many would have been caught automatically.
• Add the rule as a blocking step in the review checklist and log the number of false positives removed each day for seven days.
• Share the before-and-after rejection counts with the person who currently performs the first manual pass so they can see the time shift.
• If the rule removes fewer than 15 percent of candidates, tighten its condition and repeat the measurement for the next seven days.
• Document the final rule and the measured reduction so the same check can be reused on the next agent workflow you ship.

The same pattern of layered verification appears in The PM Who Inherited the Agent Schedule and The PM Role That Still Needs Human Judgment After Agents Ship.

I consult with product leaders building ministry coordination agents on rule-based verification layers and placement of human judgment gates. Let’s talk.

The files sat open on my laptop again at 11:17 at night. One tab held the Sermons4Kids export with its clean columns, the other the volunteer sheet someone had typed by hand, and nothing matched. I kept telling myself the next tool would just read both and figure it out.

We waited anyway. Three platforms took our money and our time while we held off on the small fixes we actually knew how to make. The curriculum stayed frozen, the same spreadsheet kept circulating, and the gap between what we needed and what anyone would sell us only grew wider.

Eventually the waiting started to feel like the real decision we were making.

The rule that forced us to stop shopping for agents

Do no harm meant we could not send raw volunteer feedback or children’s ministry notes to any external service that retained training data. Most vendors were clear about this once we asked directly. Their default position was that our content would improve their model. That single clause ended the evaluation process for every off-the-shelf option we had shortlisted.

Do good meant the agent had to reduce actual time spent by the 7-minute volunteer, not generate impressive demo outputs. Our internal logs already showed where volunteers abandoned the workflow. We did not need another dashboard. We needed a narrow script that flagged incomplete lessons before they reached the printer queue.

Stay in love with God kept the scope narrow enough that one person could maintain it. We stopped adding features once the agent performed one reliable task on our own data. Everything else waited until that task ran without intervention for a full quarter.

Where proprietary data actually lives in small teams

The useful records were never in the main database. They sat in the shared drive folder labeled “old volunteer notes,” the email threads between regional coordinators, and the margin comments on the print PDFs that came back from churches. These fragments contained the real signals about which lessons worked and which ones created extra work.

We mapped every place a decision was recorded outside the official system. The map was shorter than expected—fourteen locations total. Most were simple text files or forwarded messages. None of them matched the clean tables the commercial agents required.

Once we accepted that our data lived in those places, the build decision became mechanical. We wrote a small script that pulled from those fourteen sources, normalized the dates, and wrote a single daily summary into the existing volunteer dashboard. No new interface. No new login.

How one narrow internal agent changed our release rhythm

The first agent we kept simply checked whether every lesson in the upcoming release had a completed volunteer test record. It ran every morning and posted a one-line status in the team channel. If a lesson was missing its record, the line included the name of the last person who touched the file.

That single check removed the weekly release meeting that had been scheduled for thirty minutes every Monday. The meeting still happened, but it now started with the agent output already on screen and ended in twelve minutes instead of running long. The saved time went into testing one additional lesson per cycle.

After eight weeks the team began adding one more rule at a time. Each addition stayed inside the same narrow lane: read only from the places we already controlled, write only to the places the volunteer already looked. The pattern kept the agent maintainable by the same two people who handled content updates.

Your Turn: Apply This Today

• Pick the single workflow your current tools ignore—volunteer lesson testing, print file handoff, or regional coordinator check-ins—and list every place the decision record currently lives.
• Write a five-line script that reads only from those locations, checks one condition, and writes the result to an existing channel or dashboard you already use.
• Run that script daily for two weeks and log the exact minutes it saves or the exact error it catches before anyone else notices.
• Refuse to add a second condition until the first one has run without manual fixes for a full release cycle.
• Document the fourteen or fewer data locations in a single shared note so the next person can maintain the script without asking you.
• Compare the time spent maintaining the script against the time previously spent in the meeting or process it replaced; stop if the ratio exceeds one to five.

The pattern in “The Data Moat That Actually Matters for Small Ministry Teams” and “The Question I Kept Answering About Internal Ministry Data” shows why the smallest internal agents outlast the broadest vendor promises. I consult with ministry product leaders and faith-tech teams on building narrow agents over their own data and protecting release rhythm when external tools fall short. Let’s talk.

I used to treat ambition as something that would surface on its own once the tooling got out of the way. In the early days of rolling out agent prototypes for curriculum routing and volunteer scheduling, I kept my focus on the interfaces and the handoff logic. I figured the people around the system would naturally step into bigger questions once the repetitive work shrank. That assumption cost us six months of flat output and two key volunteers who quietly disengaged because the work no longer asked anything hard of them.

The real price showed up in the numbers we actually tracked. Completion rates stayed high, but the rate at which new ideas for ministry outcomes reached the backlog dropped by half. People stopped proposing changes to how the agents should decide what counted as a finished lesson plan or a successful follow-up sequence. They treated the agents as finished tools rather than shared instruments that still needed direction.

This is the foundational misread that causes product teams to hand execution to agents while accidentally shrinking the scope of everyone else. John Wesley’s first rule—do no harm—offers a sharper lens than most modern frameworks for agent design. Wesley meant the rule as a daily discipline, not a slogan. Applied to product work, it asks whether every handoff preserves or reduces the capacity of the people who remain in the loop.

Wesley’s rule of doing no harm applied to agent handoffs

When an agent takes over generating initial sermon outlines or children’s ministry activity sheets, the handoff decision is not neutral. If the prompt and acceptance criteria are written only by the product manager, the volunteer or staff member downstream loses the chance to define what “good enough” actually means for their context. Over time that loss registers as reduced ownership even if the output quality holds steady.

I watched this happen on a volunteer routing project at Sermons4Kids. The agent suggested assignments based on past completion data, and the handoff looked clean in the dashboard. Two months later the volunteer coordinators stopped suggesting adjustments to the rules. They assumed the system already knew what mattered. The harm was not in the suggestions themselves but in the quiet removal of their judgment from the definition of success.

Wesley’s rule forces a different question at the point of handoff: does this transfer of execution also transfer the authority to revise the criteria? If the answer is no, the design is doing harm to the ambition already present in the team.

Raising ambition inside volunteer and staff teams that now steer agents

Most ministry teams already contain people who think in outcomes rather than tasks. The shift to agents does not create that capacity; it surfaces whether the product work protects it or crowds it out. When a staff member can now edit the agent’s rules for what counts as a completed follow-up sequence, their attention moves from filling the queue to shaping what the queue should serve.

One children’s ministry director I worked with began requiring every agent-generated lesson plan to carry a short note on the single outcome the volunteer would measure in the room. The note took thirty seconds to write but forced the director to decide what mattered before the agent ran. Within three weeks her team was proposing new outcome categories the original prompts had never included.

The pattern holds across both volunteer and paid teams. Ambition grows when the product surface makes the steering decision visible and cheap to change. It shrinks when the same surface treats the agent’s defaults as settled.

The weekly practice that keeps ambition from collapsing into ticket triage

The practical discipline is a standing thirty-minute review that separates execution review from ambition review. In the first ten minutes the team looks only at what the agents completed. In the next twenty they examine one steering decision that was made or deferred the previous week and ask what would change if that decision moved one level higher in the organization.

The review stays small on purpose. It never expands into backlog grooming or metric dashboards. Its sole output is a single revised rule or outcome definition that one person will carry into the next cycle. Over time the practice keeps the team from treating agent output as the main product and their own judgment as optional commentary.

I have seen this rhythm surface decisions that would otherwise stay buried inside prompt engineering tickets. A volunteer coordinator recently used the slot to question whether the agent should optimize for lesson completion or for the number of children who asked a follow-up question after the lesson. The change moved from a minor prompt tweak to a measurable shift in how the team defined retention.

Your Turn: Apply This Today

• Pick one agent workflow you own and schedule a twenty-minute conversation this week with the person who currently only approves its output; give them the explicit task of rewriting the success rule before the next run.
• In your next 1:1, replace one execution assignment with a steering question: ask the team member to name the single outcome the agent should optimize for on their behalf and write it into the prompt themselves.
• Block thirty minutes on Friday to run the ambition review described above with one volunteer or staff member who already uses an agent you built.
• Take the most recent agent handoff you designed and surface the current success criteria to the end user in plain language so they can edit it before the next cycle.
• Identify one person whose role has shrunk to ticket triage since agents arrived and hand them authority to change one rule that affects at least three future runs.
• Document the change they make and bring it to the next weekly review so the team sees that steering decisions now belong to them, not only to the product manager.

The same pattern appears in The PM Who Inherited the Agent Schedule and The PM Role That Still Needs Human Judgment After Agents Ship. Both posts trace what happens when teams treat execution relief as the end of the work rather than the start of a new set of steering questions.

I consult with ministry product leaders on agent handoff design, weekly practices that protect team ambition, and the product decisions that keep execution relief from shrinking scope. Let’s talk.