Blog

I kept a running note in my task list for most of 2023 that read “evaluate AI tooling after Q3 metrics stabilize.” The note sat there untouched while two other teams inside the same organization shipped their first working prototypes in June. By the time our group finally opened the same tools, the core questions about user flow and content priority had already been answered without us. The real loss was not speed. It was that we no longer held the right to decide which problems counted.

Solomon’s judgment offers the clearest picture of what happened. The king did not ask which woman deserved the child. He proposed cutting the child in half and watched who refused the division. The test revealed ownership by who would rather lose the outcome than accept a broken version of it. The same test now applies to AI tooling. Teams that treated the tooling as optional were willing to let the product be split between “human work” and “machine work.” Teams that refused the split ended up shaping the actual direction.

That refusal is what I missed for two full years.

The decision point I missed while watching other teams ship

The first clear signal came during a children’s curriculum update cycle. One small group started feeding draft lesson outlines into an early model and then editing the output in under an hour. They kept the same volunteer-facing print layout we had always used, but the revision loop moved from three days to one. I watched the change in their completion rate numbers and filed it under “interesting experiment.”

Six months later their version of the flow became the default template for every new release. The change was not announced as a strategy shift. It simply arrived as the baseline. Our own team was still running the old three-day cycle and defending it with notes about quality control. By the time we tried to match their speed, the template decisions had already been locked in by the group that had treated the model as part of the same desk.

The pattern repeated on a larger content platform. A three-person pod began routing incoming support tickets through a lightweight classification model before any human read them. They did not wait for an official integration project. They just stopped routing every ticket the old way. Within eight weeks their backlog showed a different shape: fewer generic “how do I find this passage” tickets and more specific questions about print formatting for small-group leaders. The rest of us kept triaging the old way and wondered why our metrics looked unchanged.

How the model outran the person still signing off

Solomon’s test works because it forces the question of who will accept a divided outcome. In product work the divided outcome looks like one person signing off on requirements while another person runs the model that actually generates the first draft. The signer keeps the title. The model runner keeps the working version. Over time the working version drifts from the signed requirements because the model learns faster than the approval cycle.

I saw this most clearly on a volunteer resource project. The person responsible for final approval still required every new children’s lesson to pass a manual review checklist. Meanwhile a neighboring team had the model generate three variations of the same lesson, ran them with five real volunteers, and kept only the version that produced the highest completion rate. The approved checklist never changed. The actual lessons that reached volunteers did change, and they came from the team that refused to keep the work split.

The cost shows up in ownership. When the model produces the first usable artifact, the person who still signs the old document no longer controls which problems get solved. They only ratify a direction someone else has already tested.

The small teams that gained leverage by treating it as infrastructure early

The teams that moved earliest did not treat AI as a research project. They treated it as part of the same infrastructure that already included version control and the print export script. One ministry pod added a single rule to their weekly planning: every new feature ticket had to include one model-generated variant before the first human edit. The rule took fifteen minutes to write and two weeks to normalize. After that the discussion in planning meetings shifted from “what should we build” to “which of these three variants survived real use.”

Another group applied the same approach to search relevance on a large Bible platform. They stopped waiting for the next quarterly model update and instead ran small daily experiments on query classification. The experiments were not announced as AI work. They were just part of how the team kept the search results usable for users who arrived with a single verse reference and no context. Within four months the relevance numbers moved more than they had in the previous year of larger, slower projects.

These teams did not have more budget or more headcount. They simply stopped accepting the split between the tooling and the rest of the work.

Your Turn: Apply This Today

• Take the next feature ticket on your backlog and generate three model variations of the user flow before the first planning discussion.
• Run one of those variations with three actual users this week and keep only the version that produces a measurable completion improvement.
• Change the definition of “ready for review” on your team to require at least one model-assisted draft already tested.
• Pick the single metric you report most often and re-run last month’s numbers using an AI-assisted classification of the underlying tickets or lessons.
• Block thirty minutes on Friday to delete any internal process step that exists only to move work between a human and a model without adding user value.
• Write down the next approval meeting on your calendar and require the meeting owner to bring one live model output instead of a slide deck.

The teams that kept the tooling optional for another year handed the real decisions to groups that refused the split. Two of those groups now set the baseline for the products I work on most often. The posts “The Agent That Kept Running the Schedule While No One Was Watching” and “Faster Code Does Not Move the Real Ministry Bottleneck” trace what happened next in both cases.

I consult with product leaders and ministry technology teams on AI tooling integration, decision ownership, and treating models as core infrastructure. Let’s talk.

The coordinator’s phone lit up on the kitchen counter just after eleven forty. A single vibration, then the screen filled with the thread. Three automated reminders had gone out that evening to the family whose six-year-old was admitted two days earlier. The mother’s reply sat at the bottom, short and final: please stop texting us.

I stood there while the coordinator read it aloud. The room stayed quiet except for the low hum of the laptop still open on the table. The scheduler agent had done exactly what it was built to do. It checked the list, saw the names, and followed the cadence rule without pausing for any new information.

That single exchange exposed the gap. Automation treated the schedule as a fixed list of tasks. It had no way to register that a family had moved from routine attendance to crisis. The agent kept operating on yesterday’s data because nothing in its workflow required a human to confirm the context still held.

John Wesley’s first rule, do no harm, was meant for people making decisions under pressure. It forces a pause before action when the outcome might wound someone already hurting. The same logic applies to agents that touch real lives. Without an explicit checkpoint, the system defaults to motion instead of mercy.

The missing rule shows up whenever an agent crosses from logistics into personal contact. Reminders, follow-ups, and status updates all carry relational weight in ministry settings. The agent cannot weigh that weight on its own.

The missing rule that turns helpful automation into intrusion

Wesley’s rule does not ask for perfect foresight. It asks whether the next action risks harm before it is taken. An agent that sends messages without fresh confirmation has no mechanism to apply that test.

In practice this means every automated outreach that touches a person’s current life situation must stop at a human gate. The gate does not need to be complex. It only needs to surface the proposed action and the current known context so a coordinator can say yes or no in under thirty seconds.

How oversight checkpoints actually look in live ministry systems

One team built a simple queue that surfaces every scheduled message twenty-four hours before it leaves. The coordinator sees the recipient name, the message text, and any recent notes attached to that household. A single click approves or holds. Messages that sit unapproved for the full window are dropped rather than sent.

Another group added a context tag that travels with each record. When a hospitalization flag is added, the agent automatically routes every future reminder into the same review queue and marks it high priority for the coordinator. The tag stays until it is manually cleared. No additional rules engine was required.

These checkpoints mirror how volunteer workflows already function. A children’s ministry director does not hand a new volunteer the full roster without a quick check-in. The same pattern now applies to the agent.

The cost of catching the problem only after it reaches the end user

When the message lands with the family first, the coordinator loses the ability to repair the relationship on their own terms. The damage is already public inside that household. Trust erodes faster than any dashboard metric can capture.

Teams that discover the error only through complaints spend the next weeks rebuilding credibility one conversation at a time. The original time saved by the agent disappears inside those recovery hours. Retention among both volunteers and families drops when the system feels careless rather than attentive.

The fix is never retroactive. Once the message is sent, the only remaining option is apology.

Your Turn: Apply This Today

• Pick the next agent workflow you plan to ship and list every point where it would contact a person without fresh approval.
• Add one explicit review gate at the first contact point and make the queue visible only to the actual coordinator who owns that list.
• Set the default action to hold rather than send when the gate is not cleared within the defined window.
• Run the workflow against last month’s real schedule data and walk through each queued item with the coordinator in a single fifteen-minute session.
• Log every override or hold decision for two weeks and note which context tags would have prevented the original error.
• Remove the gate only after the coordinator confirms it adds no measurable delay to their normal morning routine.

The same oversight pattern appears in both The Permission Gate That Kept the Agent From Running the Schedule and The Context Window That Only Closed When a Real Coordinator Sat Down.

I consult with ministry technology leaders on autonomous agent design and human oversight protocols in scheduling systems. Let’s talk.

Someone recently asked me why an AI-assisted refactor that produced working code in two days still sat untouched in review for seven more. The answer is straightforward: the generation step solved the smallest slice of the problem. The remaining work—confirming the change against real usage, routing the right approvals, and updating the surrounding systems—still moves at human speed.

This is the foundational misread that causes product teams to celebrate velocity they cannot actually ship. When the diff appears quickly, teams assume the bottleneck has moved upstream into creation. In practice the constraint simply relocates to verification and coordination.

Teresa Torres’s continuous discovery framework makes the distinction clear. Discovery is not a phase that ends when code is written; it is the ongoing work of checking whether the proposed change still matches user needs, risk tolerance, and operational reality. AI accelerates the proposal. It does nothing to shorten the loop that tests whether the proposal holds up.

Why the generated diff is only the first 20 percent of the work

The two-day refactor touched a payment reconciliation module used by volunteer coordinators who run monthly reporting for small churches. The model produced clean, idiomatic updates to the query logic and added the missing null checks. On the surface the task looked finished.

What the model could not do was run the change against the last six months of actual reconciliation exports those coordinators keep in shared drives. Those exports contain the edge cases—duplicate donor IDs, currency conversions entered as text—that never made it into the test suite. Someone still had to pull the files, replay them, and confirm nothing broke before the change could move forward.

Continuous discovery treats this verification as first-class product work, not cleanup. The team that shipped the change eventually spent three additional days constructing a one-off script to replay the real exports. That script became the permanent regression check the original refactor had omitted.

The coordination tax that appears once code moves faster than review

Once the diff existed, four people needed to sign off: the staff engineer who owned the billing service, the ministry operations lead who understood how coordinators actually used the reports, the volunteer who maintained the print templates, and the part-time contractor who handled month-end data hygiene. None of them worked on the same calendar.

The engineer reviewed the same afternoon. The operations lead asked for a walkthrough two days later because she was traveling to a regional training. The volunteer needed the change described in a short Loom because he only checked email on Tuesdays. The contractor flagged a downstream field mapping that would affect his monthly cleanup script. Each handoff added latency that no amount of generation speed could compress.

Torres would call this the discovery loop becoming visible. The faster the code arrives, the more obvious it becomes that the organization has no standing rhythm for these conversations. Teams that treat verification as an afterthought simply accumulate calendar debt.

How ministry product teams already handle this without formal methods

Ministry teams have long worked with volunteers whose availability is measured in hours per week rather than hours per day. They already route changes through a short, explicit checklist: does the update still produce the printed report the volunteer needs, does it preserve the export the bookkeeper downloads, and does the person who will run the next cycle know the change happened.

These checklists are continuous discovery in practice. They force the proposer to name the exact artifacts and people required before any code is considered ready. When an AI refactor lands, the same checklist simply gets applied to the new diff instead of being skipped because the code “looks done.”

The teams that absorbed the two-day refactor most cleanly already kept a one-page living document titled “What must still work after any change to reporting.” Updating that document became the first verification step rather than the last. The remaining sign-offs then moved in days instead of weeks because the criteria were already written down.

Your Turn: Apply This Today

• Pick one small refactor you have been considering for an internal tool and run it through an AI coding assistant this week; commit only the generation step and nothing else.
• Immediately after the diff appears, list every person and artifact that must be checked before the change can ship; write the list in the pull request description before requesting any review.
• Schedule the first verification conversation within 24 hours of opening the pull request, even if the reviewer has not yet looked at the code; treat the meeting as discovery, not approval theater.
• Document the exact steps you took to replay real data or usage against the change; turn those steps into a reusable check that lives alongside the test suite.
• Track the elapsed time between generation and final sign-off for this single change; note which steps consumed the majority of calendar days rather than active work hours.
• Share the timing breakdown with one other product person on your team so the next refactor starts with the verification path already mapped.

See also The Context Window That Only Closed When a Real Coordinator Sat Down and The Permission Gate That Kept the Agent From Running the Schedule.

I consult with ministry product leaders and faith-tech PMs on AI refactor verification, continuous discovery loops, and coordination patterns that keep small teams shipping. Let’s talk.