I spent two years delegating judgment to the model, treating its output as the default first pass on every feature spec. The pattern has a name: automation bias, the habit of trusting a confident machine over your own checks. The assumption was simple: if the language model could surface the obvious paths and edge cases faster than my own notes, then my real work started after the first draft. That habit cost me the ability to see when a proposed flow would break for the exact users who had no one else to call.
The first time it showed up was on a children’s ministry scheduling tool. The model produced a clean volunteer assignment flow that looked efficient on paper. I shipped the review version without walking through the Sunday morning handoff myself. Two weeks later three small churches reported that the print step still required a second person to stand at the copier because the model had optimized for digital completion rates that did not exist in their rooms.
This is the foundational misread that causes product teams to treat judgment as an automatic output of scale rather than a muscle that must be exercised against real constraints. John Wesley’s three rules—do no harm, do good, attend to the means of grace—were never meant as slogans. They were weekly disciplines that forced leaders to examine whether their daily habits actually produced the character the movement needed when pressure hit.
The Rule That Reversed My Automation Bias
Wesley required class leaders to ask the same three questions of every participant each week. The questions were not abstract. They required concrete answers about money, speech, and actions that week. The discipline worked because it refused to let good intentions stand in for observable behavior.
Most agent loops in product work skip this step. The model generates a revised workflow, the PM accepts or tweaks it, and the next sprint begins. There is no recurring checkpoint that asks whether the change still does no harm to the volunteer who prints at 9:15 on Saturday night. Over months the absence of that checkpoint trains the team to accept model fluency as evidence of soundness.
The practical result shows up in error logs rather than user interviews. When the model hallucinates a step that never existed in the original paper process, the team has already lost the habit of catching it before release because nothing in the weekly rhythm forced them to rehearse the failure case.
Where Automation Bias Hides the Real Decision
Agent loops are built to optimize for task completion. They surface the next action, execute it, and report success. The hidden decision is almost never inside the loop; it sits one layer before the loop is allowed to start. That layer is the judgment about whether the task itself is worth automating for this specific user group.
On the curriculum side we once let an agent propose new lesson outlines based on engagement data. The loop ran cleanly and produced outlines that scored higher on predicted completion. Only after launch did we notice the outlines removed every hands-on activity that required a table or floor space—precisely the activities small churches without dedicated rooms relied on. The model had optimized for the metric we fed it; we had lost the weekly discipline of checking whether that metric still served the users who could not change their room.
The decision point was never inside the agent. It was the prior choice to let the agent define which activities counted as valuable. Wesley’s first rule would have required us to ask whether the change risked harm before the loop was permitted to run at all.
The Second-Order Cost When Small Churches Copy Enterprise Defaults
Psychologists describe a related effect as automation bias: people over-trust a confident machine and stop checking its work.
Enterprise teams can absorb model errors because they have staff who can override the system the next day. Small churches have one volunteer who prints on Thursday and no second person to catch the mistake on Sunday. When those churches adopt the same agent-generated defaults, the cost is not lower efficiency; it is visible failure in front of families who already feel marginal.
The pattern repeats across tools. A scheduling agent suggests automatic reminder timing based on large-church data. The small-church volunteer receives the reminder on the only evening they have child care and deletes the app. The model records lower engagement and suggests even more reminders. The loop compounds the original misread.
Wesley’s third rule—attend to the means of grace—translated into product terms means building recurring contact with the actual environment where the tool is used. Without that contact the inner skill of noticing when an optimization has become harmful atrophies, and the model is left to train on its own outputs.
Your Turn: Apply This Today
- Pick one agent-generated recommendation from this week’s work and trace it backward to the single assumption the model was allowed to make without your override.
- Log the date, the assumption, and the one-sentence reason you accepted it; keep the log in the same place you keep sprint notes so it cannot be ignored.
- Once a week, before the next agent loop runs, state out loud or in writing the exact failure that would constitute harm for the smallest user segment you support.
- Require that statement to be reviewed by one person who has used the tool in a room with no dedicated tech support.
- At the end of the month, count how many logged overrides changed the shipped version; treat a count below two as evidence the discipline has weakened.
- Schedule the next review on the calendar now rather than when the next agent output arrives.
The same pattern appears in the niche tools that still need a human print step and in the guardrail conversations that skip the question of whose judgment is actually being replaced.
I consult with product leaders and ministry tool teams on agent loop design, weekly override disciplines, and the second-order costs small-church users absorb when enterprise defaults ship unchanged. Let’s talk.