I still remember the night the curriculum agent died. A volunteer in our kids ministry pulled it up on her phone in the church basement, stared at the clean, tested output, then set the phone down and reached for the dog-eared binder she’d used for three years. Nothing in our sandbox had prepared us for that moment.
Three different pilots later, the pattern was the same. The agent handled the hard cases. The metrics looked strong. But the handoff to real calendars and real people never cleared the gap between what we built and where ministry actually happens.
The bias wasn’t in the model. It was in the quiet belief that if it worked inside our walls, it would find its way to theirs.
Where the quality metric hid the handoff failure
The first pilot scored 96 percent on accuracy for lesson-plan generation. Volunteers tested the agent in a two-week lab and finished every task. We declared success and moved to live use. What the metric never captured was the moment a volunteer had to print the output, carry it across a church parking lot, and explain it to a substitute teacher who had never seen the system.
That single transfer broke the chain. The substitute opened the printout, saw unfamiliar formatting, and defaulted to the paper curriculum already in the bin. Completion rate dropped from 94 percent in the lab to 11 percent on the actual Wednesday night. The quality score stayed untouched because we never measured the transfer itself.
The second pilot used an agent that summarized weekly sermon content into small-group questions. Internal review praised the theological tone and length. The failure point arrived when the pastor needed to route those questions to a volunteer coordinator who only checked email once a week. The coordinator never opened the file. Again the internal metric offered no warning because the handoff lived outside the product surface.
Both cases followed the same pattern: we treated the agent as a self-contained deliverable. Munger’s latticework would have required us to model the actual sequence of human movements before claiming readiness.
How distribution actually worked in the three pilots
Distribution in these settings rarely runs through app stores or dashboards. It runs through people who already carry paper, text threads, and verbal briefings on Sunday morning. The curriculum agent never reached its intended users because the only person who knew how to trigger it sat in the children’s director’s office and left at 8 p.m. on Wednesdays. No one else had both the login and the physical print step in their normal routine.
The sermon-prep pilot collapsed for a different distribution reason. Pastors liked the output when they tested it alone, but the small-group leaders who actually taught the material received it only if the pastor remembered to forward an email. That single extra step had no owner and no reminder system. The pilot recorded high usage from the pastor and zero usage from the people who needed the content most.
The third pilot required a laptop connected to a specific projector model that existed in only one classroom. Volunteers scheduled for other rooms could not run the agent at all. The distribution constraint was physical equipment, not software quality. Once we mapped the actual rooms and devices in use, the 95-percent accuracy number became irrelevant.
Each failure traced back to an unmodeled distribution path rather than a defect in the agent. The latticework approach would have flagged these paths before any code was written.
The model I now run before any new agent ships
I now force three additional models alongside the quality model before shipping. The first is a literal map of every human who must touch the output between generation and use. The second is a timing model that records when each of those humans is actually available. The third is an incentive model that asks what each person gains or loses by completing their step.
Running the map on a new agent takes less than an hour but changes the definition of done. If the map shows more than two handoffs or a timing gap longer than forty-eight hours, the pilot does not ship until the handoff is either removed or instrumented. The incentive check often reveals that one volunteer gains nothing from the extra step; that person becomes the redesign target, not the agent itself.
This process has already stopped two agents that would have repeated the earlier pattern. One required a second volunteer to re-enter data that the first volunteer had already typed. The map made the duplication visible. The other needed a Sunday-morning printer that no volunteer was willing to operate; the incentive model showed zero benefit for that role, so the feature was cut before development began.
The latticework does not slow delivery. It prevents the expensive restart that follows every silent failure on Sunday.
Your Turn: Apply This Today
- Draw the exact sequence of people who must receive output from your current AI workflow and mark the last person who actually uses it on Sunday.
- Time-stamp each handoff in that sequence against the real availability windows of the people involved, not the ideal schedule.
- Identify the single handoff with the longest gap and either automate it or assign an explicit owner this week.
- Run the incentive check on the last person in the chain: list what they gain or lose by completing their step and adjust the workflow if the gain is zero.
- Remove any step that requires equipment or access only one person possesses; redesign or drop the feature if removal is impossible.
- Re-test the revised workflow with the actual volunteers or pastors who will run it, not the internal team, before declaring the pilot ready.
The same distribution blind spot appears in “The Trust Step Kent Beck Made Non-Negotiable” and “Why Top PMs Still Route Specs Through Sunday Validation.”
I consult with faith-tech product leaders on distribution audits, pilot handoff design, and latticework application to AI workflows. Let’s talk.