Blog

Posted on

The Trust Step Kent Beck Made Non-Negotiable

Programming code on a laptop representing an expand-phase trust step

The cursor blinked after the paste command. The ops lead checked the time—9:47 p.m.—and closed the laptop. The roster looked balanced on screen. Two families would discover the overlap only when both showed up for the same classroom slot three mornings later.

The volunteer scheduler accepted the list without protest. No duplicate flag fired. The system treated the output as finished work.

Kent Beck’s explore-expand-extract cycle makes the middle step explicit: after you explore possibilities and before you extract a clean deliverable, you must expand the work into every handoff it will touch. In agent workflows the expand phase is where trust checks belong. Skip it and the extraction step simply hides the failure until a real person collides with it.

The Expand Phase That Skipped the Handoff Check

The roster generator had explored dozens of volunteer combinations and scored them for availability and past attendance. That was the explore step. The expand step never happened. No one asked the model to walk through the physical handoff: which key opens which door, which parent needs the extra car seat reminder, which two families cannot share a room because their kids have a documented conflict.

Ministry ops leads already know these constraints live in paper notebooks and whispered hallway conversations. An agent that stops at token-level coherence never sees them. The expand phase forces the model to surface those constraints as structured checks rather than hoping the final list will survive contact with reality.

Teams that treat expand as optional end up with clean JSON and broken Sunday mornings. The double-booked classroom is not an edge case; it is the predictable result of extracting before the handoff logic has been tested.

What Extract Deletes When No One Prints

Once the list reached the scheduling tool, every intermediate reasoning trace disappeared. The extraction step keeps only the final assignments. The reasoning that two families both claimed the blue room at 8:15 a.m. was discarded along with the timestamp of the generation run.

Print-first teams still catch these collisions because the printed sheet carries the implicit history of who wrote it and when. Digital extraction removes that trace. Without an explicit expand-phase log, the only remaining signal is the complaint that arrives Sunday morning.

The cost is not just the scheduling error. It is the erosion of trust between the volunteer who accepted the assignment and the system that generated it. People stop believing the tool when the first visible output contradicts what they know from lived experience.

The Logging Rule That Caught the Double Booking

One team added a single rule inside the expand phase: every generated roster must emit a line-item log of classroom, time, lead volunteer, and backup volunteer, then require a human to confirm no two line items share the same classroom-time pair. The log lives in the same repository as the final schedule.

The rule surfaced the collision before the roster ever reached volunteers. The fix took four minutes of editing instead of an 8:15 a.m. confrontation at the classroom door. The logging step cost almost nothing because it ran during expand rather than after extraction.

Kent Beck’s framework does not promise fewer errors. It promises that errors become visible while the work is still cheap to correct. Ministry contexts simply make the cost of late discovery more obvious than most product teams are willing to measure.

Your Turn: Apply This Today

  • Open the prompt that generates your next volunteer or staff roster and insert an explicit expand instruction: “List every classroom-time pair and flag any duplicate before producing the final JSON.”
  • Add a one-line logging step that writes each assignment plus its generation timestamp to a shared file the ops lead can open on a phone at 9 p.m.
  • Require the human scheduler to reply to the log with a single confirmed or rejected status before the agent is allowed to send any notifications.
  • Run the same expand-plus-log pattern on the next three agent workflows you ship, even if they have nothing to do with scheduling.
  • Measure the time between generation and first human correction for each workflow; target a reduction from days to minutes.
  • Document the single constraint that most often breaks in your context (room conflicts, skill mismatches, family overlaps) and make that constraint a required check inside every expand phase.

The Prioritization That Quietly Dropped Security Reviews showed what happens when extraction speed replaces handoff checks. The Year I Optimized for Tokens Instead of Presence traced the same pattern in a different product surface.

I consult with product leaders and ministry operations teams on agent workflow design, trust logging during the expand phase, and handoff verification that survives real-world use. Let’s talk.

Posted on

Why Top PMs Still Route Specs Through Sunday Validation

A hand marking items on a checklist, representing Sunday validation

Why do you still force new agent specs through a Sunday validation round even after the internal evals clear and the product team signs off? Because the evals stop at simulated correctness while the real ministry workflow breaks at the point where a volunteer prints the output, hands it to a child, and watches what actually happens in the room.

Jensen Huang’s argument for sovereign AI was never mainly about chips or latency. It was about refusing to outsource the final decision layer that determines whether the system serves the intended outcome. When product teams treat their own evals as sufficient, they hand that final layer to an abstraction that has never met a seven-minute volunteer or a printer jam on a Sunday morning.

This is the foundational misread that causes product teams to ship agent changes that look strong in staging and still generate downstream rework every time the output reaches actual ministry use.

The Eval That Passed Until the Print Step

A children’s ministry agent generated small-group discussion questions that scored 92 percent on factual alignment and readability metrics. The output cleared the internal review in under an hour. Three days later the volunteer printed the sheet, discovered the final question required a whiteboard the classroom did not have, and spent the entire session improvising.

The eval never tested whether the output survived the print-and-carry step. It measured token quality against a rubric that assumed ideal conditions. Real conditions include missing markers, mixed age groups, and a volunteer who will not rewrite the material at 8:15 a.m.

Teams that skip the Sunday check treat the print step as someone else’s problem. The downstream effect is the same every time: the volunteer quietly stops using the agent and reverts to whatever worksheet they already trust.

Sovereign Control Means Owning the Handoff Metric

Huang’s point was that control is meaningless if the last mile still belongs to another system. In faith-tech products the last mile is almost always physical or relational. A clean JSON response does not guarantee a clean handoff to a printed page or a live conversation.

The teams that keep sovereignty treat the handoff metric as non-negotiable. They measure whether the agent output can be used by the actual person in the actual room without additional translation. That measurement only happens when the output is placed in front of a real volunteer on a real Sunday.

Without that step, product leaders end up arbitrating taste after the fact instead of preventing the mismatch before the merge. The cost shows up in support tickets, lost weekly active users, and quiet erosion of trust among the very leaders the product claims to serve.

One Override Protocol That Survived Three Agent Updates

One product team embedded a single rule: no agent change reaches production until the most-used output of the prior week has been tested in one live ministry setting. The test is scheduled for Sunday, documented by the PM or a designated reviewer, and required before any merge that touches the prompt or the retrieval layer.

The protocol has survived three separate model upgrades because the validation target is the workflow, not the model version. When a new agent produced better-sounding small-group questions but required two extra minutes of setup, the Sunday test caught it immediately. The change was rolled back before users ever saw it.

The override is deliberately lightweight. It does not require a full user study. It only requires that one actual instance of the output travels the full distance from agent to printed page to child’s hands. That single trip has blocked more silent failures than any automated eval the team has run.

Your Turn: Apply This Today

  • Pick the single agent output used most often last week and schedule a Sunday morning test with one volunteer who normally uses it; print the output exactly as the agent produces it and observe the first ten minutes of the session.
  • Document the exact moment the output required extra work from the volunteer and log the time delta in minutes before the next sprint planning meeting.
  • Write the handoff failure as a ticket titled “Sunday validation defect” and assign it to the PM who owns the agent rather than to support.
  • Block the next production merge that touches the same agent until the Sunday validation ticket is marked resolved with evidence from an actual session.
  • Repeat the same test the following Sunday on whatever output became the new most-used item and compare the two handoff times.
  • Share the two time deltas with the engineering lead before the next roadmap review so the cost of skipped validation appears in the same document as feature estimates.

Teams that adopted similar constraints appear in both “Letter to the PM Who Now Arbitrates Taste” and “The Prioritization That Quietly Dropped Security Reviews.” The pattern is the same: the teams that kept control were the ones that refused to treat the last human step as someone else’s problem.

I consult with product leaders and ministry technology teams on agent evaluation design, workflow handoff measurement, and Sunday validation protocols. Let’s talk.

Posted on

The Open-Weight Number That Still Required a Human Gate

Abstract sphere of dots and lines representing open-weight AI models

A ministry product team reported an 8x reduction in inference costs after switching their sermon outline generator to an open-weight model. The finance review celebrated the move. The obvious reading was that cheaper tokens had solved the scaling problem for good.

The numbers told the opposite story. The cost drop masked a new choke point: every ministry-facing paragraph still passed through an unowned routing decision that determined which model handled which request and which outputs reached volunteers. The savings arrived, but the human gate did not disappear.

This is the foundational misread that causes product teams to treat inference price as the only variable that matters. Once the model itself becomes interchangeable, the latticework Charlie Munger described becomes the required tool. You stop asking which model is cheapest and start asking which mental models must sit on top of the routing layer so the output stays usable in a real church setting.

The Cost Drop That Hid the Routing Decision

The team had measured success by tokens per dollar. After the switch, that metric improved dramatically. What they had not measured was the set of rules that now decided, for any given volunteer request, whether the cheap model received the query or whether it routed to a heavier model with better guardrails.

Without a named owner, those rules lived in a shared document that no one updated after the first month. A children’s ministry outline about baptism ended up in the low-cost path and produced text that assumed the volunteer could explain credobaptism versus paedobaptism in a single paragraph. The volunteer printed it anyway. The routing decision had already been made by default.

Munger’s latticework requires holding the cost model and the ownership model at the same time. Price tells you nothing about who carries responsibility when the output reaches a 7-minute volunteer who has never seen the source material before. The cheaper model simply made the ownership gap cheaper to ignore.

When Full Control Still Leaves the Print Step Exposed

One engineering lead argued that open weights gave them full control. They could inspect weights, change system prompts, and run the model locally. That control existed on the server side. It did not extend to the moment a volunteer hit print on a physical lesson sheet.

The SermonCentral team learned this years earlier with curriculum that reached hundreds of thousands of volunteers. The print step exposed every assumption the product team had made about reading level, cultural reference, and time required. No amount of model control fixed the fact that the routing decision upstream had already chosen which version of the content reached the printer.

The latticework here pairs the control model with the distribution model. You can own the weights and still lose ownership the moment the output leaves the product and enters someone else’s workflow. The exposed print step is the signal that the routing decision was never assigned to a human in the first place.

The One Decision Rule That Survived the Model Swap

Across multiple model changes, one rule remained effective: every ministry-facing output must carry a named owner who can reverse the routing decision within one business day. The rule does not require the owner to write the content. It requires the owner to decide, after the fact, whether the cheap path was the correct path.

Teams that kept this rule also kept a simple reversal log. When a volunteer flagged an outline that felt off, the owner could see which model had handled the request and whether the routing parameters needed adjustment. Teams that dropped the rule after the cost dropped found themselves explaining the same quality issues six months later at higher volume.

The latticework makes the tension visible. Cost accounting and ownership accounting are separate models. Treating the first as a substitute for the second produces exactly the outcome observed: cheaper inference that still requires a human gate on every output that matters.

Your Turn: Apply This Today

  • Name the single product owner who can reverse any routing decision on your lowest-cost model by Friday; put their name in the runbook and notify the on-call rotation.
  • Build a one-line reversal log that records model used, request type, and whether the output reached a volunteer; require the owner to review the previous week’s entries every Monday morning.
  • Pick the three most common request categories this week and hard-code a routing preference for each that the owner can change without an engineering ticket.
  • Run a 48-hour test where every ministry-facing output from the lowest-cost model is held for owner sign-off before it reaches the print or export step.
  • Document the exact condition under which the owner would force a heavier model instead; publish that condition to the volunteer feedback channel so they know when to escalate.
  • Schedule a 30-minute calendar block next week for the owner to compare token cost against reversal frequency and decide whether the current routing rule still holds.

The 8x Number That Never Accounted for Inference Routing showed how teams celebrated savings while the real constraint moved. The PM Role That Stopped Shipping and Started Choosing described what happens when that constraint finally receives an owner.

I consult with product leaders building ministry tools on inference routing ownership, reversal processes, and output quality for volunteer workflows. Let’s talk.

View all posts