Blog

I still have the screenshot from that Tuesday. A children’s ministry coordinator in Ohio had forwarded the same signup list three times because the agent kept dropping the prior week’s commitments. No error message, just clean new suggestions that ignored everything we’d already confirmed.

I’d told the team the model would “figure it out” from recent messages. The prompt was tight, the activity feed was fresh, and we shipped. The first token bill that month was larger than the entire previous quarter combined, but the real hit was watching volunteers quietly stop trusting the tool.

That gap between what the model could see and what it needed to remember — the context layer no one had actually built — is where most of these agents quietly die.

Charlie Munger’s latticework idea helps here. He described the habit of building multiple mental models that cross-check one another instead of relying on a single narrow lens. In product terms, that means treating the context layer not as a single store but as overlapping layers—roster state, routing rules, and access history—that must remain legible to each other.

How Lost Roster State Shows Up in Real Scheduling

Take a typical Wednesday-night program that runs on thirty volunteers. The coordinator used to keep a shared spreadsheet with columns for last confirmed date, preferred role, and any notes from the previous handoff. When the agent replaced the sheet, those columns disappeared from the model’s working memory after each session.

Within two weeks the agent began assigning the same person to conflicting rooms because it had no record of the prior commitment. The coordinator spent extra hours every Monday correcting the output instead of preparing curriculum. The pattern is common: the agent optimizes for the current prompt while the team pays for the missing longitudinal view.

Teams that kept even a minimal state file alongside the agent saw fewer collisions. The file was not elegant, but it preserved the handoff chain the volunteers actually used to trust one another.

Dynamic Routing as the Practical Fix for Small Teams

Small ministry teams rarely have dedicated ops staff. Their routing decisions depend on who answered the last text thread and who owes a favor from the previous quarter. An agent that cannot surface that informal history forces the coordinator to reconstruct the social graph from scratch every week.

One team I observed added a lightweight routing table that logged every assignment with a timestamp and the name of the person who last updated it. The agent consulted the table before generating new slots. Conflicts dropped because the model now carried forward the same continuity the coordinator used to track mentally.

The table stayed small—under two hundred rows—and lived in the same database as the agent’s output. No new infrastructure was required, only the discipline to treat prior handoffs as first-class context rather than optional color.

The Context Layer Hiding Inside Your Security Review

Security reviews often push teams to strip identifiers and session history for compliance. In ministry settings that same stripping removes the exact chain of custody volunteers need when questions arise about who signed up whom.

A workable middle path logs the minimal set of prior actions required for continuity—date of last confirmation, role preference, and the name of the person who recorded it—while redacting anything outside the team’s agreed boundary. The security layer and the context layer become the same artifact instead of opposing constraints.

Teams that made this choice could still pass audits because the retained data stayed inside the organization and carried explicit expiration rules. The agent retained enough history to avoid re-asking the same availability questions every cycle.

Your Turn: Apply This Today

• Pull the last thirty days of scheduling output from one agent workflow and list every field that disappeared between sessions.
• Identify the single handoff data point volunteers mention most often in chat threads and add it to the agent’s context refresh this week.
• Write a five-row test file that replays the last three handoffs and run the agent against it before the next schedule generation.
• Tag each retained field with an expiration date so the security review can see exactly what history is kept and why.
• Share the updated context schema with the two coordinators who handle the most manual corrections and ask which row still feels missing.
• Measure the change in coordinator correction time over the next two scheduling cycles and keep only the fields that move the number.

The Token Cost Drop That Only Appeared After We Added the Knowledge Graph and The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer both trace the same pattern of missing longitudinal state.

I consult with product leaders building ministry agents on context layers, handoff preservation, and evaluation that respects volunteer history. Let’s talk.

A widely shared “Rapid AI Reorg” playbook making the rounds among ops and product leaders tells teams to redraw reporting lines around new agents inside thirty days. It assumes agents will surface the right priorities and leaders will shift straight into oversight. Advocates present this as the only way to capture the productivity gains without legacy drag.

That timeline ignores the actual work of defining acceptable output. Ministry product teams handle edge cases that sit outside any training set—curriculum that must land with a seven-minute volunteer, sermon resources that fit print-first constraints, and coverage that cannot slip on a Sunday morning. When the AI reorg happens first, the people who once carried that judgment get reassigned before anyone has built the checks that replace them.

This is the foundational misread that causes product teams to burn the judgment capacity they meant to protect. The three-week capacity log from the volunteer coordination team showed exactly where the hours moved once the new structure took hold.

The exact week the AI reorg plan collided with Sunday morning coverage gaps

The team had already moved two engineers into an “agent owner” role and flattened the rest of the group into a single pod. By Wednesday the curriculum queue showed three missing lessons for the following weekend. The engineers who used to spot those gaps during their normal review pass were now writing prompts instead.

No one had mapped the handoff. The old structure kept one person responsible for final volunteer readiness on Thursday afternoons. The new one left that check to whoever had the lightest prompt workload. Three churches reported blank pages in their packets on Saturday night.

The collision was not about the agents themselves. It was about removing the single point where a human still decided whether the output met the seven-minute test before it reached the printer.

What the capacity log revealed about diverted engineering hours

We ran the log on the volunteer coordination team for three weeks before any further changes. It captured every recurring task, the minutes spent, and who performed the judgment step. The data showed that a significant share of the week — nearly two full workdays — had shifted from direct coverage checks to prompt iteration and agent monitoring.

Those hours had previously caught most of the gaps that later reached Sunday morning. The agents handled the first draft well, but they still required the same human to decide whether a given lesson fit the age group and time limit. The reorg had moved that decision into an after-the-fact review instead of a built-in gate.

The log also tracked what happened to the diverted hours. Most went to debugging agent output that looked correct in isolation but failed when printed or handed to a volunteer with no technical background. The team had traded preventive judgment for corrective cleanup.

The narrower boundary we set instead of a full org redesign

We kept the existing reporting lines and added one new role: an evaluation owner who sat inside the current structure. That person owned the three-week capacity log, the definition of acceptable output, and the final sign-off before any agent output reached volunteers.

The boundary was simple. No reporting line changed until the evaluation layer produced consistently high pass rates on the same tasks the log had measured. The first agent stayed in a narrow lane—generating initial lesson outlines only—while the evaluation owner built the checks that would eventually let the team scale.

This kept the Sunday morning coverage intact while the team learned what the agents could and could not carry. The AI reorg stayed on the shelf until the log showed the judgment work had actually moved, not just disappeared from the original owners’ plates.

Your Turn: Apply This Today

• Choose the single team that will touch the next agent rollout and block two weeks on their calendar for a capacity log starting Monday.
• Have every member list every recurring task they perform that touches volunteer output or Sunday coverage, including the minutes and the judgment step involved.
• Track the same tasks for the two weeks after the agent ships and compare the hours that moved from direct checks to prompt work or output review.
• Define one measurable pass rate for the agent’s output on those tasks before any reporting line is allowed to change.
• Keep the existing structure until the log shows the original judgment hours have dropped substantially with no increase in coverage gaps.
• Document the exact handoff point where the evaluation owner must still sign off before any agent output reaches a volunteer.

The Evaluation Layer That Broke When We Trusted the Model Alone and The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer both walk through the same pattern on different teams.

I consult with faith-tech product leaders and ministry technology teams on AI evaluation layers, capacity logging before reorgs, and setting narrow boundaries around agent rollouts. Let’s talk.

I once signed a contract that paid an AI vendor only when their agent produced “successful small-group matches.” I believed this would keep costs honest because we would only pay after something concrete happened. The first invoice arrived thirty days later for three matches, and every one of them dissolved before the second meeting.

The real cost showed up in volunteer hours spent chasing people who never returned and in the quiet decision to run the pilot for another month anyway. We had protected the budget line but exposed the actual ministry work to metrics that rewarded volume over endurance. That invoice is the reason I now insist on agent outcome pricing that only pays for verified, lasting change — not activity dressed up as a result.

This is the foundational misread that causes product teams to accept outcome pricing that still measures activity instead of lasting change. The vendor simply redefines the outcome around whatever their model can count quickly. Ministry teams end up funding the appearance of progress while the harder work of retention and life change stays unfunded and unmeasured.

The anchor that exposed the problem was the outcome ledger we kept during that first agent pilot. We tracked every billed event against three columns: what the agent recorded, what the ministry team observed thirty days later, and whether any volunteer or participant could name a single concrete difference. The ledger made the gap visible in numbers we could not ignore.

The invoice that charged for three successful small-group matches

The line items listed three new groups formed through the agent’s outreach. Each carried a per-match fee that looked modest on paper. When we pulled the actual participant data, two of the three groups had zero attendance after week one and the third had lost its volunteer leader by week three.

We had assumed the vendor’s definition of success matched ours: a group that continued without staff intervention. Their system only needed an initial RSVP and a confirmation message. The ledger showed we paid for the confirmation step, not the continuation.

The pattern repeated in the next billing cycle. Every match that survived past two weeks had required manual follow-up from our own team. The agent’s contribution ended at the first message. We were effectively paying a finder’s fee for introductions that our volunteers still had to turn into relationships.

How the ledger forced us to rewrite the contract clause

We stopped arguing about price and started arguing about the definition column. The ledger required every billed outcome to include a thirty-day verification step that only the ministry side could complete. If the group or participant could not be confirmed by someone inside the church, the line item was clawed back.

The vendor pushed back on verification costs and latency. We held the line because the ledger already proved that unverified matches produced zero downstream activity. The revised clause tied payment to verified continuation rather than initial contact — the core move behind any real agent outcome pricing model.

That single change shifted the pilot economics. The number of billable events dropped by more than half, but the events that remained showed measurable retention and volunteer completion rates that aligned with our own historical curriculum data. The cost per lasting group became visible for the first time.

The quiet shift from seat counts to verified life change

Most ministry tools still price on seats or logins because those numbers are easy to export. The ledger experiment showed that seat counts hid the same problem we had seen with the agent matches. People created accounts after an automated nudge and never returned.

We began requiring every new pricing discussion to include a verification step tied to observable behavior thirty days out. For curriculum tools the check became whether the volunteer actually completed and returned the lesson. For discipleship agents it became whether the participant joined a recurring group or requested follow-up from a real person.

The shift forced vendors to surface data we could verify rather than data they could generate automatically. It also changed how we evaluated internal experiments. Features that drove quick sign-ups but flat thirty-day activity were deprioritized even when they looked good in the dashboard.

Your Turn: Apply This Today

• Pull the last three invoices from any agent or automation tool and list every line item the vendor called an outcome.
• For each item, add a column that only your team can fill: did the billed event produce a verifiable continuation thirty days later?
• Mark every item where the answer is unknown or negative and calculate what percentage of the bill that represents.
• Write one sentence that defines continuation for your context (group still meeting, volunteer still active, participant requested next step) and send it to the vendor as a proposed contract clause.
• Run the same thirty-day verification on your highest-volume internal feature for the next billing cycle and compare the numbers.
• Bring the ledger to the next renewal conversation and refuse to discuss price until the outcome definition matches the verification column.

The same pattern showed up when we added the knowledge graph to control token spend and again when we had to build a custom evaluation layer because off-the-shelf agents could not read our ministry data. Both posts trace the same need for teams to own the definition of progress rather than accept the one the model makes cheap to count.

I consult with faith-tech product leaders on outcome definitions for agent tools, contract clauses that protect verified ministry progress, and evaluation layers that survive real ministry data. Let’s talk.