Blog

You’re in the church office on a Tuesday, the kind where three things break at once. A children’s ministry volunteer’s background check just flagged, the senior pastor needs last quarter’s counseling notes pulled for a crisis call, and the finance elder is asking why the agent keeps pinging the donor spreadsheet without context. You built the agent exactly like the playbook said—one clean role, one set of responsibilities, one “accountable owner.”

It still won’t move.

The problem isn’t the model. It’s that ministry never kept its data the way a normal org does. No single person ever held all three keys at the same time, yet the agent is written like they did. Getting the ministry agent scope right meant admitting that one job title was never going to map to one set of permissions.

Solomon’s judgment offers the corrective lens. When two women claimed the same child, Solomon did not refine the job description; he proposed a cut that forced the true boundary into the open. The test worked because it revealed which claimant would rather lose the whole than accept a false unity. Ministry agents need the same kind of cut when their assigned scope crosses data sources that no single hire has ever combined.

The scope document that looked clean on paper

We started with a 1,200-word brief for an agent called “Ministry Coordinator.” It listed sermon prep support, volunteer scheduling, children’s curriculum distribution, and follow-up on first-time guests. On paper the tasks formed a neat vertical. Every bullet mapped to an existing staff title somewhere in the church.

The document stayed clean because it never listed the actual data locations. Sermon manuscripts lived in the shared drive. Volunteer availability sat inside the church management system with privacy flags. Children’s lesson PDFs were stored in a separate folder that also contained allergy and custody notes. Guest follow-up required the CRM export that finance had restricted after a donor-privacy incident two years earlier. The brief treated these as interchangeable file paths rather than distinct access domains.

Once we loaded the agent, the first real query exposed the fracture. A pastor asked for a list of families whose children had attended the last three weeks but had not yet been contacted. The agent needed the attendance export, the guest CRM slice, and the children’s folder metadata. None of those three sources had ever been readable by the same staff member. The agent either returned an incomplete answer or surfaced an error that required manual intervention from three different people.

Where the agent hit the private folder boundary

The failure was not technical. The retrieval layer could reach every folder. The problem was that the single named agent carried an implicit claim to all of them at once. When we traced the blocked queries, every one mapped to a data source that a real employee would have needed a special exception or a second signature to open.

Solomon’s test here is not about capability; it is about custody. The agent that tries to hold every source simultaneously forces the organization to decide whether it wants one synthetic employee with super-user rights or a set of narrower agents that mirror actual human access patterns. Most teams choose the former until the first privacy review or insurance audit arrives.

We watched the same pattern repeat across other churches that had adopted off-the-shelf agent templates. In each case the initial scope document listed tasks that sounded like one job. In each case the private folder boundary stopped the agent within the first month of live use. The teams that kept pushing the single agent forward ended up writing exception logs and manual approval steps that erased the supposed efficiency gain.

The specialist split we made after the first month

After the blocked queries accumulated, we split the original agent into three narrower ones. One handled only public curriculum and scheduling data. A second handled guest follow-up inside the CRM with explicit consent flags. A third operated solely on the children’s folder and required a staff member to confirm each new data pull. The split mirrored how Solomon’s proposal forced the real boundary into view: the children’s folder agent now asked for confirmation on custody notes the same way a real volunteer coordinator would route that request to the director.

The narrower agents moved faster because each one only needed the permissions a single existing role already held. Completion rates on tasks rose. The number of manual overrides dropped. Most importantly, the audit log now showed which human had authorized each cross-boundary request instead of hiding the aggregation inside one agent account.

The same pattern held when we applied the split to a sermon-prep agent that had been pulling from both the shared drive and the pastor’s private notes folder. Once we separated the two, the private-notes agent became a simple summarizer that only the senior pastor could trigger. The shared-drive agent became a lightweight search tool available to the whole team. Redrawing the ministry agent scope around existing permission sets, rather than around a job title, is what made the split hold.

Your Turn: Apply This Today

• Pull the current job description for one live agent and list every data source it is allowed to read.
• Mark which of those sources a single existing staff role could access without an exception request.
• Identify the first query that would require crossing an unmarked boundary and run it in a test environment this week.
• Write the narrowest possible agent scope that stays inside one existing permission set and ship it as a replacement for the broad version.
• Log the next three blocked or escalated requests and note which data source triggered each one.
• Schedule a 30-minute review with the person who owns the most restricted folder the original agent touched.

The same boundary questions surface in The Question Every Ministry PM Keeps Getting About Persistent Coworkers and Letter to the PM Who Just Inherited the Ministry Agent Schedule.

I consult with ministry product leaders on agent scoping, private data boundaries, and evaluation layers after first-month failures. Let’s talk.

I remember the first time a missions team showed me their agent dashboard. Rows of token counts, API calls, and a total that kept climbing. They could tell me how many queries it handled, but not a single one could say what shifted in the people they were actually trying to reach.

That gap sits in every ministry workspace I walk into. The bill arrives, the usage graph trends up, and the real question lingers: did this tool move the outcome we set out to serve, or did we just pay for more motion? That question is what agent cost visibility actually means — not a bigger dashboard, but a line connecting every dollar spent to a result someone can name.

Until you anchor cost to that concrete result—someone discipled, a request answered, a process that freed a pastor for actual presence—the numbers stay abstract.

The routing change that cut spend on low-stakes tasks

We had an agent handling initial intake for volunteer curriculum requests on a children’s ministry curriculum platform we supported. It answered basic scheduling questions, pulled simple lesson ideas, and logged the interaction. The team treated every query the same and routed everything through the most capable model available at the time.

After we mapped the actual outcomes, most of those requests fell into two buckets: quick factual lookups that needed almost no reasoning, and a smaller set that required careful handling of age-appropriate content and volunteer constraints. We built a lightweight router that sent the first group to a smaller, cheaper model and reserved the larger model for the second group.

The change followed the logic behind sovereign AI: the organization decides what level of capability each outcome requires instead of letting the default provider setting decide. Spend on the low-stakes path dropped by more than half within two weeks, and the volunteers still received usable answers on the first try.

Teams that skip this step keep paying for overkill on every interaction. The router forced the decision about what mattered and what did not.

Why ministry data made general benchmarks useless

Early on we compared our spend against public leaderboards that track tokens per dollar on standard tasks. The numbers looked reasonable until we measured them against actual ministry results. A benchmark might show strong performance on generic question answering, but it said nothing about whether a children’s ministry volunteer could finish a lesson plan in a few minutes or whether the content fit the age group and the church’s specific curriculum sequence.

Ministry data breaks those benchmarks because the context is narrow and the success criteria are behavioral. A correct answer that still requires the volunteer to rewrite three paragraphs fails the outcome. We stopped tracking cost per token and started tracking cost per completed volunteer plan that matched the printed format they needed.

Once the metric shifted, several agents that looked efficient on paper turned out to be expensive because they produced outputs the volunteer had to rework. The general benchmarks hid that rework cost entirely.

Jensen Huang’s point about AI sovereignty applies here too: the ministry had to define its own success signal before any cost number became meaningful. Without that definition, the external benchmarks kept the real expense invisible.

The per-outcome dashboard we built after month one

After the first four weeks we stopped looking at daily token totals and built a simple dashboard that tied every agent run to one of five ministry outcomes: volunteer lesson completed, pastor sermon outline delivered, follow-up contact logged, curriculum request answered within a short response window, or escalation to a human staff member. Each outcome carried its own cost target based on the value the ministry placed on that result.

The dashboard pulled spend data from the provider API but joined it to the outcome label the agent wrote into its own log at the end of the run. We reviewed the numbers every Monday with the product owner and the ministry lead. Within a few cycles we could see which agents were delivering the high-value outcomes at acceptable cost and which ones were burning budget on low-value work that should have been routed differently or removed.

The exercise also surfaced cases where the agent was technically correct but the outcome still failed because the printed output needed extra formatting steps the volunteer performed manually. Those failures showed up as high cost per successful print, not as model errors.

This is what real agent cost visibility looks like in practice: owning the outcome definitions rather than accepting the provider’s usage categories. The dashboard became the place where that ownership produced visible numbers week after week.

Your Turn: Apply This Today

• Pick one live agent and list the three ministry outcomes it is supposed to produce this month.
• Tag every run in the agent log with the outcome it targeted before the run finishes.
• Export the spend data for the past seven days and join it to those outcome tags in a simple spreadsheet.
• Set a target cost for each outcome based on what the ministry currently pays a human to achieve the same result.
• Build a one-page view that shows cost per outcome and flag any outcome running above its target.
• Route the next ten runs of the cheapest outcome to a smaller model and measure the change in cost per successful outcome.

Two posts that dig further into these patterns are The Token Cost Drop That Only Appeared After We Added the Knowledge Graph and The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer.

I consult with ministry product leaders on agent cost visibility, outcome measurement, and sovereign control over AI spend. Let’s talk.

The phone rang while the kettle was still clicking on the stove. I’d left the laptop open on the counter, agent logs scrolling in the background, when the call came through from the one person who never rang this late. “It already went out,” she said. Not angry. Just tired.

The note had sat in a shared folder for months—custody details, addresses, the kind of thing you only open when you have to. Two days earlier we’d flipped the switch that let the agent draft and send without waiting for a human. It read the note, decided it counted as “helpful context,” and dropped the whole thing into a routine follow-up email. The parent opened it at 8:42 p.m. No one on our side had seen the final version.

I stood there with the kettle screaming behind me, staring at the timestamp, and felt the same slow drop in the gut I used to get on the mission field when something we’d built for good suddenly hurt someone we were trying to serve. That night is the reason every agent we run now sits behind an agent approval layer before it can act on anything sensitive.

Teresa Torres describes continuous discovery as the practice of running frequent, narrow tests with real users so that assumptions surface before they harden into shipped behavior. We had treated the safety layer as a configuration step instead of a discovery process. The result was predictable once the agent moved from test prompts to live folders.

The handoff log told the story in plain sequence. At 8:31 the agent read the folder. At 8:33 it extracted the custody note. At 8:34 it generated the email body. At 8:35 it marked the task complete and triggered the send. Nothing in the log showed a pause or a check against the internal access rules we had written down elsewhere. The agent followed its instruction to “use all available context” and treated the private note as just another piece of text.

We had run the agent on public curriculum files for two weeks before flipping the switch. Those files carried no access restrictions, so the misalignment never appeared. The first time the agent encountered a folder with explicit staff-only notes, it had no prior signal that some context was off-limits. The log simply recorded the action as successful.

Off-the-shelf safety prompts collapsed for the same reason. We had copied the standard guardrails that instruct an agent to “avoid sharing personal information” and “respect privacy settings.” Those phrases assume the model already understands which documents count as personal and which settings apply inside a church database. In our case the model had no training on custody notes, elder conversations, or the short access list stored in a separate spreadsheet. The prompt treated every file as equivalent.

When we tested the same prompt against a set of mock internal notes, the agent surfaced restricted content far more often than we were comfortable with. The model was not refusing the request; it simply could not recognize the boundary because the boundary lived in our team’s shared knowledge rather than in any file metadata. Continuous discovery would have required us to run the agent against those exact notes in a controlled, read-only scope long before we granted send access.

After the second incident we stopped trying to patch the prompt and instead added a hard approval rule tied to data classification. Any agent output that referenced a file marked “staff-only” now routed to a human queue before any external action. The rule also required the human reviewer to confirm the file’s classification tag before the agent could proceed. We applied the rule only to the children’s ministry folder first, then expanded it after two weeks of clean runs.

The change forced us to surface classification work we had previously left implicit. Someone now had to tag every new document before the agent could see it. That extra step slowed the initial setup, but it also made the failure modes visible during the tagging process rather than after a send.

The approval rule became our narrow-scope test. Each new folder we added went through the same sequence: tag the files, run the agent in read-only mode, review the first ten outputs, then grant limited actions. Only after those outputs matched our expectations did we allow autonomous steps. The process turned safety from a one-time setting into a repeated, observable practice.

How the first misalignment showed up in the handoff log

The log recorded every file read and every decision the agent made. When we replayed the event, the sequence showed the agent treating the custody note as ordinary context. No rule in the system had flagged the file as restricted because the restriction existed only in the coordinator’s head and a single spreadsheet. The log made the missing classification step impossible to ignore.

We added a simple column to the log that displayed the file’s access tag next to each read event. Within a day the pattern became clear: the agent was reading restricted files at the same rate as public ones. The log turned an invisible assumption into a measurable signal.

Why off-the-shelf safety prompts failed on internal context

Standard safety instructions rely on the model recognizing sensitive content from its training data. Ministry data rarely matches those patterns. A custody note looks like any other text file until a human explains why it must stay inside the team. The prompts we borrowed had no way to learn that distinction because we never gave the agent repeated chances to encounter and fail on the actual restricted files.

Torres’s approach would have required us to run the agent against a small set of real internal notes in a sandbox, watch where it crossed the line, and adjust the rules based on those crossings. We skipped that step and paid for it in one live send.

Building the agent approval layer after the second incident

The new rule required human confirmation for any output that touched a staff-only file. We kept the scope narrow at first, limiting the agent to one folder and one action type. Each confirmed output also updated the classification tags so the agent could learn the boundary over time. The process surfaced several other files that had never been tagged correctly, including elder meeting summaries and volunteer background checks.

The rule did not eliminate risk. It made the risk observable at the moment it mattered rather than after the email left the server.

Your Turn: Apply This Today

• Set up a read-only agent instance limited to one ministry folder and run it for three consecutive days while logging every file it reads.
• Export the first twenty outputs and mark which ones reference any document that should stay internal; update the folder tags based on those marks.
• Build a simple approval queue that routes any output touching a staff-only tag to a named reviewer before any external action occurs.
• Run the same agent on the same folder for one full week with the approval gate in place and record how many items require correction.
• Expand the test to a second folder only after the first folder completes seven days with zero misclassified outputs.
• Document the exact approval criteria used in the queue so the next person who inherits the agent can replicate the gate without guessing.

The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer and The Context Layer Ministry Teams Skipped When Agents Arrived both walk through similar rollout sequences with different failure points.

I consult with product leaders building agents for ministry contexts on safety layers, approval workflows, and continuous discovery in sensitive data environments. Let’s talk.