Blog

I still have the email. The subject line just said “AI pilot next steps,” and the attachment was a contract that billed per successful run. Nobody had defined what success meant. Not the vendor, not the leadership team, not even the guy who’d been asked to run the thing.

I kept reading the same sentence: “Payment tied to measurable ministry outcomes.” I remember thinking, we can’t even agree on what counts as a pastoral visit. How are we going to price an agent that’s supposed to do the same work?

That gap sat there, quiet, while the budget line got approved anyway. What we needed was outcome-based agent pricing tied to something a human could verify, not a per-run invoice.

The ledger started as a simple shared sheet. One column for the exact output the agent produced, one for the observable change in a ministry workflow, and one for the dollar amount we would only pay if both columns were verified by a human reviewer. We added rows only after the agent ran against real curriculum data from a volunteer-curriculum provider we work with. Nothing entered the ledger until a children’s ministry coordinator confirmed the output was usable without rewrite.

The invoice that arrived before anyone agreed what counted as success

The first invoice listed thousands of successful generations at the agreed per-outcome rate. When we opened the attached log, most rows showed the agent producing sermon illustration summaries that still required a pastor to fact-check and rewrite the application points. The vendor counted them as successes because the model completed the prompt. We counted them as incomplete because no volunteer could hand the summary to a parent without additional work.

We had never written the single sentence that would have prevented the mismatch: the agent must generate a complete, print-ready small-group question set that a 7-minute volunteer can use without opening another document. Until that sentence existed, every generation was open to reinterpretation.

The mismatch showed up in the numbers first. Our internal tracking showed that only a small fraction of the billed outputs survived a single volunteer test without edits. The vendor’s own dashboard showed the vast majority as successful. The ledger made the gap visible in one view.

How the ledger changed vendor conversations

We sent the vendor the ledger instead of a revised scope document. The next call shifted immediately from feature requests to outcome definitions. The vendor’s team asked what evidence we needed for each row before payment triggered. We answered with three artifacts: the generated file, a timestamped screenshot from the volunteer who used it, and a one-sentence confirmation that the workflow step finished without further input.

That single change removed half the proposed use cases. The vendor stopped pitching broad “content creation agents” and started asking which narrow workflow step carried the highest cost when it failed. The conversation moved from model capabilities to the exact point where ministry labor was currently wasted.

The ledger also surfaced internal disagreements we had ignored. One curriculum director wanted the agent to increase the number of lessons produced per month. Another wanted the agent to reduce the time between lesson assignment and volunteer completion. Those two goals pointed to different outcome definitions and different price points. The ledger forced the choice before any new pilot began.

The outcome-based agent pricing we finally settled on

We settled on one outcome only: the agent must produce a complete, age-appropriate memory verse activity sheet that matches an existing lesson outline from our curriculum provider, passes a quick review by the volunteer coordinator, and is printed and handed to children within the same planning cycle. The dollar value attached to each verified sheet was set at the internal cost of a staff member creating the same sheet from scratch.

Payment now triggers only after the coordinator uploads the printed sheet with a date stamp and a short confirmation that no additional writing was required. The vendor accepted the definition because the success criteria are binary and the verification artifact is cheap to produce. We accepted it because the outcome directly reduces the labor that was previously blocking volunteer retention.

Everything else the agent can generate sits outside the ledger and outside the contract until we run a second, separate pilot with its own measurable finish line.

Your Turn: Apply This Today

• Write the single sentence that defines the exact output your agent must produce before any payment can trigger, then send that sentence to the vendor before the next call.
• Pick the one ministry workflow step that currently costs the most staff or volunteer time and attach a dollar value equal to the internal cost of completing that step manually.
• Build a three-column ledger in the shared project tracker with columns for output, verification artifact, and dollar amount, and add no rows until the first human confirmation arrives.
• Run the current agent against ten real records from your curriculum or scheduling data and count how many outputs meet the sentence you wrote; share that count with the vendor this week.
• Remove every use case from the pilot scope that cannot be verified with a single artifact a ministry leader already produces in their normal workflow.
• Schedule the payment trigger for the narrow outcome only after the verification artifact is uploaded, not after the model finishes generating.

The same pattern appears in the posts on evaluation layers and the agents that only became usable after we added the knowledge graph. The PM who just shipped an agent and now needs an evaluation layer ran into the identical pricing gap when the model output could not be tied to a single ministry result.

I consult with faith-tech product leaders on defining measurable agent outcomes, building verification workflows, and shifting contracts from usage to verified ministry results. Let’s talk.

I watched a volunteer delete an entire morning’s work after the general agent we’d pointed at everything—prayer requests, donor notes, facility bookings—decided to “help” with a counseling ticket. It had pulled a real name from an old database and offered the person a tidy paragraph of advice that was neither asked for nor accurate. The volunteer sat there refreshing the sent folder, realizing she now had to call someone and explain that a machine spoke on behalf of the church.

That single reply cost more time than the agent had ever saved us. The problem wasn’t malice. It was that nothing inside the system told it where its job stopped. What we needed instead were named specialist agents, each built around one job and nothing else.

Solomon didn’t split the difference between two mothers. He named the one thing that actually mattered and refused every other claim. Every specialist agent that has worked for us since has followed the same logic: name one job, and refuse everything outside it.

The named specialist agent that only handled roster updates

One team replaced a broad scheduling agent with a single specialist whose only job was to propose roster changes and nothing else. The specialist could read availability data, suggest swaps, and flag conflicts, but it had no ability to send notifications or update the calendar. Every proposal landed in a human queue.

The restriction forced the model to stay precise about dates and roles instead of guessing at downstream effects. Volunteers noticed the difference immediately. They stopped receiving messages that assumed they could cover a shift they had already declined. Acceptance of the suggestions rose because they arrived already vetted for basic accuracy.

The same team had previously tried to give the general agent more context and better prompting. Those changes reduced errors only marginally. The hard boundary around roster updates cut the error rate roughly in half without any increase in model size.

The approval gate that saved the finance team from three bad decisions

A different group added an explicit approval step before any agent could move budgeted funds between categories. The agent could draft the transfer, calculate the impact on remaining line items, and write the justification, but the actual ledger change required a named human to approve.

In the first month the gate caught three separate proposals that would have left a children’s event underfunded because the agent had pulled stale enrollment numbers. Each case was small enough that a human reviewer might have missed it under time pressure. The gate made the mismatch visible before the transfer executed.

The finance lead later said the real value was not the catches themselves but the change in how the team talked about the agent. Instead of treating it as an autonomous decision maker, they treated it as a drafter whose work required sign-off. That shift removed the background anxiety that had accompanied every automated suggestion.

How multi-account support actually worked when each agent stayed narrow

A ministry platform supporting several hundred small churches tried one agent for all account questions and watched response quality collapse. The agent mixed data between accounts, offered features that belonged to a different pricing tier, and occasionally referenced another church’s custom fields.

They split the work into three named specialists: one that only read public plan details, one that only handled data export requests, and one that only opened tickets for human review. Each specialist carried its own job title and its own approval gate for any action that touched account data.

Support staff reported that the volume of follow-up corrections dropped sharply. When a user asked about exporting attendance records, only the export specialist responded, and it refused to touch any other data type. The narrow scope made the handoff to a human reviewer obvious rather than hidden inside a long, plausible-sounding reply.

Your Turn: Apply This Today

• Open your current agent list and rename the broadest one with a single, real job title such as “Roster Change Drafter” instead of “Ministry Assistant.”
• Pick one meaningful action that agent currently performs and route it through an explicit human approval step this week.
• Write the refusal message the agent will show when a user asks it to do anything outside its new title.
• Test the renamed agent against the last three requests it handled incorrectly and confirm it now declines the ones that fall outside its lane.
• Share the job title and approval rule with the two people who review its output most often and ask them to flag any drift.
• Schedule a thirty-minute review next Friday to decide whether the second agent on your list also needs its own title and gate.

The same pattern of narrow titles and visible gates appears in both “The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer” and “The Evaluation Layer That Broke When We Trusted the Model Alone.”

I consult with product leaders building AI agents for ministry contexts on specialist design, approval workflows, and trust boundaries in multi-agent systems. Let’s talk.

I sat there with the volunteer coordinator while the printer jammed for the third time that morning. She didn’t swear, just reached over, tugged the half-printed sheet free, and wrote the activity on the back of her hand with a Sharpie. We had four weeks on the calendar and one working laptop between us. The agent was supposed to spit out usable ideas. Instead it kept asking questions we already knew the answers to.

That small failure kept bothering me long after the test ended. We hadn’t built the wrong tool on purpose. We’d simply started from the assumption that ministry teams needed more information and better structure, when what a working ministry agent build actually needed was something that could survive a broken printer and a folding table.

The printed job description that actually mattered

The sheet on the wall listed five duties and nothing else. The agent had to pull the right lesson from the curriculum folder, check the supply list against what the church already owned, suggest one adjustment for the current group size, and stop. No extra explanations. No new ideas unless the coordinator asked.

That single card kept the scope from drifting. When the model tried to generate a full service order or suggest new songs, the team rejected it on the spot because it violated the printed duties. The rule “do no harm” translated directly into refusing outputs that added work the volunteer would later have to undo.

The card also made evaluation simple. At the end of each day the coordinator marked yes or no next to each duty. After two weeks the team could see exactly which duties the model handled and which ones still needed guardrails. No abstract accuracy score. Just five lines on paper that matched the real job.

What happened when the children’s director sat in the room

She did not join as a reviewer after the fact. She sat at the table while the tests ran and said out loud when an answer would waste her time or create extra steps. One response suggested printing three extra pages. She shook her head and the team added a constraint that every answer had to reference only materials already in the supply closet.

Her presence turned the model from a general tool into a narrow assistant that understood her actual constraints. The rule “do good” stopped being theoretical. It became the moment she said “that works” and the agent moved to the next test case without further changes.

The team also noticed when she stayed quiet. If she had to re-read an answer twice, they treated the silence as a signal that the output had failed the “stay in love” test. It did not serve the person doing the work on Sunday morning.

Why a narrow ministry agent build beats a six-month roadmap

A larger committee would have started with questions about model choice, data privacy standards, and long-term hosting. This team started with the printed card and one user. They shipped a working version in twenty-six days because every decision stayed inside the five duties on the wall.

The narrow scope forced shared understanding. When something broke, all three people could point to the same line on the card and agree on the fix. No one argued about future features because the calendar on the door said the build ended on day twenty-six.

The same pattern shows up in other ministry tools. When scope stays small and the user stays close, the real constraint turns out to be clarity between the people in the room, not the power of the model.

Your Turn: Apply This Today

• Choose one named ministry role, such as the children’s coordinator at your church, and write their exact job on a single printed page before you open any code editor.
• Block one four-week window on the calendar with the end date written in pen and commit to shipping only what fits on that page.
• Run every test with the actual person in the chair and require them to say yes or no to each output against the printed duties that same day.
• Reject any model response that adds work the user would have to fix later, even if the response looks impressive.
• At the end of week two, review only the marks on the printed page and cut any duty the model still fails more than once.
• Document the final working agent by the job card it serves, not by the model or framework it uses, then move the card to the next narrow task.

The same narrow-build pattern appears in both “The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer” and “The Custom Agent We Had to Build After Off-the-Shelf Tools Hit Ministry Data Limits.”

I consult with faith-tech product leaders on running tight four-week agent builds and keeping scope narrow with actual ministry users. Let’s talk.