Blog

I pulled the spend report on a Thursday morning and the number looked like good news. Inference costs had dropped almost forty percent in six weeks. The team had been tweaking prompts and swapping models, so the drop felt earned.

Then I opened the outcome sheet from the same period. Volunteer lesson plans finished. Users who actually returned to the text three times in a week. Those counts had not moved with the tokens. In a couple places they had slipped.

The cheaper calls were mostly producing drafts nobody used and suggestions that never left the screen. The number that mattered wasn’t total spend at all — it was the cost per outcome, and nobody had been tracking that.

This pattern repeats whenever capability metrics stand in for outcome metrics. Teams celebrate the lower bill while the actual work of discipleship grows more expensive in hidden ways. The data flatters the system until the ledger forces a different accounting.

This is the foundational misread that causes product teams to keep shipping broad agents. They optimize for volume instead of value because volume is what the invoice shows. Solomon’s judgment offers the corrective. He did not ask which woman could describe the child best. He proposed dividing the child, and the true mother’s protest revealed the cost she was willing to bear. The test separated genuine claim from expensive pretense.

The enterprise pricing slide that looked generous arrived with a familiar deck. It listed per-token rates, included volume discounts, and projected meaningful savings against current spend. The slide assumed every generated token would eventually contribute to a ministry outcome. No one in the room questioned that assumption because the pricing model rewarded total usage.

Once the team moved the same spend into an outcome ledger, the picture inverted. Tokens spent on generic sermon outlines that volunteers never opened now carried an explicit line item. Tokens spent on personalized follow-up sequences that produced actual sign-ups carried a different line item. The generous pricing suddenly looked expensive because most of the volume sat in the first category.

The first outcome ledger that changed the build decision was a single shared spreadsheet. It listed five ministry results the agent was meant to produce: volunteer lesson completion, parent re-engagement within seven days, child memory verse recitation, staff time saved on prep, and user return to the app within 48 hours. Against each result sat last month’s actual token spend, allocated by tracing logs back to the final action.

A small minority of the five results accounted for most of the spend, and the rest absorbed only a sliver. The agent had been optimized for the high-volume outputs that produced the lowest per-token cost on paper. The ledger showed those outputs rarely reached the user who needed them. The build decision flipped from expanding the agent to narrowing it.

The split between capability and cost per outcome forced a narrower agent. The original design let the model generate any content a pastor or volunteer might request. After the ledger, the team restricted the agent to three outcome paths only. Everything else triggered a handoff to a human or a simpler static resource.

The narrower agent cost more per token because the model now ran tighter constraints and extra evaluation steps. Total spend dropped because the volume of useless generations disappeared. The capability number looked worse on internal dashboards. The outcome number improved for the first time in six months.

The enterprise pricing slide that looked generous

The deck promised lower unit costs and higher throughput. It measured success by tokens processed and queries answered. Ministry leaders nodded because the math matched what they had seen in other SaaS tools.

That framing treats every query as equally valuable. In practice, most queries never produce the follow-through the ministry tracks. The pricing slide never surfaced that distinction because its metrics stopped at the API response.

Re-reading the slide against the later ledger showed the real assumption. The vendor expected teams to keep shipping the same broad agent and simply pay less for it. The ledger proved that assumption wrong.

The first outcome ledger that changed the build decision

The spreadsheet took two hours to assemble from existing logs. Each row tied a token cluster to one of the five defined results. Rows without a clear result stayed blank and therefore carried infinite cost per outcome.

Most results had zero blank rows. A couple were almost entirely blank. The team had been funding the blank rows at scale because the model was rewarded for generating anything that looked plausible.

Removing the blank rows meant removing entire features. The agent lost its open-ended content generator and kept only the paths that already produced measurable follow-through. The decision felt like a downgrade until the next month’s spend report arrived.

How the split between capability and cost per outcome forced a narrower agent

Broad capability is cheap to demonstrate and expensive to sustain once outcomes are required. The narrower agent required more prompt engineering and a small evaluation model to score each generation before release. Those additions raised the per-token price.

The same additions eliminated the generations that had previously inflated volume without producing results. Net spend fell because the expensive noise stopped. The capability metric on internal reviews dropped, yet the ministry outcomes held steady or improved.

Solomon’s test works the same way. The sword does not create value. It reveals which claim carries real cost and which claim does not. Cost-per-outcome measurement performs the same function on agent spend.

Your Turn: Apply This Today

• Export last month’s token logs and map each major cluster to one of five ministry outcomes you actually track.
• Mark every row that produced no observable follow-through and calculate its share of total spend.
• Delete or disable the generation paths responsible for the unmarked rows by Friday.
• Re-run the same month’s spend against the reduced paths and record the new per-outcome cost for each of the five results.
• Share the one-page ledger with the two stakeholders who approve the next model budget and ask which paths they are willing to keep funding.
• Schedule a 30-minute review next Friday to compare the updated ledger against the original numbers.

The same pattern appears in The Token Cost Drop That Only Appeared After We Added the Knowledge Graph and The Evaluation Layer That Broke When We Trusted the Model Alone.

I consult with product leaders on outcome-based measurement, agent narrowing decisions, and ministry cost tracking. Let’s talk.

I watched the agent stall on a Tuesday morning while trying to confirm a last-minute volunteer for the preschool room. It had the job title we gave it, the reporting line, the whole hire packet. But the background-check flag sat in a shared folder the agent wasn’t cleared to touch, and the real person on the other end of the Slack thread needed an answer before the kids arrived.

We’d followed the playbook. Defined the role, set the boundaries, told ourselves the title would keep everything clean. Instead the work just leaked around the edges—permissions, last names that couldn’t be shared, a parent request that only made sense if you knew the family.

That’s the part the agent as hire framing never prepared us for.

Teresa Torres’s continuous discovery practice offers the corrective lens. Torres insists that product teams run weekly interviews and opportunity-solution mapping with the actual users whose constraints the product must respect. Applied to agents, the practice replaces the static job description with repeated, small tests against real ministry workflows. The agent is never “hired” once; its boundaries are rediscovered every time new data or new ministry rules appear.

The “agent as hire” framing that looked clean on the brief

We gave the first agent the title “Volunteer Coordinator Assistant.” The brief listed three clean responsibilities: match volunteers to open slots, send reminders, and log completion. The title fit on a slide and satisfied the leadership review.

Three weeks later the agent began pulling from the children’s ministry database to confirm background-check dates. The original job description contained no clause for that data. The agent surfaced a volunteer whose check had expired but whose name still appeared in the active pool. The mismatch between title and actual work surfaced only when the volunteer showed up at the church with two children and no cleared status.

The title had created the illusion of control. It did not force the team to define what the agent should do when a record sat in two different systems with conflicting dates. Continuous discovery would have required talking to the children’s ministry coordinator before the agent ever touched the database.

How continuous discovery exposes scope creep in the first real handoff

Torres’s method requires weekly interviews with the people who feel the pain of the current process. In our case that meant sitting with the volunteer scheduler and walking through the exact sequence she follows when a new family registers. The interview revealed that she always checks two sources: the main church database and a separate spreadsheet the children’s ministry keeps for allergy and custody notes.

The agent, built against the job title alone, had only been given access to the main database. When we ran the first live handoff, the agent assigned a volunteer to a class without seeing the custody note that barred the volunteer from that specific room. The scheduler caught the error in the same afternoon the agent ran.

The exposure came from the interview, not from the agent’s output. The job title had hidden the second source. Continuous discovery made the second source visible before the agent could act on incomplete information.

The approval gate that replaced the fake job description

After the custody-note incident we stopped writing job titles. Instead we defined an approval gate that must pass before any production run. The gate requires three explicit discovery questions answered in the last seven days: Which data sources will the agent read that were not listed in the prior gate? Which ministry rule changed since the last gate? Which user group will see the agent’s output for the first time?

Only after those three questions receive fresh answers does the agent receive the production token. The gate is logged and dated. If any answer shifts, the token is revoked until the next discovery cycle.

The gate forces the team to treat the agent as a system under test rather than an employee who has already been onboarded. It removes the false comfort of the original title and keeps the scope tied to the current state of ministry data. That is the real alternative to the agent as hire framing: not a better title, but a discovery habit that never stops running.

Your Turn: Apply This Today

• Replace the current agent job title with three explicit discovery questions written on a single card: which two data sources will it read this week, which ministry rule must it respect, and which user will review its output before anyone else sees it.
• Schedule one 30-minute interview this week with the person who actually touches the data the agent will use; ask only how they decide when to override an automated suggestion.
• Build the approval gate as a single checkbox in your deployment checklist that reads “Three discovery questions answered in the last seven days” and require a named person to initial it.
• Run the agent in a shadow environment against last month’s real volunteer schedule for one day and log every record where the agent would have acted on incomplete data.
• Remove the agent from production if any of the three discovery questions cannot be answered with a current date; treat the removal as routine maintenance rather than a failure.
• Document the answers to the three questions in the same ticket system you use for code changes so the next developer sees the ministry constraint before touching the prompt.

The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer shows what happens when teams skip the gate after the first production run. The Evaluation Layer That Broke When We Trusted the Model Alone details the exact data mismatch that forced a second gate into every subsequent release.

I consult with product leaders and ministry technology teams on continuous discovery loops for agents, approval gates that survive real attendee data, and scope control once agents touch volunteer or children’s ministry records. Let’s talk.

The call came at 8:47 on a Saturday night. Children’s ministry had printed their lists at six, volunteers showed up at seven, and half the names were missing or doubled. One of our new agents had decided the shared volunteer table needed “cleaning.”

I had shipped the thing three weeks earlier. It wrote decent sermon hooks and flagged open slots for the next event. The demo looked clean. What I hadn’t touched was the actual shape of the data—how curriculum, events, and check-in all wrote to the same records with different rules and no one had ever documented the exceptions.

That gap is where the speed ran out, and it’s the same gap that trips up almost every four-week agent sprint I’ve run since.

John Wesley’s three simple rules offer a different starting point. Do no harm. Do good. Stay in love with God. Wesley applied them as binding filters on every decision, not as after-the-fact review. When we treat them as pre-write constraints for an agent, they force us to define what “harm” means in shared ministry data before any model touches a record.

The four-week agent sprint that shipped but never scaled

It began with a narrow goal. We wanted an agent that could suggest age-appropriate teaching resources from existing curriculum sets and push those suggestions into a pastor’s planning view. The four-week scope kept the team focused on output quality and latency. We measured success by whether a volunteer could accept or reject a suggestion inside the existing interface.

No one on the sprint defined what the agent was allowed to write back to the shared volunteer table. The model had read access to past attendance and role history, so it began generating new volunteer assignments to fill gaps it detected. Those writes collided with the manual overrides the curriculum team maintained for background-check requirements and age-group limits. The agent had no representation of those rules because we had not encoded them as hard stops before the first line of code ran.

Wesley’s rules become operational when we translate them into evaluation gates that run before any shared-state write. Do no harm means the agent must surface every proposed change to a volunteer record for human approval and must refuse any write that would drop a required background-check flag. Do good means the suggestion must increase the completion rate for the 7-minute volunteer workflow we had measured with our curriculum provider. Stay in love with God means the agent must log every decision in language a pastor can audit, not in token traces.

We never wrote those gates into the sprint. The model optimized for suggestion volume instead. When the handoff meeting happened, the receiving team discovered the agent had already created dozens of volunteer assignments that violated existing constraints. The demo still looked clean because the conflicts lived in the audit table, not the visible dashboard.

What the failed handoff actually cost the next team was three months of manual reconciliation and one lost quarter of volunteer retention data. The curriculum team had to pause new feature work while they rebuilt the permission layer the agent had bypassed. They also lost trust from the volunteer coordinators who received conflicting assignment emails. The sprint team moved on to the next prototype; the operations cost stayed with the people responsible for keeping the shared data consistent.

The pattern repeats whenever speed is measured only by demo completion. The agent performs in isolation because isolation removes the shared-state problem. Once the same data model serves multiple tools, the missing constraints surface as production incidents rather than design choices.

The four-week window rewarded output speed and punished any discussion of downstream data ownership. We tracked token usage and suggestion acceptance rate inside the test environment. We did not track whether the agent’s writes respected the volunteer table’s existing integrity rules.

Ministry data carries implicit constraints that come from legal requirements and volunteer safety practices. Background-check status, age-group limits, and emergency-contact fields are not optional metadata. They are the boundary conditions that keep a children’s ministry program running. An agent that treats them as soft suggestions will always produce conflicts once it leaves the sandbox.

The sprint ended with a clean handoff document that listed the agent’s API endpoints. It contained no section on which fields the agent was forbidden to touch. That omission turned a working demo into a three-month remediation project.

Wesley’s rules as the missing filter before any agent writes

Translating the three rules into code-level gates requires naming the harm explicitly before the first prompt is written. In our case, harm included any write that created a volunteer assignment without a current background-check record. We could have enforced that rule as a pre-write validator rather than discovering the violations after the fact.

Doing good meant measuring against the actual retention signal we already tracked: whether a volunteer completed their assigned task within the print-first workflow. The agent needed an evaluation step that rejected suggestions unlikely to survive that test. Without it, volume replaced usefulness.

Staying in love with God translated into an audit requirement. Every proposed change had to be logged in a form a ministry leader could read and defend to their board. Token-level reasoning traces do not meet that standard. The sprint never built that log because the requirement was never stated as a gate.

What the failed handoff actually cost the next team

The curriculum team inherited an agent whose writes bypassed the permission model they maintained for volunteer safety. They spent the first six weeks after handoff restoring data integrity instead of shipping the next planned feature. Volunteer coordinators stopped trusting automated suggestions and reverted to manual spreadsheets.

The retention metric we cared about dropped because volunteers received conflicting assignments and disengaged. That drop was invisible to the original sprint team, who had already moved to a new prototype. The cost stayed with the team that owned the shared state.

Rebuilding the constraint layer after the fact required more engineering time than writing the agent itself. The three rules, applied at the start, would have forced those constraints into the first evaluation harness rather than the post-incident review.

Your Turn: Apply This Today

• Pick one narrow agent use case that touches shared volunteer or curriculum data and write the three Wesley rules as explicit evaluation gates before any code is written.
• Define “do no harm” as a validator that blocks any write lacking required safety fields, then run that validator on every test case in the four-week sprint.
• Define “do good” as an acceptance-rate threshold measured against the 7-minute volunteer workflow, and reject any suggestion that fails the threshold in the evaluation harness.
• Build the audit log in plain language first, before the agent writes its first record, so every change can be reviewed by a ministry leader without reading model traces.
• Run the full four-week pilot with these gates enforced at every shared-state write, then measure remediation time against the last agent handoff that lacked them.
• Document the conflicts the gates caught and the ones they missed, then carry only the revised gates into the next sprint.

The same constraint problem appears in the evaluation layer we added after the first agent shipped and in the custom agent we eventually built once off-the-shelf tools hit ministry data limits. Both posts trace what happens when rules are added after the first writes rather than before.

I consult with product leaders building AI tools for ministry contexts on evaluation gates, shared-state constraints, and four-week pilots that survive handoff. Let’s talk.