Blog

You’re staring at the renewal email on your laptop, cursor hovering over the seat count. It says 62 active users. You know the real number is closer to eight people who still show up and actually do the work.

The rest logged in once for training and never again. The software doesn’t know the difference, and the invoice doesn’t care.

Outcome pricing changes that math. It waits to get paid until the number on the screen matches what actually happened in someone’s life.

Why Seat Pricing Hid the Real Mission Failure

Seat pricing treats every active account as equal value. A children’s ministry coordinator who logs in once to print a lesson counts the same as the volunteer who returns every week and eventually trains two replacements. The contract never distinguishes between the two, so the product never has to either.

Teams that built for the seven-minute volunteer learned this gap early. Print-first interfaces and single-screen lesson plans increased completion rates because the metric was whether the material reached the kids, not whether the leader created an account. Those same teams watched enterprise renewals stall once buyers started asking how many of those accounts produced a second or third volunteer on the roster.

The hidden failure was not bad design. It was a pricing model that made the harder measurement optional. Without financial pressure to instrument roster state changes, the product stayed at the surface of engagement.

What Changes When Outcome Pricing Pays for Retained Volunteers

Outcome pricing requires the product to observe the exact moment a handoff occurs. That observation forces instrumentation the seat model never demanded. A team must now track when a volunteer record moves from “active” to “trainer” and then to “inactive with replacement logged.”

The evaluation layer becomes non-negotiable. An off-the-shelf analytics package that only reports logins will not survive the first contract negotiation. The team that already built custom retention signals around volunteer completion rates holds the advantage; they can price against the metric they already measure.

Buyers also change. A church that once bought one hundred seats to cover every name on a spreadsheet now faces a narrower, more expensive question: which volunteers actually produced a handoff last quarter? The conversation moves from budget line items to documented behavior change.

Protecting Small Teams When Pricing Follows Results

Small ministry teams lose when pricing only rewards scale. Outcome contracts can amplify that risk if the definition of success assumes large cohorts. A metric that requires a large number of handoffs per year will price out the church plant running a dozen volunteers.

The safeguard is to price against the percentage of a roster that reaches a handoff state rather than absolute numbers. A team of eight that moves three volunteers into trainer roles in six months meets the same contract threshold as a team of eighty that moves thirty. The ratio keeps the agreement viable for both.

Product leaders who have already instrumented roster transitions at the volunteer level can support this model without new engineering. Those who only tracked aggregate logins must now decide whether to invest in the state-change layer or concede the outcome-priced segment of the market to teams that did.

Your Turn: Apply This Today

• Choose one roster transition—volunteer to trainer, or active to replaced—and write the exact database state change that records it.
• Add the field and the timestamp to the volunteer object this week so the transition can be queried without manual export.
• Build a simple query that returns the count and percentage of volunteers who reached the chosen state in the last ninety days.
• Present that single number to your renewal or pricing conversation instead of total active seats.
• Define the payment trigger as “X percent of roster reaches the state within Y days of onboarding” and test the language with one customer.
• Document the edge cases—volunteers who leave without a recorded replacement—and decide how the contract treats them before any negotiation begins.

The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer and The Evaluation Layer That Broke When We Trusted the Model Alone both trace the same requirement: metrics must survive contact with real behavior before anyone agrees to pay for them.

I consult with ministry product leaders and faith-tech PMs on outcome metric design, volunteer retention instrumentation, and pricing tied to roster state changes. Let’s talk.

I sat with the volunteer coordinator while she pulled the roster for the Tuesday crew. The list looked right at first. Then I noticed the guy whose wife had left in July was still marked “active,” and the couple who moved to another city two weeks back had a fresh phone number listed like nothing changed.

She stared at it for a long second, then scrolled again, slower this time. The screen gave no sign anything was off.

That quiet gap between what the system thinks is true and what’s actually happening in people’s lives is where most of these tools quietly break.

The same pattern shows up whenever product teams treat agent output as the final artifact instead of raw material. Torres’s framework requires repeated, side-by-side review of what the tool generated and what the human team kept or changed. Skip that evaluation loop and the system quietly drifts toward the last good snapshot it saw.

Watching the Exact State Loss Happen

The PM opened the edit history. The agent had generated the list at 9:17 a.m. from cached data pulled three days earlier. The staff coordinator had then spent twelve minutes correcting six entries before the final version went to the small group leaders. None of those corrections fed back into the agent.

Three days later the same agent ran again on the Tuesday night schedule. It produced the uncorrected version because the feedback loop never existed. The PM could see the two versions side by side only because he had saved both manually. The production system kept only the agent output.

This is not a data-pipeline bug. It is a discovery failure. The team never built the habit of checking what actually shipped against what the agent proposed. Without that weekly comparison, every subsequent run reinforced the stale state.

The Weekly Edit Comparison That Builds the Evaluation Loop

Torres’s continuous discovery asks teams to treat every shipped artifact as new input for the next iteration. In agent work that means saving the human-edited version and feeding the differences back into the prompt or the retrieval layer. The comparison has to happen on a fixed cadence, not when something breaks.

One team started requiring every agent-generated roster to be saved as both the raw draft and the final corrected file. On Monday mornings they opened the two versions, highlighted every line that changed, and updated the agent’s context window with those exact changes. After six weeks the agent stopped repeating the weeks-old names because the corrected data had become the new baseline.

The cost is small. The discipline is not. Most teams still review agent output only when a volunteer complains. By then the model has already repeated the error several more times.

Ministry Rosters as the Hardest Test Case

Rosters expose the problem faster than almost any other church dataset. People change roles, phones, and availability every week. The ground truth moves constantly. An agent that produces a clean-looking list from last month’s snapshot will look correct until someone tries to use it.

We ran the same test on curriculum assignments and event sign-ups. The pattern held. The agent output matched the final human version only when the team had forced a side-by-side review the previous week. Without it, the model drifted toward whatever snapshot it had last seen.

The lesson is not that ministry data is uniquely messy. The lesson is that any domain with high change frequency reveals the missing evaluation loop sooner. Continuous discovery works here because the cost of an undetected error is immediate and visible to real people trying to run programs.

Your Turn: Apply This Today

• Set a recurring Monday 30-minute block on your calendar to open last week’s agent draft and the final human version side by side for one recurring output.
• Save both the raw agent file and the corrected file in the same folder with date stamps before any further edits occur.
• Highlight every line that changed and paste the differences into a running “context updates” document the agent can reference next time.
• Run the agent again on the same task the following week and note whether the number of corrections drops.
• After four weeks, calculate the reduction in manual edits and decide whether to keep the loop or adjust the prompt structure.
• Share the four-week edit log with the rest of the product team so the pattern becomes visible beyond the single PM who noticed the roster problem.

The same evaluation gap appears in The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer and The Evaluation Layer That Broke When We Trusted the Model Alone. Both posts walk through the concrete mechanics of closing the loop after an agent ships.

I consult with ministry product leaders on agent evaluation loops and roster data integrity. Let’s talk.

In the ministry organizations we’ve watched adopt early coworker agents, paid user seats dropped fast within the first quarter. The obvious reading—that teams simply needed fewer logins and could downgrade their plans—turns out to be the opposite of what actually happened. The seats disappeared, yet the real human work did not shrink by the same margin. Instead it concentrated at the narrow points where agents handed tasks back to staff for judgment calls on tone, doctrine, or edge-case pastoral context.

That concentration exposed the flaw in any pricing model built around headcount. Per-seat pricing assumes steady usage across all users. Once agents handle the repeatable volume, the remaining work sits with a smaller group that still carries the full weight of oversight. The cost structure no longer matches the actual distribution of effort.

The outcome ledger we kept during the first agent pilot made this visible. Every day for fourteen days we logged the exact moment an agent completed a task and the moment a human either accepted, edited, or rejected it. The ledger did not track model accuracy in the abstract. It tracked the handoff.

The 14-day ledger that showed where agents actually replaced seats

The ledger ran on three products we had already shipped to ministry teams. One handled weekly curriculum outlines for children’s ministry volunteers. Another generated first-draft sermon illustrations. The third routed prayer requests to the right follow-up category. Each time the agent finished its portion we recorded the clock time and the category of the next required human step.

By day four the pattern was consistent. The agent replaced most of the original seat-minutes on curriculum outlines—well over two-thirds of the work. On prayer routing it replaced even more. Sermon illustrations saw the smallest shift, because pastors still rewrote most of the theological framing. The seat count dropped everywhere, but the remaining work clustered around the smaller slice of tasks that still required a named staff member to sign off.

We had assumed the savings would scale evenly. The ledger proved the savings were lumpy. One volunteer coordinator could now manage what used to require four people for the first pass, yet that same coordinator spent more time on the final review layer than before. The pricing conversation with the vendor therefore had to shift from “how many seats” to “which tasks still need a human override and how often.”

Why per-seat pricing hid the real cost of persistent coworker agents

Per-seat pricing treats every active login as interchangeable. It does not distinguish between the volunteer who opens the tool once a week to print a lesson and the staff member who now monitors agent output every morning. When the agent becomes a persistent coworker, the second role grows heavier while the first role shrinks or disappears. The contract still bills both the same.

In one midsize church the children’s ministry director had previously paid for 22 volunteer seats. After the agent went live she dropped to six seats for the volunteers who still needed direct access. The director herself, however, added two hours of daily review time that had not existed before. The vendor’s renewal quote assumed the drop from 22 to six seats represented a straight 73 percent reduction in cost. It did not account for the new, higher-value work now concentrated in the remaining seats.

The mismatch is structural. Per-seat pricing was designed for static software. Persistent agents change the ratio of machine output to human judgment inside the same workflow. Without a ledger that names the handoff points, finance teams continue to optimize for the wrong variable.

How to renegotiate the next contract around verified task completion instead

Start by exporting the ledger categories that actually required human intervention. Turn each category into a line item with a verified completion count rather than a seat count. The curriculum product, for example, moved from a per-seat license to a per-approved-outline fee once the ledger showed that most outlines needed only a single staff check before release.

Next, define the override threshold in the contract. If the agent produces work that still requires a human to change more than 15 percent of the output, the task does not count as completed for billing purposes. This clause forces the vendor to improve the agent at the exact points where judgment remains necessary instead of claiming broad seat reductions.

Finally, schedule a quarterly ledger review with the vendor. The review compares the previous quarter’s handoff data against the current quarter. If the share of tasks needing human override drops, the per-task price can adjust downward. If it rises, the price holds until the agent improves. The contract now tracks the actual division of labor rather than the number of named users.

Your Turn: Apply This Today

• Pick the single agent or automation your team uses most and export its last 30 days of task logs.
• Mark every task that required a staff member to edit, reject, or add context before release.
• Calculate the percentage of total tasks that still needed that human step.
• Map your current per-seat or subscription cost against only those tasks that cleared the human step without changes.
• Write one sentence that states the price you would pay if the vendor charged solely for verified completions.
• Send that sentence and the supporting ledger numbers to the vendor as the starting point for the next renewal discussion.

The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer walks through the exact handoff metrics that make outcome pricing workable. The Evaluation Layer That Broke When We Trusted the Model Alone shows what happens when those metrics stay internal and never reach the contract.

I consult with faith-tech product leaders on outcome-based pricing models and agent evaluation layers. Let’s talk.