The Cost Number That Only Appeared After We Measured Per-Outcome Spend

I pulled the spend report on a Thursday morning and the number looked like good news. Inference costs had dropped almost forty percent in six weeks. The team had been tweaking prompts and swapping models, so the drop felt earned.

Then I opened the outcome sheet from the same period. Volunteer lesson plans finished. Users who actually returned to the text three times in a week. Those counts had not moved with the tokens. In a couple places they had slipped.

The cheaper calls were mostly producing drafts nobody used and suggestions that never left the screen. The number that mattered wasn’t total spend at all — it was the cost per outcome, and nobody had been tracking that.

This pattern repeats whenever capability metrics stand in for outcome metrics. Teams celebrate the lower bill while the actual work of discipleship grows more expensive in hidden ways. The data flatters the system until the ledger forces a different accounting.

This is the foundational misread that causes product teams to keep shipping broad agents. They optimize for volume instead of value because volume is what the invoice shows. Solomon’s judgment offers the corrective. He did not ask which woman could describe the child best. He proposed dividing the child, and the true mother’s protest revealed the cost she was willing to bear. The test separated genuine claim from expensive pretense.

The enterprise pricing slide that looked generous arrived with a familiar deck. It listed per-token rates, included volume discounts, and projected meaningful savings against current spend. The slide assumed every generated token would eventually contribute to a ministry outcome. No one in the room questioned that assumption because the pricing model rewarded total usage.

Once the team moved the same spend into an outcome ledger, the picture inverted. Tokens spent on generic sermon outlines that volunteers never opened now carried an explicit line item. Tokens spent on personalized follow-up sequences that produced actual sign-ups carried a different line item. The generous pricing suddenly looked expensive because most of the volume sat in the first category.

The first outcome ledger that changed the build decision was a single shared spreadsheet. It listed five ministry results the agent was meant to produce: volunteer lesson completion, parent re-engagement within seven days, child memory verse recitation, staff time saved on prep, and user return to the app within 48 hours. Against each result sat last month’s actual token spend, allocated by tracing logs back to the final action.

A small minority of the five results accounted for most of the spend, and the rest absorbed only a sliver. The agent had been optimized for the high-volume outputs that produced the lowest per-token cost on paper. The ledger showed those outputs rarely reached the user who needed them. The build decision flipped from expanding the agent to narrowing it.

The split between capability and cost per outcome forced a narrower agent. The original design let the model generate any content a pastor or volunteer might request. After the ledger, the team restricted the agent to three outcome paths only. Everything else triggered a handoff to a human or a simpler static resource.

The narrower agent cost more per token because the model now ran tighter constraints and extra evaluation steps. Total spend dropped because the volume of useless generations disappeared. The capability number looked worse on internal dashboards. The outcome number improved for the first time in six months.

The enterprise pricing slide that looked generous

The deck promised lower unit costs and higher throughput. It measured success by tokens processed and queries answered. Ministry leaders nodded because the math matched what they had seen in other SaaS tools.

That framing treats every query as equally valuable. In practice, most queries never produce the follow-through the ministry tracks. The pricing slide never surfaced that distinction because its metrics stopped at the API response.

Re-reading the slide against the later ledger showed the real assumption. The vendor expected teams to keep shipping the same broad agent and simply pay less for it. The ledger proved that assumption wrong.

The first outcome ledger that changed the build decision

The spreadsheet took two hours to assemble from existing logs. Each row tied a token cluster to one of the five defined results. Rows without a clear result stayed blank and therefore carried infinite cost per outcome.

Most results had zero blank rows. A couple were almost entirely blank. The team had been funding the blank rows at scale because the model was rewarded for generating anything that looked plausible.

Removing the blank rows meant removing entire features. The agent lost its open-ended content generator and kept only the paths that already produced measurable follow-through. The decision felt like a downgrade until the next month’s spend report arrived.

How the split between capability and cost per outcome forced a narrower agent

Broad capability is cheap to demonstrate and expensive to sustain once outcomes are required. The narrower agent required more prompt engineering and a small evaluation model to score each generation before release. Those additions raised the per-token price.

The same additions eliminated the generations that had previously inflated volume without producing results. Net spend fell because the expensive noise stopped. The capability metric on internal reviews dropped, yet the ministry outcomes held steady or improved.

Solomon’s test works the same way. The sword does not create value. It reveals which claim carries real cost and which claim does not. Cost-per-outcome measurement performs the same function on agent spend.

Your Turn: Apply This Today

• Export last month’s token logs and map each major cluster to one of five ministry outcomes you actually track.
• Mark every row that produced no observable follow-through and calculate its share of total spend.
• Delete or disable the generation paths responsible for the unmarked rows by Friday.
• Re-run the same month’s spend against the reduced paths and record the new per-outcome cost for each of the five results.
• Share the one-page ledger with the two stakeholders who approve the next model budget and ask which paths they are willing to keep funding.
• Schedule a 30-minute review next Friday to compare the updated ledger against the original numbers.

The same pattern appears in The Token Cost Drop That Only Appeared After We Added the Knowledge Graph and The Evaluation Layer That Broke When We Trusted the Model Alone.

I consult with product leaders on outcome-based measurement, agent narrowing decisions, and ministry cost tracking. Let’s talk.