An 8x reduction in inference costs showed up in the first quarter after several faith-tech teams shifted to open-weight models. The number came from pooled benchmarks on common English queries, yet the same dashboards revealed that low-volume requests from non-English ministry contexts now routed through multiple fallback paths and produced effective costs two to three times higher than the prior proprietary baseline.
The reversal sits in the routing layer, not the model weights. Once cost became the primary signal for model selection, teams stopped measuring whether a particular workflow ever completed for the users who needed it most.
This pattern repeats because product groups treat the headline metric as stable across contexts. The assumption collapses once routing rules begin prioritizing volume and latency over coverage.
Teresa Torres’s continuous discovery approach names the mistake directly. Teams that stop interviewing users and observing workflows after the initial model switch lose sight of how inference decisions actually shape which ministries can participate. Continuous discovery requires ongoing contact with the real sequence of steps a volunteer or pastor takes, not just the token counts logged in the backend.
The Cost Shift That Teams Celebrated Too Early
Early wins came from high-frequency sermon-prep queries in English. Routing logic sent these to the cheapest available open-weight model and reported the expected savings. The same logic, however, sent queries in Spanish, Swahili, or regional dialects to secondary paths because the primary model lacked sufficient training data.
Those secondary paths often invoked a larger model or an external service, wiping out the per-token advantage. Ministry leaders running children’s curriculum in those languages saw longer wait times and occasional failures without any visible change in the cost dashboard.
The 8x figure therefore described only the subset of traffic that already matched the model’s strengths. Everything else moved into a hidden cost bucket that grew as more teams adopted the routing pattern.
Data Residency Decisions That Now Sit With Product
Routing choices now determine where user prompts are stored and processed. Some open-weight endpoints keep logs in regions that meet basic compliance but lack the data-processing agreements required by certain mission organizations.
Product teams that previously deferred residency questions to infrastructure now face them weekly. A routing rule that favors the lowest-cost provider can send a pastor’s draft sermon outline across borders without explicit consent, creating friction that discovery interviews later surface as reluctance to adopt the tool.
Continuous discovery surfaces these constraints when product managers sit with users while they prepare materials rather than reviewing aggregated logs after the fact. The residency decision stops being an infrastructure detail and becomes a product constraint that affects who can safely use the system.
Tying Routing Logs Back to One Observable Workflow
Most dashboards still separate cost data from workflow completion data. A single children’s ministry volunteer preparing a lesson may trigger three distinct inference calls across two models, yet the system reports only the summed token count.
When logs are joined to the actual sequence—select curriculum, adapt for age group, print handout—the product team sees which routing decisions cause the volunteer to abandon the task. The 8x savings on the first call become irrelevant if the third call fails or delays the print step beyond the seven minutes the volunteer has available.
Reconnecting logs to the observable workflow is the minimum requirement for Torres-style discovery. Without that link, cost metrics continue to celebrate savings that never reach the users whose contexts differ from the benchmark set.
Your Turn: Apply This Today
- Select one agent workflow your team currently runs and list every inference call it triggers in order.
- Tag each call with the exact user context it serves, including language and ministry type.
- Calculate the real cost per completed workflow, not per token, using last week’s actual traffic.
- Identify the single routing rule that most often diverts low-volume contexts to higher-cost or lower-quality paths.
- Schedule three 20-minute calls this week with users who match the diverted contexts and watch them run the workflow live.
- Adjust the routing rule for one context based on what the calls reveal, then measure completion rate and effective cost for the next seven days.
The Quality Erosion No 8x Claim Ever Mentions and The PM Who Inherited the 8x Target both trace how headline metrics drift from actual ministry outcomes once routing and residency choices accumulate.
I consult with product leaders in faith-tech on inference routing decisions, data residency constraints for ministry tools, and continuous discovery practices that reconnect logs to observable workflows. Let’s talk.