Blog

A product manager asked me last month how to respond when an agent begins pulling pricing data from their site. The answer is that machine-readable pricing pages only matter if you still own the last-mile decision about whether that price reflects your actual constraints and values.

Jensen Huang has argued that companies must maintain sovereign control over their AI infrastructure rather than cede it to external providers. The same principle applies here. Pricing data becomes another surface where control can slip away if the final judgment about what a number means stays with the model instead of the team that set it.

The question every PM faces is not whether to publish pricing in a format agents can read. It is whether the organization keeps the authority to override or reinterpret that data once it leaves the page.

The real question behind the .md pricing file

Most teams publish pricing in markdown or structured JSON because an agent asked for it or because leadership wanted to look forward-leaning. The file itself is not the problem. The problem is that the file usually contains only the public numbers without the internal logic that produced them.

I watched one ministry platform publish tiered pricing for volunteer curriculum access. The .md file listed the rates clearly. Three weeks later an agent surfaced the lowest tier to a small church in a high-cost region and auto-generated a pilot proposal. The team had not encoded the rule that discounted rates required a minimum active volunteer count. The agent had no way to know the constraint existed.

The file answered what the price was. It never answered why that price existed for that user at that moment. That gap is where teams lose ground.

Where sovereign control breaks when agents pull the data

Sovereign control fails at the handoff point between data retrieval and decision. Once an agent extracts pricing, the next step is usually a recommendation or a generated offer. If the recommendation engine sits outside the product team, the original constraints travel only as far as the file format allows.

Huang’s point about sovereign AI is useful here because it separates ownership of the model from ownership of the outcome. A team can host its own pricing model or fine-tune an open one, yet still lose control if no one reviews the output before it reaches a pilot user. The model runs, the price appears in a proposal, and the organization discovers the mismatch only after the user has already seen it.

This pattern shows up in both commercial and ministry settings. A curriculum product I worked with let an agent suggest annual licenses to children’s ministry volunteers. The agent used public rates without checking that most volunteers needed print options that changed the effective cost. The pilot launched with incorrect expectations and required three weeks of manual correction.

The ownership step that survives automation

The step that survives is the internal gate that sits between agent output and any external user. This gate does not need to block every action. It needs to surface the pricing data alongside the original constraints so a human can confirm the match before anything ships.

In practice this means every agent-generated pricing suggestion carries a short record of the rules that applied at the time of retrieval. The record travels with the suggestion. A reviewer checks the record against the current context in under two minutes. If the context has shifted, the suggestion stops.

The gate does not slow velocity when the rules remain stable. It only activates when the situation deviates from what the file alone can express. Teams that build this step early keep the last-mile decision inside the organization even as agents handle more of the retrieval work.

Your Turn: Apply This Today

• Export your current pricing constraints into a one-page internal document that lists every rule an agent cannot infer from the public file alone.
• Build a simple review queue that captures every agent-generated pricing suggestion before it reaches a test user or pilot.
• Assign one PM to spend thirty minutes each Friday checking the queue against the constraint document and noting any mismatches.
• Add a required field in the queue for the original pricing rule that the agent would have needed to know.
• Run one existing pilot through the queue retroactively to surface any pricing decisions already made without the gate.
• Schedule a thirty-minute meeting with the agent owner to agree on the exact format the constraint record must take.

Two earlier posts explored adjacent problems. Letter to the PM Handed an AI Agent Budget looked at budget ownership after agents enter the workflow, and The PM Who Stopped Writing Specs and Started Shipping Prototypes examined how to keep decision rights when automation speeds up delivery.

I consult with product leaders on agent integration, pricing data ownership, and sovereign control in product decisions. Let’s talk.

Teams running automated LLM judges on ministry content report 78 percent agreement with human preferences on standard benchmarks. The same teams, when forced to score outputs against the exact prompts their volunteers actually submit, see agreement fall to 38 percent. The obvious reading—that the models are mostly reliable and just need occasional human spot-checks—is backwards. The models perform well only on the narrow slice of tasks that already match the generic training data; they diverge sharply on the messy, context-heavy work that defines real ministry output.

That divergence is the signal. Teresa Torres’s continuous discovery framework treats every production artifact as a fresh opportunity to interview the user, not a finished deliverable. Applied to AI features, it means the evaluation step cannot be automated away once and for all. The benchmark must remain a living record of how actual users phrase their requests and what “good enough” looks like in their setting.

Why the 45-minute human eval cut through the noise

The first time we froze a set of real volunteer prompts and scored the model outputs by hand, the process took forty-five minutes for a batch of twelve items. That single run surfaced three failure modes the LLM judge had rated as passing: a children’s lesson that used vocabulary three grade levels too high, a small-group guide that assumed the leader already knew the theological term “sanctification,” and a prayer prompt that ended with a corporate-sounding closing no volunteer would ever say aloud.

The 45-minute constraint mattered. It forced us to score quickly and against a frozen input list rather than rewriting prompts on the fly. Speed kept the exercise from turning into another round of prompt engineering. The fixed inputs kept the comparison honest; every model variant faced the same volunteer language, not an improved version of it.

Most teams skip this step because it feels slower than letting the model judge itself. The data shows the opposite result: the human pass reveals the exact places where the model’s average-case training collides with the long-tail reality of ministry work. Once those collisions are written down, later automated checks can be calibrated against them instead of generic rubrics.

How the same workflow applies to baptism outlines and photo policies

The same frozen-input method works for tasks that never touch a sermon manuscript. One ministry team ran it on baptism service outlines. They captured the exact wording parents used when requesting date options, then scored the generated outlines for pastoral tone and legal clarity. The LLM judge had passed every sample. The human review flagged two outlines that omitted the requirement for a follow-up visit with the family—something every real coordinator treats as non-negotiable.

Another team applied it to photo policies for children’s events. They froze twelve real parent emails asking whether photos could be posted on the church app. The model outputs sounded reasonable until scored against the actual policy language the church had already published. Three outputs used phrases the policy explicitly prohibited, yet the automated judge had scored them as compliant.

In both cases the 45-minute human pass did not replace the model; it created a narrow, reusable test that later model versions could be run against. The test stayed small enough to repeat after every major model update without requiring a full product review cycle.

The checkpoint that keeps the benchmark alive after launch

Continuous discovery requires a recurring trigger, not goodwill. The teams that maintained the benchmark after launch tied it to an existing release gate rather than adding a new meeting. They added one line to the pre-launch checklist: “Run current frozen test set; record any score drop greater than 10 percent.” The line took less than five minutes to verify once the test set existed.

When the score did drop, the team did not rerun the full 45-minute exercise immediately. They first checked whether the drop came from changed volunteer language or from model drift. Only the latter triggered a fresh human scoring round. This kept the cost bounded while still surfacing the moment when the original benchmark stopped reflecting reality.

The checkpoint also surfaced when the product itself had changed the volunteer task. One team noticed their new intake form collected an extra field; the frozen prompts no longer matched how volunteers now described the request. Updating the frozen set became part of the same release process rather than a separate research project.

Your Turn: Apply This Today

• Pick one volunteer task that already has a written prompt or email template—children’s lesson requests, baptism date inquiries, or photo permission language.
• Copy the last ten real submissions exactly as they arrived, including typos and abbreviations, into a frozen list.
• Run your current model or agent against that list and save the outputs with no further prompt changes.
• Score each output yourself or with one volunteer using a simple three-point scale: usable as-is, needs light edit, or unusable.
• Write the two most common failure reasons on a single index card and tape it to your monitor.
• Schedule a five-minute calendar block the week after your next model update to rerun the same frozen list and note any score change.

The pattern in “Letter to the PM Handed an AI Agent Budget” and “The Bias That Quietly Killed Three Ministry Pilots” is the same: teams that treated evaluation as a one-time gate later discovered their models had drifted from the actual work. The same drift appears here once the human benchmark is allowed to go stale.

I consult with product leaders and ministry technology teams on frozen-input evaluation design, continuous discovery checkpoints, and volunteer-task benchmarks. Let’s talk.

I once spent nine months building an AI sermon-prep assistant that scored 92 percent on internal accuracy benchmarks. I assumed the quality metric would carry it across the finish line into volunteer workflows. Three pilots later the tool sat unused in two of the churches and barely touched in the third, while the teams reverted to their old copy-paste routines. The cost was not just the wasted engineering hours; it was the lost credibility with ministry leaders who had cleared their calendars for the pilot and then watched nothing change in their actual week.

That outcome forced me to admit the assumption I had carried into every ministry AI project up to that point: if the model performed well in isolation, the rest would sort itself out. The assumption was convenient because it kept the scope of my work narrow. It also proved expensive each time distribution, positioning, and the handoff to non-technical users were left as afterthoughts.

How the bias showed up in three pilots I ran

The first pilot placed the assistant inside a children’s ministry platform similar to the one I helped shape for Sermons4Kids users. The model could generate story outlines in seconds. Yet the volunteer onboarding flow still required a logged-in account, a desktop browser, and a five-step export to the print queue the volunteers actually used on Sunday morning. After six weeks the completion rate for the AI-generated outlines sat at 11 percent while the older static curriculum pages held steady at 68 percent.

In the second pilot, aimed at small-church pastors, we positioned the tool as an “AI co-preacher” inside an existing sermon resource site. The language in the marketing email framed it as a breakthrough. Pastors who clicked through landed on a clean interface that still demanded they learn a new prompt syntax and then manually reconcile the output with their denominational review process. Two months in, only four of the thirty invited pastors had run the tool more than once.

The third pilot tried to shortcut distribution by embedding the model directly into an existing church management system. The integration passed every technical test. What the logs later showed was that the handoff step—moving an AI-generated pastoral care note from the system into the actual follow-up workflow—still required a separate spreadsheet the staff kept on a shared drive. The model stayed technically sound and practically invisible.

Each time the pattern repeated: strong model performance inside the sandbox, weak or nonexistent movement once the artifact reached the person who had to act on it by nine o’clock Tuesday morning.

Munger’s model that forces the other factors into view

Charlie Munger described a latticework of mental models as the habit of pulling ideas from multiple disciplines so no single narrow frame dominates the decision. The quality-only frame is itself a model—one borrowed from software engineering where a better algorithm often does win. Ministry workflows, however, sit at the intersection of volunteer time scarcity, print-first constraints, and relational trust that must survive a failed output.

When the latticework includes Munger’s own inversion model—ask what would guarantee failure—the missing pieces surface quickly. An AI tool fails in a church not because its perplexity score is too high but because the distribution path never intersects the actual decision point, the positioning triggers the wrong mental category (“another dashboard”), and the handoff leaves the non-technical user holding raw text with no next action defined.

Applying the latticework means treating distribution, positioning, and handoff as first-order variables measured with the same rigor once reserved for model accuracy. It also surfaces second-order effects: a poorly positioned tool burns relational capital with gatekeepers who then block later experiments, even better ones.

What changes when you stop optimizing the model first

The shift begins with mapping the exact physical or digital surface where the output must appear. In the children’s ministry case that surface was the single sheet of paper handed to a volunteer at 8:15 a.m. Once that surface was named, the pilot moved from prompt engineering to a print-queue integration that cut three clicks and removed the login requirement.

Positioning changes next. Instead of announcing an “AI co-preacher,” the revised framing described a “first-draft generator that still needs your voice.” The language matched the existing mental model pastors already held about sermon prep and lowered the activation energy required to try it.

Handoff receives the same concrete treatment. The pastoral care pilot added an explicit “copy to follow-up list” button that wrote directly into the spreadsheet already in use. Adoption among the same staff group moved from near zero to consistent weekly use within four weeks.

The model itself was not degraded; its scope simply stopped expanding until the surrounding lattice was in place. Accuracy became a supporting metric rather than the lead indicator.

Your Turn: Apply This Today

• Open your current AI experiment’s project doc and list the three non-quality factors—distribution path, positioning language, and handoff step—on a single line each.
• For the distribution path, write the exact URL, email subject line, or physical handoff the end user will encounter first and schedule a test with one actual user this week.
• For positioning, draft the one-sentence description you will use in the first message to that user and run it past a non-technical colleague for clarity before sending.
• For the handoff, identify the exact next artifact the user must produce after seeing the AI output and add one automated or one-click bridge to that artifact.
• Log the three factors plus the corresponding test steps in a shared note visible to the full pilot team by end of day tomorrow.
• Review the log in your next stand-up and mark which factor moved closest to the end user’s actual workflow surface.

The same lattice that exposed the bias in my earlier pilots now shapes every new experiment. Two recent posts on the blog explore adjacent angles: “The Bias That Quietly Killed Three Ministry Pilots” and “The Routing Layer No Ministry Pilot Logged”.

I consult with product leaders and ministry technology teams on AI pilot design, distribution mapping, and non-technical user handoffs. Let’s talk.