Blog

A product leader asked me last month how to decide what parts of the job to hand off to agents now that every promotion comes with an agent mandate. The answer is simple but rarely stated: keep the judgment calls where the downside lands on people you know by name, and let the models handle the rest. Everything else is secondary. The old playbook never prepared anyone for this question.

Charlie Munger spent decades telling investors to stay inside their circle of competence. The rule was not about avoiding hard work. It was about refusing to make calls when you could not see the full set of consequences. In product work that rule now collides with live agents that produce first drafts at scale. The circle does not shrink because the tools got faster. It moves because the cost of being wrong now hits real users and real teammates before you finish reviewing the output.

The promotion the old playbook never described

The title change arrived with a new expectation. I was told the team would run on agents for initial scoping, first-pass requirements, and early user-story drafts. The headcount stayed flat, but the mandate was to ship more surface area with the same number of humans. The old promotion playbook measured scope by how many features I could personally shepherd. The new one measured it by how many decisions I could safely review after an agent had already moved them forward.

At first the arrangement felt efficient. The agents produced clean documents that looked ready for stakeholder review. Then a set of stories reached the volunteer build team for children’s ministry curriculum. The agent had optimized for completion rate metrics that made sense on paper but ignored the seven-minute volunteer who prints the lesson on Sunday morning and discovers the activity requires materials that are not in the supply closet. The error was small on screen and expensive in real rooms. That was the moment the circle of competence stopped being a theory.

Where the old playbook breaks once agents write the first draft

Munger’s test was never whether you could understand the output. It was whether you could still trace the second-order effects on the actual people who would use it. When agents write the first draft, that trace becomes harder because the draft already carries the appearance of finished work. The temptation is to treat the model’s version as the default and only edit around the edges. The boundary moves inward the moment you accept that default.

I watched the same pattern on two other teams. One handled global discipleship tools where traffic numbers are easy to celebrate. The agent suggested a flow that increased session time for logged-in users but quietly dropped first-time visitors from low-bandwidth regions. The other team managed sermon resources for pastors who prepare on Friday night. The agent optimized for search ranking and removed the plain-text print option that those pastors relied on when their internet dropped. In both cases the humans reviewing the work had never held the downstream roles themselves. The competence gap was not technical. It was relational.

The two questions the old playbook left out

Research like Harvard Business Review on AI and product roles confirms the old playbook never accounted for managing agents that draft the work.

The first question is whether I can name the three people who will feel the downside if this decision is wrong. If I cannot name them, I do not own the call. The second question is whether those same three people would recognize the failure mode inside the first five minutes of using the product. If the answer is no, the scope stays with me until I can answer both questions clearly. These two questions have become the practical edge of my circle.

Applying them has changed what I accept. I still review agent output, but I now require a short “who gets hurt” paragraph written by a human who has done the actual job the agent is modeling. The paragraph must be attached to every story that reaches review. It is not a process document. It is the minimum signal that the judgment call has not yet been outsourced.

Your Turn: Apply This Today

• Block thirty minutes this week and write down the three judgment calls you currently own that you would refuse to hand to an agent even if the output looked perfect.
• For each of those three calls, name the two people downstream who would absorb the cost if the call is wrong.
• Write one sentence for each person describing how they would first notice the failure in their actual workflow.
• Schedule a fifteen-minute conversation with one of those people and read the sentence out loud to confirm it matches their experience.
• Add a standing rule in your next planning doc that any agent-generated story must include a “who gets hurt” paragraph before it reaches review.
• Review the last five stories your team accepted from agents and mark which ones would have failed the two questions above.

The PM Who Just Inherited the Fable Agent Mandate and The Compensation Split No Playbook Prepared PMs For both trace the same boundary problem from different angles.

I consult with product leaders and ministry technology teams on agent scope decisions, judgment call ownership, and downstream user impact reviews. Let’s talk.

The proven-better pattern approach, drawn directly from the scaling tactics in books like “Inspired” by Marty Cagan, directs teams to locate high-performing flows from mature products and transplant them into smaller offerings. The method treats completion rates and delight signals from large user bases as reliable signals worth replicating. It promises faster iteration by skipping the need to invent from scratch. The proven-better pattern can still miss the people you most want to serve.

This method breaks for ministry tools aimed at volunteer-led congregations because the original delight signals emerged under conditions of paid staff, reliable devices, and repeated weekly exposure. A pattern built for daily app users cannot assume the same friction profile when the user opens the tool once a month on a shared tablet during set-up time. The transplant therefore lands without evidence that the underlying job still exists in the new setting.

The result is shipped features that look finished on internal demos yet produce zero measurable movement in the actual retention metric that matters for these teams: whether the volunteer completes the task before the first child arrives.

This is the foundational misread that causes product teams to treat proven patterns as context-neutral assets rather than hypotheses that still require validation. The pattern itself is not the problem. The decision to bypass the step that checks fit is.

Teresa Torres’s continuous discovery habits supply the missing lens here. Torres insists that discovery work must continue after a pattern has shown results elsewhere. The habits force teams to test the assumption that the same outcome will appear in a different environment rather than declaring the pattern “proven” once it succeeds at scale.

Where the proven-better pattern first led me wrong

I took a three-screen welcome sequence that had lifted activation numbers in a consumer reading app and dropped it into a planning tool for children’s ministry volunteers. The sequence used progress bars, a quick profile photo upload, and a one-tap “start your first lesson” button. Internal testing showed the flow completed in under ninety seconds.

When the first twenty-person church installed the update, the photo upload step failed on the shared device because the tablet camera permission had been locked down by the church IT volunteer. The progress bar never advanced for anyone who reached that screen. Completion rate for the new onboarding dropped to 12 percent within the first month.

The copy had preserved every pixel of the original pattern. It had preserved none of the conditions that made the pattern work.

Why the proven-better pattern missed the smallest churches

The original pattern produced delight because users returned daily and the app could remember their last state across sessions. In the small church setting the tool is opened once, used for ten minutes, and then closed until the next rotation of volunteers. No memory of prior sessions survives.

The progress bar that signaled momentum to a daily user now signaled an unfinished task to a volunteer who would not return for four weeks. The metric that looked like delight in the source context became an abandonment signal in the new one.

Teams measuring only the source metric never see this reversal. They ship the pattern and move to the next proven item on the list.

The discovery interview the proven-better pattern skips

A single thirty-minute conversation with a volunteer coordinator from a rural church would have surfaced the camera permission constraint and the four-week gap between uses. That conversation never occurred because the pattern had already cleared internal success criteria at the source company.

Torres’s continuous discovery habits require exactly this interview even after the pattern has data behind it. The habit is not additional work; it is the work that prevents the pattern from becoming expensive technical debt in the new context.

Your Turn: Apply This Today

• Pick the smallest user segment your product currently serves and pull the last three patterns you copied from larger products.
• Schedule one thirty-minute call this week with a single user from that segment; ask only what changed the last time they tried to complete the flow.
• Record the exact constraint they name that did not exist in the original pattern’s environment.
• Write one sentence that states whether the copied pattern still solves the job under that constraint.
• Remove or replace the pattern element that fails the test before the next sprint planning meeting.
• Repeat the interview with one new user from the same segment every month for the next quarter.

The same continuous discovery habit that would have caught the onboarding mismatch appears again in “Trust Logs Beat the Next Feature Sprint” and “The v1 Where the Model Handed Me the Safe Path.”

I consult with product leaders and ministry technology teams on continuous discovery for small-segment users, pattern transplantation risks, and retention metrics that survive volunteer constraints. Let’s talk.

I spent eight months sending every product spec to the largest model we could afford. The assumption was simple: more parameters meant fewer gaps for the volunteer teams who would later implement the feature. Prayer request flows, curriculum upload paths, even the small print-first forms for Sermons4Kids all went through the same pipeline. Output looked finished. Token counts stayed under budget. Launches still slipped because the teams had to rebuild the exact judgment calls the model had flattened. What finally fixed it was a set of routing rules, not a bigger model.

The cost showed up in volunteer completion rates. A children’s ministry coordinator would open the generated story, stare at the polished steps, and realize the model had removed the three places where a tired volunteer needed to decide whether to skip, adapt, or hand off. Those decisions never appeared in the original ticket. They reappeared as rework after the first test with real users.

This pattern is the foundational misread that causes product teams to treat model scale as a substitute for visible ownership. The bigger model does not eliminate judgment work; it buries it inside language that feels authoritative. Teams then discover the missing pieces only when the artifact reaches the people who actually run the program on Sunday morning.

John Wesley’s three rules offer a practical lens here. Do no harm. Do good. Stay in love with the disciplines that keep the work honest. Applied to model routing, the first rule means refusing any output that hides the exact decisions volunteers must still make. The second rule means routing only the portions of work that genuinely reduce repetition without erasing context. The third rule means keeping a running record of what each tier of model forces the team to restore by hand.

When Routing Rules Were Missing for a Prayer Request Flow

The spec described a simple intake form that let parents submit a request and receive a follow-up from a small-group leader within forty-eight hours. Fable 5 produced ten acceptance criteria, four edge-case branches, and a suggested notification schedule. Every sentence read as if someone had already tested it with real families.

The ministry lead opened the ticket and immediately flagged three problems the model had resolved without stating its assumptions. It had decided that every request would route to the same leader pool. It had assumed the parent would receive an automated confirmation that included the leader’s name. It had removed the step where the volunteer could mark a request as “needs pastoral review” without logging a separate ticket.

Those three decisions were the actual product. Restoring them took longer than writing the original spec because the language in the ticket made the choices look settled. The team had to reopen the conversation with the volunteer coordinators to learn what they actually needed to retain control over.

The Rewrite Cost Routing Rules Would Have Caught

Every time we routed a full artifact through the largest model, the rewrite work happened after the ticket reached the volunteer side. The model produced clean flows that assumed perfect information and consistent volunteer availability. Real ministry calendars contain neither.

One Sermons4Kids update required the volunteer to choose between printing a single sheet or a full lesson packet depending on how many kids showed up that week. The model had defaulted to the full packet because that option looked more complete. The volunteer had to add back the decision point and the accompanying print instructions. That single addition required three rounds of internal review because the original ticket no longer contained the rationale.

The token bill recorded only generation time. It never captured the hours spent reintroducing the places where a human had to exercise judgment under time pressure. Those hours appeared later as missed launch dates and lowered volunteer retention metrics.

Routing Rules That Actually Kept the Work Visible

Anthropic’s guidance on choosing the right model for the task reinforces why routing rules beat defaulting to the biggest model every time.

We eventually replaced the single-pipeline approach with three explicit tiers. Tier one handled only repetitive formatting and placeholder text. Tier two generated options with the decision points left explicit and bracketed. Tier three was reserved for narrow slices where the model had already demonstrated consistent accuracy against logged volunteer feedback.

Each tier carried a required log entry: what the model produced and what the team had to restore before the artifact reached the volunteer test. The log lived in the same ticket system the ministry teams used, not in a separate research document.

The discipline forced us to notice when the largest model was smoothing over exactly the friction that made a feature usable by a seven-minute volunteer. It also made the cost of each routing choice visible to the rest of the product group before the next planning cycle.

Your Turn: Apply This Today

• Pick the next three product artifacts on your roadmap and assign each one a model tier before any generation begins.
• For the artifact assigned tier one, log every sentence the model adds that removes a volunteer decision point.
• For the artifact assigned tier two, require the team to restore at least two explicit judgment steps before the ticket moves to volunteer review.
• After the third artifact ships, compare the three logs and mark which tier produced the highest volume of hidden rewrite work.
• Share the marked log with the two volunteer coordinators who will actually use the feature and ask them to confirm the restored steps match what they need.
• Update the routing rules document with the single sentence that best describes when tier three is now off-limits for your current volunteer contexts.

The v1 Where the Model Handed Me the Safe Path and Trust Logs Beat the Next Feature Sprint both trace the same pattern of hidden judgment costs across different product surfaces.

I consult with product leaders and ministry technology teams on explicit model tiering, restoring volunteer judgment points, and keeping rewrite costs visible in the ticket system. Let’s talk.