Blog

The question I keep hearing from product teams is how to move from shipping AI features to actually building agents that improve over time. The answer is not better model selection or tighter prompt engineering. It is shifting from feature velocity to systems that accumulate team judgment.

This question surfaces because most roadmaps still treat agents as a new category of deliverable. Teams list capabilities like “handles scheduling conflicts” or “summarizes user feedback” and then measure progress by how many of those items ship. The result is agents that perform isolated tasks but never learn from the context the rest of the team already holds.

Teresa Torres’s continuous discovery framework shows why this pattern repeats. Torres argues that discovery must run in parallel with delivery, not in advance of it. When teams apply that lens to agents, the problem becomes clear: most current agent work still runs on a delivery-first cadence. The agent gets defined, built, and released before anyone checks whether the underlying assumptions about team knowledge have changed.

Why agent roadmaps keep collapsing into feature lists

Feature lists reward visible output. An agent that books rooms looks like progress even if it books the wrong room for the same reasons the last three tools did. The list does not track whether the agent knows why a certain room was rejected last quarter or which stakeholder vetoed it. Over time the agent repeats the same errors because nothing in the roadmap forces the team to record those reasons.

Continuous discovery changes the unit of work. Instead of adding another agent behavior, the team runs weekly interviews focused on recent decisions the agent should have influenced. The output is not a new feature but an updated set of assumptions that get fed back into the agent’s context layer. This keeps the roadmap from drifting into isolated capabilities.

One team I watched tried to build an agent for volunteer scheduling across multiple church sites. Their initial roadmap listed twenty behaviors. After running discovery interviews with the actual coordinators, they dropped twelve of those behaviors and instead built a simple way to capture why a volunteer was declined. The agent improved faster because the team now had a living record of past judgments rather than a growing list of functions.

Discovery loops that actually capture institutional knowledge

Most teams run discovery as a research phase that happens before build. Torres’s approach treats it as an ongoing loop that runs alongside delivery. For agent work this means the discovery questions must target how past decisions get reused, not just what users say they want next.

The practical version looks like this: every week the product manager pulls the three most recent agent failures or overrides. They sit with the people who made the override and ask what information the agent was missing. That single piece of context gets turned into a structured note that the agent can reference. Over months the notes form a lightweight institutional memory that no feature list ever captured.

Without this loop the agent stays frozen at the knowledge level of its first release. With the loop the agent slowly absorbs the pattern of choices the team has already made. The discovery work is not separate from the agent; it is the mechanism that makes the agent accumulate judgment.

The org design move most teams skip

Continuous discovery requires someone to own the loop, not just the output. Most agent teams assign ownership to an engineer or a single product manager who already carries delivery pressure. The discovery work gets deprioritized the moment shipping dates slip.

The teams that make progress create a rotating discovery owner role that sits outside the delivery sprint. That person runs the weekly interviews, maintains the decision notes, and updates the agent’s context rules. Delivery still happens, but the discovery owner protects the accumulation of judgment from getting lost in velocity metrics.

This role does not require a new headcount. It requires a temporary shift in how existing time gets allocated. One team I worked with gave the discovery owner four protected hours each week. Those hours produced more lasting agent improvement than the previous three months of feature additions.

Your Turn: Apply This Today

• Pick the agent your team ships most often and list the last five times a human overrode its output.
• Schedule three thirty-minute conversations this week with the people who made those overrides; ask only what context the agent lacked.
• Turn the clearest missing piece from each conversation into a one-sentence note and add it to whatever context store the agent already uses.
• Assign one person on the team to own the weekly override review for the next four weeks and protect those four hours on their calendar.
• Measure whether the same override pattern appears again after the notes are added; track it for two releases.
• Write a short internal note on what the agent now handles differently and share it with the broader product group.

Two earlier posts on the same thread walk through how judgment shows up in product systems and why most discovery work fails to stick once delivery pressure returns.

I consult with product leaders on AI agent roadmaps, continuous discovery loops, and building systems that accumulate team judgment. Let’s talk.

The captain texted at 9:47. None of the three families had heard a thing about tonight. The agent had already marked the shift complete, names in a row, times locked in. But the one person who used to know which mom forgets her phone in the car and which dad needs a second nudge never got the hand-off.

I kept staring at the list like it should explain itself. It didn’t. The work got assigned, the thread just disappeared.

Captains Without a Rulebook

The scheduling agent you deployed does not carry Wesley’s first rule. It books a volunteer who has already told three people she cannot serve that week because the agent only sees the calendar field, not the conversation that happened after the last service. The captain who used to catch that detail no longer sees the assignment until the volunteer shows up confused or angry. Harm happens quietly, one missed text at a time.

The second rule, do good, requires the captain to notice when a family needs an extra phone call or a different role. The agent optimizes for coverage, not care. When the captain stops making those small interventions, the families who needed them drift. The system reports high fill rates while the actual ministry weakens.

The third rule is the one most teams ignore. Staying in love with the work means the captain keeps a living record of why certain volunteers are placed where they are. That record lives in the captain’s head and in a few scribbled notes. When the agent takes the scheduling task, the captain stops writing the notes. Three months later no one can explain why the rotation changed or who promised to train the new helper. The knowledge does not travel because the person who held it no longer has a reason to keep it.

The Knowledge OS That Actually Travels

Wesley’s classes worked because every leader performed the same three actions in the same order every week. The repetition created a portable operating system. A new leader could step in and know exactly how to begin because the pattern was visible and personal.

When you remove the personal loop, the pattern disappears. The agent produces schedules faster than any captain could, but it does not produce the weekly habit of checking in with the person who just served. That habit is what turns one captain’s experience into the next captain’s starting point. Without it, each new leader starts from zero even though the agent has perfect historical data.

The same pattern shows up in the children’s ministry curriculum work I watched last year. The team that kept a captain responsible for reviewing every volunteer’s completed lesson sheet before the next week retained 40 percent more volunteers than the team that let the platform send the next lesson automatically. The difference was not the content. It was the captain who still had to sign off and therefore still had to know what had actually happened.

Why Agent Speed Breaks the Hand-off Chain

Agent speed removes the friction that once forced the captain to touch the schedule. That friction was the hand-off. When the captain had to open the sheet, review the names, and send the reminder herself, she noticed the gaps. When the agent sends the reminder, the captain’s attention moves to the next task. The knowledge that used to move forward with the schedule now stays in the agent log that no one opens.

The break compounds across reorgs. A new ministry director arrives and asks why certain volunteers keep getting skipped. The old captain has already moved on. The agent has no record of the conversation that happened in the hallway after the service two years ago. The only remaining trace is a blank field that used to hold a captain’s note.

Teams that keep the captain accountable for the loop, even after the agent takes the mechanical work, preserve the chain. The captain still reviews the agent’s output every Monday, still calls the one name that does not match the pattern, and still writes the single sentence that explains the change. The agent handles volume. The captain carries the rule.

Your Turn: Apply This Today

• Pick the single scheduling agent output that lands on your desk this week and add a mandatory captain review step that must be completed before any reminder goes out.
• Write the three Wesley rules on a single index card and tape it to the monitor the captain uses when checking the agent’s list.
• Require the captain to add one sentence to a shared note after each review explaining one change the agent would not have caught.
• Block thirty minutes on the captain’s calendar every Monday for that review and treat the block as non-negotiable for the next four weeks.
• At the end of week four, ask the captain to forward the accumulated notes to the person who will hold the role next quarter.
• Remove the agent’s ability to send reminders directly to volunteers until the captain has logged the review.

The posts “The Agent That Kept Running the Schedule While No One Was Watching” and “The Ministry PM Who Handed Off Captain Duties Without the Handoff Log” trace the same pattern in two other teams.

I consult with ministry product leaders on agent hand-off loops, captain accountability systems, and knowledge retention after automation. Let’s talk.

I shipped the refactor on a Tuesday afternoon. Two days of prompts and quick reviews, and the whole module sat there cleaned up, tested, ready. By Wednesday morning the first approval request had already bounced back with a question from someone two layers up who’d never touched the repo.

That was the part nobody warned us about. The code moved fast. Everything else moved like it always had—slow, cautious, built for a world where changes arrived one at a time and every line carried human weight. Now the lines arrive in batches, and the weight still has to be carried the old way.

Teresa Torres’s continuous discovery framework treats opportunity identification and solution testing as ongoing activities rather than staged events. Applied here, it reveals that verification gates function as the new discovery layer. When teams keep generation fast but leave validation infrequent and heavyweight, the system simply moves the bottleneck downstream without removing it.

The Number Everyone Celebrates

The two-day refactor number gets reported in sprint reviews and leadership updates. It reflects actual engineering hours spent on code changes. What the number hides is that the same teams now generate three times the volume of proposed changes per cycle.

That volume lands in queues designed for slower, human-authored diffs. Reviewers face context switching across dozens of small, AI-suggested edits rather than a handful of deliberate ones. The celebration of the initial number prevents anyone from measuring the cumulative time spent re-explaining intent across each verification step.

Ministry platforms that produce children’s curriculum or sermon outlines see the same pattern. A volunteer-facing lesson set that once took a content editor two weeks now reaches first draft in three days. The subsequent review with doctrinal reviewers, accessibility specialists, and print production still consumes the original calendar time because no one adjusted the handoff criteria.

Where the Six Weeks Actually Hide

The delay lives in repeated clarification loops rather than in the code itself. Each approver asks questions that the original prompt to the AI never surfaced: edge cases for age groups, regional translation needs, or integration with existing print workflows. Because the generation step moved fast, these questions arrive late and in batches.

Continuous discovery, in Torres’s terms, would have surfaced those constraints through regular exposure to users and stakeholders before the refactor began. Instead, the fast generation step creates the appearance that the work is mostly complete. The verification gate then becomes the place where discovery finally occurs, but without the lightweight interview cadence that makes discovery cheap.

One platform that manages global sermon resources found that legal and translation sign-offs now require separate meetings for each AI-generated variant. The meetings themselves had not changed frequency; only the number of variants had increased. The six weeks therefore represent accumulated coordination cost, not review effort per se.

Redesigning the Verification Gate

The fix begins by moving small validation checks earlier and more often. Rather than waiting for a complete refactor to reach the full approver list, teams can run targeted opportunity solution trees with the actual users of the output—volunteers, pastors, or translators—while generation is still happening. This turns verification from a final gate into a running discovery process.

Concrete changes include shortening the artifact that enters review. Instead of shipping an entire refactored module, teams surface one user-facing difference at a time through a simple prototype or print sample. Approvers respond to the observable impact rather than the full technical diff. This matches Torres’s emphasis on testing solutions against real opportunities instead of internal completeness criteria.

Another adjustment is to embed the same people who will ultimately sign off into the prompt refinement stage itself. When a doctrinal reviewer sees the initial prompt and flags a constraint before generation runs, the subsequent cycle avoids the six-week surprise. The calendar time shrinks because the verification criteria travel with the work rather than arriving after it.

Teams that adopted this pattern on curriculum tools reported that total elapsed time dropped from six weeks to under three, not because generation got faster but because the verification criteria became explicit and stable earlier in the loop.

Your Turn: Apply This Today

• Map the exact sequence of people and artifacts that must approve any AI-generated change in your current workflow, listing each step by name and average days elapsed.
• Pick the longest single verification step and run a 30-minute conversation with that approver this week focused only on the three constraints they care about most.
• Insert one lightweight check—such as a shared prompt log or single-user test—before the refactor reaches the full gate, then measure whether the later step shortens.
• Reduce the review artifact size by half for the next cycle and require approvers to respond to observable user impact rather than technical completeness.
• Schedule a recurring 15-minute slot with one downstream stakeholder to review emerging prompts rather than finished outputs.
• Document the revised gate criteria in a single shared paragraph and circulate it before any new generation task begins.

The same verification trap shows up in the workflow described in The Refactor That Finished in Two Days and Then Sat for a Week and in the coordination overhead tracked in Faster Code Does Not Move the Real Ministry Bottleneck.

I consult with product leaders in faith-tech organizations on verification workflows, AI output governance, and shifting from generation speed to discovery loops. Let’s talk.