You’re the product manager who just inherited the 8x target for an AI sermon-prep tool inside a mid-sized denomination. The previous lead left three weeks ago. The only brief you received was a slide deck that promised the model would cut prep time for volunteer coordinators from forty minutes to five. No one on the pastoral staff has used an LLM for anything beyond a quick email draft. Before you touch the model, the job you actually inherited is AI agent oversight: deciding what the agent may do alone and where a person must sign off.
That target sits on your desk next to a login for an internal instance of Claude and a folder labeled “prompt library v2.” Your first instinct is to open the folder and start testing.
This is the foundational misread that causes product teams to ship agents that look productive in demos and then quietly fail on Sunday morning. The mistake is treating the model as the primary actor and the human as a reviewer who will simply catch whatever slips through. Teresa Torres’s continuous discovery framework shows why this breaks. Torres insists the team must keep returning to the real user in context, updating the problem statement each time the solution changes. When the solution is an agent, the problem statement must include the exact moment the work leaves the agent and returns to the person who still carries the outcome.
You cannot discover that moment by iterating on prompts alone. You discover it by watching the handoff itself.
AI Agent Oversight Starts With a Written Job Description
The first artifact you build should not be a system prompt. It should be a one-page job description for the agent that names the exact deliverable, the exact constraints, and the exact person who will sign off. At Sermons4Kids we learned this the hard way when we tried to automate volunteer curriculum summaries. The model produced clean text, yet the volunteer still had to rewrite every summary because the tone did not match the actual kids in their room. We had optimized for readability instead of for the handoff to a person who still had to teach the lesson.
Write the job description the same way you would write one for a part-time ministry assistant. Include the Sunday outcome the human still owns. Include the non-negotiables that cannot be automated—pastoral voice, local context, last-minute room changes. Only after that page exists do you test whether an agent can produce work that survives the handoff.
Teams that skip this step keep discovering the same gap in user interviews six months later. Torres would call that a failure of continuous discovery. The problem statement never updated to include the human checkpoint.
AI Agent Oversight Means Logging the Escalation Path
Every agent you ship will eventually surface something it cannot resolve. The question is whether that moment arrives as a quiet failure or as a clear, logged escalation to the right human. Most teams log token counts and latency. Few log the precise condition under which the agent must stop and hand the thread back.
Map the escalation path in the same format you would use for a customer-support workflow. Define the trigger, the data that travels with the handoff, and the single owner who receives it. Test the path with a real coordinator who has never seen the agent before. If the coordinator cannot act on the handoff in under two minutes, the path is not ready.
The 8x target collapses when three escalations arrive on the same Sunday and no one knows who owns them. Continuous discovery requires that you treat the escalation path as part of the product, not as a support ticket that appears after launch.
Measure the handoff, not the tokens
This mirrors NIST’s guidance in its AI Risk Management Framework, which treats accountability and human review as core controls, not afterthoughts.
The metric that matters is the percentage of agent output that reaches the Sunday outcome without requiring rework by the human who owns it. Track how many minutes the volunteer or pastor still spends correcting or completing the work. Track how often the escalation path is used and whether the outcome improves after the handoff.
At Bible Gateway scale we watched thousands of users open a reading plan generated by an early recommendation model, then immediately close it because the plan did not account for their actual reading rhythm. The model had high completion inside the sandbox. The handoff to the real reader failed. Only after we measured time-to-first-real-read did the metric move.
Token usage and latency will look strong in the dashboard. They will not tell you whether the children’s ministry volunteer finished the lesson on time. That is the only number that converts into durable adoption inside a faith community.
Your Turn: Apply This Today
- Pick the single agent loop you are most tempted to ship this month and write its one-page job description before you open the prompt editor.
- Define the exact reversal step that returns control to a named human role—children’s director, worship pastor, volunteer coordinator—and log the data package that travels with it.
- Run one 15-minute observation with a real user performing the handoff on a printed copy of the job description; note every place they hesitate.
- Build a two-row dashboard this week: one row for agent output volume, one row for minutes the human still spends after the handoff.
- Schedule the first follow-up discovery call for the same user forty-eight hours after they first use the agent in a live setting.
- Write the escalation trigger condition in plain language on a single index card and tape it to your monitor before you write the next prompt.
The posts on measuring what survives the handoff and on running continuous discovery inside ministry tools both explore the same pattern from different angles.
I consult with product leaders shipping AI inside mission-driven organizations on workflow handoffs, escalation design, and discovery metrics that survive Sunday morning. Let’s talk.