The Afternoon the Agent Dashboard Showed Last Month’s Volunteers

I sat with the volunteer coordinator while she pulled the roster for the Tuesday crew. The list looked right at first. Then I noticed the guy whose wife had left in July was still marked “active,” and the couple who moved to another city two weeks back had a fresh phone number listed like nothing changed.

She stared at it for a long second, then scrolled again, slower this time. The screen gave no sign anything was off.

That quiet gap between what the system thinks is true and what’s actually happening in people’s lives is where most of these tools quietly break.

The same pattern shows up whenever product teams treat agent output as the final artifact instead of raw material. Torres’s framework requires repeated, side-by-side review of what the tool generated and what the human team kept or changed. Skip that evaluation loop and the system quietly drifts toward the last good snapshot it saw.

Watching the Exact State Loss Happen

The PM opened the edit history. The agent had generated the list at 9:17 a.m. from cached data pulled three days earlier. The staff coordinator had then spent twelve minutes correcting six entries before the final version went to the small group leaders. None of those corrections fed back into the agent.

Three days later the same agent ran again on the Tuesday night schedule. It produced the uncorrected version because the feedback loop never existed. The PM could see the two versions side by side only because he had saved both manually. The production system kept only the agent output.

This is not a data-pipeline bug. It is a discovery failure. The team never built the habit of checking what actually shipped against what the agent proposed. Without that weekly comparison, every subsequent run reinforced the stale state.

The Weekly Edit Comparison That Builds the Evaluation Loop

Torres’s continuous discovery asks teams to treat every shipped artifact as new input for the next iteration. In agent work that means saving the human-edited version and feeding the differences back into the prompt or the retrieval layer. The comparison has to happen on a fixed cadence, not when something breaks.

One team started requiring every agent-generated roster to be saved as both the raw draft and the final corrected file. On Monday mornings they opened the two versions, highlighted every line that changed, and updated the agent’s context window with those exact changes. After six weeks the agent stopped repeating the weeks-old names because the corrected data had become the new baseline.

The cost is small. The discipline is not. Most teams still review agent output only when a volunteer complains. By then the model has already repeated the error several more times.

Ministry Rosters as the Hardest Test Case

Rosters expose the problem faster than almost any other church dataset. People change roles, phones, and availability every week. The ground truth moves constantly. An agent that produces a clean-looking list from last month’s snapshot will look correct until someone tries to use it.

We ran the same test on curriculum assignments and event sign-ups. The pattern held. The agent output matched the final human version only when the team had forced a side-by-side review the previous week. Without it, the model drifted toward whatever snapshot it had last seen.

The lesson is not that ministry data is uniquely messy. The lesson is that any domain with high change frequency reveals the missing evaluation loop sooner. Continuous discovery works here because the cost of an undetected error is immediate and visible to real people trying to run programs.

Your Turn: Apply This Today

• Set a recurring Monday 30-minute block on your calendar to open last week’s agent draft and the final human version side by side for one recurring output.
• Save both the raw agent file and the corrected file in the same folder with date stamps before any further edits occur.
• Highlight every line that changed and paste the differences into a running “context updates” document the agent can reference next time.
• Run the agent again on the same task the following week and note whether the number of corrections drops.
• After four weeks, calculate the reduction in manual edits and decide whether to keep the loop or adjust the prompt structure.
• Share the four-week edit log with the rest of the product team so the pattern becomes visible beyond the single PM who noticed the roster problem.

The same evaluation gap appears in The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer and The Evaluation Layer That Broke When We Trusted the Model Alone. Both posts walk through the concrete mechanics of closing the loop after an agent ships.

I consult with ministry product leaders on agent evaluation loops and roster data integrity. Let’s talk.