The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer

I shipped the agent on a Tuesday afternoon. By Thursday it had booked a regional meeting on the same day a volunteer coordinator only checks email, then followed up with the wrong person when nobody showed.

The team kept saying we’d add evaluation later. But later showed up the moment the agent started acting on calendars that weren’t really ours to touch. I kept wondering what would happen if we never built a way to see which decisions were worth keeping and which ones needed the sword.

Solomon didn’t run a model on the two mothers. He watched what each one was willing to lose. That distinction still feels like the only test that matters once an agent is loose in the system.

Where the first wave of failures showed up

The first problems appeared in the follow-up tasks the agent created after weekend services. It correctly read a pastor’s note about a family in crisis, then assigned the visit to the nearest available elder without checking whether that elder had already declined similar requests twice in the past quarter. The elder accepted the task in the tool, then missed it because the timing conflicted with an existing commitment the agent could not see.

A second failure hit the shared resource calendar. The agent moved a youth room booking to accommodate a last-minute counseling session. The booking system allowed the change, but the children’s ministry team that had printed materials for that slot only learned about it when volunteers showed up to an empty room. The model had judged the request reasonable in isolation.

These were not hallucination errors. They were judgment errors. The agent optimized for the immediate request without the historical friction that experienced coordinators carry in their heads.

How the hybrid split actually works in a small team

We stopped asking the model to score every proposed change. Instead we built three narrow rule engines that run first. Any change that passes all three still receives a lightweight model review only on the free-text justification the requester supplied. Everything else routes to a human queue.

The rules themselves are small and version-controlled. One checks whether the person assigned has accepted a comparable task in the last fourteen days. Another verifies the resource is not already committed during the proposed window according to the printed schedule that still governs most rooms. The third confirms the change does not touch any record marked “pastor only” in the directory.

Because the rules are explicit, the team can update them the same week a new failure pattern appears. The model review step only runs on the subset that clears the rules, which keeps the token spend low and the review list short enough for one person to scan in twenty minutes.

The one rule you enforce before any agent touches shared state

Before the agent is allowed to write anything, every proposed update must include a one-sentence human-readable reason that can be shown to the person whose calendar or task list is being changed. If that sentence cannot be generated or is empty, the change is rejected outright.

This single requirement surfaces most of the coordination problems that pure LLM judging misses. It forces the agent to state its assumption in language the recipient will recognize. When the assumption is wrong, the recipient can correct it quickly instead of discovering the error after the fact.

The rule also creates an audit trail that does not depend on model logs. When leadership asks why a particular elder was double-booked, the answer is the sentence the agent wrote, not a reconstructed chain of prompts.

Your Turn: Apply This Today

• Write the three rules as executable checks in the language your current database already supports and commit them to the same repo as the agent code this week.
• Build a simple rejection message that returns the exact rule that failed plus the one-sentence reason field so the requester sees the block immediately.
• Identify the single person who will receive the daily list of changes that passed the rules but still need the light model review, and give that person a 15-minute time block on their calendar.
• Add the human-readable reason requirement as a required field in the agent’s output schema and test it against last month’s failed task assignments.
• Schedule a 30-minute review with the two staff members who handle the most shared calendars to surface any existing edge cases the first three rules miss.
• Version the rules file with a one-line changelog entry every time you adjust a threshold so the next PM can see why the bar moved.

The same pattern shows up in The PM Who Inherited the Agent Schedule and The PM Role That Still Needs Human Judgment After Agents Ship. Both posts trace what happens when teams treat evaluation as a modeling problem instead of a governance problem.

I consult with product leaders at ministry organizations on agent evaluation layers, rule-based governance for shared ministry data, and the handoff points that still require human judgment. Let’s talk.