The Evaluation Layer That Broke When We Trusted the Model Alone

I watched the model hand back a clean volunteer roster one Tuesday morning. Every shift covered, no obvious clashes on the calendar. Then the first reply hit my phone—an elder who’d been double-booked into both the food-bank run and his own kid’s surgery.

The schedule looked balanced because the model only scored what it could see. It never felt the weight of someone having to choose between two people who both needed them.

We had treated the output like a finished decision instead of raw material that still needed an outside rule to catch what the model couldn’t afford to break.

Why pure model scoring failed on real schedules

Model scoring systems rate candidate schedules on metrics such as total coverage, average travel time, and stated preference match. Those numbers look decisive until a children’s ministry volunteer who agreed to two consecutive weeks is assigned a third week that conflicts with a prior commitment the model never saw. The score remains high because the conflict lives outside the prompt context.

Observed cases show the model consistently preferring the schedule that balances load across names rather than protecting the single volunteer whose absence would collapse an entire age-group rotation. The numeric optimum therefore produces the outcome that requires the most manual rescue calls the following week.

The deeper issue is that model scoring treats every constraint as soft. A hard rule such as “no volunteer may receive two consecutive weeks in the same role without explicit confirmation” never enters the evaluation function unless it is injected as code rather than left to the model’s judgment.

The simple rule check that caught the three-way fork

One team added a deterministic pre-filter that rejected any schedule containing a volunteer assigned to three distinct locations within a fourteen-day window. The filter ran after the model produced its top three candidates and before any human opened the file. In the first month the filter discarded seventeen schedules that the model had ranked highest.

The three-way fork surfaced when the same volunteer name appeared in the output for a Wednesday rehearsal, a Saturday outreach, and a Sunday service that required travel across town. The model had scored the combination acceptable because each individual assignment matched stated availability. The rule check flagged the cumulative load that no single prompt had surfaced.

Adding the filter required fewer than twenty lines of code and ran in under a second. It removed the need for coordinators to discover the conflict only after the volunteer had already been contacted and had to decline.

Where human review still has to sit in the loop

Human review belongs after the rule layer has already eliminated the mechanically invalid options. The reviewer then faces a short list that contains only candidates that satisfy every hard constraint. At that point the reviewer’s task narrows to weighing relational factors the rule set cannot encode, such as whether two volunteers who share a carpool should be kept on the same rotation.

Placing the human earlier, before rules run, wastes attention on schedules that will be rejected anyway. Placing the human later, after the schedule has already been published, forces recovery work that could have been avoided. The Solomon pattern requires the costly test to occur before the decision is finalized, not after.

Teams that moved the human into the final gate reported fewer last-minute replacements and higher volunteer retention on the same set of assignments. The model still generated the initial candidates; the added layers simply prevented the model from declaring victory on its own output.

Your Turn: Apply This Today

• Pick the single agent output your team reviews most often and write one deterministic rule that must pass before any human sees the file.
• Run last week’s rejected outputs through that rule and count how many would have been caught automatically.
• Add the rule as a blocking step in the review checklist and log the number of false positives removed each day for seven days.
• Share the before-and-after rejection counts with the person who currently performs the first manual pass so they can see the time shift.
• If the rule removes fewer than 15 percent of candidates, tighten its condition and repeat the measurement for the next seven days.
• Document the final rule and the measured reduction so the same check can be reused on the next agent workflow you ship.

The same pattern of layered verification appears in The PM Who Inherited the Agent Schedule and The PM Role That Still Needs Human Judgment After Agents Ship.

I consult with product leaders building ministry coordination agents on rule-based verification layers and placement of human judgment gates. Let’s talk.