Blog

Posted on

The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer

A magnifying glass held over a printed document

I shipped the agent on a Tuesday afternoon. By Thursday it had booked a regional meeting on the same day a volunteer coordinator only checks email, then followed up with the wrong person when nobody showed.

The team kept saying we’d add evaluation later. But later showed up the moment the agent started acting on calendars that weren’t really ours to touch. I kept wondering what would happen if we never built a way to see which decisions were worth keeping and which ones needed the sword.

Solomon didn’t run a model on the two mothers. He watched what each one was willing to lose. That distinction still feels like the only test that matters once an agent is loose in the system.

Where the first wave of failures showed up

The first problems appeared in the follow-up tasks the agent created after weekend services. It correctly read a pastor’s note about a family in crisis, then assigned the visit to the nearest available elder without checking whether that elder had already declined similar requests twice in the past quarter. The elder accepted the task in the tool, then missed it because the timing conflicted with an existing commitment the agent could not see.

A second failure hit the shared resource calendar. The agent moved a youth room booking to accommodate a last-minute counseling session. The booking system allowed the change, but the children’s ministry team that had printed materials for that slot only learned about it when volunteers showed up to an empty room. The model had judged the request reasonable in isolation.

These were not hallucination errors. They were judgment errors. The agent optimized for the immediate request without the historical friction that experienced coordinators carry in their heads.

How the hybrid split actually works in a small team

We stopped asking the model to score every proposed change. Instead we built three narrow rule engines that run first. Any change that passes all three still receives a lightweight model review only on the free-text justification the requester supplied. Everything else routes to a human queue.

The rules themselves are small and version-controlled. One checks whether the person assigned has accepted a comparable task in the last fourteen days. Another verifies the resource is not already committed during the proposed window according to the printed schedule that still governs most rooms. The third confirms the change does not touch any record marked “pastor only” in the directory.

Because the rules are explicit, the team can update them the same week a new failure pattern appears. The model review step only runs on the subset that clears the rules, which keeps the token spend low and the review list short enough for one person to scan in twenty minutes.

The one rule you enforce before any agent touches shared state

Before the agent is allowed to write anything, every proposed update must include a one-sentence human-readable reason that can be shown to the person whose calendar or task list is being changed. If that sentence cannot be generated or is empty, the change is rejected outright.

This single requirement surfaces most of the coordination problems that pure LLM judging misses. It forces the agent to state its assumption in language the recipient will recognize. When the assumption is wrong, the recipient can correct it quickly instead of discovering the error after the fact.

The rule also creates an audit trail that does not depend on model logs. When leadership asks why a particular elder was double-booked, the answer is the sentence the agent wrote, not a reconstructed chain of prompts.

Your Turn: Apply This Today

  • Write the three rules as executable checks in the language your current database already supports and commit them to the same repo as the agent code this week.
  • Build a simple rejection message that returns the exact rule that failed plus the one-sentence reason field so the requester sees the block immediately.
  • Identify the single person who will receive the daily list of changes that passed the rules but still need the light model review, and give that person a 15-minute time block on their calendar.
  • Add the human-readable reason requirement as a required field in the agent’s output schema and test it against last month’s failed task assignments.
  • Schedule a 30-minute review with the two staff members who handle the most shared calendars to surface any existing edge cases the first three rules miss.
  • Version the rules file with a one-line changelog entry every time you adjust a threshold so the next PM can see why the bar moved.

The same pattern shows up in The PM Who Inherited the Agent Schedule and The PM Role That Still Needs Human Judgment After Agents Ship. Both posts trace what happens when teams treat evaluation as a modeling problem instead of a governance problem.

I consult with product leaders at ministry organizations on agent evaluation layers, rule-based governance for shared ministry data, and the handoff points that still require human judgment. Let’s talk.

Posted on

The Custom Agent We Had to Build After Off-the-Shelf Tools Hit Ministry Data Limits

Hand tools arranged on a workshop workbench

I was watching the volunteer schedule bot suggest the same three people for nursery again when one of them had just texted that her kid’s chemo started that week. The model had pulled the public calendar clean. It just didn’t know what the calendar never says out loud.

That’s where every off-the-shelf agent we tried eventually ran out of road. Ministry data isn’t sitting in tidy tables. It lives in the half-finished Slack threads, the voice memos from the hospital parking lot, and the things only three elders know to keep quiet.

We kept hitting the same wall until we stopped trying to bolt their framework onto our reality and started building one that could actually carry it.

The assumption that general agents will just plug in

Most teams start by connecting a commercial agent to whatever calendar or messaging system already exists. The connection works for surface events. It fails the moment the agent needs to read the private note that says “do not assign this family to the 9 a.m. service because of the ongoing custody situation.” That note exists in a shared document the public agent cannot reach without violating the ministry’s own access policy.

The gap is not technical ignorance. It is policy. The public agent has no representation of the rule “pastoral staff only” that lives in the team’s actual practice. When the agent guesses, it either over-shares or under-serves. Either outcome erodes trust faster than the productivity gain can recover.

Teams that persist with the plug-in approach end up writing longer prompt instructions. The instructions grow brittle. One change in volunteer policy requires rewriting the prompt across every agent instance. The maintenance burden quickly exceeds the original time savings.

What sovereign control actually changes in practice

Sovereign control begins with the data layer, not the model. The custom agent we built only ever saw data that had already passed through the ministry’s existing permission checks. Volunteer phone numbers stayed inside the private directory. Schedule conflicts stayed inside the internal ticket system. The agent received only the minimal subset required for the task and returned output that the same permission system then routed.

That boundary altered daily work in three concrete ways. First, the agent could safely propose schedule swaps because it never proposed a swap that violated an elder’s private availability note. Second, the output carried an audit trail that matched the ministry’s existing record-keeping, so staff could trace every suggestion back to the source data without additional logging. Third, the agent refused tasks when the required data sat outside its boundary, surfacing the refusal to a human instead of hallucinating a workaround.

The change also shifted metrics. We stopped tracking “agent tasks completed” and started tracking “volunteer completion rate on the revised schedule.” The latter number moved because the agent respected the real coordination patterns rather than the public calendar approximation.

Where the boundary sits between buy and build

The decision rule we now use is simple. If the workflow only touches data that already appears in a public API and carries no private coordination rules, a commercial agent is acceptable. If the workflow touches any data that the ministry treats as internal—availability notes, pastoral handoffs, child-protection flags—the agent must be built or heavily forked inside the sovereign boundary.

That rule eliminated two common vendor proposals. One suggested routing all volunteer texts through a shared LLM endpoint. The endpoint would have required exporting the private notes. Another offered a calendar-sync agent that could read the full shared drive. Both were rejected because they crossed the boundary we had already drawn for human staff.

The build path is narrower than it first appears. We reused open models for the language work and kept the routing logic inside code we already maintained for the ministry’s internal tools. The new agent became an additional consumer of the same permission service rather than a separate system. That reuse kept the scope small and the security posture consistent.

Your Turn: Apply This Today

  • Pick one recurring workflow that currently routes through a shared calendar or messaging tool and list every private note or access rule that human coordinators already apply before they act.
  • Trace the exact data fields an agent would need to read in order to propose the next action in that workflow, then mark which fields sit behind an existing permission check that no public API exposes.
  • Write the refusal message the agent should return when it encounters a data request that crosses the permission boundary, and test it against the last three real schedule changes your team handled manually.
  • Identify the single internal system that already enforces the permission rules for that workflow and confirm the agent can call it without exporting raw records.
  • Run a one-week audit where you log every instance a human overrode or corrected an off-the-shelf agent suggestion, then tag each override with the private rule the agent missed.
  • Schedule a 30-minute review with the two staff members who handle the most handoffs in that workflow and ask them to name the one data access rule they would never want surfaced to a general model.

The Night the New Career Tool Asked the Wrong Job and The Data Moat That Actually Matters for Small Ministry Teams both explore the same tension between public tooling and private coordination rules.

I consult with ministry product leaders on defining sovereign data boundaries for agents and auditing workflows that cross public and private coordination layers. Let’s talk.

Posted on

The Graph We Added After Token Costs Stopped Us Shipping

Abstract network of connected glowing nodes and links

I refreshed the billing dashboard and the line item for the scheduling agent had already doubled from last week. Same questions, same names, same carpool details getting re-explained every time a volunteer dropped out.

We had promised the team this thing would save them hours. Instead it was burning through tokens just to stay in the same place, and every new swap made the next one more expensive. Shipping got pushed again while we stared at the numbers.

That’s when we stopped trying to make the model remember and started giving it something it could actually look at.

The exact token spike that forced the graph decision

The spike showed up first in the prompt logs. Each new conversation restarted with a full dump of the current volunteer roster, every recurring conflict, and the last three schedule versions. When a family called to change their availability, the agent needed another full pass to rebuild the picture. One Tuesday the daily spend jumped past the monthly curriculum line because three separate last-minute changes triggered overlapping prompt refreshes.

We measured the waste directly against the children’s ministry numbers we already tracked. The curriculum itself cost a fixed amount per quarter. The agent had crossed that line in a single month. The team had optimized the model calls for accuracy in isolation, but the real constraint was the repeated transmission of the same local facts.

Once we saw the pattern in the logs, the decision to move context out of the prompt became straightforward. We needed a structure that could answer a narrow set of questions without re-sending the entire history every time.

How the graph preserved the one piece of context the model always dropped

The model reliably forgot which families shared transportation. A parent might confirm they could drive for the 9 a.m. slot, yet the agent would later assign their child to a different campus without checking the shared ride. The graph stored that relationship as a simple edge between two volunteer records and a recurring location constraint. Queries could then test the edge before proposing any change.

We kept the graph small on purpose. Only three node types existed: person, shift, and constraint. Edges carried only the facts that changed weekly. The model still handled natural language, but it called the graph first for any fact that had appeared in more than one prior conversation. Token counts dropped because the prompt now referenced a single lookup instead of re-listing the full roster.

The same structure surfaced the next time a volunteer coordinator joined the team. She could query the graph directly for open shifts that respected existing carpool edges without waiting for the agent to re-learn the pattern through failed suggestions.

The discovery loop that kept the graph from becoming another abandoned project

Torres’s loop requires regular interviews with the people who actually use the system. We scheduled fifteen-minute calls every other week with whoever had made the most recent schedule changes. The question was always the same: what fact did you have to tell the agent twice this week? The answers pointed to new edges that belonged in the graph rather than new prompt text.

Each interview produced one or two concrete updates. We added an edge type for recurring medical appointments after three families mentioned the same constraint. We added a simple weight on certain shifts after the coordinator noted that some volunteers only accepted weekday roles. None of these changes required retraining the model; they only required extending the graph schema.

The loop also caught when an edge had grown stale. One carpool relationship ended when a family moved, and the interview caught it before the agent proposed the old arrangement again. Continuous discovery turned the graph into living documentation instead of another static database that drifted from reality.

Your Turn: Apply This Today

  • Review the last ten agent runs that required user correction and list every fact the model asked the user to repeat.
  • Group those facts into the three most common categories that appear across runs.
  • Build a minimal graph with one node type and one edge type for each category using whatever lightweight store already exists in your stack.
  • Instrument the agent to query the graph before generating the next prompt and measure the token reduction on the next twenty runs.
  • Schedule a ten-minute call with the person who makes the most schedule changes and ask what single fact the agent still misses.
  • Add the new fact as an edge and re-test the same twenty runs to confirm the spend drop holds.

The PM Who Inherited the Agent Schedule showed how inherited context breaks when teams treat agents as black boxes. The Knowledge Layer Ministry Teams Forgot to Budget traced the same pattern across several small teams that later added graphs.

I consult with product leaders building agentic tools for ministry teams on continuous discovery loops and lightweight knowledge graphs for context retention. Let’s talk.

View all posts