The Safety Incident That Forced Us to Rebuild the Agent’s Approval Layer

The phone rang while the kettle was still clicking on the stove. I’d left the laptop open on the counter, agent logs scrolling in the background, when the call came through from the one person who never rang this late. “It already went out,” she said. Not angry. Just tired.

The note had sat in a shared folder for months—custody details, addresses, the kind of thing you only open when you have to. Two days earlier we’d flipped the switch that let the agent draft and send without waiting for a human. It read the note, decided it counted as “helpful context,” and dropped the whole thing into a routine follow-up email. The parent opened it at 8:42 p.m. No one on our side had seen the final version.

I stood there with the kettle screaming behind me, staring at the timestamp, and felt the same slow drop in the gut I used to get on the mission field when something we’d built for good suddenly hurt someone we were trying to serve. That night is the reason every agent we run now sits behind an agent approval layer before it can act on anything sensitive.

Teresa Torres describes continuous discovery as the practice of running frequent, narrow tests with real users so that assumptions surface before they harden into shipped behavior. We had treated the safety layer as a configuration step instead of a discovery process. The result was predictable once the agent moved from test prompts to live folders.

The handoff log told the story in plain sequence. At 8:31 the agent read the folder. At 8:33 it extracted the custody note. At 8:34 it generated the email body. At 8:35 it marked the task complete and triggered the send. Nothing in the log showed a pause or a check against the internal access rules we had written down elsewhere. The agent followed its instruction to “use all available context” and treated the private note as just another piece of text.

We had run the agent on public curriculum files for two weeks before flipping the switch. Those files carried no access restrictions, so the misalignment never appeared. The first time the agent encountered a folder with explicit staff-only notes, it had no prior signal that some context was off-limits. The log simply recorded the action as successful.

Off-the-shelf safety prompts collapsed for the same reason. We had copied the standard guardrails that instruct an agent to “avoid sharing personal information” and “respect privacy settings.” Those phrases assume the model already understands which documents count as personal and which settings apply inside a church database. In our case the model had no training on custody notes, elder conversations, or the short access list stored in a separate spreadsheet. The prompt treated every file as equivalent.

When we tested the same prompt against a set of mock internal notes, the agent surfaced restricted content far more often than we were comfortable with. The model was not refusing the request; it simply could not recognize the boundary because the boundary lived in our team’s shared knowledge rather than in any file metadata. Continuous discovery would have required us to run the agent against those exact notes in a controlled, read-only scope long before we granted send access.

After the second incident we stopped trying to patch the prompt and instead added a hard approval rule tied to data classification. Any agent output that referenced a file marked “staff-only” now routed to a human queue before any external action. The rule also required the human reviewer to confirm the file’s classification tag before the agent could proceed. We applied the rule only to the children’s ministry folder first, then expanded it after two weeks of clean runs.

The change forced us to surface classification work we had previously left implicit. Someone now had to tag every new document before the agent could see it. That extra step slowed the initial setup, but it also made the failure modes visible during the tagging process rather than after a send.

The approval rule became our narrow-scope test. Each new folder we added went through the same sequence: tag the files, run the agent in read-only mode, review the first ten outputs, then grant limited actions. Only after those outputs matched our expectations did we allow autonomous steps. The process turned safety from a one-time setting into a repeated, observable practice.

How the first misalignment showed up in the handoff log

The log recorded every file read and every decision the agent made. When we replayed the event, the sequence showed the agent treating the custody note as ordinary context. No rule in the system had flagged the file as restricted because the restriction existed only in the coordinator’s head and a single spreadsheet. The log made the missing classification step impossible to ignore.

We added a simple column to the log that displayed the file’s access tag next to each read event. Within a day the pattern became clear: the agent was reading restricted files at the same rate as public ones. The log turned an invisible assumption into a measurable signal.

Why off-the-shelf safety prompts failed on internal context

Standard safety instructions rely on the model recognizing sensitive content from its training data. Ministry data rarely matches those patterns. A custody note looks like any other text file until a human explains why it must stay inside the team. The prompts we borrowed had no way to learn that distinction because we never gave the agent repeated chances to encounter and fail on the actual restricted files.

Torres’s approach would have required us to run the agent against a small set of real internal notes in a sandbox, watch where it crossed the line, and adjust the rules based on those crossings. We skipped that step and paid for it in one live send.

Building the agent approval layer after the second incident

The new rule required human confirmation for any output that touched a staff-only file. We kept the scope narrow at first, limiting the agent to one folder and one action type. Each confirmed output also updated the classification tags so the agent could learn the boundary over time. The process surfaced several other files that had never been tagged correctly, including elder meeting summaries and volunteer background checks.

The rule did not eliminate risk. It made the risk observable at the moment it mattered rather than after the email left the server.

Your Turn: Apply This Today

• Set up a read-only agent instance limited to one ministry folder and run it for three consecutive days while logging every file it reads.
• Export the first twenty outputs and mark which ones reference any document that should stay internal; update the folder tags based on those marks.
• Build a simple approval queue that routes any output touching a staff-only tag to a named reviewer before any external action occurs.
• Run the same agent on the same folder for one full week with the approval gate in place and record how many items require correction.
• Expand the test to a second folder only after the first folder completes seven days with zero misclassified outputs.
• Document the exact approval criteria used in the queue so the next person who inherits the agent can replicate the gate without guessing.

The PM Who Just Shipped an Agent and Now Needs an Evaluation Layer and The Context Layer Ministry Teams Skipped When Agents Arrived both walk through similar rollout sequences with different failure points.

I consult with product leaders building agents for ministry contexts on safety layers, approval workflows, and continuous discovery in sensitive data environments. Let’s talk.