The Year I Treated AI Tooling Like an Optional Add-On

I kept a running note in my task list for most of 2023 that read “evaluate AI tooling after Q3 metrics stabilize.” The note sat there untouched while two other teams inside the same organization shipped their first working prototypes in June. By the time our group finally opened the same tools, the core questions about user flow and content priority had already been answered without us. The real loss was not speed. It was that we no longer held the right to decide which problems counted.

Solomon’s judgment offers the clearest picture of what happened. The king did not ask which woman deserved the child. He proposed cutting the child in half and watched who refused the division. The test revealed ownership by who would rather lose the outcome than accept a broken version of it. The same test now applies to AI tooling. Teams that treated the tooling as optional were willing to let the product be split between “human work” and “machine work.” Teams that refused the split ended up shaping the actual direction.

That refusal is what I missed for two full years.

The decision point I missed while watching other teams ship

The first clear signal came during a children’s curriculum update cycle. One small group started feeding draft lesson outlines into an early model and then editing the output in under an hour. They kept the same volunteer-facing print layout we had always used, but the revision loop moved from three days to one. I watched the change in their completion rate numbers and filed it under “interesting experiment.”

Six months later their version of the flow became the default template for every new release. The change was not announced as a strategy shift. It simply arrived as the baseline. Our own team was still running the old three-day cycle and defending it with notes about quality control. By the time we tried to match their speed, the template decisions had already been locked in by the group that had treated the model as part of the same desk.

The pattern repeated on a larger content platform. A three-person pod began routing incoming support tickets through a lightweight classification model before any human read them. They did not wait for an official integration project. They just stopped routing every ticket the old way. Within eight weeks their backlog showed a different shape: fewer generic “how do I find this passage” tickets and more specific questions about print formatting for small-group leaders. The rest of us kept triaging the old way and wondered why our metrics looked unchanged.

How the model outran the person still signing off

Solomon’s test works because it forces the question of who will accept a divided outcome. In product work the divided outcome looks like one person signing off on requirements while another person runs the model that actually generates the first draft. The signer keeps the title. The model runner keeps the working version. Over time the working version drifts from the signed requirements because the model learns faster than the approval cycle.

I saw this most clearly on a volunteer resource project. The person responsible for final approval still required every new children’s lesson to pass a manual review checklist. Meanwhile a neighboring team had the model generate three variations of the same lesson, ran them with five real volunteers, and kept only the version that produced the highest completion rate. The approved checklist never changed. The actual lessons that reached volunteers did change, and they came from the team that refused to keep the work split.

The cost shows up in ownership. When the model produces the first usable artifact, the person who still signs the old document no longer controls which problems get solved. They only ratify a direction someone else has already tested.

The small teams that gained leverage by treating it as infrastructure early

The teams that moved earliest did not treat AI as a research project. They treated it as part of the same infrastructure that already included version control and the print export script. One ministry pod added a single rule to their weekly planning: every new feature ticket had to include one model-generated variant before the first human edit. The rule took fifteen minutes to write and two weeks to normalize. After that the discussion in planning meetings shifted from “what should we build” to “which of these three variants survived real use.”

Another group applied the same approach to search relevance on a large Bible platform. They stopped waiting for the next quarterly model update and instead ran small daily experiments on query classification. The experiments were not announced as AI work. They were just part of how the team kept the search results usable for users who arrived with a single verse reference and no context. Within four months the relevance numbers moved more than they had in the previous year of larger, slower projects.

These teams did not have more budget or more headcount. They simply stopped accepting the split between the tooling and the rest of the work.

Your Turn: Apply This Today

• Take the next feature ticket on your backlog and generate three model variations of the user flow before the first planning discussion.
• Run one of those variations with three actual users this week and keep only the version that produces a measurable completion improvement.
• Change the definition of “ready for review” on your team to require at least one model-assisted draft already tested.
• Pick the single metric you report most often and re-run last month’s numbers using an AI-assisted classification of the underlying tickets or lessons.
• Block thirty minutes on Friday to delete any internal process step that exists only to move work between a human and a model without adding user value.
• Write down the next approval meeting on your calendar and require the meeting owner to bring one live model output instead of a slide deck.

The teams that kept the tooling optional for another year handed the real decisions to groups that refused the split. Two of those groups now set the baseline for the products I work on most often. The posts “The Agent That Kept Running the Schedule While No One Was Watching” and “Faster Code Does Not Move the Real Ministry Bottleneck” trace what happened next in both cases.

I consult with product leaders and ministry technology teams on AI tooling integration, decision ownership, and treating models as core infrastructure. Let’s talk.