I spent fourteen months refusing to greenlight an AI outline generator for children’s ministry because the projected usage numbers never materialized in our internal dashboards. Every sprint review I asked the same question: show me the cohort that actually opens the tool more than once. The answer stayed flat. We had real volunteer feedback in the form of one-off emails and hallway comments, but I treated those as noise until the telemetry caught up. The mistake was treating usage data as the only signal that counted and the qualitative input as noise.
The cost showed up later. Two other teams shipped lighter versions of the same idea while we waited. Their adoption came through the exact channels we had dismissed as anecdotal. By the time our version reached the same directors, the window for shaping how they used AI had already closed.
Taste as the first filter before any usage data
Volunteer tools rarely produce clean usage signals until the workflow already matches how people actually prepare on Tuesday nights. The children’s director prints the lesson at the kitchen table, marks it with a highlighter, and hands the marked copy to the small-group leader on Wednesday. No login, no repeat session recorded. The data stays silent because the real use happens offline.
I used to treat this silence as a verdict against the feature. Now I treat it as evidence that the initial test must happen through direct observation rather than aggregated logs. One director’s reaction to a printed sample carries more weight than a month of anonymous click data that never arrives.
The practical shift is to route early decisions through taste first. Build the smallest artifact that one practitioner can hold and judge inside their existing rhythm. If it survives that single judgment, then instrument it. Not the other way around.
How Wesley’s rules expose where data stays silent
John Wesley gave the early Methodists three simple rules: do no harm, do good, and attend to the means of grace. Applied to product work, the first rule means refusing to ship something that adds friction to an already thin volunteer margin. The second means looking for concrete help that fits the actual preparation window. The third means paying attention to the repeated practices that sustain the work week after week.
Data dashboards are excellent at measuring harm once it scales. They are nearly useless at detecting whether a tool does good inside a single, unrepeated Tuesday night routine. That judgment requires the same kind of attentive presence Wesley expected of class leaders who visited homes rather than waiting for reports.
The rules therefore reorder the sequence. Taste informed by direct contact becomes the first screen. Quantitative thresholds come later, once the practice has already taken root in one location.
When to ship the version that feels right to one children’s director
We eventually released a minimal outline generator after a single director walked us through her exact Sunday morning handoff. She showed us the folder she carries, the three places she writes notes, and the moment she decides the material will not work for her group. That thirty-minute walkthrough replaced six months of waiting for cohort metrics.
The version we shipped matched her folder structure and printed cleanly on standard paper. Usage numbers appeared only after the tool was already in rotation with three other directors who heard about it through the same informal channel. The telemetry finally moved because the fit had already been tested in one real context.
Shipping on that kind of single-point confirmation feels reckless until you accept that volunteer contexts keep most usage invisible by design. The alternative is to keep polishing a tool that never enters the only workflow that matters.
Your Turn: Apply This Today
- Pick one AI feature decision currently stalled on usage projections and write the single sentence that describes the exact moment a children’s director would first hold the output in her hands.
- Find three volunteers who match the target context and schedule a thirty-minute walkthrough this week where they show you their current preparation materials without any new tool present.
- Print or mock the smallest possible version of the feature that fits inside their existing folder or notebook and watch where they mark it or set it aside.
- Note the exact point in their rhythm where the artifact either reduces a step or adds one, then adjust the mock before showing it to anyone else.
- Send the revised mock to those same three volunteers with one question: does this replace something you already do or sit on top of it?
- If two of the three say it replaces a step, ship the narrow version to their group only and measure what actually changes in their handoff rather than waiting for broader telemetry.
The same pattern shows up in The Tuesday the Children’s Director’s Inbox Became the Real Product Spec and The 1997 Lesson AI Product Teams Keep Missing.
I consult with AI product leaders and ministry tool teams on taste-driven early decisions, volunteer workflow observation, and when to release before metrics appear. Let’s talk.