The request comes in through Slack, dressed up as a technical conversation: “We need more training data before we can improve the model.” Sometimes it’s true. More often, it’s a symptom of something upstream that no amount of data will fix.
I’ve sat in product reviews where teams requested months of data collection before they could ship a meaningful AI feature — only to discover, after collecting the data, that they’d been optimizing for the wrong output the whole time. The model got better at predicting something. It just wasn’t predicting the right thing.
The AI Feature Problem That Isn’t a Data Problem
Most AI feature failures I’ve seen in product organizations aren’t engineering failures. They’re problem definition failures. The team knew what they wanted to build. They knew which model architecture they’d use. They had a rough sense of the training data they’d need. What they hadn’t done was write a crisp one-paragraph answer to a deceptively simple question: What specific decision are we trying to improve, for whom, and how will we know the improvement is real?
That question sounds basic. In practice, it’s brutally hard to answer precisely — and most AI feature discussions move past it without ever settling it. Instead, teams anchor on the technical implementation (which model, which API, which pipeline) and skip the product fundamentals that determine whether any of it will matter.
The result is a common and expensive failure mode: you ship an AI feature, it’s technically impressive, and adoption is flat. Users aren’t hostile to it — they just don’t reach for it. Because the feature is solving a problem that wasn’t sharp enough to justify its existence in the first place.
Problem Definition Before Model Selection
There’s a sequence that high-performing AI product teams follow — often implicitly — that lower-performing teams skip. It goes: problem definition → success metric → data requirements → model selection. Most teams invert this. They start with a model or an API capability, then work backward to find a use case. The resulting features are technically coherent but strategically thin.
Problem definition, done right, answers four things before any engineering begins:
Who has the problem? Not “our users” — a specific persona, role, or workflow. The tighter your answer, the more precise your feature can be. Vague personas produce vague AI features that try to be useful to everyone and end up indispensable to no one.
What decision are they currently making badly? AI is most useful when it augments or automates a decision that a human is already making but making slowly, inconsistently, or with incomplete information. If you can’t identify the decision, you can’t define what “better” looks like.
What does “better” actually mean? Faster? More accurate? More consistent? Less effortful? These aren’t the same, and they don’t optimize for the same outputs. A feature that reduces decision time by 40% but introduces 15% more errors may be worse than no feature at all — depending on the stakes of the decision.
How will you measure the improvement? Before you define your data requirements, you need your success metric. If you can’t articulate a measurable outcome that would tell you the feature worked, you’re not ready to scope the data pipeline.
Why Teams Skip This Step
Skipping problem definition isn’t laziness. It’s usually organizational pressure. There’s a demo to prep for. There’s a competitor who just shipped something. There’s an executive who read about the technology and wants to see it in the product. In those conditions, “we need to define the problem more precisely” sounds like a stall tactic. It isn’t — but it can feel like one.
The reframe I use with teams: problem definition doesn’t slow down AI feature development. It accelerates it. When the problem is crisp, data requirements become obvious. Model selection becomes obvious. Evaluation criteria become obvious. The team spends less time rebuilding the pipeline because they didn’t spec it wrong the first time.
I’ve watched teams spend 12 weeks collecting data for a feature that took 4 weeks to build — only to realize at launch that they’d defined the output variable incorrectly. A week of problem definition work at the front would have surfaced that misalignment long before the data collection started.
The Practical Test: Can You Write the Prompt?
Here’s a heuristic I’ve started using with teams evaluating AI features: can you write the prompt that would produce the output you want, and does that output solve the problem as you’ve defined it? If you’re using a generative model, this is literal — write the prompt. If you’re using a predictive model, translate: can you describe in plain English what you’re asking the model to predict, and does that prediction map to a real decision that real users need help making?
If you can’t write the prompt — or if the prompt output doesn’t map clearly to a user decision — you don’t have a data problem. You have a problem definition problem. And that’s a conversation to have in a product review, not a sprint planning session.
Your Turn: Apply This Today
Before your team writes another line of code or collects another batch of training data, run your AI feature through this problem definition checklist:
- Write the one-paragraph problem statement. Include: who has the problem, what decision they’re currently making badly, what “better” means in measurable terms, and how you’ll know the feature worked. If your team can’t agree on this paragraph in one working session, that’s your first sprint — not the model.
- Identify the specific decision your AI feature augments or automates. If you can’t name a decision, you don’t have an AI problem — you may have a search problem, a dashboard problem, or an information architecture problem. AI is not the right hammer for every nail.
- Define your success metric before scoping data requirements. What measurable outcome will tell you the feature improved the user’s decision? Write it down. Lock it. Then — and only then — work backward to determine what data you need to produce that outcome reliably.
- Run the “prompt test.” Can you write the prompt (or describe the prediction) in plain language? Does that output directly address the decision you identified? If there’s a gap between the model output and the user’s decision, that gap is your product design problem.
- Interview three users about the decision before building. Ask them to walk you through the last time they made this decision. What information did they use? Where did they get stuck? What would have made it faster or easier? Their workflow, not your intuition, should drive your data spec.
- Set a problem definition review gate. Before any AI feature moves from ideation to data collection, require a written problem definition document. One page. Four questions answered. This gate will save you more engineering cycles than any model optimization will.
Related reading: The Product Leader’s AI Infrastructure Blind Spot covers the organizational patterns that cause AI investments to underperform, and Opportunity Solution Trees Meet AI Reality at Scale addresses how to adapt discovery frameworks when AI is the solution space you’re exploring.
Working through AI product strategy and finding that your team keeps building features that don’t land? I consult with product leaders on AI feature scoping, problem definition frameworks, and the organizational dynamics that cause AI investments to underdeliver. Let’s talk.