Blog

The story of Solomon’s judgment is one of the oldest decision-making frameworks in recorded history. Two women claim the same child. No witnesses, no evidence, no algorithmic output that could resolve the dispute. Solomon’s solution — propose dividing the child, then watch who objects — is a masterclass in what optimization cannot do: use the decision itself as the test that reveals the truth.

I’ve been thinking about this story a lot while working on AI recommendation systems. We’ve gotten very good at optimization. We can recommend content, predict churn, personalize experiences, and rank options with impressive accuracy. What we haven’t gotten good at — and what I’m not sure AI will ever be good at — is recognizing when the optimization is solving the wrong problem entirely.

The Algorithm vs. the Test

A recommendation algorithm optimizes for a signal. Click-through rate, completion rate, return visits, explicit ratings — whatever you tell it to optimize, it will optimize. The problem is that the signals we can measure are often proxies for the outcomes we actually care about, and at some point the proxy diverges from the outcome.

I’ve seen this play out in digital content platforms repeatedly. A reading plan recommendation algorithm optimizes for completion rates. It gets good at predicting which plans users will finish. It starts recommending plans that feel familiar, comfortable, and achievable — because those are the ones that get completed. Completion rates go up. The metric is green. But users who choose plans that “mismatch” their stated preferences — the challenging, unfamiliar ones — show better long-term engagement. The algorithm optimized for completion and selected against growth.

Solomon’s test is the thing the algorithm can’t run. He couldn’t optimize his way to the truth. He had to create a condition that would reveal the truth through the parties’ responses. That kind of judgment — knowing that the right test will surface the right answer — is what AI cannot replicate, and what product leaders need to understand as an irreplaceable capability.

The Limits of AI Optimization

There’s a specific category of decisions where optimization fails systematically, and it maps precisely to the conditions of Solomon’s judgment: when the right answer requires understanding something about a party’s underlying motivation or stake in the outcome that isn’t visible in any behavioral signal.

In product terms: the algorithm can tell you what users do. It cannot tell you what users are trying to become. It can tell you what content users complete. It cannot tell you whether completion served their actual goal. It can tell you what users click. It cannot tell you whether the click reflects genuine engagement or habit, genuine interest or boredom-driven curiosity.

These are not problems of insufficient data. They’re problems of category — you cannot optimize your way to answers about meaning, motivation, and stake without reducing those human realities to signals that don’t actually capture them. This is what HBR’s research on AI decision limits has consistently found: AI excels at decisions that can be fully specified by their optimization criteria. It fails at decisions where the criteria themselves are contested or where the right answer depends on understanding what’s at stake for the parties involved.

Creating Space for Judgment

The practical implication for product teams isn’t “stop using optimization.” It’s “design your system so that optimization handles what optimization is good at, and humans handle what optimization can’t reach.”

This requires being explicit about the decisions your AI system is making and auditing them for the category of problem they represent. Can the decision be fully specified by a measurable outcome? Optimization is appropriate. Does the decision depend on understanding a user’s underlying motivation, growth trajectory, or stake in the outcome? Optimization needs a human checkpoint.

In practice, this means building escalation paths. Decisions that an algorithm can handle with confidence: automated. Decisions that carry high downstream consequence and depend on judgment the algorithm can’t access: escalated to a human. The challenge is designing the system so that the escalation happens before the consequence rather than after it.

The Wisdom Gap in AI Product Teams

Most product teams have become very good at evaluating AI performance — accuracy rates, precision, recall, business metric lift. What they haven’t systematically developed is the capacity to evaluate AI judgment — the ability to recognize when the system is optimizing confidently toward the wrong objective.

This is the wisdom gap. Wisdom, in the traditional sense, isn’t just knowledge or capability — it’s knowing when to apply which capability, and knowing when the right answer can’t come from a formula at all. Solomon wasn’t impressive because he had access to more information than anyone else. He was impressive because he understood that the right test would reveal what no available information could.

Building that capacity into a product team means creating explicit processes for questioning whether AI systems are optimizing for the right things — not just whether they’re optimizing well. It means reviewing decisions at the system level, not just the feature level. It means asking, regularly: what would the right answer look like if we couldn’t use a metric to find it? If the answer changes when you remove the metric, the metric is the wrong one.

The Practical Applications

Three specific places where Solomon’s framework changes how I evaluate AI product decisions:

Recommendation systems: Don’t just optimize for completion or engagement. Build qualitative research into your cadence that asks users whether the AI-recommended path served their actual goal. The behavioral signal and the goal-based signal often diverge. The divergence is where the product insight lives.

Personalization: The most effective personalization isn’t always the most comfortable personalization. Before concluding that an algorithm is working because engagement is up, ask whether the algorithm is serving users’ stated goals or optimizing toward the path of least resistance. Users often engage more with what’s familiar than with what’s growth-oriented. The engagement metric can’t distinguish between the two.

High-consequence decisions: Any AI-assisted decision with significant downstream consequences for a specific user — content restrictions, account actions, eligibility determinations — needs a human checkpoint. Not because the AI will always be wrong, but because the cost of confident wrongness in these cases is high enough that the judgment of a human who understands what’s at stake is worth the operational overhead.

Your Turn: Apply This Today

Build the judgment layer into your AI product process:

• Audit your AI system’s decisions for decision category. List the top 10 decisions your AI system makes. For each one, ask: can this decision be fully specified by its optimization criteria? If yes, automation is appropriate. If the right answer depends on understanding user motivation or stake, build a human checkpoint.
• Separate your “proxy metrics” from your “outcome metrics.” Identify which metrics your AI optimizes directly and which outcomes you actually care about. Map the relationship between them. Where the proxy and the outcome diverge, you have a place where optimization is working against you.
• Build qualitative research into your AI evaluation cadence. Once per quarter, interview users whose behavior your AI system has most significantly influenced. Ask whether the AI-driven experience served their actual goal. The divergence between behavioral signals and goal-based answers is where your most important product insights live.
• Design a “Solomon test” for your highest-stakes AI decisions. For the AI-assisted decisions with the highest downstream consequence, design a test that would reveal whether the AI’s recommendation was right — not just whether users accepted it. Acceptance and correctness are not the same.
• Create an escalation path for judgment-dependent decisions. Identify the category of decisions that require understanding user motivation or stake. Build an explicit escalation path so that those decisions reach a human before the consequence rather than after. Make it part of your system design, not an emergency procedure.
• Hold a “what if we removed the metric?” review annually. For each of your AI system’s optimization targets, ask: what would the right answer look like if we couldn’t use this metric? If the answer changes significantly, the metric needs to be reconsidered. This is the most uncomfortable product review you will have and the most valuable.

The judgment-vs-optimization tension shows up across AI product decisions — the Kahneman System 1 paradox addresses the cognitive load dimension of the same problem, and why product decisions are never just product decisions explores the ethical dimension of designing AI systems that optimize toward the right ends.

Building AI systems and trying to design judgment into the process rather than optimizing it away? I consult with product teams on AI product strategy, human-AI decision frameworks, and building systems that know when optimization is the right tool — and when it isn’t. Let’s talk.

John Wesley reduced the entire Methodist movement to three rules: do no harm, do good, and stay in love with God. Not a comprehensive theology. Not thirty-seven principles. Three simple constraints that anyone in a Methodist society could remember, apply, and teach.

I’ve been thinking about Wesley’s framework — specifically his instinct that constraint creates more impact than capability — as product teams struggle with AI implementation. Claude can write, code, analyze, reason, and create. GPT-4 handles everything from customer support to strategic planning. The temptation is to throw AI at every problem and see what sticks.

Wesley understood something we keep relearning: the most powerful systems aren’t the ones that can do everything. They’re the ones that do specific things exceptionally well within clear boundaries. Here’s what his three-rule framework looks like as an AI product design principle.

Do No Harm: The First Constraint in AI Prompt Design

In AI product work, “do no harm” translates to a specific discipline: define what the AI system should not do before you define what it should do.

Most teams design AI features by expanding capability — adding more things the AI can handle, more use cases it supports, more outputs it can generate. The Wesley approach inverts this: start by naming the exclusions. What should this AI never say? What user requests should it decline to fulfill? What outputs would be harmful, misleading, or counterproductive even if technically achievable?

The constraint changes the architecture of the feature. Teams that start with capability tend to build AI that does many things adequately. Teams that start with harm prevention tend to build AI that does specific things with genuine care for the user’s actual outcome. The difference in user trust is significant and compounds over time.

In practice: before your next AI feature ships, write down three things it should never do. Make those constraints explicit in your system prompt, your evaluation criteria, and your launch checklist. The discipline of naming the harms before the capabilities will change what you ship.

Do Good: The Second Constraint

“Do good” seems obvious — of course the goal is to do good. But Wesley’s point wasn’t that doing good is obvious. His point was that you have to choose it actively and specifically. You have to name the good you’re trying to do, not assume it follows from capability.

For AI product teams, this translates to specificity of purpose. The most dangerous AI features are the ones built with vague positive intent — “help users be more productive,” “improve the experience,” “make things easier.” These goals aren’t wrong; they’re insufficiently specific. They don’t constrain what the AI does well enough to actually do good reliably.

The teams building the most impactful AI features are the ones who can answer precisely: for which specific user, in which specific situation, doing which specific task, does this AI feature create genuine value? The narrower the answer, the more intelligence you can focus on serving that specific case. AI amplifies specificity. A vague AI feature serves everyone adequately. A specific AI feature genuinely transforms a particular user’s experience.

The prompt design corollary: every AI system prompt should contain an explicit statement of the specific good the AI is designed to do. Not “help users.” Something precise enough that you could evaluate whether a given output advances it or not.

Stay in Love With the Mission: The Third Constraint

Wesley’s third rule is the hardest to translate into product terms, but the translation is important. “Stay in love with God” was Wesley’s constraint against letting the institution — the methodology, the system, the practice — become the end rather than the means. The movement existed to serve spiritual formation, not itself. When the structure started serving itself, Wesley’s third rule was the corrective.

For AI product work, this is the constraint against losing the mission in the mechanics. It’s easy to become so focused on what the AI can do — what’s technically impressive, what gets stakeholder attention, what generates engagement metrics — that you lose track of whether the AI is actually serving the user’s underlying goal.

I’ve watched this happen. A team builds an AI writing assistant that users engage with heavily — and then realize the engagement is driven by the novelty of watching the AI write, not by the quality of what users are actually producing. The metric is green. The mission has drifted. The AI became the point rather than the tool.

“Stay in love with the mission” is a standing question for every AI feature review: is this feature advancing the user’s actual goal, or is it becoming the goal itself? The discipline of asking it consistently is what keeps AI features from becoming impressive demonstrations that quietly fail the user. More on this from a product ethics perspective: the Interaction Design Foundation’s framework on design patterns is a useful lens for evaluating where AI features cross from tool to substitution.

Why Constraint Beats Capability in AI Product Design

The Methodist movement succeeded not because Wesley had the most sophisticated theology, but because he had the most practical framework. Three rules anyone could remember, apply, and teach — that’s what scaled the movement across 18th-century Britain.

The teams building the most impactful AI products right now aren’t the ones deploying every capability. They’re the ones who’ve chosen clear constraints that channel AI capability toward specific, valuable outcomes. Their prompts are shorter and more focused. Their AI features do fewer things with more precision. Their users trust the outputs because the product was clearly designed with the user’s outcome in mind rather than the AI’s capability ceiling.

In a world where AI can do almost anything, the crucial question isn’t what’s possible. It’s what’s wise. Wesley figured out how to answer that question with three rules. Your AI prompts need the same discipline.

Your Turn: Apply This Today

Apply the three-constraint framework to your current AI features:

• Write the “do no harm” list for your highest-traffic AI feature. Name three things this AI should never output, regardless of what the user asks for. Write them as explicit constraints in your system prompt. If you’ve never done this exercise, do it before your next feature review — it will surface assumptions nobody has articulated.
• Rewrite your AI feature’s purpose statement as a specific user outcome. Replace “help users be more productive” with a sentence specific enough to evaluate: “Help [specific user type] accomplish [specific task] in [specific context] faster and with better results.” If you can’t write the specific version, the feature isn’t ready to ship.
• Add a “mission drift” check to your sprint review. Ask once per sprint: are users engaging with this AI feature because it’s helping them accomplish their goal, or because it’s impressive to interact with? The engagement metric doesn’t distinguish between these. Your qualitative research should.
• Test the three-rule prompt structure on your next system prompt. Write a system prompt that has three sections: what the AI must never do, what specific good the AI is designed to do, and what the user’s underlying goal is that the AI should always serve. Compare the outputs to your current prompt. The discipline almost always improves them.
• Audit your AI feature’s constraint density. Count the constraints in your current system prompt vs. the capabilities you’ve described. If capabilities dramatically outnumber constraints, you’ve built an AI feature by expansion rather than by design. Add at least one constraint for every three capabilities you’ve described.
• Apply the “Wesley test” before your next AI feature launch. Ask: does this feature do no harm? Does it advance a specific, named good for a specific user? Does it serve the user’s mission rather than substituting for it? If you can’t answer yes to all three, the feature needs more design work before it ships.

The constraint-beats-capability principle shows up in other forms too — the choice overload paradox in AI features is the consumer-facing version of the same dynamic, and Munger’s inversion principle is a complementary framework for designing around constraints rather than toward capabilities.

Building AI features and trying to avoid the “impressive but useless” trap? I consult with product teams on AI prompt architecture, constraint-based feature design, and building AI that serves users’ actual goals rather than demonstrating capability. Let’s talk.