Home / The Question Problem

The Question Problem.

Chat works best when the user knows what to ask. Many useful products have the opposite problem: finding the question they would not have known to type.

It's the dominant pattern in new AI products, and the assumption is easy to miss.

During World War II the US military studied bullet holes on bombers returning from missions and planned to armour the most damaged areas. Abraham Wald, a statistician advising the effort, pointed out that they were only seeing the planes that made it back. The fatal hits were on the ones that didn't. Chat interfaces have a similar blind spot. Conversation can help a user sharpen a vague question, but it can't surface what they'd never think to ask. The questions a user types are the planes that returned. The rest never show up.

For products where users arrive with a clear question, this isn't a problem. Summarise a document, draft a reply, look up a number. Chat handles all of that well. The problem shows up in products where finding the right question is what the user needs help with.

A dashboard already shows you when a single metric is trending badly. But a user who doesn't know that two metrics are connected, that a drop in one explains a spike in another, won't think to ask about the relationship. The interface returns a confident answer to whatever was asked. The connection stays invisible. If they want to track something over time, they have to remember to come back. The product won't tell them anything changed.

The design challenge is recognising when a problem calls for something other than chat. A product built around monitoring or understanding data probably shouldn't sit there waiting to be asked. One direction worth exploring is treating the history of what a user has queried as a signal. Track what they've looked at, compare it against what's available, and surface the gaps as suggestions.