The metrics trap
analysis paralysis for product managers 2026-07-07 #product
Data-driven product management brings a double-edged sword to software product development. Data that illuminates a customer’s messy reality can save us from our simplified models and simplistic assumptions. But sometimes, we still need to make a decision, without reassuring supporting data. Asking for data can prevent bad decisions, but it can also prevent decisions.
Requirements analysis paralysis
In a way, we had a more straightforward software development experience in the past. Before modern product management, neither shame nor evidence influenced our assumptions, and we called them requirements.
Unfortunately, we trained generations of programmers, myself included, to ask for ever more detailed ‘requirements’ (in scare quotes), in an attempt to reduce uncertainty. Never mind that however detailed these so-called requirements, someone had just made them up.
Even then, we knew about analysis paralysis: the danger of overthinking that creative act of making up requirements. ????
Agile software development
Agile software development nearly solved the problem of sham requirements, by replacing them with user stories. I say nearly, because upon learning that user stories had become fashionable, the whole industry immediately forgot (or had never actually learned) that:
- a real user would plausibly actually tell the user story, e.g. to a colleague over lunch
- you’d know that they might do this because you had actually talked to them.
Without these ingredients, replacing sham requirements with sham user stories only gave us a lot less text. Now we could describe software capabilities on sticky notes, instead of in hundred-page specification documents. But we had’t dealt with uncertainty. That would need real data.
Behavioural analytics
The rise of e-commerce led to the discipline of behavioural analytics: using data about consumer behaviour to understand it. Wikipedia explains:
Behavioral analytics utilizes the massive volumes of raw user event data captured during sessions in which consumers use applications, games, or websites, including traffic data like navigation paths, clicks, social media interactions, purchasing decisions and marketing responsiveness.
Crucially, this concerns consumers and massive volumes of raw user event data. Business software, however, typically features neither.
Behavioural analysis paralysis
Business software with hundreds of daily active users, instead of consumer software’s millions, yields little user behaviour data. You can’t do A/B testing when it would take months or years to gather enough data to show a significant effect. Meanwhile, off-the-shelf shopping funnel analysis tools don’t work with complex business process workflows. But you can look at feature usage rates and success rates. Even small amounts of data can reveal big problems.
Behavioural analysis paralysis happens when you want to continue with data-driven product management, don’t have enough data, and can’t reasonably get it. You can insist on measuring behaviour before making product decisions, but then you risk getting bogged down in complex workflows, getting too little data, and not making a decision.
Data-informed project management
In practice, data-driven product management yields to data-informed decisions. In practice, data provides evidence, often circumstantial, not conclusions.
As a product manager, you have to become comfortable with uncertainty. But more than that, you have to handle other team members’ data discomfort, and their analysis paralysis.
