A dashboard shows 742 payment errors. Before treating that count as an incident, the team needs the number of payment attempts, the period covered and a definition of what counts as an error. Failed attempts, customers unable to pay and completed orders are different units of analysis.
The same count can describe very different situations depending on the volume of activity behind it.
Let’s investigate the context of the metric
If the product processed 120,000 payments yesterday, 742 errors represent around 0.6% of all payments. If it processed 5,000 payments, the same 742 errors represent almost 15%. The number hasn’t changed, but the story has.
This happens all the time when we work with product metrics. We see a number and our brain immediately tries to interpret it. Conversion went up, good. Retention went down, bad. We have 50,000 active users, impressive. There were 742 errors, worrying. The problem is that a metric rarely tells us enough on its own.
Take something as common as “conversion increased from 20% to 30%”. It sounds like great news, but what conversion are we talking about? Visit to signup? Signup to activation? Trial to paid? Which users are included? What period are we comparing? Did traffic change during that period? Has the way we calculate the metric remained the same?
The same thing happens when someone says that the product has 50,000 active users. Before deciding whether that is good, we need to know what “active” actually means. Sending a message could be meaningful activity for a messaging app. Creating or updating a task might make more sense for a project management tool. For software that people only need a few times a year, weekly activity might be almost meaningless.
There is no universal definition because the metric is supposed to represent something happening in the product. That’s why defining a metric is already a product decision. We are deciding which behavior matters, which users count and over what period we want to observe it.
A checkout completion rate of 81% needs a comparable baseline. In a hypothetical product whose usual range is 75% to 85%, it may be consistent with recent behaviour. In another that has stayed between 96% and 98%, the same result warrants investigation. The journey definition and measurement window must remain comparable.
This is why I rarely find an isolated data point very useful. I want to know what happened yesterday, last week or before the last release. I want to know what the normal range looks like. Sometimes I also want to know whether Mondays behave differently from Fridays or whether there is some obvious seasonal pattern.
A number becomes much more useful when we can place it next to something. And even then, the overall number can hide what is really happening. Imagine that the payment error rate for our product is 4.1%. We investigate a little further and discover that it is 1.8% on Web, 2% on iOS and 18.4% on Android. Now the conversation is different. We don’t really have a general payment problem. Something seems to be happening on Android.
So we look again. Android 8.3 has an error rate of 2.1%, while Android 8.4 has an error rate of 31.7%. This is one of the reasons segmentation is so useful in Product Management. An average combines many different users into a single number, and sometimes the experience of one group can disappear inside that average. Looking at the metric by platform, country, plan, acquisition channel, customer type or app version can reveal a completely different story.
That doesn’t mean every metric needs twenty different breakdowns. Most of the time, the overall number is a perfectly reasonable place to start. But when something changes, we shouldn’t stop there.
The same caution applies to anomalies. If orders per minute suddenly increase by 150%, something unusual has happened, but that doesn’t automatically mean something is broken. Maybe Marketing has launched a campaign. Maybe a large customer has started using the product. Maybe demand is simply higher than usual.
Investigating an unexpected result
I think this is where dashboards can sometimes give us a false sense of certainty. They make numbers look definitive. There is a line, a percentage, perhaps a red arrow pointing down. Everything looks very precise, so it is tempting to think the interpretation must be precise too.
But the dashboard doesn’t know why activation dropped. It doesn’t know whether a conversion increase came from the change we released last week. It doesn’t know whether 742 errors are catastrophic or completely normal. It is showing us evidence. We still have to understand it.
For a Product Manager, that means getting comfortable with not jumping immediately from a number to a conclusion. When a metric moves, I want to know what exactly we’re measuring, who is included, what the denominator is, what period we’re looking at and what we’re comparing it with. If the change looks interesting, I want to know whether it affects everyone or whether a particular segment is driving it.
Then there is another question that is surprisingly useful: what would I need to know before making a decision based on this number? These checks give an investigation a direction: confirm the definition, compare the relevant baseline and identify the affected segments before attributing a cause.
A metric belongs in a regular product review when it helps the team assess an outcome or decide where to investigate. Additional breakdowns can remain available for diagnosis without competing for attention in every meeting.
For the payment example, the next useful figures are the total attempts, the comparable error rate and the distribution by platform or payment method.
Product Metrics for Product Managers develops these questions through metric definitions, segmentation and product decisions.
