Business metrics that make no sense

Business metrics that make no sense

What happens when you give people a metric to improve? They will find a way to improve it with the least effort possible.

Sometimes it gives you what you want. After all, the business wisdom says that you should hire smart people, tell them what you want, and get out of their way. On the other hand, often such an approach ends with “metric hacking.” You get an improvement in the measurement, but it hurts your business overall.

False metrics

There are many kinds of wrong metrics. The first kind of them is what Seth Godin calls “false metrics.”

In his blog post, he describes metrics that promote cheating. For example, he describes car dealers who are so preoccupied with getting a 5-star rating from the customer that they spend more time persuading people to give them five stars than providing five-star service.

If you are using OKRs, you may be familiar with the concepts of health checks. It is an additional value that is supposed to prevent “metric hacking.” The goal is to achieve the “key result” without breaking the rule defined by the “health metric.”

If one of the “key results” is to achieve a 30% increase in the number of sales, “creative” salespeople may decide to give everyone a 50% discount. That is a proven way of increasing sales, but for sure nobody wants such a solution. In this case, the health metric may be “The average revenue per client does not decrease more than 5 %”.

Vanity metrics

The second problematic kind of metric are values that can only improve.

The authors of the Lean Analytics book, Alistair Croll and Ben Yoskovitz, call them vanity metrics. It makes no sense to track things like the total number of registrations. Even if you do everything wrong, this value will not decrease. You can tell, that you can always see that it does not increase either.

That is the point! The parameter you should track is the number of users who registered in a given period. When you do it, you can derive even more useful metrics like the ratio between registrations last month and the current month (to check if you are “speeding up”).

Once you start gathering the right metrics, you can build up on that. The next stage may be tracking churn (the number of people who stop using your service every month).

Short-term success

The third kind of harmful metrics are the ones that promote short term solutions. If you start giving special offers to customers who are dissatisfied with your service to prevent churn, it may give you short term improvements.

For sure, you are going to prevent churn. Guess what is going to happen, when people start giving each other tips online. The next year you are going to see both a considerable increase in the number of complaints and a growing ineffectiveness of your churn prevention method.

Health metrics will not help you prevent negative long-term effects. The second problem is the behavior you promote. If you give employees bonuses for achieving the goal, they will achieve that goal even if they must destroy the business in order to do it. That is why it always good to have someone on the team who plays the role of a “devil’s advocate” and tries to think of unwanted second-order effects.

Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you watch programming live streams, check out my YouTube channel.
You can also follow me on Twitter: @mikulskibartosz

For business inquiries, send me a message on LinkedIn or Twitter.

Bartosz Mikulski
Bartosz Mikulski * data scientist / software engineer * conference speaker * organizer of School of A.I. meetups in Poznań * co-founder of Software Craftsmanship Poznan & Poznan Scala User Group