In this article, I am going to show you three kinds of metrics that make no sense. Those metrics are susceptible to “metric hacking,” and can be improved with little effort even if the overall condition of the company keeps getting worse. Ideally, we should avoid using such metrics and never make any of them our goal.

False metrics

The first kind of them is what Seth Godin calls “false metrics.”

What is a false metric? A false metric is a metric that promotes cheating, and gaming the system instead of improving the quality of work.

For example, Seth Godin talks about car dealers who are so preoccupied with getting a 5-star rating from the customer that they spend more time persuading people to give them five stars than providing five-star service.

If you are using OKRs, you may be familiar with the concepts of health checks. It is an additional value that is supposed to prevent “metric hacking.” The goal is to achieve the “key result” without breaking the rule defined by the “health metric.”

If one of the “key results” is to achieve a 30% increase in the number of sales, “creative” salespeople may decide to give everyone a 50% discount. That is a proven way of increasing sales, but for sure nobody wants such a solution. In this case, we can define the health metric as: “The average revenue per client does not decrease more than 5 %”.

Vanity metrics

A vanity metric is a measurement that can only improve, such as the number of registered users or the total value of transactions.

I have learned about this concept from the Lean Analytics book, by Alistair Croll and Ben Yoskovitz. It makes no sense to track such metrics because even if we do everything wrong, this value will not decrease.

On the other hand, if we do everything wrong, it will also not increase. This observation gives us a hint about what is crucial. In this case, the parameter we should track is the number of users who registered in a given period. When we do it, we can derive even more useful metrics like the ratio between registrations last month and the current month (to check if we are “speeding up”).

The next stage may be tracking churn (the number of people who stop using your service every month). Once we start gathering the right metrics, we can build up on top of that. Vanity metrics don’t give us any hints about what we should track next.

Short-term success

The third kind of harmful metric is the ones that promote short term solutions. If we start giving special offers to customers who are dissatisfied with your service to prevent churn, it may give us short term improvements.

Certainly, we are going to prevent churn, but guess what is going to happen when people start giving each other tips online. The next year, we will see both a considerable increase in the number of complaints and a growing ineffectiveness of our churn prevention method.

Health metrics will not help us prevent adverse long-term effects. The second problem is the behavior you promote. If we give employees bonuses for achieving the goal, they will accomplish that goal even if they must destroy the business to do it. That is why it always good to have someone on the team who plays the role of a “devil’s advocate” and tries to think of unwanted second-order effects.

Business metrics that make no sense

False metrics

Vanity metrics

Short-term success

Nested cross-validation in time series forecasting using Scikit-learn and Statsmodels

Predicting customer churn using the Pareto/NBD model

Business metrics that make no sense

False metrics

Vanity metrics

Short-term success

Nested cross-validation in time series forecasting using Scikit-learn and Statsmodels

Predicting customer churn using the Pareto/NBD model

Related Posts

I worked as a data scientist and that was the worst job I have ever had.

How to remove outliers from Seaborn boxplot charts

Pandas stack and unstack explained