Probability plot - visually compare probability distributions
Data Science

Probability plot - visually compare probability distributions

How to visually check whether your sample is normally distributed?
Bartosz Mikulski
Bartosz Mikulski – 2 min read

Probability plot is a method of comparing probability distribution which can be easily interpreted by humans. It is not some weird number which is denoted by some Greek letter you can’t even pronounce correctly. Probability plot is just a picture.

Probability plot with Scikit learn

Let’s plot the probability plot of the age variable from the Titanic dataset. I am going to check whether it fits to a normal distribution.

from scipy.stats import probplot
import seaborn as sea
import matplotlib.pyplot as plt
titanic = sea.load_dataset("titanic")
probplot(titanic.age, dist="norm", plot = plt)
Probability plot of the age column from the Titanic dataset.
Probability plot of the age column from the Titanic dataset.

What do we know? The data is not normally distributed, that we know for sure. If it were the probability plot would look like this:

from scipy.stats import norm
example = norm.rvs(200, size = 200)
probplot(example, dist="norm", plot = plt)
A probability distribution that matches the given distribution. In this case, a normal distribution.
A probability distribution that matches the given distribution. In this case, a normal distribution.

What if we want to compare the data to a different kind of probability distribution? Just change the value of the “dist” parameter. The interpretation of the output stays the same.

Interpretation

If the distribution of given sample fits the given probability distribution, the plot displays a line that follows the line x = y (the second plot in this blog post).

What does the first plot mean? An arced line (or even S-shaped line) indicates that the given sample distribution is skewed or has a “heavy tail”. Is it true? Let’s look at the age variable histogram:

The histogram of the age variable
The histogram of the age variable

As expected the distribution is skewed.

Why the probability plot is useful? You can quickly draw the plot, look at it, and know whether the sample has the expected distribution. You don’t need a statistical test. You don’t need to think about the significance level. You don’t need to think about the null hypothesis and the alternative hypothesis. You don’t need to decide whether you want to use a Shapiro–Wilk test, Kolmogorov–Smirnov test or something else. You just make a plot and look at it ;)

Older post

Count unique elements of an infinite stream of objects

HyperLogLog - probabilistic counting algorithm

Newer post

[book review] James Whittaker's Little Book of the Future

Read this book if you believe we can use A.I. and IoT to build a bright future.

Related Posts

Career Data Science

I worked as a data scientist and that was the worst job I have ever had.

I believed in the Sexiest Job of 21 Century hype. I was wrong.
Published on: 04 Mar 2022

Data Science

How to remove outliers from Seaborn boxplot charts

Hide outliers when displaying boxplot in Seaborn
Published on: 14 Oct 2019

Data Science

Pandas stack and unstack explained

How to use the stack and unstack functions in Pandas
Published on: 23 Sep 2019