How to deal with underfitting and overfitting in deep learning

How to deal with underfitting and overfitting in deep learning

In this article, I am going to summarize the facts about dealing with underfitting and overfitting in deep learning which I have learned from Andrew Ng’s course.

Ideal model

Before we start, we must decide what the best possible performance of a deep learning model is. We may find the best possible result by checking the accuracy achieved by human experts when working on the same test dataset.

We need it because that information allows us to set reasonable goals and have expectations that can be achieved. After all, if human experts make mistakes in 5% of the cases, and we use a dataset prepared by those experts for training the model, we can’t expect 100% accuracy (or even 95% accuracy).

Underfitting

The first kind of problem is underfitting. It occurs when we observe that the model’s training set error is significantly larger than the expected error of an ideal model. In machine learning, when the model performs poorly even on the training set, we say that the model has a high bias.

According to Andrew Ng, the best methods of dealing with an underfitting model is trying a bigger neural network (adding new layers or increasing the number of neurons in existing layers) or training the model a little bit longer.

Why does he give such advice? An underfitting neural network may be not powerful enough to capture the patterns in the training dataset. In this case, adding more layers or increasing the number of neurons may help overcome the problem.

Increasing the training time is advised because it is possible that the underperforming model has not found the most optimal values of its parameters yet.

Do you want to show your product/service to 25000 data science enthusiasts every month? I am looking for companies which would like to become a partner of this blog.

Are you interested? Is your employer interested? Here are the details of the offer.

Overfitting

Overfitting occurs when the model performs well when it is evaluated using the training set, but cannot achieve good accuracy when the test dataset is used. This kind of problem is called “high variance,” and it usually means that the model cannot generalize the insights from the training dataset.

Andrew Ng suggests that the best solution to overfitting is getting more data and using regularization.

Such a solution is suggested, because the model may not get enough training examples to learn the patterns properly, so adding news observations to the training dataset may increase the chance of getting a better model.

On the other hand, it is possible that the neural network is too complicated and because of that, it can deal adequately only with the training set examples. If it were a human writing an exam at school, we would say that he/she has memorized the homework, but he/she did not learn the concept. The same may happen to a machine learning model.

Because of that, we use regularization techniques to force the model to generalize the insights. In deep learning, we can choose a few such techniques:

  • L1 and L2 methods are used to modify the cost function in a way that penalizes more complex models.

  • Dropout is used to prevent the model from relying too much on one input because the input may disappear

  • We use early stopping to prevent “memorization” of training examples

  • Data augmentation — to make the training dataset more diverse

Andrew Ng also makes a critical comment on using dropout. He tells that we should first focus on finding the proper hyperparameters and don’t bother about dropout. After we have good hyperparameters, we may start adding and tuning dropout. Those two tasks can be done separately and to avoid complexity we should not try to do them at the same time.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you watch programming live streams, check out my YouTube channel.
You can also follow me on Twitter: @mikulskibartosz

If you want to hire me, send me a message on LinkedIn or Twitter.


If this article was helpful, consider donating to WWF or any other charity of your choice.
Bartosz Mikulski
Bartosz Mikulski * data scientist / software engineer * conference speaker * organizer of School of A.I. meetups in Poznań * co-founder of Software Craftsmanship Poznan & Poznan Scala User Group