How to choose the right mini-batch size in deep learning
Deep learning

How to choose the right mini-batch size in deep learning

Andrew Ng recommendation about mini batch size
Bartosz Mikulski
Bartosz Mikulski – 1 min read

This blog post contains a summary of Andrew Ng’s advice regarding choosing the mini-batch size for gradient descent while training a deep learning model. Fortunately, this hint is not complicated, so the blog post is going to be extremely short ;)

Andrew Ng recommends not using mini-batches if the number of observations is smaller then 2000. In all other cases, he suggests using a power of 2 as the mini-batch size. So the minibatch should be 64, 128, 256, 512, or 1024 elements large.

The most important aspect of the advice is making sure that the mini-batch fits in the CPU/GPU memory! If data fits in CPU/GPU, we can leverage the speed of processor cache, which significantly reduces the time required to train a model!

Older post

How to deal with underfitting and overfitting in deep learning

The lessons learned from Andrew Ng’s online course

Newer post

Which hyperparameters of deep learning model are important and how to find them

How to speed up finding the right hyperparameters of a machine learning model

Related Posts

Deep learning

Why do we use dropout in artificial neural networks?

How does dropout work in artificial neural networks?
Published on: 12 Mar 2021

TensorFlow Deep learning

How to automatically select the hyperparameters of a ResNet neural network

Training ResNet network for multiclass image classification using keras-tuner
Published on: 12 Aug 2019

TensorFlow Deep learning

Using Hyperband for TensorFlow hyperparameter tuning with keras-tuner

Tuning TensorFlow with Hyperband
Published on: 09 Aug 2019