How to retry a Python function call

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (38/100)

The worst thing about making calls to external service is that those calls sometimes fail because of network errors or because the service is temporarily unavailable. It makes sense to retry the function call a few times and hope that the problem disappears in those situations.

The retry package gives us an easy method to implement retries in Python. All we need is a decorator added to the function we want to retry.

First, we have to install the package. If we use pip, the installation looks like this: pip install retry.

After that, we must find the definition of the function we want to retry, for example, this one:

1
2
def a_function_that_fails_for_no_reason():
    ...

Now, we add the decorator to the function. To define a decorator, we need the names of the errors that should cause a retry. All other errors will be propagated to the caller without retrying the function.

Additionally, we need the number of retries and the delay between the subsequent function calls.

1
2
3
4
5
from retry import retry

@retry((HTTPError, DNSError), tries = 3, delay = 2, backoff = 2)
def a_function_that_fails_for_no_reason():
    ...

Did you enjoy reading this article?
Would you like to learn more about software craft in data engineering and MLOps?

Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!

Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.

Bartosz Mikulski

Bartosz Mikulski

  • Data/MLOps engineer by day
  • DevRel/copywriter by night
  • Python and data engineering trainer
  • Conference speaker
  • Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
  • Twitter: @mikulskibartosz
Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.