Use LatestOnlyOperator to skip some tasks while running a backfill in Airflow

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (24/100)

Occasionally, when we use Airflow, we have a DAG which always works on the most recent snapshot of data even if we run a backfill for days in the past. It happens when at least one of the tasks downloads a snapshot of the current state from an external API, uploads data into another service (using an API, FTP upload, sending events to a message queue, or anything else), or pushes some information to users.

The last case is the worst. Certainly, we don’t want to send old versions of newsletters to all of the subscribers just because we had to backfill some values in a report.

How do we make sure that such time-sensitive tasks are executed only in the most recent DAG run and don’t start when we run a backfill? We can do that by using the LatestOnlyOperator in Airflow!

It is simple to use it. All we need to do is importing the operator, creating a new instance, and adding it to the DAG:

from airflow.operators.latest_only_operator import LatestOnlyOperator

is_this_latest_dag_run = LatestOnlyOperator(task_id="dont_send_newsletters_during_backfills", dag=dag)

upstream_task >> is_this_latest_dag_run >> send_newsletters >> other_downstream_task

When the operator runs, it checks whether the current time is between the most recent execution time and the next scheduled execution time. If yes, it lets the downstream tasks run. If no, it means that we are running a backfill, and the downstream tasks get skipped.

Would you like to help fight youth unemployment while getting mentoring experience?

Develhope is looking for tutors (part-time, freelancers) for their upcoming Data Engineer Courses.

The role of a tutor is to be the point of contact for students, guiding them throughout the 6-month learning program. The mentor supports learners through 1:1 meetings, giving feedback on assignments, and responding to messages in Discord channels—no live teaching sessions.

Expected availability: 15h/week. You can schedule the 1:1 sessions whenever you want, but the sessions must happen between 9 - 18 (9 am - 6 pm) CEST Monday-Friday.

Check out their job description.

(free advertisement, no affiliate links)

Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Bartosz Mikulski
Bartosz Mikulski * MLOps Engineer / data engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.