Use HttpSensor to pause an Airflow DAG until a website is available

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (58/100)

What should I do if I wanted to pause an Airflow DAG until one of my articles gets publishes?

Let’s assume that I have already scheduled publication of the text you are reading right now, and I know the article URL. It is going to be https://mikulskibartosz.name/use-httpsensor-to-wait-for-website

Now, I have to define an HTTP connection in my Airflow connections settings. The connection should point to the root address of my website: https://mikulskibartosz.name.

After that, I add a HttpSensor to my DAG. It is going to use the connection defined earlier and wait until the resource is available:

1
2
3
4
5
6
sensor = HttpSensor(
    task_id='http_sensor_check',
    http_conn_id='mikulskibartosz_name',
    endpoint='use-httpsensor-to-wait-for-website',
    poke_interval=60,
    dag=dag)


The Airflow DAG will wait until the website returns a status code, which indicates success. When you look at this sensor’s code, you will see that it returns False only when the failure was due to 404 Not Found status. In a case of any other failure, it propagates the exception to Airflow.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group


This website DOES NOT use cookies
but you may still see the cookies set earlier if you have already visited it.