Use HttpSensor to pause an Airflow DAG until a website is available
What should I do if I wanted to pause an Airflow DAG until one of my articles gets publishes?
Let’s assume that I have already scheduled publication of the text you are reading right now, and I know the article URL. It is going to be https://mikulskibartosz.name/use-httpsensor-to-wait-for-website
Now, I have to define an HTTP connection in my Airflow connections settings. The connection should point to the root address of my website: https://mikulskibartosz.name.
After that, I add a HttpSensor to my DAG. It is going to use the connection defined earlier and wait until the resource is available:
1 2 3 4 5 6 sensor = HttpSensor( task_id='http_sensor_check', http_conn_id='mikulskibartosz_name', endpoint='use-httpsensor-to-wait-for-website', poke_interval=60, dag=dag)
The Airflow DAG will wait until the website returns a status code, which indicates success. When you look at this sensor’s code, you will see that it returns False only when the failure was due to 404 Not Found status. In a case of any other failure, it propagates the exception to Airflow.
Did you enjoy reading this article?
Would you like to learn more about software craft in data engineering and MLOps?
Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!
You may also like
- Data/MLOps engineer by day
- DevRel/copywriter by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz