Doing data quality checks using the SQLCheckOperator

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (28/100)

SQLCheckOperator is an Airflow operator that executes a SQL query, expects to receive a single row in the response, and attempts to cast every value in the row to a boolean. It succeeds when all returned values can be cast to true, so the query may return those values:

  • a boolean True

  • a non-zero numeric value (including negative values!)

  • a non-empty string

  • a non-empty list, set, or dictionary

In addition to failing when any of the values is False, the SQLCheckOperator operator also fails when the query returns no rows.

For example, we can use that operator to check the count of values in a table:

1
2
3
4
5
from airflow.operators.sql import SQLCheckOperator

operator = SQLCheckOperator(
     sql="SELECT COUNT(*) FROM some_table WHERE some_column='{{ yesterday_ds_nodash }}'"
)



Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group


This website DOES NOT use cookies
but you may still see the cookies set earlier if you have already visited it.