Doing data quality checks using the SQLCheckOperator
SQLCheckOperator is an Airflow operator that executes a SQL query, expects to receive a single row in the response, and attempts to cast every value in the row to a boolean. It succeeds when all returned values can be cast to true, so the query may return those values:
-
a boolean
True
-
a non-zero numeric value (including negative values!)
-
a non-empty string
-
a non-empty list, set, or dictionary
In addition to failing when any of the values is False
, the SQLCheckOperator operator also fails when the query returns no rows.
For example, we can use that operator to check the count of values in a table:
1
2
3
4
5
from airflow.operators.sql import SQLCheckOperator
operator = SQLCheckOperator(
sql="SELECT COUNT(*) FROM some_table WHERE some_column='{{ yesterday_ds_nodash }}'"
)

You may also like
Remember to share on social media! If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.
If you want to contact me, send me a message on LinkedIn or Twitter.
Would you like to have a call and talk? Please schedule a meeting using this link.