Doing data quality checks using the SQLCheckOperator
SQLCheckOperator is an Airflow operator that executes a SQL query, expects to receive a single row in the response, and attempts to cast every value in the row to a boolean. It succeeds when all returned values can be cast to true, so the query may return those values:
-
a boolean
True
-
a non-zero numeric value (including negative values!)
-
a non-empty string
-
a non-empty list, set, or dictionary
In addition to failing when any of the values is False
, the SQLCheckOperator operator also fails when the query returns no rows.
For example, we can use that operator to check the count of values in a table:
1
2
3
4
5
from airflow.operators.sql import SQLCheckOperator
operator = SQLCheckOperator(
sql="SELECT COUNT(*) FROM some_table WHERE some_column='{{ yesterday_ds_nodash }}'"
)
Did you enjoy reading this article?
Would you like to learn more about leveraging AI to drive growth and innovation, software craft in data engineering, and MLOps?
Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!
You may also like

Bartosz Mikulski
- MLOps engineer by day
- AI and data engineering consultant by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz
- Mastodon: @mikulskibartosz@mathstodon.xyz