Featured

PySpark-Check - data quality validation for PySpark 3.0.0

Last week, I was testing whether we can use AWS Deequ for data quality validation. I ran into a few problems. First of all, it...
Bartosz Mikulski
06 Jul 2020

The problem with software testing in data engineering

What if we found a bug in our data pipelines? What if that bug were easy to fix, but it would require a lot of...
Bartosz Mikulski
15 Jun 2020

Data flow - what functional programming and Unix philosophy can teach us about data streaming

What does stream processing have in common with functional programming and Unix?
Bartosz Mikulski
04 May 2020

Four books to boost your programmer career in 2020

I quit my dream job because of a book
Bartosz Mikulski
06 Jan 2020

All Stories

Send a Slack message from an Airflow DAG

How to use the SlackAPIPostOperator to send a templated message to a Slack channel
Bartosz Mikulski
29 Oct 2020

How to delay an Airflow DAG until a given hour using the DateTimeSensor

How to use the DateTimeSensor in Airflow
Bartosz Mikulski
28 Oct 2020

How to run PySpark code using the Airflow SSHOperator

How to submit a PySpark job using SSHOperator in Airflow
Bartosz Mikulski
27 Oct 2020

How to add a manual step to an Airflow DAG using the JiraOperator

How can you add a human action to an Airflow DAG?
Bartosz Mikulski
26 Oct 2020

How Data Mechanics can reduce your Apache Spark costs by 70%

Stop wasting time and money tuning Apache Spark parameters
Bartosz Mikulski
26 Oct 2020

Conditionally pick an Airflow DAG branch using an SQL query

How to use the BranchSQLOperator to choose a DAG branch to execute
Bartosz Mikulski
25 Oct 2020

How to trigger an Airflow DAG from another DAG

How to trigger another DAG from an Airflow DAG
Bartosz Mikulski
24 Oct 2020

Why does the ExternalTaskSensor get stuck?

How to fix the stuck ExternalTaskSensor
Bartosz Mikulski
23 Oct 2020

How to render an Airflow template for testing

How to generate the code of an Airflow task from a template and a given execution date
Bartosz Mikulski
22 Oct 2020

How to check the next execution date of an Airflow DAG

How to use Airflow CLI to get the next execution date of a DAG
Bartosz Mikulski
21 Oct 2020

Doing data quality checks using the SQLCheckOperator

How to use SQLCheckOperator to verify that the database contains an expected number of rows
Bartosz Mikulski
20 Oct 2020

How to deal with the jinja2 TemplateNotFound error in Airflow

How to fix the TemplateNotFound error while using a custom Airflow operator
Bartosz Mikulski
19 Oct 2020