All Stories

Why does the ExternalTaskSensor get stuck?

How to fix the stuck ExternalTaskSensor
Bartosz Mikulski
23 Oct 2020

How to render an Airflow template for testing

How to generate the code of an Airflow task from a template and a given execution date
Bartosz Mikulski
22 Oct 2020

How to check the next execution date of an Airflow DAG

How to use Airflow CLI to get the next execution date of a DAG
Bartosz Mikulski
21 Oct 2020

Doing data quality checks using the SQLCheckOperator

How to use SQLCheckOperator to verify that the database contains an expected number of rows
Bartosz Mikulski
20 Oct 2020

How to deal with the jinja2 TemplateNotFound error in Airflow

How to fix the TemplateNotFound error while using a custom Airflow operator
Bartosz Mikulski
19 Oct 2020

How to postpone Airflow DAG until files get uploaded into an S3 bucket

How to use Airflow sensors to detect that files have been uploaded into an S3 bucket
Bartosz Mikulski
18 Oct 2020

What is the difference between a transformation and an action in Apache Spark?

What is an action in Apache Spark? What do you understand as transformations in Apache Spark?
Bartosz Mikulski
17 Oct 2020

Use LatestOnlyOperator to skip some tasks while running a backfill in Airflow

How to skip some tasks when backfilling a DAG in the past
Bartosz Mikulski
16 Oct 2020

Christopher Bergh - How the DataOps principles help data engineers make data pipelines trustworthy

An interview with Christopher Bergh who explains how the DataOps principles help data engineers make data pipelines trustworthy
Bartosz Mikulski
16 Oct 2020

How to retrieve the statuses of the recent DAG executions from Airflow database

How to make a dashboard that displays Airflow DAG statuses
Bartosz Mikulski
15 Oct 2020

How to find and terminate an idle Redshift session

How to find the idle session that is blocking the connection pool in Redshift
Bartosz Mikulski
14 Oct 2020

How to configure Spark to maximize resource usage while using AWS EMR

How to configure EMR to use all available resources when running a Spark cluster
Bartosz Mikulski
13 Oct 2020