Featured

PySpark-Check - data quality validation for PySpark 3.0.0

Last week, I was testing whether we can use AWS Deequ for data quality validation. I ran into a few problems. First of all, it...
Bartosz Mikulski
06 Jul 2020

The problem with software testing in data engineering

What if we found a bug in our data pipelines? What if that bug were easy to fix, but it would require a lot of...
Bartosz Mikulski
15 Jun 2020

Data flow - what functional programming and Unix philosophy can teach us about data streaming

What does stream processing have in common with functional programming and Unix?
Bartosz Mikulski
04 May 2020

Four books to boost your programmer career

I quit my dream job because of a book
Bartosz Mikulski
06 Jan 2020

All Stories

How to use AWS Batch to run a Python script

How to build a Docker image, define an AWS Batch job using Terraform, and run the AWS Batch job using Airflow
Bartosz Mikulski
26 Feb 2021

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Bartosz Mikulski
12 Feb 2021

How to test REST API contract using BDD

Testing a REST API using Behave in Python
Bartosz Mikulski
05 Feb 2021

Testing data products: BDD for data engineers

How to use BDD to test PySpark code
Bartosz Mikulski
29 Jan 2021

Definition of done for data engineers

When can data engineers be sure that they have done the task?
Bartosz Mikulski
14 Jan 2021

Don't learn another programming language

Should you learn a new programming language this year?
Bartosz Mikulski
07 Jan 2021

How to read from SQL table in PySpark using a query instead of specifying a table

Fetching data using a SQL query in PySpark
Bartosz Mikulski
01 Jan 2021

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Bartosz Mikulski
31 Dec 2020

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Bartosz Mikulski
30 Dec 2020

Send SMS from an Airflow DAG using AWS SNS

How to configure SNS subscription to send SMS messages and use Airflow to send them
Bartosz Mikulski
29 Dec 2020

How to emulate temporary tables in Athena

Use CTAS to create a temporary table in Athena
Bartosz Mikulski
28 Dec 2020

How to enable S3 bucket versioning using Terraform

How to configure Define S3 bucket versioning in Terraform
Bartosz Mikulski
27 Dec 2020