All Stories

Broadcast variables and broadcast joins in Apache Spark

How to speed up joins of small DataFrames by using the broadcast join
Bartosz Mikulski
28 Sep 2020

How to use the window function to get a single row from each group in Apache Spark

How to group values by a key and extract a single row from each group in Apache Spark
Bartosz Mikulski
27 Sep 2020

How to make a pivot table in AWS Athena or PrestoSQL

How to make a pivot table in AWS Athena, and why the pivot function does not exist
Bartosz Mikulski
26 Sep 2020

What is the difference between repartition and coalesce in Apache Spark?

When should you use coalesce instead of repartition in Apache Spark
Bartosz Mikulski
25 Sep 2020

How to pivot an Apache Spark DataFrame

How to turn an Apache Spark or PySpark DataFrame into a pivot table.
Bartosz Mikulski
24 Sep 2020

What is the difference between cache and persist in Apache Spark?

When should you use the cache, and when you should use the persist function
Bartosz Mikulski
23 Sep 2020

Why your company should use PrestoSQL

Should your team use PrestoSQL?
Bartosz Mikulski
16 Sep 2020

Is counting rows all we can do?

How to detect problems in data pipelines before they turn into hard to debug bugs? I wish I knew.
Bartosz Mikulski
08 Sep 2020

How to Speed Up AWS Athena Queries Using Partition Projection

How to define partition projection while creating an Athena table
Bartosz Mikulski
30 Aug 2020

How to send a customized Slack notification when an Airflow task fails

How to customize a Slack notification before sending it to the Slack incoming webhook.
Bartosz Mikulski
27 Aug 2020

How to use one SparkSession to run all Pytest tests

How to speed us Pytest tests by reusing the same SparkSession in all of them
Bartosz Mikulski
20 Jul 2020

How to send AWS CloudWatch Alerts to a Slack channel using Terraform

How to use Terraform to configure a CloudWatch alert and send the message to a Slack channel.
Bartosz Mikulski
13 Jul 2020
About me | Newsletter