All Stories

How to check whether a regular expression matches a string in Hive

What is the equivalent of Athena/Presto regexp_like in Hive
Bartosz Mikulski
10 Dec 2020

How to check whether a YARN application has finished

How to use Airflow PythonSensor to check whether a YARN application finished running

Bartosz Mikulski
09 Dec 2020

How to use WHEN CASE queires in AWS Athena

Using conditions in AWS Athena queries
Bartosz Mikulski
08 Dec 2020

How to decode base64 to text in AWS Athena

How to use from_base64 in AWS Athena
Bartosz Mikulski
07 Dec 2020

How to combine two DataFrames with no common columns in Apache Spark

Use full outer join to combine two Apache Spark DataFrames with no common columns
Bartosz Mikulski
06 Dec 2020

How to get names of columns with missing values in PySpark

How to get the names of missing properties for every row in a PySpark Dataframe
Bartosz Mikulski
05 Dec 2020

How to set a different retry delay for every task in an Airflow DAG

How to use a different retry delay in every Airflow task
Bartosz Mikulski
04 Dec 2020

How to find the Hive partition closest to a given date

How to use Airflow to find the Hive partition closest to a given date
Bartosz Mikulski
03 Dec 2020

Get the date of the previous successful DAG run in Airflow.

Get the start time or the execution date of the previous successful DAG run in Airflow
Bartosz Mikulski
02 Dec 2020

How to prevent Airflow from backfilling old DAG runs

How to disable backfilling of an Airflow DAG or skip a part of the DAG during a backfill
Bartosz Mikulski
01 Dec 2020

What is s3:TestEvent, and why does it break my event processing?

S3 sends s3:TestEvent to SQS after setting up the bucket notifications
Bartosz Mikulski
30 Nov 2020

Making OFFSET LIMIT queries in AWS Athena

How to use OFFSET in AWS Athena queries
Bartosz Mikulski
29 Nov 2020