All Stories

How to select a random sample of rows using Athena

How to use a window function to select random rows from Athena
Bartosz Mikulski
21 Nov 2020

Use HttpSensor to pause an Airflow DAG until a website is available

Pause an Airflow DAG until an HTTP endpoint returns 200 OK
Bartosz Mikulski
20 Nov 2020

How to add an EMR step in Airflow and wait until it finishes running

How to use AwsHook and EmrStepSensor to add an EMR step and wait until it finishes running
Bartosz Mikulski
19 Nov 2020

How to use Virtualenv to prepare a separate environment for Python function running in Airflow

How to use the PythonVirtualenvOperator in Airflow
Bartosz Mikulski
18 Nov 2020

Remove a directory from S3 using Airflow S3Hook

How to remove files with a common prefix from S3
Bartosz Mikulski
17 Nov 2020

Run a command on a remote server using SSH in Airflow

how to use the SSHHook in a PythonOperator to connect to a remote server from Airflow using SSH and execute a command.
Bartosz Mikulski
16 Nov 2020

Use the ROW_NUMBER() function to get top rows by partition in Hive

How to calculate row number by partition in Hive and use it to filter rows
Bartosz Mikulski
15 Nov 2020

How to configure both core and spot instances in EMR using Terraform

Use EMR instance group to add spot instances to an EMR cluster
Bartosz Mikulski
14 Nov 2020

How to temporarily disable an AWS Lambda function using AWS CLI without removing the function

Disable an AWS Lambda using AWS CLI
Bartosz Mikulski
13 Nov 2020

How to add an EMR step from AWS Lambda

How to configure a new EMR step using AWS Lambda in Python
Bartosz Mikulski
12 Nov 2020

Send event to AWS Lambda when a file is added to an S3 bucket

Trigger AWS Lambda when a file is created in an S3 bucket
Bartosz Mikulski
11 Nov 2020

Select Serverless configuration variables using the stage parameter

Bartosz Mikulski
10 Nov 2020
About me | Newsletter