mikulskibartosz.name
Career Coaching for Data Professionals
Speaker
Bartosz Mikulski
Building trustworthy data pipelines because AI cannot learn from dirty data
All Stories
Use HttpSensor to pause an Airflow DAG until a website is available
Pause an Airflow DAG until an HTTP endpoint returns 200 OK
How to add an EMR step in Airflow and wait until it finishes running
How to use AwsHook and EmrStepSensor to add an EMR step and wait until it finishes running
How to use Virtualenv to prepare a separate environment for Python function running in Airflow
How to use the PythonVirtualenvOperator in Airflow
Remove a directory from S3 using Airflow S3Hook
How to remove files with a common prefix from S3
Run a command on a remote server using SSH in Airflow
how to use the SSHHook in a PythonOperator to connect to a remote server from Airflow using SSH and execute a command.
Use the ROW_NUMBER() function to get top rows by partition in Hive
How to calculate row number by partition in Hive and use it to filter rows
How to configure both core and spot instances in EMR using Terraform
Use EMR instance group to add spot instances to an EMR cluster
How to temporarily disable an AWS Lambda function using AWS CLI without removing the function
Disable an AWS Lambda using AWS CLI
How to add an EMR step from AWS Lambda
How to configure a new EMR step using AWS Lambda in Python
Send event to AWS Lambda when a file is added to an S3 bucket
Trigger AWS Lambda when a file is created in an S3 bucket
Select Serverless configuration variables using the stage parameter
Use a custom function in Airflow templates
How to add a custom function to Airflow and use it in a template
« Prev
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Next »