mikulskibartosz.name
Consulting
Speaker
Bartosz Mikulski
I help data engineering tech leads #makeDataTrustworthy because AI cannot learn from dirty data
Featured
PySpark-Check - data quality validation for PySpark 3.0.0
Last week, I was testing whether we can use AWS Deequ for data quality validation. I ran into a few problems. First of all, it...
The problem with software testing in data engineering
What if we found a bug in our data pipelines? What if that bug were easy to fix, but it would require a lot of...
Data flow - what functional programming and Unix philosophy can teach us about data streaming
What does stream processing have in common with functional programming and Unix?
Four books to boost your programmer career in 2020
I quit my dream job because of a book
All Stories
What is s3:TestEvent, and why does it break my event processing?
S3 sends s3:TestEvent to SQS after setting up the bucket notifications
Making OFFSET LIMIT queries in AWS Athena
How to use OFFSET in AWS Athena queries
How to get an alert if an AWS lambda does not get invoked during the last 24 hours
How to get a notification when AWS Lambda stops begin used
How to set Airflow variables while creating a dev environment
How to use command-line to set Airflow variables
How to check when an Athena table was updated
How to track the time when an Athena table was updated
How to run an Airflow DAG in a loop
How to keep running a DAG indefinitely
How to use xcom_pull to get a variable from another DAG
Get an XCOM variable from another DAG
What to do when Airflow BashOperator fails with TemplateNotFound error
How to fix TemplateNotFound error when using Airflow BashOperator
Copy directories in S3 using s3-dist-cp
How to copy files in S3 and preserve the directory structure
How to select a random sample of rows using Athena
How to use a window function to select random rows from Athena
Use HttpSensor to pause an Airflow DAG until a website is available
Pause an Airflow DAG until an HTTP endpoint returns 200 OK
How to add an EMR step in Airflow and wait until it finishes running
How to use AwsHook and EmrStepSensor to add an EMR step and wait until it finishes running
« Prev
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Next »