mikulskibartosz.name
Start here
About me
Twitter
Mastodon
mlops.today
Bartosz Mikulski
Building trustworthy data pipelines because AI cannot learn from dirty data
All Stories
How to render an Airflow template for testing
How to generate the code of an Airflow task from a template and a given execution date
How to check the next execution date of an Airflow DAG
How to use Airflow CLI to get the next execution date of a DAG
Doing data quality checks using the SQLCheckOperator
How to use SQLCheckOperator to verify that the database contains an expected number of rows
How to deal with the jinja2 TemplateNotFound error in Airflow
How to fix the TemplateNotFound error while using a custom Airflow operator
How to postpone Airflow DAG until files get uploaded into an S3 bucket
How to use Airflow sensors to detect that files have been uploaded into an S3 bucket
What is the difference between a transformation and an action in Apache Spark?
What is an action in Apache Spark? What do you understand as transformations in Apache Spark?
Use LatestOnlyOperator to skip some tasks while running a backfill in Airflow
How to skip some tasks when backfilling a DAG in the past
Christopher Bergh - How the DataOps principles help data engineers make data pipelines trustworthy
An interview with Christopher Bergh who explains how the DataOps principles help data engineers make data pipelines trustworthy
How to retrieve the statuses of the recent DAG executions from Airflow database
How to make a dashboard that displays Airflow DAG statuses
How to find and terminate an idle Redshift session
How to find the idle session that is blocking the connection pool in Redshift
How to configure Spark to maximize resource usage while using AWS EMR
How to configure EMR to use all available resources when running a Spark cluster
How to use AWSAthenaOperator in Airflow to verify that a DAG finished successfully
How to check that an AWS Athena table contains data after running an Airflow DAG.
« Prev
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Next »