Building trustworthy data pipelines because AI cannot learn from dirty data
How to add dependencies to AWS lambda
The process of adding dependencies to an AWS Lambda consists of two steps. First, we have to install the dependencies in the source code directory. Later, we have to package...
08 Jan 2020
Four books to boost your programmer career
I quit my dream job because of a book
06 Jan 2020
What is the difference between data lake, data warehouse, and data mart
We can easily distinguish between them by focusing on three qualities: data structure (schema), data quality, and ownership.
18 Dec 2019
Three biggest traps to avoid while setting Spark executor memory
What happens when you set the executor memory of a Spark worker which uses YARN as the cluster resource manager? Does it get exactly the amount of memory you requested?...
16 Dec 2019
How to use Airflow backfill to run DAGs for a specified date in the past?
Have you created a new Airflow DAG, but now you have to run it using every data snapshot created during the last six months? Don’t worry. You don’t need to...
11 Dec 2019
What do you need to know about storing passwords in AWS?
How to use the AWS Secrets Manager
09 Dec 2019
Apache Spark: should we use RDD, Dataset, or DataFrame?
Is there a difference between Dataset and DataFrame? Why do we even have both?
04 Dec 2019
What a data engineer can learn from The Unicorn Project?
Have you ever seen a novel about developers? Reading such a book seems to be a massive waste of time, doesn’t it? After all, the internet is full of stories...
02 Dec 2019
AI in production: Roobits Events360
What would you do if you were writing an application which had to process one billion events per day?
18 Nov 2019
#AI in production
AI in production: Carta Healthcare
11 Nov 2019
#AI in production
Using Exponentially Weighted Moving Average for anomaly detection
05 Nov 2019
Using Boltzmann distribution as the exploration policy in TensorFlow-agent reinforcement learning models
There is a whole spectrum of exploration strategies between random and greedy policies.
04 Nov 2019