I help data engineering tech leads #makeDataTrustworthy because AI cannot learn from dirty data
How to concatenate multiple MySQL rows into a single field?
How to concatenate multiple rows into a string in MySQL
07 Oct 2020
How to get an array/bag of elements from the Hive group by operator?
How to get an array of elements from one column when grouping by another column in Hive
06 Oct 2020
Working with dates and time in Apache Spark
How to get relative dates (yesterday, tomorrow) in Apache Spark, and how to calculate the difference between two dates
05 Oct 2020
How to save an Apache Spark DataFrame as a dynamically partitioned table in Hive
How to use the saveAsTable function to create a partitioned table
04 Oct 2020
When to cache an Apache Spark DataFrame?
Should we cache everything in Apache Spark or are there any rules?
03 Oct 2020
How to flatten a struct in a Spark DataFrame?
How to convert struct fields into separate columns.
02 Oct 2020
What is the difference between CUBE and ROLLUP and how to use it in Apache Spark?
Desc: How to use the cube and rollup functions in Apache Spark or PySpark. What is the difference between a cube and a rollup.
01 Oct 2020
How to concatenate columns in a PySpark DataFrame
How to use the concat and concat_ws functions to merge multiple columns into one in PySpark
30 Sep 2020
How to derive multiple columns from a single column in a PySpark DataFrame
Extract multiple columns from a single column using the withColumn function and a PySpark UDF
29 Sep 2020
Broadcast variables and broadcast joins in Apache Spark
How to speed up joins of small DataFrames by using the broadcast join
28 Sep 2020
How to use the window function to get a single row from each group in Apache Spark
How to group values by a key and extract a single row from each group in Apache Spark
27 Sep 2020
How to make a pivot table in AWS Athena or PrestoSQL
How to make a pivot table in AWS Athena, and why the pivot function does not exist
26 Sep 2020