How to get names of columns with missing values in PySpark
When we do data validation in PySpark, it is common to need all columns’ column names with null values. In this article, I show how to get those names for every row in the DataFrame.
First, I assume that we have a DataFrame
df and an array
all_columns, which contains the names of the columns we want to validate.
We have to create a column containing an array of strings that denote the column names with null values. Therefore, we have to use the
when function to check whether the value is null and pass the column names as the literal value. We use the
* to unpack the array produced by for comprehension into a Spark array:
1 2 3 missing_column_names = array(*[ when(col(c).isNull(),lit(c)) for c in all_column ])
After that, we assign the values to a new column in the DataFrame:
1 df = df.withColumn("missing_columns", missing_column_names)
You may also like
- How to use one SparkSession to run all Pytest tests
- How to configure Spark to maximize resource usage while using AWS EMR
- What is the difference between cache and persist in Apache Spark?
- Apache Spark: should we use RDD, Dataset, or DataFrame?
- Check-Engine - data quality validation for PySpark 3.0.0