How to get names of columns with missing values in PySpark
Apache Spark

How to get names of columns with missing values in PySpark

How to get the names of missing properties for every row in a PySpark Dataframe
Bartosz Mikulski
Bartosz Mikulski – 1 min read

When we do data validation in PySpark, it is common to need all columns’ column names with null values. In this article, I show how to get those names for every row in the DataFrame.

First, I assume that we have a DataFrame df and an array all_columns, which contains the names of the columns we want to validate.

We have to create a column containing an array of strings that denote the column names with null values. Therefore, we have to use the when function to check whether the value is null and pass the column names as the literal value. We use the * to unpack the array produced by for comprehension into a Spark array:

missing_column_names = array(*[
    when(col(c).isNull(),lit(c)) for c in all_column
])

After that, we assign the values to a new column in the DataFrame:

df = df.withColumn("missing_columns", missing_column_names)
Older post

How to set a different retry delay for every task in an Airflow DAG

How to use a different retry delay in every Airflow task

Newer post

How to combine two DataFrames with no common columns in Apache Spark

Use full outer join to combine two Apache Spark DataFrames with no common columns

Related Posts

Data Engineering Apache Spark

What is shuffling in Apache Spark, and when does it happen?

When does an Apache Spark cluster perform the shuffle operation?
Published on: 20 May 2022

Apache Spark

How to measure Spark performance and gather metrics about written data

How to track Spark metrics in AWS CloudWatch
Published on: 05 Mar 2021

Apache Spark

How to combine two DataFrames with no common columns in Apache Spark

Use full outer join to combine two Apache Spark DataFrames with no common columns
Published on: 06 Dec 2020