How to retrieve the table descriptions from Glue Data Catalog using boto3
AWS

How to retrieve the table descriptions from Glue Data Catalog using boto3

How to get the comments from the create table statements when the metadata is stored in the Glue Data Catalog
Bartosz Mikulski
Bartosz Mikulski – 1 min read

It is not a common use-case, but occasionally we need to create a page or a document that contains the description of the Athena tables we have. It is relatively easy to do if we have written comments in the create external table statements while creating them because those comments can be retrieved using the boto3 client.

In this article, I am going to show you how to do it.

First, we have to create a glue client using the following statement:

import boto3

glue_client = boto3.client('glue',
            region_name=region_name,
            aws_access_key_id=aws_access_key_id,
            aws_secret_access_key=aws_secret_access_key)

To retrieve the tables, we need to know the database name:

glue_tables = glue_client.get_tables(DatabaseName=db_name, MaxResults=1000)

Now, we can iterate over the tables and retrieve the data such as the column names, types, and the comments added when the table was created:

for table in glue_tables['TableList']:
    for column in table['StorageDescriptor']['Columns']:
        column_name = column['Name']
        comment = column.get('Comment', '')
        column_type = column['Type']

We have to remember that the code above does not return the columns used for data partitioning. To get the partition keys, we need the following code:

for table in glue_tables['TableList']:
    for partition_key in table.get('PartitionKeys', []):
        column_name = partition_key['Name']
        comment = partition_key.get('Comment', '')
        column_type = partition_key['Type']
Older post

How to perform a batch write to DynamoDB using boto3

How to write multiple DynamoDB objects at once using boto3

Newer post

How to start an AWS Glue Crawler to refresh Athena tables using boto3

How to create and start an AWS Glue Crawler from Python code using boto3

Related Posts

MLOps AWS

How to deploy a Transformer-based model with custom preprocessing code to Sagemaker Endpoints using BentoML

Deploy a machine learning model with custom inference code to a Sagemaker Endpoint using BentoML
Published on: 01 Oct 2021

AWS MLOps

Multimodel deployment in Sagemaker Endpoints

How to deploy multiple models in a single Sagemaker Endpoint?
Published on: 28 May 2021

AWS

How to deploy a REST API AWS Lambda using Chalice and AWS Code Pipeline

How to create a REST API Endpoint using AWS Lambda, Chalice, and AWS Code Pipeline
Published on: 16 Apr 2021