How to define an AWS Athena view using Airflow

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (80/100)

In this brief tutorial, I will show how to define an AWS Athena view using Airflow. We will need two things:

  • AWSAthenaOperator
  • the SQL query that defines the view

Let’s start with the query. To define the view, we have to call the CREATE VIEW statement. However, to make it work flawlessly in Airflow, we should make the statement pass even if the view already exists, so I suggest using CREATE OR REPLACE VIEW. Here is an example SQL query that creates a view:

1
create_view_sql = 'CREATE OR REPLACE VIEW the_view_name AS SELECT t1.* FROM some_table t1;'

Of course, such a view makes no sense, but it is good enough to use in a tutorial.

Now, we have to import the AWSAthenaOperator:

1
from airflow.contrib.operators.aws_athena_operator import AWSAthenaOperator

After that, we can create a new instance of the operator and add it to a dag. Note that I have to define the AWS connection id, which refers to a connection configured in Airflow and the database in which I want to create the view. In addition to that, I have to pass the S3 location where I want to store Athena queries’ results because Athena (being a Presto-based service) stores the results in files:

1
2
3
4
5
6
7
8
create_view = AWSAthenaOperator(
    task_id='create_the_view',
    query=create_view_sql,
    aws_conn_id='aws_connection_id',
    database='athena_database',
    output_location='the_output_location_of_athena_queries',
    dag=dag
)

Subscribe to the newsletter and join the free email course.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.

Do you want to work with me at riskmethods?

REMOTE position (available in Poland or Germany)