Send event to AWS Lambda when a file is added to an S3 bucket

Trigger AWS Lambda when a file is created in an S3 bucket

Bartosz Mikulski 11 Nov 2020 – 2 min read

In this blog post, I will show you how to configure S3 bucket notification, AWS Lambda permission, and the Lambda trigger to receive a notification when a file is added to an S3 bucket. Handling the event in the Lambda function is out of the scope of this article. I will use Terraform to configure the notifications and permissions.

The first thing we have to do is configuring the bucket notifications. Note that it is impossible to define notifications that have overlapping filter prefixes when the filter suffix is the same!

resource "aws_s3_bucket_notification" "bucket-events" {
  bucket = "bucket_name"

  queue {
    events = ["s3:ObjectCreated:*"]
    queue_arn = aws_sqs_queue.queue_name.arn
    filter_prefix = "file_key_prefix"
    filter_suffix = "file_key_suffix"
  }
}

After that, we have to give the bucket_name bucket permission to send events to the queue, and the Lambda function needs permission to read the events:

resource "aws_sqs_queue_policy" "bucket-events-policy" {
  queue_url = aws_sqs_queue.queue_name.id
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Id": "${aws_sqs_queue.queue_name.arn}",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:SendMessage",
      "Resource": "${aws_sqs_queue.queue_name.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::bucket_name"
        }
      }
    },
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:ReceiveMessage",
      "Resource": "${aws_sqs_queue.queue_name.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn of the lambda function"
        }
      }
    }
  ]
}
EOF
}

In the end, we have to add the SQS ARN as the source of the events in the Serverless configuration of the Lambda function:

# Put this in the function part in the Serverless configuration
events:
  - sqs: 'SQS ARN'

Send event to AWS Lambda when a file is added to an S3 bucket

Select Serverless configuration variables using the stage parameter

How to add an EMR step from AWS Lambda

Send event to AWS Lambda when a file is added to an S3 bucket

Select Serverless configuration variables using the stage parameter

How to add an EMR step from AWS Lambda

Related Posts

How to deploy a Transformer-based model with custom preprocessing code to Sagemaker Endpoints using BentoML

Multimodel deployment in Sagemaker Endpoints

How to deploy a REST API AWS Lambda using Chalice and AWS Code Pipeline