What is s3:TestEvent, and why does it break my event processing?
AWS

What is s3:TestEvent, and why does it break my event processing?

S3 sends s3:TestEvent to SQS after setting up the bucket notifications
Bartosz Mikulski
Bartosz Mikulski – 1 min read

When you setup SQS notifications in an S3 bucket, for example, when you want to get a message when a new file is uploaded into the bucket, the first thing you get is an s3:TestEvent which looks like this:

{
    "Service":"Amazon S3",
    "Event":"s3:TestEvent",
    "Time":"2020-11-30T00:00:00.000Z",
    "Bucket":"the_bucket_name",
    "RequestId":"some_id",
    "HostId":"another_id"
}

Its structure differs from all of the other notifications sent by S3, so most likely, it will break your event processing code. Don’t worry. AWS will send that only once to test whether S3 can publish messages to your SQS queue.

Unfortunately, they will not automatically remove that message after the test. There are three ways to deal with that problem:

  • wait until the message expires - unprofessional, but totally valid solution
  • setup a dead letter queue - most likely, you will have another function that processes the messages in DLQ, and it may fail too
  • poll messages from the queue, locate the s3:TestEvent, and remove it - probably the best option, unless you plan to change the configuration of S3 notifications multiple times a day and keep getting new test events.
Older post

Making OFFSET LIMIT queries in AWS Athena

How to use OFFSET in AWS Athena queries

Newer post

How to prevent Airflow from backfilling old DAG runs

How to disable backfilling of an Airflow DAG or skip a part of the DAG during a backfill

Related Posts

MLOps AWS

How to deploy a Transformer-based model with custom preprocessing code to Sagemaker Endpoints using BentoML

Deploy a machine learning model with custom inference code to a Sagemaker Endpoint using BentoML
Published on: 01 Oct 2021

AWS MLOps

Multimodel deployment in Sagemaker Endpoints

How to deploy multiple models in a single Sagemaker Endpoint?
Published on: 28 May 2021

AWS

How to deploy a REST API AWS Lambda using Chalice and AWS Code Pipeline

How to create a REST API Endpoint using AWS Lambda, Chalice, and AWS Code Pipeline
Published on: 16 Apr 2021