What is s3:TestEvent, and why does it break my event processing?

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (68/100)

When you setup SQS notifications in an S3 bucket, for example, when you want to get a message when a new file is uploaded into the bucket, the first thing you get is an s3:TestEvent which looks like this:

1
2
3
4
5
6
7
8
{
    "Service":"Amazon S3",
    "Event":"s3:TestEvent",
    "Time":"2020-11-30T00:00:00.000Z",
    "Bucket":"the_bucket_name",
    "RequestId":"some_id",
    "HostId":"another_id"
}

Its structure differs from all of the other notifications sent by S3, so most likely, it will break your event processing code. Don’t worry. AWS will send that only once to test whether S3 can publish messages to your SQS queue.

Unfortunately, they will not automatically remove that message after the test. There are three ways to deal with that problem:

  • wait until the message expires - unprofessional, but totally valid solution
  • setup a dead letter queue - most likely, you will have another function that processes the messages in DLQ, and it may fail too
  • poll messages from the queue, locate the s3:TestEvent, and remove it - probably the best option, unless you plan to change the configuration of S3 notifications multiple times a day and keep getting new test events.

Did you enjoy reading this article?
Would you like to learn more about software craft in data engineering and MLOps?

Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!

Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.

Bartosz Mikulski

Bartosz Mikulski

  • Data/MLOps engineer by day
  • DevRel/copywriter by night
  • Python and data engineering trainer
  • Conference speaker
  • Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
  • Twitter: @mikulskibartosz
Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.