How to reset the consumer offset in Apache Kafka topic

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (88/100)

In Kafka, we can easily reset the consumer group offset to any value we want. It allows us to reprocess some messages or skip the messages we don’t want to process. All we need is the

For example, if I want to reset the offset of the topic my_topic accessed by the consumer group called the_consumers to the earliest available offset, I have to run the following command:

2 --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-earliest --topic my_topic --execute

Note that I must add the --execute parameter at the end of the command. Otherwise, it will print the result of the operation, but it will not perform it!

There are a few more options regarding resetting the offsets. In my opinion, the most useful ones are:

  • to the latest, which makes the consumers skip all of the existing messages and start with the last one added to the topic:
2 --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-latest --topic my_topic --execute
  • to datetime, which sets the offset to the offset of the first message added to the topic after the datetime:
2 --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-datetime 2020-12-20T00:00:00.000 --topic my_topic --execute

Note that it uses the date in the ISO8601 format.

Would you like to help fight youth unemployment while getting mentoring experience?

Develhope is looking for tutors (part-time, freelancers) for their upcoming Data Engineer Courses.

The role of a tutor is to be the point of contact for students, guiding them throughout the 6-month learning program. The mentor supports learners through 1:1 meetings, giving feedback on assignments, and responding to messages in Discord channels—no live teaching sessions.

Expected availability: 15h/week. You can schedule the 1:1 sessions whenever you want, but the sessions must happen between 9 - 18 (9 am - 6 pm) CEST Monday-Friday.

Check out their job description.

(free advertisement, no affiliate links)

Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Bartosz Mikulski
Bartosz Mikulski * MLOps Engineer / data engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.