How to reset the consumer offset in Apache Kafka topic

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (88/100)

In Kafka, we can easily reset the consumer group offset to any value we want. It allows us to reprocess some messages or skip the messages we don’t want to process. All we need is the kafka-consumer-groups.sh.

For example, if I want to reset the offset of the topic my_topic accessed by the consumer group called the_consumers to the earliest available offset, I have to run the following command:

1
2
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-earliest --topic my_topic --execute

Note that I must add the --execute parameter at the end of the command. Otherwise, it will print the result of the operation, but it will not perform it!

There are a few more options regarding resetting the offsets. In my opinion, the most useful ones are:

  • to the latest, which makes the consumers skip all of the existing messages and start with the last one added to the topic:
1
2
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-latest --topic my_topic --execute
  • to datetime, which sets the offset to the offset of the first message added to the topic after the datetime:
1
2
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-datetime 2020-12-20T00:00:00.000 --topic my_topic --execute

Note that it uses the date in the ISO8601 format.


Subscribe to the newsletter and join the free email course.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.