Pods does not recover from failure: abandoned subscription: was taking too long

[alok87]

Batcher needs to be manually restarted when the following errors happen. When Kafka faces downtime, it is seen some batcher get stuck with the following errors. They need a restart from recovering from this.

[sarama] 2021/09/20 06:41:38 consumer/broker/2 abandoned subscription to ts.db.table/0 because consuming was taking too long

They should recover without restarts, or fatal and restart on its own.

[alok87]

The first short term fix should be to make the error Fatal so that pod restarts and do not stay like this. This is very much required in Main Sink Group where multiple topics are loaded together in one pod to save connection to Redshift.

Happens for the loader as well.