i have set up a program to consume from a kafka topic.
it has been running fine, till ~ 25% of the messages (out of 2/3mil rows) are consumed, there would be a lot of reblance started happening.
i have tried to increase the `
and reduce the `
and to asynchronously process the messages. have tried to set up the `
`method as well. however, the rebalance is still keep happening, and the program starts stucking at 25%.
after a lot of investigation and trials, turns out that was due to the memory footprints of the program.
as more messages are put onto heap, several snap * 25% * 2/3 million rows, JVM starts busy running the GC thread, which eventually blocked the consumer to send back the heartbeat in time.
filter the message, or run this on a server would fix the issue.