Kpow v89.3 introduces the ability to disable the log.cleaner.enable=true broker configuration check that Kpow requires by default.
What is Log Cleaner Enable?
log.cleaner.enable is a Kafka broker configuration setting that governs data deletion, it is true by default.
When the log cleaner is enabled your Kafka brokers will periodically delete TTL'd data. If you change this configuration to false then no data will ever be physically deleted from your cluster, regardless of whether it has expired or not.
Why Does Kpow Require Log Cleaner Enable?
Kpow writes data into internal topics in your primary cluster, this data helps Kpow perform long-term analysis on the health of your clusters. We drastically reduced the amount of data written to internal topics in a recent release, but Kpow still relies on TTLd data being periodically deleted from your cluster.
Given a long enough period of operation without any data deletion Kpow's internal topics could grow to a considerable size. Being conservative, previous releases of Kpow will refuse to start if the brokers of your primary cluster are not configured to delete data.
How Do I Disable the Log Cleaner Check?
Update to v89.3 and start Kpow with the new environment variable setting: DISABLE_LOG_CLEANER_CHECK=true.
Is It Safe to Disable the Log Cleaner Check?
In the short-term, we believe so. Kpow has significantly reduced the amount of data written to internal topics such that this has become a less significant issue. The next major release of Kpow (v90) has an ephemeral mode where you can choose to run Kpow without writing any data to internal topics.
From v90, Kpow will detect where your primary cluster does not support data deletion and automatically switch into ephemeral mode.