Release 89.3: Optional Log Cleaner Check Disable

Thursday, August 11th 2022 by Factor House

Kpow v89.3 introduces the ability to disable the log.cleaner.enable=true broker configuration check that Kpow requires by default.

See the full Kpow Changelog: https://github.com/operatr-io/kpow/blob/main/CHANGELOG.md

What is Log Cleaner Enable?

log.cleaner.enable is a Kafka broker configuration setting that governs data deletion, it is true by default.

When the log cleaner is enabled your Kafka brokers will periodically delete TTL'd data. If you change this configuration to false then no data will ever be physically deleted from your cluster, regardless of whether it has expired or not.

Why Does Kpow Require Log Cleaner Enable?

Kpow writes data into internal topics in your primary cluster, this data helps Kpow perform long-term analysis on the health of your clusters. We drastically reduced the amount of data written to internal topics in a recent release, but Kpow still relies on TTLd data being periodically deleted from your cluster.

Given a long enough period of operation without any data deletion Kpow's internal topics could grow to a considerable size. Being conservative, previous releases of Kpow will refuse to start if the brokers of your primary cluster are not configured to delete data.

How Do I Disable the Log Cleaner Check?

Update to v89.3 and start Kpow with the new environment variable setting: DISABLE_LOG_CLEANER_CHECK=true.

Is It Safe to Disable the Log Cleaner Check?

In the short-term, we believe so. Kpow has significantly reduced the amount of data written to internal topics such that this has become a less significant issue. The next major release of Kpow (v90) has an ephemeral mode where you can choose to run Kpow without writing any data to internal topics.

From v90, Kpow will detect where your primary cluster does not support data deletion and automatically switch into ephemeral mode.

Changelog v89.3

  • Provide DISABLE_LOG_CLEANER_CHECK configuration option

Get this release on Dockerhub, on the AWS Marketplace, on ArtifactHub (Helm Charts), or Download as a JAR File.

Pull the latest Kpow Docker Container

docker pull operatr/kpow:89.3

Download the latest Kpow Java JAR

[https://releases.operatr.io/operatr-2022-08-11-17-58.jar?AWSAccessKeyId=AKIAXNNKGCAKDUQSTKG4\&Signature=%2FmjZNFHGKn4wrnq%2BTujgvKb%2FZ70%3D\&Expires=1675756765](https://releases.operatr.io/operatr-2022-08-11-17-58.jar?AWSAccessKeyId=AKIAXNNKGCAKDUQSTKG4&Signature=%2FmjZNFHGKn4wrnq%2BTujgvKb%2FZ70%3D&Expires=1675756765)

Download the Kpow Java 8 JAR (uses Jetty 9, we recommend updating to Java 11+ and using our standard JAR)

https://releases.operatr.io/operatr-2022-08-11-18-00.jar?AWSAccessKeyId=AKIAXNNKGCAKDUQSTKG4&Signature=6ku61%2B%2BD00xl9rxD8KNrSrEvvRI%3D&Expires=1675757028

Tags:

Internal Topics
Log Cleaner