Kpow v88.2 introduces improved support for monitoring Flink consumers, the ability to encrypt your Kpow configuration to avoid passwords in plaintext, new configuration options for connecting to Confluent Schema Registries that require mutual TLS for authentication, and more.
Simple Consumers
Kpow provides great insight into Kafka Consumer Groups.
Until v88.2 Kpow provided little insight into the earlier implementation of Kafka consumers - Simple Consumers. That's because simple consumers have no group membership or assignment information, often the only information we have on them is a name and an understanding of their consumer offsets.
Simple consumers are often anonymous, ad-hoc, or bespoke consumption workloads that manage their own state. One prominent user of simple consumers is Flink, the popular stream processing framework.
In Kpow v88.2 we introduce a new 'Simple Consumers' tab in the Consumers UI, under this tab you will find information about any simple consumer that we can identify consuming from your cluster - including Flink workloads.
Note: If hundreds of ad-hoc or simple groups degrade your Kpow performance, turn this feature off with SNAPSHOT_SIMPLE_GROUPS=false.
Simple Consumer UI
Simple consumers metrics are held separately from our existing consumer group metrics as simple consumers have a different lifecycle from consumer groups. They don't have a group state and tend to exist for as long as their offsets are commit to the consumer_offsets topic. That means old/idle simple consumers can have a prolonged period where they are considered lagging - as such we don't intermingle aggregate simple and consumer group metrics.
The following simple consumer metrics have been added to Kpow's Prometheus endpoint:
- "simple.group.offset.lag.total" - the total lag of all simple consumers
- "simple.group.offset.delta.total" - the total read/s of all simple consumers
- "simple.topic.consumption.inactive.mins" - the total minutes a topic has seen zero consumption by simple consumers
- "simple.topic.offset.delta" - the total read/s of all simple consumers to a topic
- "simple.topic.offset.lag" - the total lag of all simple consumers to a topic
- "simple.broker.offset.delta" - the total read/s of all simple consumers to a broker
- "simple.broker.offset.lag" - the total lag of all simple consumers to a broker
In a future release we will introduce aggregate simple consumer metrics to the Overview UI.
Kpow Encrypted Configuration
Kpow is deployed by our customers in every cloud and on premises. Often times sensitive configuration is protected by vaults or secret managers.
Kpow requires standard Kafka configuration to connect to and manage your Kafka resources, this can include plaintext passwords for keys, keystores, truststores, and even the bindPassword for LDAP if you are using our JAAS LDAP integration. Kpow v88.2 includes support for obfuscating and encrypting plaintext configuration.
For more information on how to secure Kpow configuration see the Kpow Secure Configuration Guide. Use the Java API to secure your own client configuration.
We have chosen to open-source our implementation as the kpow-secure project, available under the Apache 2.0 license.
Confluent Schema TLS Configuration
In this release we introduce support for connecting to Confluent Schema Registry with mutual TLS authentication.
The following environment variables can be used to configure registry connections with mutual TLS:
- SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION
- SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD
- SCHEMA_REGISTRY_SSL_KEY_PASSWORD
- SCHEMA_REGISTRY_SSL_KEYSTORE_TYPE
- SCHEMA_REGISTRY_SSL_KEYMANAGER_ALGORITHM
- SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION
- SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD
- SCHEMA_REGISTRY_SSL_TRUSTSTORE_TYPE
- SCHEMA_REGISTRY_SSL_TRUSTMANAGER_ALGORITHM
- SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM
- SCHEMA_REGISTRY_SSL_PROVIDER
- SCHEMA_REGISTRY_SSL_CIPHER_SUITES
- SCHEMA_REGISTRY_SSL_PROTOCOL
- SCHEMA_REGISTRY_SSL_ENABLED_PROTOCOLS
- SCHEMA_REGISTRY_SSL_SECURE_RANDOM_IMPLEMENTATION
- SCHEMA_REGISTRY_SSL_KEYSTORE_KEY
- SCHEMA_REGISTRY_SSL_KEYSTORE_CERTIFICATE_CHAIN
- SCHEMA_REGISTRY_SSL_TRUSTSTORE_CERTIFICATES
- SCHEMA_REGISTRY_SSL_ENGINE_FACTORY_CLASS