Release 91.1: ksqlDB UI and Broker Disk Telemetry

Monday, March 6th 2023 by Factor House

Release Info

Kpow can be found on Dockerhub

docker pull factorhouse/kpow-ee:91.1

View our Docker quick start here.

Kpow v91.1 introduces a new ksqlDB UI, new disk usage telemetry, schema registry improvements, wildcard filtering, and closes a number of minor issues.


ksqlDB User Interface

Kpow now provides ksqlDB integration paired with Kpow's enterprise features such as role-based access control and data-masking.

ksqlDB features seamlessly integrate with other Kpow managed resources such as connect clusters and schema registries.

Like all actions performed within Kpow, an audit log is kept for data governance when issuing any ksqlDB action.

See the ksqlDB setup guide for configuration details, or read our ksqlDB feature guide for more information about Kpow's new ksqlDB capabilities.

image

Broker Disk Telemetry

Kpow v91.1 provides deeper insight into broker disk usage, all the way down to a directory level. Simply navigate to Brokers > Disk.

Where your Kafka brokers are v3.3.0+ you will see Total Disk, Usable Disk, and Disk Remaining % information.

image

Broker Disk Telemetry in Prometheus

Each of these new fields is available via Kpow's prometheus endpoint at a broker and broker directory level.

Broker total and usable bytes

# HELP broker_bytes_total The total disk size in bytes of a broker
# TYPE broker_bytes_total gauge
broker_bytes_total{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="2",} 1.05491914752E11
broker_bytes_total{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="3",} 1.05491914752E11
broker_bytes_total{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="1",} 1.05491914752E11

# HELP broker_bytes_usable The usable disk size in bytes of a broker
# TYPE broker_bytes_usable gauge
broker_bytes_usable{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="3",} 1.00089417728E11
broker_bytes_usable{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="1",} 1.00089393152E11
broker_bytes_usable{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="2",} 1.00089397248E11

Broker directory total and usable bytes

# HELP broker_dir_bytes_total The total disk size in bytes of a broker directory
# TYPE broker_dir_bytes_total gauge
broker_dir_bytes_total{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="1",dir="_kafka_datalogs_logs",} 1.05491914752E11
broker_dir_bytes_total{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="3",dir="_kafka_datalogs_logs",} 1.05491914752E11
broker_dir_bytes_total{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="2",dir="_kafka_datalogs_logs",} 1.05491914752E11

# HELP broker_dir_bytes_usable The usable disk size in bytes of a broker directory
# TYPE broker_dir_bytes_usable gauge
broker_dir_bytes_usable{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="3",dir="_kafka_datalogs_logs",} 1.00089417728E11
broker_dir_bytes_usable{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="2",dir="_kafka_datalogs_logs",} 1.00089397248E11
broker_dir_bytes_usable{domain="cluster",id="CuxsifYVRhSRX6iLTbANWQ",target="1",dir="_kafka_datalogs_logs",} 1.00089393152E11

Schema Registry Improvements

Kpow v91.1 introduces improvements to schema registry authentication and observation.

Confluent Cloud Schema Registry OAuth Authentication

Kpow supports authenticating with Confluent Cloud schema registry via OAuth.

See the Confluent Cloud guide to Schema Registry OAuth and the Kpow Schema Registry Configuration Guide for more information.

Confluent Schema Registry Observation Version

Kpow provides a configurable OBSERVATION_VERSION that allows you to choose a more efficient mode of observering Confluent schema registry, at the small cost of showing schema compatibility information only at the drill-down level in the Kpow UI.

In time we will default the OBSERVATION_VERSION to '2', the current default is '1' (no change from existing).

Read more in the Kpow Schema Registry Configuration Guide.

Custom Serdes and Message Headers

Message Headers are now passed to the serialize and deserialize methods of custom serdes configured with Kpow.

See the Kpow Custom Serdes Guide for more information.

Note: this change introduces a minor breaking change in that your custom serdes must be compiled with Kafka v2.1.0+.


Changelog v91.1

  • Introduce new ksqlDB integration
  • Introduce new disk usage telemetry
  • Introduce support for Confluent Cloud Schema Registry OAuth authentication
  • Improve schema registry observation with configurable OBSERVATION_VERSION
  • Improve custom serdes with headers support
  • Improve top-level filtering with wildcard support
  • Improve topic and broker configuration UI with fast filters and table preferences
  • Improve bulk actions with bulk group delete support
  • Fix ACL bug related to comma-separated principals
  • Fix intellisense bug in key entry field when producing messages
  • Fix minor dark mode UI bugs
  • Bump to Kafka 3.4.0 dependencies
  • Bump to Confluent 7.3.1 dependencies

See the full Kpow Changelog.


Get this release on Dockerhub, on the AWS Marketplace, on ArtifactHub (Helm Charts), or Download as a JAR File.

Pull the latest Kpow Docker Container

docker pull factorhouse/kpow-ee:91.1

Download the Kpow Java 11+ JAR

https://releases.operatr.io/kpow-2023-03-06-21-12.jar?AWSAccessKeyId=AKIAXNNKGCAKDUQSTKG4&Signature=5lpJiZvCcl8JQkfiU78i7DWR6IM%3D&Expires=1693649667

Download the Kpow Java 8 JAR

https://releases.operatr.io/kpow-2023-03-06-21-14.jar?AWSAccessKeyId=AKIAXNNKGCAKDUQSTKG4&Signature=IZsyPX9XR26udtAjyU6MI0CyGeg%3D&Expires=1693649786

Tags:

ksqlDB UI
Disk Telemetry