Kafka Stream UI Representation

[MSeal]

Goals

1. Enable visibility into the relationship between topics involved in a workspace's Kafka Stream application

Non-Goals

1. Manipulating existing project code around KStream relationships
2. Deviating from KStream Open Source models or requirements (maybe a KIP or smaller change would be needed to better enable better associations down the road)
3. Remote KStream application tracking (maybe a follow-up or outcome of how to overcome identification, but currently this is easier to solve for local systems first?)

Background

Kafka Streams is a nearly completely client side abstraction which manages topics to a higher abstraction level. Understanding how these topics are being managed for an application under development would be a great tool assist for building these programs. Currently visibility in all contexts about KStreams applications is limited or at the discretion of the program developer.

Proposal

This has more open questions than clear routes to success atm. Since the Kafka Stream applications don't export their model or scope of impact to logs or remote stores it's challenging to associate topics to a given stream. One could implement a naming convention or possibly sniff topic message headers to pull out an attribute? Or maybe we would need to have the application log it's associations explicitly for the extension to parse. Extracting this information from the source code itself would not be easily feasible in all situations, leading to error prone reporting. Ideas and discussion welcome on best potential approaches here to take, though we can lean on the locality of the KStream program and the extension being on the same machine here for solutioning.

[derek1ee]

@d8joseph may have some ideas around this area.