Question to Kafka Metrics

[bartman64]

So we are using grafana to get a glance of our services and kafka metrics. Those services were implement on quarkus base using kafka-streams.
Now with the switch to the parallel consumers the metrics are no longer visible in grafana. Currently I don't know, if with some config properties I could make them visible again or those aren't implemented/usable yet since I saw a open issues regarding metrics (#27)
I understand that kafka-streaming differs to the kafka low-level-api approach used from the parallel consumer, but I don't know how to access those metrics.

If someone could give me some insight would be great, thx 😄

[astubbs]

#27 is just about adding extra metrics specific to parallel consumer.

You should still be able to access all the metric exposed by the consumer and producer instances - can you not see those?

[bartman64]

Hmm ok...
So we have a grafana dashboard with all those metrics and before switching to the parallel consumer, we were able to see the metrics.
After that the graphs stay empty.
So I was just wondering what happend there, but havn't looked deeper into that myself

[amolkamble17]

We are having quarkus streaming application with pre distribution pipeline which produces messages to multiple topics
and parallel consumer (non streaming application) consumes that message and writes to output topic. We are not able to see metrics in Grafana dashboard of non streaming application when it consumes messages produced by Kafka streaming.

[astubbs]

Sorry @amolkamble17, I missed your question. I'm not sure I follow - are you saying you can't see any JMX sourced metrics from PC's consumer or producer instances?

FYI, keep an eye on:

• Feature/health metrics micrometer [WIP] #485

[amolkamble17]

Thank you @astubbs for reply.
Yes, we are using parallel consumer as below.

var options = ParallelConsumerOptions.<String, String>builder()
.ordering(KEY)
.commitMode(ParallelConsumerOptions.CommitMode.PERIODIC_CONSUMER_ASYNCHRONOUS)
.maxConcurrency(Integer.parseInt(10))
.consumer(kafkaConsumer)
.producer(kafkaProducer)
.build();

    ParallelStreamProcessor<String, String> eosStreamProcessor = ParallelStreamProcessor.createEosStreamProcessor(options);
    eosStreamProcessor.subscribe(of(inputTopic + "." + randomNumber));

1. How do we get metrics of parallel consumer to show on Grafana dashboard.

We are having quarkus stream application which already expose kafka metrics for streaming application but it is not showing metrics of parallel consumer. Currently we are using 0.5.2.4 version of parallel-consumer-core.
Could you please give some example how to extract metrics for parallel consumer and that would be available with existing metrics.

2. Which version and when this enhancement of PC metrics will be complete.