diff --git a/docs/content/docs/connectors/datastream/kafka.md b/docs/content/docs/connectors/datastream/kafka.md index 8f2c56ed04a51..d49303ab460c9 100644 --- a/docs/content/docs/connectors/datastream/kafka.md +++ b/docs/content/docs/connectors/datastream/kafka.md @@ -212,26 +212,82 @@ Note that Kafka source does **NOT** rely on committed offsets for fault toleranc is only for exposing the progress of consumer and consuming group for monitoring. ### Monitoring -Kafka source exposes metrics in Flink's metric group for monitoring and diagnosing. + +Kafka source exposes the following metrics in the respective [scope]({{< ref "docs/ops/metrics" >}}/#scope). + #### Scope of Metric -All metrics of Kafka source reader are registered under group ```KafkaSourceReader```, which is a -child group of operator metric group. Metrics related to a specific topic partition will be registered -in the group ```KafkaSourceReader.topic..partition.```. -For example, current consuming offset of topic "my-topic" and partition 1 will be reported in metric: -```.operator.KafkaSourceReader.topic.my-topic.partition.1.currentOffset``` , + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ScopeMetricsUser VariablesDescriptionType
OperatorcurrentEmitEventTimeLagn/aThe time span from the record event timestamp to the time the record is emitted by the source connector¹: currentEmitEventTimeLag = EmitTime - EventTime.Gauge
watermarkLagn/aThe time span that the watermark lags behind the wall clock time: watermarkLag = CurrentTime - WatermarkGauge
sourceIdleTimen/aThe time span that the source has not processed any record: sourceIdleTime = CurrentTime - LastRecordProcessTimeGauge
pendingRecordsn/aThe number of records that have not been fetched by the source. e.g. the available records after the consumer offset in a Kafka partition.Gauge
KafkaSourceReader.commitsSucceededn/aThe total number of successful offset commits to Kafka, if offset committing is turned on and checkpointing is enabled.Counter
KafkaSourceReader.commitsFailedn/aThe total number of offset commit failures to Kafka, if offset committing is + turned on and checkpointing is enabled. Note that committing offsets back to Kafka + is only a means to expose consumer progress, so a commit failure does not affect + the integrity of Flink's checkpointed partition offsets.Counter
KafkaSourceReader.committedOffsetstopic, partitionThe last successfully committed offsets to Kafka, for each partition. + A particular partition's metric can be specified by topic name and partition id.Gauge
KafkaSourceReader.currentOffsetstopic, partitionThe consumer's current read offset, for each partition. A particular + partition's metric can be specified by topic name and partition id.Gauge
-and number of successful commits will be reported in metric: -```.operator.KafkaSourceReader.commitsSucceeded``` . -#### List of Metrics +¹ This metric is an instantaneous value recorded for the last processed record. This metric is provided because latency histogram could be expensive. The instantaneous latency value is usually a good enough indication of the latency. -| Metric Name | Description | Scope | -|:----------------:|:-----------------------------------------------:|:-----------------:| -| currentOffset | Current consuming offset of the topic partition | TopicPartition | -| committedOffset | Committed offset of the topic partition | TopicPartition | -| commitsSucceeded | Number of successful commits | KafkaSourceReader | -| commitsFailed | Number of failed commits | KafkaSourceReader | #### Kafka Consumer Metrics All metrics of Kafka consumer are also registered under group ```KafkaSourceReader.KafkaConsumer```. diff --git a/docs/content/docs/ops/metrics.md b/docs/content/docs/ops/metrics.md index 53b59d97ef766..1580136b81ce9 100644 --- a/docs/content/docs/ops/metrics.md +++ b/docs/content/docs/ops/metrics.md @@ -1345,52 +1345,7 @@ Certain RocksDB native metrics are available but disabled by default, you can fi ### Connectors #### Kafka Connectors - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ScopeMetricsUser VariablesDescriptionType
OperatorcommitsSucceededn/aThe total number of successful offset commits to Kafka, if offset committing is turned on and checkpointing is enabled.Counter
OperatorcommitsFailedn/aThe total number of offset commit failures to Kafka, if offset committing is - turned on and checkpointing is enabled. Note that committing offsets back to Kafka - is only a means to expose consumer progress, so a commit failure does not affect - the integrity of Flink's checkpointed partition offsets.Counter
OperatorcommittedOffsetstopic, partitionThe last successfully committed offsets to Kafka, for each partition. - A particular partition's metric can be specified by topic name and partition id.Gauge
OperatorcurrentOffsetstopic, partitionThe consumer's current read offset, for each partition. A particular - partition's metric can be specified by topic name and partition id.Gauge
+Please refer to [Kafka monitoring]({{< ref "docs/connectors/datastream/kafka" >}}/#monitoring). #### Kinesis Connectors