Skip to content

Agent just stopped working. #132

@willipl1972

Description

@willipl1972

Running Red Hat 9.6, Cassandra 4.1.9, Openjdk 11. The agent was working fine and it just stopped working with the following in the logs:

WARN [prometheus-netty-pool-0] 2025-07-18 11:46:20,295 DefaultChannelPipeline.java:1152 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.nio.BufferOverflowException: null
at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:249)
at com.zegelin.prometheus.exposition.NioExpositionSink.writeBytes(NioExpositionSink.java:27)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter$MetricVisitor.writeLabels(TextFormatMetricFamilyWriter.java:111)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter$MetricVisitor.writeLabelSets(TextFormatMetricFamilyWriter.java:123)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter$MetricVisitor.writeMetric(TextFormatMetricFamilyWriter.java:141)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter$MetricVisitor.lambda$null$3(TextFormatMetricFamilyWriter.java:185)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter$MetricVisitor.lambda$visit$4(TextFormatMetricFamilyWriter.java:184)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter$MetricVisitor.lambda$metricWriter$0(TextFormatMetricFamilyWriter.java:155)
at com.zegelin.prometheus.exposition.text.TextFormatMetricFamilyWriter.writeMetric(TextFormatMetricFamilyWriter.java:227)
at com.zegelin.prometheus.exposition.text.TextFormatExposition.nextSlice(TextFormatExposition.java:81)
at com.zegelin.prometheus.exposition.FormattedByteChannel.read(FormattedByteChannel.java:24)
at io.netty.handler.stream.ChunkedNioStream.readChunk(ChunkedNioStream.java:112)
at io.netty.handler.stream.ChunkedNioStream.readChunk(ChunkedNioStream.java:31)
at io.netty.handler.codec.http.HttpChunkedInput.readChunk(HttpChunkedInput.java:102)
at io.netty.handler.codec.http.HttpChunkedInput.readChunk(HttpChunkedInput.java:43)
at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:230)
at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808)
at com.zegelin.cassandra.exporter.netty.HttpHandler.sendMetrics(HttpHandler.java:303)
at com.zegelin.cassandra.exporter.netty.HttpHandler.channelRead0(HttpHandler.java:94)
at com.zegelin.cassandra.exporter.netty.HttpHandler.channelRead0(HttpHandler.java:39)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:829)

Receive the following with curl:

curl http://localhost:9500/metrics
curl: (18) transfer closed with outstanding read data remaining

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions