Expose NodeState Getter from ApacheHttpClient5Transport for node metrics

[001mertiya]

Is your feature request related to a problem?

Currently, the internal denylist in ApacheHttpClient5Transport is not accessible, making it difficult to expose client-side node health metrics. This limits observability of node status from the client perspective.

What solution would you like?

Provide a public getter (e.g., getDenylist()) to access the denylist so users can integrate it with metrics systems and monitor node availability.

What alternatives have you considered?

Reflection (unsafe and brittle)

Do you have any additional context?

Exposing the denylist would allow better integration with monitoring tools and improve visibility into OpenSearch node health from the client side.

[001mertiya]

Hi @reta, Could you please take a look at this when you have time? What are your thoughts? #1698
Thanks.

[reta]

Hi @reta, Could you please take a look at this when you have time? What are your thoughts? #1698 Thanks.

The ApacheHttpClient5Transport has dedicated FailureListener which could be used to collect the node failures / health. Why do we need to expose the deny list? This is internal to the client implementation, the host management is not exposed and arguably should not

[001mertiya]

Hi @reta,
I was trying to retrieve all the metrics that were added in this PR: #1574 (comment)

We can get request metrics using the out-of-the-box instrumentation. However, to track active and inactive nodes, I was thinking of adding a getter in ApacheHttpClient5Transport so that users can easily integrate it with a meter registry.

If we try to achieve the same using FailureListener, the user would need to manually maintain a set of active and inactive nodes and update them within the onFailure.

One more thing - what’s your recommendation for collecting network-level and more detailed metrics? I believe we should have comprehensive metrics that allow us to pinpoint exactly where the issue lies just by analyzing them.

[reta]

Thanks @001mertiya

We can get request metrics using the out-of-the-box instrumentation. However, to track active and inactive nodes, I was thinking of adding a getter in ApacheHttpClient5Transport so that users can easily integrate it with a meter registry.

Could you elaborate a bit how you are going to convert such lists into metrics? The FailureListener is, in my opinion, the straight abstraction to collect and expose per-node failures as the come.

One more thing - what’s your recommendation for collecting network-level and more detailed metrics? I believe we should have comprehensive metrics that allow us to pinpoint exactly where the issue lies just by analyzing them.

What exactly you mean by network-level? The operating system network stack?

[001mertiya]

Could you elaborate a bit how you are going to convert such lists into metrics? The FailureListener is, in my opinion, the straight abstraction to collect and expose per-node failures as the come.

As you can see in the dashboard attached in this PR commentt, I'm planning to plot the number of active nodes and inactive nodes.
If we can get access to the denylist getter added in this PR #1698, or something similar, then we can plug it in meterRegistry as shown below to retrieve the metric.
Gauge.builder("os.dead.node.count", () -> openSearchClientTransport.getDenyList().size()) .tag("total_nodes", String.valueOf(size)) .description("Dead host state count") .register(meterRegistry);

And if we want to achieve the same using the FailureListener, then we need to maintain a set of active and inactive nodes. On failure, we should add the node to the inactive set and remove it from the active set. Similarly, we need access to onResponse(Through a wrapper, but it's not possible because it's private.) to determine when to move a previously inactive (dead) node back to the active set.

What exactly you mean by network-level? The operating system network stack?

I’m referring to a detailed breakdown of end-to-end latency, including stages like authentication/SSL handshake time, connection wait time, transport time, and OpenSearch server time. Ideally, there would be a diagnostic log context that records each stage’s name, the time the client spent on that stage, along with the status code, exceptions, and other relevant details.

[reta]

@001mertiya I see your point, so I think we should spend some time to expose nodes and their state property:

• in general, the list of all available nodes is not known, when using sniffer, the initial set of nodes will be changed
• out of those nodes, some could be marked as temporarily unavailable (the OpenSearch Java borrows RestCient's DeadHostState but those are not dead nodes, it is confusing, they are excluded temporarily and may be added back)

I think introducing something like NodesChangeListener would cover the needs:

interface NodesChangeListener {
    void onNodeAdded(Node node);
    void onNodeRemoved(Node node);
    void onNodeStateChanged(Node node, NodeState from, NodeState to);
}

enum NodeState {
    Active,
    Unavailable,
    ...
}

Alternatively, as more simple solution, we could expose:

Map<Node, NodeState> getNodes();

I’m referring to a detailed breakdown of end-to-end latency, including stages like authentication/SSL handshake time, connection wait time, transport time, and OpenSearch server time.

This is mostly what Apache HttpClient / HttpCore 5 instrumentation should provide if such details are available.

[001mertiya]

I think introducing something like NodesChangeListener would cover the needs:

I didn't fully get your context. Do you mean that we can plug in NodesChangeListener with the sniffer to keep track of all node states?

enum NodeState {
Active,
Unavailable,
...
}

Do you think we should have NodeState in opensearch-java? But how would we track the node state, considering that the nodes we’re seeing aren’t actually dead?

Alternatively, as more simple solution, we could expose:

That's what we did in the linked PR #1698, we can improve the PR to add NodeState

[reta]

I didn't fully get your context. Do you mean that we can plug in NodesChangeListener with the sniffer to keep track of all node states?

We could keep sniffer aside from now, just focus on the fact that the node list could change any time

Do you think we should have NodeState in opensearch-java? But how would we track the node state, considering that the nodes we’re seeing aren’t actually dead?

This is what we could by combining deny list + nodes

That's what we did in the linked PR #1698, we can improve the PR to add NodeState

Yes, the change is in right direction but we need to expose the whole picture (nodes + states) instead

[001mertiya]

Hi @reta , thanks for your inputs.

We could keep sniffer aside from now, just focus on the fact that the node list could change any time

Yes, and if we use ApacheHttpClient5Transport, which is the recommended way to create a client. Sniffer support is not available for ApacheHttpClient5Transport, but it is available for RestClientTransport, which is provided for legacy support. What would you suggest for sniffer like support when creating a client using ApacheHttpClient5Transport? Is there any plan to add sniffer support for ApacheHttpClient5Transport?

This is what we could by combining deny list + nodes.
Yes, the change is in right direction but we need to expose the whole picture (nodes + states) instead

Can you review the PR #1698 ?

This is mostly what Apache HttpClient / HttpCore 5 instrumentation should provide if such details are available.

Is it possible to fetch latency information by adding execution interceptors at different stages? For example, if we add one before MAIN_TRANSPORT and one at addExecInterceptorFirst, then the time between addExecInterceptorFirst and MAIN_TRANSPORT could represent the time taken for the SSL handshake + authentication. What are your thoughts?

[reta]

Is there any plan to add sniffer support for ApacheHttpClient5Transport?

Definitely, I think we should much featuresets

Is it possible to fetch latency information by adding execution interceptors at different stages?

I don't know to be fair, worth trying