Add listenable TransportRequestHandler in TransportNodesAction

[zane-neo]

Description

Add listenable TransportRequestHandler in TransportNodesAction.

Use case:
In ml-commons, a scenario is to deploy model to the cluster, this action needs to read the model metadata from index first and then do time-consuming operation for locally running model and non time-consuming operations for remote running model.

The thing is the current TransportNodesAction limited the API that developers have to return a response when performing nodeOperation, so we have to workaround this by forwarding the deploy result to a listener to update the model status, the workflow looks like below:

This works fine for the time-consuming models(run locally) but not for remote models because deploying remote models is lightweight which only needs create several object in memory after metadata read. Once we enhanced this API to support listenableTransportRequestHandler, then we can pass the listener to the noOperation and return actual model status instead of deploying.

Also recently we found an issue of model deployment which is caused by the forwarding deploy result to listener part, simply put, if a node crash during the model deploy, then the listener will not able to receive all responses and not updating the model status, please refer: opensearch-project/ml-commons#2970.

So enhancing this class offering a straightforward solution to this scenario and I believe it can benefit other similar cases which accept a listener during the nodeOperation.

Related Issues

#15165

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 11c214f: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 72.00%. Comparing base (a81b868) to head (2e6023c).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...rch/action/support/nodes/TransportNodesAction.java	83.33%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15166      +/-   ##
============================================
+ Coverage     71.94%   72.00%   +0.05%     
- Complexity    64612    64734     +122     
============================================
  Files          5298     5301       +3     
  Lines        301952   302191     +239     
  Branches      43627    43654      +27     
============================================
+ Hits         217247   217591     +344     
+ Misses        66884    66722     -162     
- Partials      17821    17878      +57     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for f17a5c0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 3c68c3f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[zane-neo]

@peternied @gaobinlong @dblock Can you help review this PR?

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[github-actions]

❌ Gradle check result for cdd58e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[gaobinlong]

@zane-neo Could you give some use cases about this change to help understanding the intention, thanks!

[zane-neo]

@zane-neo Could you give some use cases about this change to help understanding the intention, thanks!

@gaobinlong Thanks, Binlong. I'll add more cases in the description and the issue.

[github-actions]

❌ Gradle check result for 2e6023c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 2e6023c: SUCCESS

[gaobinlong]

LGTM.

[gaobinlong]

@rajiv-kv @shwetathareja could you also help to take a look at this PR, thanks!