Asynchronous FlightServerChannel to avoid thread contention per request

[rishabhmaurya]

Description

Addresses contention per request when multiple threads are flushing the response to the same FlightServerChannel. This can happen in search path when concurrent segment search is enabled where multiple threads will try to flush corresponding batches to the FlightServerChannel. This can cause thread to be blocked as it is currently protected under synchronized block.

Solution
The better way to handle it is to process the flushing of response, lets call it BatchTask, asynchronously. This can be done by having single threaded executor per request, which can keep adding BatchTask from multiple threads for the given request to its queue and process them linearly.

• Added event loop group which maintains set of single threaded executors with configurable count.
• one event loop from event loop group is assigned to the FlightServerChannel in a round robin way when a request is received at the server. This guarantees linear processing.
• One event loop can be associated with multiple channels. This is to avoid too many single threaded executor service to be created.
• All type of events on FlightServerChannel i.e. regular batch flush, complete stream and error are handled asynchronously now.
• Since its asynchronous, we need to pass the FlightTransportChannel to the BatchTask to release it when the stream is either completed or results in an error, which was not the case before and could be released in the finally block.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
  - [ ] API changes companion pull request created, if applicable.
  - [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 24384a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 24384a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rishabhmaurya]

@andrross It would great if you can review this one, thank you

[github-actions]

❌ Gradle check result for 0fc1213: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0fc1213: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6a90b41: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6a90b41: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6a90b41: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6a90b41: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

I rebased from main to get the latest test fixes

[github-actions]

❌ Gradle check result for 6fe1cb7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6fe1cb7:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6fe1cb7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6fe1cb7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 40b621b: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 59.82143% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.89%. Comparing base (0c882c4) to head (40b621b).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
.../arrow/flight/transport/FlightOutboundHandler.java	56.94%	25 Missing and 6 partials ⚠️
...arrow/flight/transport/FlightTransportChannel.java	47.05%	9 Missing ⚠️
...search/arrow/flight/transport/FlightTransport.java	66.66%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19403      +/-   ##
============================================
- Coverage     72.90%   72.89%   -0.02%     
+ Complexity    69967    69922      -45     
============================================
  Files          5676     5676              
  Lines        321143   321224      +81     
  Branches      46429    46441      +12     
============================================
+ Hits         234138   234159      +21     
- Misses        68092    68140      +48     
- Partials      18913    18925      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.