[dask] Aggregate distributed validation metrics, fix dask early stopping

[szjunma]

Summary

This PR adds support to distributed evaluation metrics so every worker sees the globally-aggregated result, which fixes two interrelated problems:

1. Wrong best_score_ ([python-package][dask] DaskLGBMClassifier.evals_result_ returns evals_result_ of one worker #6963): Each worker only saw its local data slice, so early stopping could fire based on whichever worker's eval metric stops improving first, not the true global metric.
2. Socket error / hang on early stop ([dask] [python-package] Early stopping causes DaskLGBMClassifier to hang #6351, DaskLGBMRegressor early stopping cause socket error 104 #6197, [python-package] [dask] version 4.6.0 use dask early_stopping hang on #7009): When one worker's local early stopping fired first, that worker exited the loop and closed its network connection. Remaining workers hit Socket recv error (code: 104) or hung waiting for the exited worker at the next collective.

After this PR, all workers see the same global metric and agree on when to stop.

One design choice to make is how to handle workers that receive no eval chunk. In this PR, workers with no local rows for a validation set still participate in evaluation: they register an empty local validation dataset and contribute zero local numerator / denominator to the distributed metric reduction. Because workers with an empty local eval set still enter the same aggregation calls, the distributed collective stays synchronized and no worker hangs waiting for peers that never participate.

What changed

C++ metric aggregation

• Additive metrics (binary_logloss, l2, multi_logloss, cross_entropy, etc.): sum aggregation via GlobalSyncUpBySum. Cheap, no extra memory.
• Rank-based metrics (NDCG, MAP): per-query weighted sums aggregated via GlobalSyncUpBySum before final division, preserving query-level semantics.
• Gather-based metrics (AUC, AveragePrecision, AUC-μ): new GatherEvalData helper in src/metric/gather_eval_data.hpp Allgathers the full (label, score, weight) arrays peer-to-peer; each worker recomputes the global result.

Python / Dask

• Workers always register eval placeholders so zero-data workers participate in collectives and synchronization stays balanced.
• Eval padding gated on not is_ranker to avoid C++ NDCG throwing on zero-row data.
• eval_class_weight is filtered per-worker to classes actually present in the local eval slice — avoids ValueError: Class label N not present on older sklearn (1.0.x).
• Distributed-eval note added to Dask estimator fit docstrings.

Tests

• 48 parametrized validation metric scenarios across additive, rank-based, and gather-based metrics.
• test_early_stopping_uses_global_metric (binary / regression / multiclass): asserts best_score_ equals the recorded global metric at best_iteration_ and that best_iteration_ is the argmin of the loss series. Test completion without socket errors also covers [dask] [python-package] Early stopping causes DaskLGBMClassifier to hang #6351/DaskLGBMRegressor early stopping cause socket error 104 #6197/[python-package] [dask] version 4.6.0 use dask early_stopping hang on #7009.
• test_auc_mu_matches_pairwise_auc_ground_truth: compares C++ auc_mu against sklearn.roc_auc_score over each OvO pair.
• Shared helpers _per_query_mean, _ndcg_at_k, _map_at_k replace inline ranking metric logic.

Known out of scope

Custom Python eval callables still see per-worker local values.

[jameslamb]

Thanks for your interest in LightGBM. I'm open to you exploring this (I do see how it could be valuable) but there is much to figure out for this to work well.

keep ordering/query-dependent metrics like AUC, average precision, AUC-Mu, NDCG, and MAP out of scope

Thanks for calling this out. Please remove "fixes" from the PR description... this PR in its current state is not enough to close the issue. It doesn't cover:

• the metrics you listed
• custom metrics implemented as Python functions

I think being inconsistent there will be confusing for users. Without looking at the code how would they know whether evals_result contains a global or a local-to-one-worker metric?

This should at a minimum be documented, maybe somewhere in the constructor docstrings for the Dask estimators.

## Test Plan

Did you test these changes yourself? It looks to me like you might have pointed an LLM at #6963, said "make a PR", and had it just commit and push whatever it came up with without review. I see strong evidence of that... common LLM-generated-PR template in the description, confusing choices in the code, and overly-verbose tests.

If so, please don't do that! If you're interested in contributing to LightGBM I'm happy to work with you, but I refuse to talk to an LLM through you.

If you'd like to continue, please:

• address all the requests in my first round of comments
• re-write the PR description in your own words
• from this point forward, write all review discussion in your own words (no copying in LLM responses)
• review every line of code yourself before committing (no automatically pushing LLM-generated code to the PR branch)

[jameslamb]

It looks to me like this change also breaks some of the Dask tests. These look relevant:

FAILED tests/python_package_test/test_dask.py::test_eval_set_no_early_stopping[specified-eval_sizes1-array-binary-classification] - lightgbm.basic.LightGBMError: Socket recv error, Connection reset by peer (code: 104)
FAILED tests/python_package_test/test_dask.py::test_eval_set_no_early_stopping[specified-eval_sizes1-array-multiclass-classification] - lightgbm.basic.LightGBMError: vector::_M_default_append
FAILED tests/python_package_test/test_dask.py::test_eval_set_no_early_stopping[specified-eval_sizes1-array-regression] - lightgbm.basic.LightGBMError: Socket recv error, Connection reset by peer (code: 104)
...

https://github.com/lightgbm-org/LightGBM/actions/runs/26325496087/job/77528016327?pr=7290#step:6:9821

Before pushing your next round of commits, please try testing locally.

cmake -B build -S .
cmake --build build --target _lightgbm -j4
sh build-python.sh -install --precompile
pytest tests/python_package_test/test_dask.py

[szjunma]

@jameslamb Thanks for the review. I do plan to work on the dask early stopping issue - this PR is the first part of it.

I have put the PR to draft and will continue working on addressing your comments + other improvements.

[jameslamb]

Oh good to know! We'd be grateful for the help on the Dask integration.

Sure, you're welcome to work in draft here for a bit and @ if you have any questions.

[szjunma]

@jameslamb This PR is ready for a re-review. The changes can be grouped into 4 parts:

1. add aggregation for supported metrics
2. handle empty eval set at worker level
3. gate non-supported metrics to existing behavior
4. add tests

Since the last review, I've also added docstring documentation about the aggregation limitations, simplified the tests, and fixed CI failures.

[szjunma]

Hey @jameslamb, circling back on this — the scope has grown a bit since the last review. It now also covers non-additive metrics, which means all built-in metrics are supported and early stopping in Dask is working (without extra effort). Ready for another look whenever you have time.