Fix MSE calibration distributed amax sync and add multi‑GPU test

[CedricHwong]

What does this PR do?

Type of change: Bug fix

Overview:

• Synchronizes MSE calibration amax across distributed groups (DP/EP/TP) after calibration finishes.
• Adds a multi‑GPU test that verifies amax values match when distributed_sync=True and differ when distributed_sync=False.

Usage

  import copy
  import modelopt.torch.quantization as mtq

  # Build a quantization config that uses MSE calibration
  cfg = copy.deepcopy(mtq.INT8_DEFAULT_CFG)
  cfg["algorithm"] = {
      "method": "mse",
      "distributed_sync": True,
  }

#Run quantization + calibration (forward_loop feeds calibration data)
  model = mtq.quantize(model, cfg, forward_loop)

Testing

PYTHONPATH=/root/epfs/workspace/code/personal_repos/Model-Optimizer pytest -q tests/gpu/torch/quantization/test_mse_calibrate_sync.py
      - Result: 3 passed, 1 skipped (skip: 1‑GPU case)

• Make sure you read and follow Contributor guidelines (https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed.
• Is this change backward compatible?: Yes
• Did you write any new necessary tests?: Yes
• Did you add or update any necessary documentation?: No
• Did you update Changelog (https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: YES

Additional Information

• New test validates distributed amax synchronization for MSE calibration under NCCL.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[realAsma]

Can we hold off on this PR. We are currently running into issues related to Megatron MoE max calibration. See #752

In addition, amax calibration across mse is not as straight forward as implemented in this PR.
To be correct, mse distributed sync should sync the mse_losses across the relevant distributed parallelism.

Considering that we are still getting mse calibration results, this feature is too early in my opinion.