align input scale for moe experts for fp8

[SwekeR-463]

Fixes #1194

• Aligned the act_max for the first linear projections (gate and up) across all experts in MoE blocks to ensure a single input_scale for FP8 dispatch.
• Added unit test with DeepSeek-V2-Lite-Chat.

[yiliu30]

Thanks for the PR!
Overall LGTM. Left a few comments.

[yiliu30]

@chensuyue @XuehaoSun Looks like the XPU test failed, and it doesn’t seem related to this PR. Could you take a look?

[WeiweiZhang1]

LGTM

[yiliu30]

LGTM, thanks!

[xin3he]

@SwekeR-463 @yiliu30 I think here is a missing change, set_amax_for_all_moe_layers is not applied for fp8 during tuning.

auto-round/auto_round/compressors/base.py

Line 1326 in 46aec78

if is_nv_fp(self.act_data_type) or is_static_wfp8afp8(self):

auto-round/auto_round/compressors/base.py

Line 2763 in 46aec78

if is_nv_fp(self.act_data_type):

[xin3he]

BTW, if my understanding is correct that vLLM requires AR_ENABLE_UNIFY_MOE_INPUT_SCALE, I think the default value should be True since vLLM is our main target.

[yiliu30]

@SwekeR-463 @yiliu30 I think here is a missing change, set_amax_for_all_moe_layers is not applied for fp8 during tuning.

auto-round/auto_round/compressors/base.py

Line 1326 in 46aec78

if is_nv_fp(self.act_data_type) or is_static_wfp8afp8(self):

auto-round/auto_round/compressors/base.py

Line 2763 in 46aec78

if is_nv_fp(self.act_data_type):

Hi @xin3he ,this seems to be a general gap in FP8_STATIC rather than something related to this enhancement. Since this PR focus is the RTN case, it’s fine to ignore it here. Please feel free to create another PR to fix that part.

[yiliu30]

BTW, if my understanding is correct that vLLM requires AR_ENABLE_UNIFY_MOE_INPUT_SCALE, I think the default value should be True since vLLM is our main target.

Since FP8 dispatch is primarily for extreme inference speed, and it’s disabled by default in vllm‑gaudi.
And sharing input scales across all experts may degrade accuracy; I prefer to keep it disabled by default.

[xin3he]

Thanks for your reply, LGTM since it's for RTN only.

[yiliu30]

Hi @XuehaoSun looks like the CI was blocked by CodeQLExpected test? Could you please take a look, thx!