GLM-4.5-Air quantized model failed to load in VLLM - with fix

[avtc]

I have quantized GLM-4.5-Air model with autoround and with GPTQModel.
During quantization I have skipped shared_experts layers in both configurations.
The GPTQModel quant loads in VLLM.
The autoround quant does not load into VLLM with errors.

The initial error:

(VllmWorker rank=3 pid=35201) INFO 08-26 12:47:02 [gptq_marlin.py:266] Using ExllamaLinearKernel for GPTQMarlinLinearMethod
Loading safetensors checkpoint shards:   0% Completed | 0/23 [00:00<?, ?it/s]
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511] WorkerProc failed to start.
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511] Traceback (most recent call last):
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/v1/executor/multiproc_executor.py", line 485, in worker_main
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/v1/executor/multiproc_executor.py", line 382, in __init__
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     self.worker.load_model()
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/v1/worker/gpu_worker.py", line 201, in load_model
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/v1/worker/gpu_model_runner.py", line 1876, in load_model
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     self.model = model_loader.load_model(
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     self.load_weights(model, model_config)
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/model_loader/default_loader.py", line 259, in load_weights
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     loaded_weights = model.load_weights(
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]                      ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/models/glm4_moe.py", line 673, in load_weights
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     return loader.load_weights(weights)
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     yield from self._load_module(prefix,
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     loaded_params = module_load_weights(weights)
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]   File "/home/ubuntu/git/avtc/vllm/vllm/model_executor/models/glm4_moe.py", line 565, in load_weights
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]     param = params_dict[name]
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511]             ~~~~~~~~~~~^^^^^^
(VllmWorker rank=0 pid=35198) ERROR 08-26 12:47:03 [multiproc_executor.py:511] KeyError: 'layers.29.mlp.shared_experts.down_proj.weight'

1. I have compared the quants metadata and have found that autoround quant has model.layers.1.mlp.gate.e_score_correction_bias (and all other modules with e_score_correction_bias) of type torch.bfloat16, while the GPTQModel quant has type torch.float32.

2. I have compared the configs, and tried to use GPTQModel quantize config for autoround quant. And it works.
   In fact adding this property made it work:

    "dynamic": {
      "-:.*shared_experts": {},
      "-:.*shared_head": {},
      "-:lm_head.weight": {},
      "-:model.embed_tokens.weight": {}
    },

P.S. I have padded intermediate size to run with tp8. Also use --enable-expert-parallel --dtype=float16 for vllm. --enable-expert-parallel can be omitted if moe intermediate size is also padded.

[wenhuach21]

Thanks for the information.

1 We were not aware of this issue, good catch. We will try to fix it . But converting e_score_correction_bias to bf16 does not seems to affect the loading in vLLM, as shown in https://huggingface.co/Intel/GLM-4.5-int4-AutoRound/blob/main/model-00009-of-00038.safetensors, though it probably affect accuracy.

2 This is a vLLM issue rather than ours. I assume this model could run on transformers. And we are able to quantize most of the layers you listed for GLM-4.5. Since we don’t pre-cache any quantization configuration for any model and vllm may solve this issue in the future, we prefer to keep it this way for now.

[wenhuach21]

#770

[wenhuach21]

@avtc Sorry, I may have missed something. As you mentioned:

I have quantized the GLM-4.5-Air model with AutoRound and with GPTQModel.
During quantization, I skipped the shared_experts layers in both configurations.
The GPTQModel quantized version loads in vLLM, but the AutoRound quantized version fails to load with errors.

Do you mean that for both GPTQModel and AutoRound you applied the same configuration, but only the AutoRound model cannot load in vLLM?

"dynamic": {
"-:.*shared_experts": {}
}

If so, this is not the expected behavior. As you also mentioned that the gptqmodel exclude some other layers, like shared_head, I am a little confused.

Anyway, @n1ck-guo please have a try and make sure we have the same combability as gptqmodel

[avtc]

@wenhuach21
I have taken the quantization GPTQ dynamic setup from another successfully loaded quant as an example:
https://huggingface.co/QuantTrio/GLM-4.5-Air-GPTQ-Int4-Int8Mix/blob/main/config.json

And used for autoround:

layer_config = {
    ".*shared_experts": {"bits": 16},
    "model.embed_tokens": {"bits": 16},
    "lm_head": {"bits": 16}
}

for gptqmodel:

dynamic = {
    r"-:model.embed_tokens.weight": {},
    r"-:.*shared_experts": {},
    r"-:.*shared_head": {},
    r"-:lm_head.weight": {},
    }

In fact there are no shared_head weight and autoround does not allow to place it in layer config. So I assume in GPTQModel it is ignored.

The autoround quant successfully loaded into vllm after adding dynamic section to the quantization config.

[wenhuach21]

@wenhuach21 I have taken the quantization GPTQ dynamic setup from another successfully loaded quant as an example: https://huggingface.co/QuantTrio/GLM-4.5-Air-GPTQ-Int4-Int8Mix/blob/main/config.json

And used for autoround:

layer_config = {
    ".*shared_experts": {"bits": 16},
    "model.embed_tokens": {"bits": 16},
    "lm_head": {"bits": 16}
}

for gptqmodel:

dynamic = {
    r"-:model.embed_tokens.weight": {},
    r"-:.*shared_experts": {},
    r"-:.*shared_head": {},
    r"-:lm_head.weight": {},
    }

In fact there are no shared_head weight and autoround does not allow to place it in layer config. So I assume in GPTQModel it is ignored.

The autoround quant successfully loaded into vllm after adding dynamic section to the quantization config.

Thanks for the detailed information. We’ll investigate to see if this is a bug.

[n1ck-guo]

Thank you for the information and found the issue. I conducted relevant experiments on our A100 machine, but could not find the problem you mentioned.
The following is the log of my experiment.

First, I using main branch auto_round and latest release version of vllm (pip install -U vllm) to quant the GLM-4.5-Air with layer_config:

layer_config = {
    ".*shared_experts": {"bits": 16},
    "model.embed_tokens": {"bits": 16},
    "lm_head": {"bits": 16}
}

This cause a bug of auto_round because of setting model.embed_tokens. This bug will be fixed soon.
After change layer_config to

layer_config = {
    ".*shared_experts": {"bits": 16},
    "lm_head": {"bits": 16}
}

I can sucess to export the quantized model.

However, I met some bugs when loading this model with vllm. This is because the intermediate_size (10944) is not divisible by default group_size (128).

Therefore, I regenerate a w4g64 quantized model, and vllm can load it successfully.

Here is the script for quantization and loading, could you please try it and see if it works?

Quantize:

from auto_round import AutoRound

layer_config = {
    ".*shared_experts": {"bits": 16},
    "lm_head": {"bits": 16}
}

ar = AutoRound(model="GLM-4.5-Air/", layer_config=layer_config, iters=0, group_size=64)
ar.quantize_and_save(output_dir="GLM-4.5-Air-auto-round-g64")

vLLm cmd:

VLLM_WORKER_MULTIPROC_METHOD=spawn python -m vllm.entrypoints.openai.api_server --port 8001 --trust-remote-code --tensor-parallel-size 4 --gpu-memory-utilization 0.90 --model GLM-4.5-Air-auto-round-g64/ --enable-expert-parallel --max-model-len 32768 --max-seq-len-to-capture 32768 --dtype float16

If there is a bug when loading vllm:
TypeError: MoeWNA16Method.get_weight_loader..moe_wna16_weight_loader() got an unexpected keyword argument 'return_success'

This is a known issue cause by vLLM, please try vllm-project/vllm#22797

[avtc]

@n1ck-guo can you please share resulting quantization_config? Does it have "dynamic" section or something like that?

Here is a script to automatically pad the intermediate size for required group-size and tensor-parallel size (the device_map prepared for 4 GPU):
quantize-glm4.5-air-autoround-padding.py

[n1ck-guo]

Here is the quantization_config, I used a 4-layers model for test.

{
"bits": 4,
"group_size": 64,
"sym": true,
"data_type": "int",
"iters": 0,
"autoround_version": "0.6.1.dev",
"quant_method": "auto-round",
"packing_format": "auto_round:auto_gptq",
"extra_config": {
"model.layers.1.mlp.shared_experts.gate_proj": {
"bits": 16
},
"model.layers.1.mlp.shared_experts.up_proj": {
"bits": 16
},
"model.layers.1.mlp.shared_experts.down_proj": {
"bits": 16
},
"model.layers.2.mlp.shared_experts.gate_proj": {
"bits": 16
},
"model.layers.2.mlp.shared_experts.up_proj": {
"bits": 16
},
"model.layers.2.mlp.shared_experts.down_proj": {
"bits": 16
},
"model.layers.3.mlp.shared_experts.gate_proj": {
"bits": 16
},
"model.layers.3.mlp.shared_experts.up_proj": {
"bits": 16
},
"model.layers.3.mlp.shared_experts.down_proj": {
"bits": 16
}
}

[avtc]

Thanks I see vllm has parser for extra_config, so should work, will check today.

[avtc]

@n1ck-guo here is my quantize_config.json of resulting model:

{
  "bits": 4,
  "group_size": 128,
  "sym": true,
  "data_type": "int",
  "seqlen": 512,
  "iters": 1,
  "autoround_version": "0.6.1.dev",
  "quant_method": "gptq",
  "desc_act": false,
  "true_sequential": false,
  "damp_percent": 0.01
}

I have used the script I attached above, the main branch of auto-round and layer_config:

layer_config = {
    ".*shared_experts": {"bits": 16},
    "lm_head": {"bits": 16}
}

I was using: autoround.quantize_and_save(OUTPUT_DIR, format="auto_gptq") in the end to save the model, as saving to format=auto_round did not work for me (caused by presence "model.embed_tokens": {"bits": 16}, in layer_config).

I have quantized with format=auto_round, and on load in vllm got the TypeError: MoeWNA16Method.get_weight_loader..moe_wna16_weight_loader() got an unexpected keyword argument 'return_success'.
After building from vllm-project/vllm#22797 - it works.

So the issue is in not saving the "dynamic" section for "auto_gptq" format.

[JartX]

@wenhuach21 merged PR vllm-project/vllm#22797 into main