Add trust_remote_code to GRPOConfig

[muupan]

What does this PR do?

This PR adds trust_remote_code to GRPOConfig and makes GRPOTrainer use it when making the model and its related objects to support custom models.

Fixes #4129

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[qgallouedec]

Suggested change

	metadata={"help": "Whether to trust remote code when loading custom models e.g. from the Hugging Face Hub."},
	metadata={"help": "Whether to trust remote code when loading custom models from the Hugging Face Hub."},

[muupan]

trust_remote_code matters when loading things from local files too. The docs of vLLM describe it similarly.

Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer.

https://docs.vllm.ai/en/latest/api/vllm/index.html#vllm.LLM

That said, the docs of transformers only mention the hub. Do you think e.g. should be deleted? I don't have a strong opinion so will follow your preference.

Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.

https://huggingface.co/docs/transformers/v4.56.2/en/model_doc/auto#transformers.AutoModel.from_pretrained.trust_remote_code

[qgallouedec]

When using remote code, the idea is that that model is not included in transformers, right? So maybe you need something like this instead:

if hasattr(transformers, config.architectures[0]):
    architecture = getattr(transformers, config.architectures[0])
    model = architecture.from_pretrained(model_id, **model_init_kwargs)
else:
    model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

[muupan]

Thanks, you are right. I haven't tested the code path of AutoConfig as I was passing an already loaded model to GRPOTrainer. I will try rewriting it like your code.

[muupan]

I fixed it. Now it works when model passed to GRPOTrainer is str.