Loss calculation of `RewardTrainer` may be inaccurate when performing gradient accumulation?

[jue-jue-zi]

trl/trl/trainer/reward_trainer.py

Line 250 in 7e20753

num_items_in_batch=None,

It seems that RewardTrainer does not use num_items_in_batch in the loss calculation. According to the transformers Trainer documentation, self.model_accepts_loss_kwargs = False must be set to ensure the loss is computed correctly when performing gradient accumulation?