The implementation of dynamic batch loading code seems inconsistent with the pseudo-code in the paper

[YWMditto]

For example, truncating the loss difference to 0 does not seem to be implemented.

LLM-Shearing/llmshearing/callbacks/dynamic_loading_callback.py

Line 34 in 1386c8f

diff = torch.tensor(losses) - torch.tensor(self.target_loss)

And, what is the purpose of this line?

LLM-Shearing/llmshearing/callbacks/dynamic_loading_callback.py

Line 41 in 1386c8f

updated_domain_weights = (1-c) * updated_alpha + c / self.n_domains

[xiamengzhou]

Hi, sorry for the late reply!

In the implementation, we added a small uniform proportion (c=1e-4/7) to each domain when updating the weights. It's simply a smoothing factor, and does not in practice affect results much.