Optimizer

[lucellent]

Is it possible to add adamw8bit?
I tried changing in the config, but it looks unsupported

[JusperLee]

Here's a suggested reply for your issue:

---

Thank you for bringing this up. It seems the current configuration doesn't support AdamW8bit by default. To resolve this, you'll need to modify the look2hear/system/optimizers.py file.

1. First, ensure that the bitsandbytes library is installed, as it provides support for 8-bit optimizers like AdamW8bit:

   pip install bitsandbytes
   

2. Then, in the optimizers.py file, import the necessary module:

   from bitsandbytes.optim import AdamW8bit

3. Update the optimizer function to include AdamW8bit. For example, you can modify the section where optimizers are initialized:

   if optimizer_name == 'adamw8bit':
       optimizer = AdamW8bit(model.parameters(), lr=learning_rate, weight_decay=weight_decay)

4. Finally, make sure your configuration supports the adamw8bit entry.

Once these changes are made, you should be able to run the model using AdamW8bit. Let me know if you encounter any issues during this process.

---

This should guide them in resolving the unsupported optimizer issue!

[Manoa1911]

does this do any good for CPU inference ?

[JusperLee]

I'm not entirely sure either, but from what I know, AdamW8bit is primarily optimized for GPU usage, and it may not provide the same benefits for CPU inference.