A different discriminator?

[ehsan-soe]

Hi,

I really enjoyed your paper.
In this regard, I have following questions that I appreciate your reply,

1. I wonder if it possible to replace the current discriminator which is build on top of the LMHead with any other trained discriminator? In other words, does it really need to be built on top of the LM (p(x)) itself, or it can be any discriminator?
2. If the answer to previous is yes, then is it possible to fine-tune gpt-2 on our own data and then generate from that with our specific discriminator (attr model)?

[mshlis]

@ehsan-soe

1. It needs to be built on top of the LM to get gradients for backprop (if otherwise you would have to use some form of expensive/noisy gradient approximation scheme to use the same approach)
2. You can still fine tune this approach on your own data, just fine tune the descriminator ontop of the LM

(Note: i have no actual association to this project)

[srulikbd]

@mshlis so can I just replace GPT2 with like "distilbert-base-multilingual-cased"?

[mshlis]

@srulikbd sure but for what result? The goal of this is to do text generation which you can’t do effectively with distilbert. You probably want a backbone that is trained on next word/letter prediction.

[srulikbd]

OK, you are right.
I saw experiments that tried to use BERT like LM but got bad results, because of the bidirectionality training.
so I think I should find GPT2 that was trained on my specific language, or train one myself..

thanks for the quick answer!

[liya-gafurova]

Hello! Thank you for your great work!

In your article you have mentioned that instead of GPT2 it could be used another transformer-based text generator ("but the method applies in any representation space from any transformer-based text generator").
Is it possible to replace GPT2 with CTRL? And how to manage using CTRL with its different codes (like "Links", "Wikipedia", "Rewies", "Reddit " etc) within PPLM codebase?