Is PPO really better than SFT (in general)? under the condition of same amount of data

[allanj]

For example, if we ask the model to generate a program, rather than simply continuation.

If we do not fine-tune them, RL does not even know what to generate I believe.

Do you have more thoughts on this?

[allanj]

I'm really confused that, when you run PPO without SFT, for example, in Narrative QA? How do the (quite-small) model knows it should generate an answer?