Pytorch Profiler

[AndreSlavescu]

Description

Please provide a brief summary of the changes in this pull request.

Checklist

Before submitting this PR, ensure the following steps have been completed:

• Run the slash command /verifyruns on your own server.
  • Run the cluster bot on your server:

    python discord-bot.py

  • Start training runs with the slash command /verifyruns.
  • Verify that the bot eventually responds with:

    ✅ All runs completed successfully!
    

    (It may take a few minutes for all runs to finish. In particular, the GitHub
    runs may take a little longer. The Modal run is typically quick.)
    For more information on running a cluster bot on your own server, see
    README.md.

[alexzhang13]

@AndreSlavescu The only problem I see with this is that it implicitly just wraps a profiler around the top of a file, but this can be problematic if someone just submits a standard kernel file that's formatted as a bunch of functions (you'll have to be explicit about this). Like basically this only works if someone submits a file that doesn't run.

I need to give this more thought, @S1ro1 maybe you have better thoughts.

Can you instead make sure for now that profiling function goes into say the if name == "main"?

[alexzhang13]

For now, can you make it so the PyTorch profiler goes inside say if __name__ == '__main__'? also make sure it indents properly so it actually runs.