Refactor project structure to python proj

[pancak3]

/resolve #290

[pancak3]

Hey @maugustosilva @Vezio, this is sadly a huge pr :(

Although I tried my best to validate changes locally, I must not have covered many things due to environment limits. Testing this PR in your environment should be required as well. Please let me know what issues, and enjoy the mess : D

In general, this PR includes pdm as the package manager. Ruff is configured as the formatter and linter. The main source files are moved to the folder llm_d_benchmark under the root. The following are the related changes I made:

0. Updated main readme
1. Removed installations of py deps from install_deps.sh since pdm manages them
2. Fixed pre-commit
3. Added formatter and linter in pre-commit hooks
4. Removed unnecessary go entries from the makefile
5. Fixed bugs that do not respect dry-run in some steps
6. Changed & fixed Python imports
7. Added formatter and linter in workflow
8. Changed necessary paths in the Dockerfile. The container image was built successfully, but I doubt the Dockerfile has some bugs. I will pass this to the person who owns it
9. others, see changes

Most of the py file changes are from the formatter and linter and are thus safe to skip in your review.

Quick check cmds

pip install pdm && pdm install && pdm run $SHELL

pdm run format-check
pdm run lint-check
pdm run test

pdm run pre-commit run --all 

Cheers.

[maugustosilva]

@pancak3 Oh well :-). First of all thanks for the contributions, but with 165 files changed, I need to ask about the possibility of splitting it on smaller PRs.

[Vezio]

@pancak3 I'd propose the following integration path:

• Open a PR with changes for integrating the linter to automation (but not directly using it, yet)
• Open a PR with changes for the changes made by the linter (then enable the linter automation)
  • For this one - I'd prefer if we made the changes in chunks so we can transparently view what was changed
• Open a PR with changes for the Python Package restructure

Thank you for such great efforts!

[pancak3]

@pancak3 I'd propose the following integration path:

• Open a PR with changes for integrating the linter to automation (but not directly using it, yet)

• Open a PR with changes for the changes made by the linter (then enable the linter automation)

  • For this one - I'd prefer if we made the changes in chunks so we can transparently view what was changed

• Open a PR with changes for the Python Package restructure

Thank you for such great efforts!

That sounds pretty reasonable to me. I will check out the first todo item within the week and others after. Got some research ddls to catch this month. Thanks! @Vezio @maugustosilva

[Vezio]

@pancak3 awesome thanks! I sense there will be a LOT of contributions this week and a bit of refactoring, so it will probably be good timing for your research you have going on so the code takes a breather 😄