Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge disc backend to acc 2.3 #3

Merged
merged 12 commits into from
Oct 11, 2024
Merged

Merge disc backend to acc 2.3 #3

merged 12 commits into from
Oct 11, 2024

Conversation

yitongh
Copy link

@yitongh yitongh commented Aug 8, 2024 •
edited
Loading

Open this PR to track the modifications on disc backend.

@CLAassistant
Copy link

CLAassistant commented Aug 8, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@eedalong @yitongh
Disc computation (#2)
85927c6 
Support Disc as backend
Co-authored-by: yancey.yx <[email protected]>
Co-authored-by: wangang.wa <[email protected]>
Yancey1989 and others added 5 commits August 8, 2024 14:32
@Yancey1989 @yitongh
add bazel flag to disable disc backend (#23)
ff96e9b 
* add flag to disable disc backend in bazel workspace
@Yancey1989 @yitongh
support disc debug mode to dump mhlo and logs (#25)
2d70a88 
support disc backend debug mode to dump DISC compilation logs
@eedalong @yitongh
support flash attention in disc (pytorch#34)
d14788b
@baoleai @yitongh
fix disc flag when complie python (pytorch#39)
29d1ae6 
* fix bazel flag when complie python

* fix lint.
@Yancey1989 @yitongh
support bf16 on disc backend (pytorch#40)
e1b8ed7 
add float-norm pass to support bf16 amp training
@yitongh
Copy link
Author

yitongh commented Aug 8, 2024

The prs that have not been merged yet:

  • e5470e3d50d528f49f287cfbb015e0d10897b897
  • cceb7f76fa2c1dcdc3260cef75d3cf34e0744126
  • 3fa4fe284dacef4ae447271356d4719cd223c2e2
  • c820ca030a51429a97654178afdefaf5f88d4485
  • 4c7fbd81383bbd05de2aca7b3c84731f197e9628

cceb7f76fa2c1dcdc3260cef75d3cf34e0744126 should be merged before support disc debug mode to dump mhlo and logs because "//torch_xla/csrc/runtime:env_vars" and "//torch_xla/csrc/runtime:sys_util" can be linked.

FA 256 support: #4

@yitongh
Copy link
Author

yitongh commented Aug 8, 2024 •
edited by anw90
Loading

TODO:

  • third_party/nccl/nccl.h not found
  • ENABLE_DISC=0 not work

yitongh and others added 4 commits August 12, 2024 14:45
@yitongh
Support Flash Attention 2.5.6 for disc backend (#4)
41baac7
@Yancey1989
fix build failed with NCCL (#5)
1e499d9 
* fix build failed on nccl

* using nccl hdrs
@yitongh
Use the value of DISC_DEVICE as the device type of disc backend (#8)
aa2c40e 
* change the device type of disc to cuda to make amp work properly

* Use the value of DISC_DEVICE as the device type of disc backend
@yitongh
disable compilation of DISC by default (#15)
d094086
@anw90 anw90 merged commit fab18e0 into acc Oct 11, 2024
2 checks passed
@anw90 anw90 deleted the merge_disc branch October 11, 2024 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anw90 anw90 anw90 approved these changes

@Yancey1989 Yancey1989 Awaiting requested review from Yancey1989

@Seventeen17 Seventeen17 Awaiting requested review from Seventeen17

@eedalong eedalong Awaiting requested review from eedalong

@baoleai baoleai Awaiting requested review from baoleai

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@yitongh @CLAassistant @anw90 @Yancey1989 @Seventeen17 @eedalong @baoleai