New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于CPU通信的一点疑问 #343

Open

Nemo-HelloWorld opened this issue Feb 18, 2025 · 1 comment

Nemo-HelloWorld commented Feb 18, 2025 •

edited

Loading

请问在megatron/megatron/training/checkpointing.py 246-250行中：

    # Get the max iteration retrieved across the ranks.
    if torch.distributed.is_initialized():
        iters_cuda = torch.tensor([iteration], dtype=torch.long, device='cuda' if 'nccl' in torch.distributed.get_backend() else 'cpu')
        torch.distributed.all_reduce(iters_cuda, op=torch.distributed.ReduceOp.MAX)
        max_iter = iters_cuda[0].item()

通信张量iters_cuda的device的判断条件为什么不是：

device='cpu' if 'gloo' in torch.distributed.get_backend() else 'cuda'

The text was updated successfully, but these errors were encountered:

Author

Nemo-HelloWorld commented Feb 18, 2025

再就是为什么flagscale/train/train.py 2084-2088行中：

        flags = torch.tensor(
            [int(do_train), int(do_valid), int(do_test)],
            dtype=torch.long, device=get_device_type_for_comm())
    else:
        flags = torch.tensor([0, 0, 0], dtype=torch.long, device=get_device_type_for_comm())

为什么get_device_type_for_comm函数的参数为空？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment