Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ray Dashboard] The format of the GPU profile file downloaded from the dashboard is incorrect #46240

Open
yx367563 opened this issue Jun 25, 2024 · 0 comments
Labels
bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component)

Comments

@yx367563
Copy link

What happened + What you expected to happen

Hi All. I ran GPU profiling on Ray using Nsight and the report was also generated. However, I am not able to view it on Nsight Systems on WSL. I got the following error:
/dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/Analysis/Clients/ReportFile.cpp(339): Throw in function void QuadDAnalysis::ReportFile::verifyVersionTag() Dynamic exception type: boost::wrapexceptQuadDCommon::CorruptedDataException std::exception::what: CorruptedDataException [QuadDCommon::tag_message*] = Invalid version prefix: 1NVIDIA Tegra Profiler Report [QuadDAnalysis::tag_report_file_name*] = “/Users/rajesh.dalai/Downloads/nsight_worker_process_854.nsys-rep”
Can anyone help with this? It seems that there is a problem with the format downloaded from the dashboard. Maybe some escapes are not processed properly. Because there is a 1 in front of NVIDIA, and the subsequent lines have similar problems

Versions / Dependencies

Ray: 2.23.0
Nsight: 2024.4.1.61-244134315967v0

Reproduction script

Follow the instructions in the Ray official documentation: https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#run-nsight-on-ray

Issue Severity

High: It blocks me from completing my task.

@yx367563 yx367563 added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Development

No branches or pull requests

1 participant