Is there a way to profile functions called from a kernel?

[cadop]

I'm a novice at the profiling, so with that in mind;

I am using nvtx by setting nvidia nsights to run my virtual environment python.exe calling my script. I use with wp.ScopedTimer("Simulation Step", use_nvtx=True, color="yellow", detailed=True): , which gives me some view of the kernels within the nsights report.

However, I can't see any of my @wp.func's , nor warp calls.

Maybe I am using the wrong approach. Essentially I have a kernel that takes 1 second and another that takes 50ms. The 1 second one uses a bunch of functions, but none of them are showing.

[shi-eric]

The Warp functions are compiled into the kernel, so you won't get granularity finer than that with Nsight Systems (afaik). If you are trying to answer the question why is this 1 second kernel taking so long then your question comes at a great time since I just added a section to the docs talking about kernel-level profiling with Nsight Compute: https://nvidia.github.io/warp/profiling.html#nsight-compute-profiling

[shi-eric]

Maybe start with seeing if C/SASS correlation works in example_sph.py. I just added:

wp.config.lineinfo = True
wp.config.line_directives = False

Then ran ncu --open-in-ui -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 python example_sph.py --stage_path None --num_frames 10

[cadop]

Full disclosure, I haven't used nsight compute before, so if there is a simple button I am missing please let me know. But below is what I have going on. When running the terminal command, this UI pops up. I click on "Source", and try to view as source and sass, but only sass shows.

[shi-eric]

No worries, I don't think it's something you're doing incorrectly. Your view is the same one I get if I don't compile with line information, but I know you aren't doing that due to your first screenshot.

One more thing I'll add is set WARP_DEBUG="True" before compiling.

No line info:

WARP_DEBUG="True" ncu --open-in-ui -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 python example_sph.py --stage_path None --num_frames 10
Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=20, releaselevel='final', serial=0)
==PROF== Connected to process 18194 (/home/eshi/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/bin/python3.8)
Warp 1.7.0 initialized:
   CUDA Toolkit 12.8, Driver 12.8
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 3090" (24 GiB, sm_86, mempool enabled)
     "cuda:1"   : "NVIDIA GeForce RTX 3090" (24 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Supported fully (all-directional)
   Kernel cache:
     /home/eshi/.cache/warp/1.7.0
NVRTC version 12.8
NVRTC options:
--gpu-architecture=compute_86
--include-path=/home/eshi/code-projects/warp/warp/native
--std=c++17
--define-macro=NDEBUG
--undefine-macro=WP_VERIFY_FP
--define-macro=WP_ENABLE_MATHDX=1
--fmad=true
--device-as-default-execution-space
--extra-device-vectorization
--restrict
Writing 150878 B to /home/eshi/.cache/warp/1.7.0/wp___main___74a237a_p18194/wp___main___74a237a.sm86.ptx (wt)
Module __main__ 74a237a load on device 'cuda:0' took 852.66 ms  (compiled)
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
step took 7222.58 ms
step took 18.37 ms
step took 17.74 ms
step took 19.18 ms
step took 20.06 ms
step took 19.15 ms
step took 18.45 ms
step took 17.75 ms
step took 18.20 ms
step took 17.87 ms
==PROF== Disconnected from process 18194
==PROF== Report: /home/eshi/code-projects/warp/warp/examples/core/example_sph.ncu-rep

Note that in my case, NCU didn't complain that I didn't have line information since I'm not using the source import option, but when I open the report, I'm not given an ability to correlate the SASS code.

Then I edit my example_sph.py file to have the two config commands:

WARP_DEBUG="True" ncu --open-in-ui -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 python example_sph.py --stage_path None --num_frames 10
Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=20, releaselevel='final', serial=0)
==PROF== Connected to process 18687 (/home/eshi/.local/share/uv/python/cpython-3.8.20-linux-x86_64-gnu/bin/python3.8)
Warp 1.7.0 initialized:
   CUDA Toolkit 12.8, Driver 12.8
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 3090" (24 GiB, sm_86, mempool enabled)
     "cuda:1"   : "NVIDIA GeForce RTX 3090" (24 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Supported fully (all-directional)
   Kernel cache:
     /home/eshi/.cache/warp/1.7.0
NVRTC version 12.8
NVRTC options:
--gpu-architecture=compute_86
--include-path=/home/eshi/code-projects/warp/warp/native
--std=c++17
--define-macro=NDEBUG
--generate-line-info
--undefine-macro=WP_VERIFY_FP
--define-macro=WP_ENABLE_MATHDX=1
--fmad=true
--device-as-default-execution-space
--extra-device-vectorization
--restrict
Writing 282970 B to /home/eshi/.cache/warp/1.7.0/wp___main___db68d0b_p18687/wp___main___db68d0b.sm86.ptx (wt)
Module __main__ db68d0b load on device 'cuda:0' took 878.51 ms  (compiled)
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
step took 7008.10 ms
step took 20.05 ms
step took 31.86 ms
step took 18.28 ms
step took 17.69 ms
step took 18.14 ms
step took 18.03 ms
step took 18.23 ms
step took 18.20 ms
step took 17.93 ms
==PROF== Disconnected from process 18687
==PROF== Report: /home/eshi/code-projects/warp/warp/examples/core/example_sph.ncu-rep

Note the extra printout in the NVRTC options: --generate-line-info. I assume you'll see the same thing.

Maybe we're running into a bug with Windows or the CUDA 12.6 driver... If your really want to go to basics, you could forget about Warp and see if you can compile a super simple one-kernel CUDA program with --generate-line-info and profile it with NCU.

[cadop]

Is WARP_DEBUG a shortcut you have for bash?

I put wp.config.mode = "debug" in the file, but i'm not getting NVRTC info displayed. Maybe something more is wrong

[shi-eric]

It's not a shortcut, but just an environment variable you need to set. On Windows powershell it might be like $env:WARP_DEBUG="true"

[cadop]

I'm not able to get that output. Am I missing an entire step?

Basically all i've done is:

1. make a projects folder
2. make a venv
3. pip install warp-lang[extras]

Then part 1 of debugging was:
5. pip install nvtx
6. install nsight systems
7. open nsight systems
8. Create a project with target application using venv python and example_sphy.py. set working directory to that projects folder
9. Run, then see the compute kernels listed in the "CUDA API" line, below "NVTX" line

I couldn't see the kernel functions, so posted this question. Then;
11. install nsight compute from website
12. copy example_sph.py from github into my folder
13. Run the command below

Are there some debug environments I needed to setup?

(.venv310) PS D:\Projects> $env:WARP_DEBUG = "True"; ncu --open-in-ui -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 python example_sph.py --stage_path None --num_frames 10
==PROF== Connected to process 45348 (C:\Program Files\Python310\python.exe)
Warp 1.7.0 initialized:
   CUDA Toolkit 12.8, Driver 12.6
   Devices:
     "cpu"      : "AMD64 Family 25 Model 24 Stepping 1, AuthenticAMD"
     "cuda:0"   : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled)
     "cuda:1"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Not supported
   Kernel cache:
     C:\Users\cadop3\AppData\Local\NVIDIA\warp\Cache\1.7.0
Module __main__ 6c38d92 load on device 'cuda:0' took 1.91 ms  (cached)
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
step took 9549.59 ms
step took 16.09 ms
step took 17.91 ms
step took 17.38 ms
step took 18.08 ms
step took 18.29 ms
step took 18.19 ms
step took 18.36 ms
step took 18.21 ms
step took 17.99 ms
==PROF== Disconnected from process 45348
==PROF== Report: D:\Projects\example_sph.ncu-rep

[shi-eric]

Yes, you didn't clear your cache, so we aren't going down the code path that would print the NVRTC compile options.

[cadop]

Sorry. I added wp.clear_kernel_cache() right after my import in the example_sph.py file. (I had been using that in my original code, and was using a specific cache dir, but hadn't included it when starting from scratch on the sph example. )

It seems like the --generate-line-info is showing.

(.venv310) PS D:\Projects> $env:WARP_DEBUG = "True"; ncu --open-in-ui -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 .\.venv310\Scripts\python.exe example_sph.py --stage_path None --num_frames 10
==PROF== Connected to process 42416 (C:\Program Files\Python310\python.exe)
Warp 1.7.0 initialized:
   CUDA Toolkit 12.8, Driver 12.6
   Devices:
     "cpu"      : "AMD64 Family 25 Model 24 Stepping 1, AuthenticAMD"
     "cuda:0"   : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled)
     "cuda:1"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Not supported
   Kernel cache:
     C:\Users\cadop3\AppData\Local\NVIDIA\warp\Cache\1.7.0
NVRTC version 12.8
NVRTC options:
--gpu-architecture=sm_89
--include-path=D:\Projects\.venv310\Lib\site-packages\warp\native
--std=c++17
--define-macro=NDEBUG
--generate-line-info
--undefine-macro=WP_VERIFY_FP
--define-macro=WP_ENABLE_MATHDX=1
--fmad=true
--device-as-default-execution-space
--extra-device-vectorization
--restrict
Writing 340608 B to C:\Users\cadop3\AppData\Local\NVIDIA\warp\Cache\1.7.0\wp___main___2131e76_p42416\wp___main___2131e76.sm89.cubin (wb)
Module __main__ 2131e76 load on device 'cuda:0' took 609.65 ms  (compiled)
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
step took 8693.64 ms
step took 16.02 ms
step took 18.50 ms
step took 15.37 ms
step took 15.41 ms
step took 18.88 ms
step took 21.50 ms
step took 31.17 ms
step took 26.24 ms
step took 17.97 ms
==PROF== Disconnected from process 42416
==PROF== Report: D:\Projects\example_sph.ncu-rep

It doesn't show the line info warning, but also it doesn't show the source (in the documentation it shows the source getting opened in the figure, before the --import-source is explained, so I am assuming it should be).

I do get the warning, and likewise still does not show source. Using the --import-source directive:

(.venv310) PS D:\Projects> $env:WARP_DEBUG = "True"; ncu --open-in-ui --import-source 1 -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 .\.venv310\Scripts\python.exe example_sph.py --stage_path None --num_frames 10
==PROF== Connected to process 73764 (C:\Program Files\Python310\python.exe)
Warp 1.7.0 initialized:
   CUDA Toolkit 12.8, Driver 12.6
   Devices:
     "cpu"      : "AMD64 Family 25 Model 24 Stepping 1, AuthenticAMD"
     "cuda:0"   : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled)
     "cuda:1"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Not supported
   Kernel cache:
     C:\Users\cadop3\AppData\Local\NVIDIA\warp\Cache\1.7.0
NVRTC version 12.8
NVRTC options:
--gpu-architecture=sm_89
--include-path=D:\Projects\.venv310\Lib\site-packages\warp\native
--std=c++17
--define-macro=NDEBUG
--generate-line-info
--undefine-macro=WP_VERIFY_FP
--define-macro=WP_ENABLE_MATHDX=1
--fmad=true
--device-as-default-execution-space
--extra-device-vectorization
--restrict
Writing 340608 B to C:\Users\cadop3\AppData\Local\NVIDIA\warp\Cache\1.7.0\wp___main___2131e76_p73764\wp___main___2131e76.sm89.cubin (wb)
Module __main__ 2131e76 load on device 'cuda:0' took 603.84 ms  (compiled)
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
step took 8759.48 ms
step took 26.85 ms
step took 16.96 ms
step took 16.47 ms
step took 32.60 ms
step took 16.48 ms
step took 32.79 ms
step took 18.68 ms
step took 17.94 ms
step took 18.29 ms
==PROF== Disconnected from process 73764
==WARNING== Failed to import the following source files:
==WARNING==  D:/Projects/wp___main___2131e76.cu
==PROF== Report: D:\Projects\example_sph.ncu-rep

Initialize code used for the last run:

Q: Would using WSL conflate things? I could try on linux through that, but if it hasn't been tested either, I could try on a linux machine if you are out of ideas.

[cadop]

Okay, sorry for double messages. I have the SPH example working, so it seems to not be a total system failure problem.

I realized the warning was actually different, and was having an issue with the source files. I'm not sure why it was looking in that directory but the cache was written somewhere else. So I specified the cache directory in the script.

The "Kernel cache" is still printing it is from appdata, maybe warp is initialized earlier than the directory being set?

(.venv310) PS D:\Projects> $env:WARP_DEBUG = "True"; ncu --open-in-ui -f -o example_sph -k get_acceleration_a9fb4286_cuda_kernel_forward --set full -c 5 .\.venv310\Scripts\python.exe example_sph.py --stage_path None --num_frames 10
==PROF== Connected to process 67712 (C:\Program Files\Python310\python.exe)
Warp 1.7.0 initialized:
   CUDA Toolkit 12.8, Driver 12.6
   Devices:
     "cpu"      : "AMD64 Family 25 Model 24 Stepping 1, AuthenticAMD"
     "cuda:0"   : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled)
     "cuda:1"   : "NVIDIA RTX A6000" (48 GiB, sm_86, mempool enabled)
   CUDA peer access:
     Not supported
   Kernel cache:
     C:\Users\cadop3\AppData\Local\NVIDIA\warp\Cache\1.7.0
NVRTC version 12.8
NVRTC options:
--gpu-architecture=sm_89
--include-path=D:\Projects\.venv310\Lib\site-packages\warp\native
--std=c++17
--define-macro=NDEBUG
--generate-line-info
--undefine-macro=WP_VERIFY_FP
--define-macro=WP_ENABLE_MATHDX=1
--fmad=true
--device-as-default-execution-space
--extra-device-vectorization
--restrict
Writing 340608 B to D:\Projects\warpcache\wp___main___2131e76_p67712\wp___main___2131e76.sm89.cubin (wb)
Module __main__ 2131e76 load on device 'cuda:0' took 625.73 ms  (compiled)
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
==PROF== Profiling "get_acceleration_a9fb4286_cuda_kernel_forward": 0%....50%....100% - 40 passes
step took 8721.41 ms
step took 27.97 ms
step took 28.18 ms
step took 14.81 ms
step took 16.34 ms
step took 16.94 ms
step took 16.68 ms
step took 22.85 ms
step took 36.50 ms
step took 16.21 ms
==PROF== Disconnected from process 67712
==PROF== Report: D:\Projects\example_sph.ncu-rep

[shi-eric]

Resorting to WSL probably wouldn't be necessary and I haven't tested that either. Seems like it should be supported, though: https://docs.nvidia.com/nsight-compute/ReleaseNotes/index.html#system-requirements

I'd suggest going to a simpler case and seeing if you can profile a basic CUDA program. Maybe one of the programs from the nsight-training repo like https://github.com/NVIDIA/nsight-training/blob/master/cuda/2020_ncu_smem/t5_1.cu

[shi-eric]

In your other message, the calling of wp.clear_kernel_cache() on line 40 also calls wp.init() behind the scenes (assuming wp.init() has not been called before thsi point) and leads to that initialization message showing the cache at AppData\Local\NVIDIA\warp\Cache\1.7.0.

[cadop]

Thanks @shi-eric ! I got it working. If others come by this, might help to read the thread. My thoughts on what might help the docs, or others:

• A BIG problem I had when moving from SPH to my own code, was that apparently some kernels just don't have any source file that can be viewed. For example, compute_mesh_edge_lengths. When I moved to my own code, for awhile I hadn't set the number of kernels high enough, so it should have been working, but it would give a warning that no source could be found, try to compile with --get-line-info. It was compiled with it, the reason the source couldnt be found was simply I needed to increase kernels to >50 before my own kernels showed up and started being recorded. The warning message was confusing things.
• With multiple files, it's not clear from the examples how directives should be placed. I have a main run file, which imports a class in another file. That class launches kernels, that are in a 3rd file. The line directives need to be placed in the file that has the kernels, not the main file and not the file that does the launching.
• I suggest making a best practice to manually set the kernel cache in your own python code, and setting it as part of the source-files argument in ncu
• The kernel cache clearing is almost the last comment in the profiling section. If it was in a note environment in the md file, and put at the start of the nsight compute section, it would help when going through the example with sph
• I wasn't clear in the docs if I was supposed to/or need to have the scopeddevice or scopedtimer (with or without nvtx). Earlier examples use it, but doesn't seem to have an impact with nsight compute

If anything else comes to mind ill followup here, but hopefully some of these comments will help someone

[shi-eric]

Thanks for the updates and feedback. Why wasn't profiling example_sph.py working for you? That's still not clear to me.